Update: I think I have this figured out: SoX will do pretty much everything I need. It’ll even do intelligent crossfades and tempo adjustment. w00t!

The Problem—Short Form

Given a database of MP3 files and their intro and outro times, I need a piece of software with a text-based file format that will allow me to programmatically produce a workout mix, crossfading between tracks at the specified intervals. Conceptually:

[Track 1]
file    = /path/to/track01.mp3
startAt = 00:00:00
endAt   = 00:03:21

[Track 2]
file    = /path/to/track2.mp3
startAt = 00:00:17
endAt   = 00:02:57

I’m unconcerned with the input file format, as long as it’s something I can produce with, for example, Perl. My one effect requirement is the ability to crossfade, but I don’t need beat-matching or other magic. (I’d like to also be able to pan, but that’s a want, not a need.)

I’ve tried:

GarageBand

The input .band file format is binary. (Wrapped in xml to look friendly, but binary nonetheless.)

Audacity

The .aup project file format is XML and readily suits my needs … but the audio has to be pre-converted into chunked .au files.

SoundBooth

The .stmp file format is binary.

So, Lazyweb, any suggestions?

The Problem—Long Form

As the weather gets warmer and the runs get longer, I find myself in need of more customized music for my workouts. Sometimes I want an interval mix, sometimes a gentle climb. Point being, I’ve got a hundred gigabytes of music and I’d like to put it to good use. I’ve got it BPM-tagged, as well as ancillary data on how long the intros and outros are.

But, mixing the music takes time. I’ve gotten it down to about 2:1 time invested to playback length—a 30-minute mix takes about an hour to prepare—but I’d like a bit more randomization.

I’d really, really like to programmatically generate my own playlists, with instructions for where to start and end each song. But, in all my poking around, I have yet to find a program with a plain text input file that will also take MP3s.

I’ve started looking into Csound, but that seems like it might be like hiring a defense contractor to build a bird house. It may just come down to Perl → mp3splt → madplay → faac, presuming I can figure out a way to hack in little crossfades between tracks.

Any suggestions?

My project today was to take this SQL statement:

SELECT "goalType",
  SUM("distancekm") AS "totalkm",
  COUNT(*) AS "workouts",
  COUNT("powerSongAlbum") AS "songcount",
  AVG("distancekm") AS "avgkm",
  MAX("distancekm") AS "maxkm",
  MIN("distancekm") AS "minkm"
FROM "workouts"
GROUP BY "goalType"

… and create a set of ColdFusion components to transform that query into a MapReduce function that would run on MongoDB. Mapping basic SQL to MongoDB MapReduce isn’t too hard. Getting the SQL parser from Derby working within ColdFusion was significantly harder.

But I did it. This is the completely automated result:

db.runCommand({
mapreduce: "workouts",
map: function () { emit(
  this.goalType,
  {
    '_cfcount': 1,
    'distancekm_cfsum': isNaN(this.distancekm) ? null : this.distancekm,
    'distancekm_cfnum': isNaN(this.distancekm) ? 0 : 1,
    'powerSongAlbum_cfcount': (this.powerSongAlbum == null) ? 0 : 1,
    'distancekm_cfmax': isNaN(this.distancekm) ? null : this.distancekm,
    'distancekm_cfmin': isNaN(this.distancekm) ? null : this.distancekm
  }
); },
reduce: function (key,vals) {
  var ret = {
    'distancekm_cfmax': null,
    'distancekm_cfsum': null,
    'distancekm_cfmin': null,
    'distancekm_cfnum': 0,
    'powerSongAlbum_cfcount': 0,
    '_cfcount': 0
  };
  for(var i = 0; i < vals.length; i++) {
    var v = vals[i];
    ret['distancekm_cfnum'] += v['distancekm_cfnum'];
    if(!isNaN(v['distancekm_cfmax'])) ret['distancekm_cfmax'] = (ret['distancekm_cfmax'] == null) ? v['distancekm_cfmax'] : (ret['distancekm_cfmax'] > v['distancekm_cfmax']) ? ret['distancekm_cfmax'] : v['distancekm_cfmax'];
    ret['_cfcount'] += v['_cfcount'];
    if(!isNaN(v['distancekm_cfmin'])) ret['distancekm_cfmin'] = (ret['distancekm_cfmin'] == null) ? v['distancekm_cfmin'] : (v['distancekm_cfmin'] > ret['distancekm_cfmin']) ? ret['distancekm_cfmin'] : v['distancekm_cfmin'];
    ret['powerSongAlbum_cfcount'] += v['powerSongAlbum_cfcount'];
    if(!isNaN(v['distancekm_cfsum'])) ret['distancekm_cfsum'] = v['distancekm_cfsum'] + (ret['distancekm_cfsum'] == null ? 0 : ret['distancekm_cfsum']);
  }
  return ret;
},
finalize: function (key,val) {
  return {
    'totalkm'   : val['distancekm_cfsum'],
    'workouts'  : val['_cfcount'],
    'songcount' : val['powerSongAlbum_cfcount'],
    'avgkm'     : (isNaN(val['distancekm_cfnum']) || isNaN(val['distancekm_cfsum'])) ? null : val['distancekm_cfsum'] / val['distancekm_cfnum'],
    'maxkm'     : val['distancekm_cfmax'],
    'minkm'     : val['distancekm_cfmin']
  };
},
out: "s2mr",
verbose: true
});

And here’s the output when I run that against my MongoDB collection:

{ "_id" : null, "value" : {
    "totalkm" : 451.6752000000001,
    "workouts" : 54,
    "songcount" : 53,
    "avgkm" : 8.364355555555557,
    "maxkm" : 19.7502,
    "minkm" : 0.0194
} }
{ "_id" : "Distance", "value" : {
    "totalkm" : 304.76879999999994,
    "workouts" : 27,
    "songcount" : 27,
    "avgkm" : 11.287733333333332,
    "maxkm" : 26.2581,
    "minkm" : 4.0486
} }
{ "_id" : "Time", "value" : {
    "totalkm" : 19.221,
    "workouts" : 2,
    "songcount" : 2,
    "avgkm" : 9.6105,
    "maxkm" : 9.9224,
    "minkm" : 9.2986
} }

w00t!

I’ve left the code modular enough that I can make a CouchDB version almost as easily.

Update: Better yet, I can now get all the way to an actual query data type:

(Note for potential cynics: no, I’m not missing the point. This is not meant as a production tool—it’s a learning tool. If my students can start with SQL and see the end result in MapReduce, then they have that much better chance of grokking all of it.)

1. Checkout Derby from SVN.

2. Apply a patch:

   Index: java/engine/org/apache/derby/impl/sql/compile/BinaryOperatorNode.java
   ===================================================================
   --- java/engine/org/apache/derby/impl/sql/compile/BinaryOperatorNode.java	(revision 911671)
   +++ java/engine/org/apache/derby/impl/sql/compile/BinaryOperatorNode.java	(working copy)
   @@ -732,6 +732,26 @@
    	}
   
    	/**
   +	 * Get the operator
   +	 *
   +	 * @return String The operator.
   +	 */
   +	public String getOperatorString()
   +	{
   +		return operator;
   +	}
   +
   +	/**
   +	 * Get the methodName
   +	 *
   +	 * @return String The methodName.
   +	 */
   +	public String getMethodName()
   +	{
   +		return methodName;
   +	}
   +
   +	/**
    	 * Categorize this predicate.  Initially, this means
    	 * building a bit map of the referenced tables for each predicate.
    	 * If the source of this ColumnReference (at the next underlying level)
   Index: java/engine/org/apache/derby/impl/sql/compile/SelectNode.java
   ===================================================================
   --- java/engine/org/apache/derby/impl/sql/compile/SelectNode.java	(revision 911671)
   +++ java/engine/org/apache/derby/impl/sql/compile/SelectNode.java	(working copy)
   @@ -360,6 +360,36 @@
    	}
   
    	/**
   +	 * Return the groupByList for this SelectNode.
   +	 *
   +	 * @return GroupByList	The groupByList for this SelectNode.
   +	 */
   +	public GroupByList getGroupByList()
   +	{
   +		return groupByList;
   +	}
   +
   +	/**
   +	 * Return the havingClause for this SelectNode.
   +	 *
   +	 * @return HavingClause	The havingClause for this SelectNode.
   +	 */
   +	public ValueNode getHavingClause()
   +	{
   +		return havingClause;
   +	}
   +
   +	/**
   +	 * Return the orderByList for this SelectNode.
   +	 *
   +	 * @return OrderByList	The orderByList for this SelectNode.
   +	 */
   +	public OrderByList getOrderByList()
   +	{
   +		return orderByList;
   +	}
   +
   +	/**
    	 * Find colName in the result columns and return underlying columnReference.
    	 * Note that this function returns null if there are more than one FromTable
    	 * for this SelectNode and the columnReference needs to be directly under
   Index: java/engine/org/apache/derby/impl/sql/compile/QueryTreeNodeVector.java
   ===================================================================
   --- java/engine/org/apache/derby/impl/sql/compile/QueryTreeNodeVector.java	(revision 911671)
   +++ java/engine/org/apache/derby/impl/sql/compile/QueryTreeNodeVector.java	(working copy)
   @@ -46,7 +46,7 @@
    		return v.size();
    	}
   
   -	QueryTreeNode elementAt(int index)
   +	public QueryTreeNode elementAt(int index)
    	{
    		return (QueryTreeNode) v.elementAt(index);
    	}
   Index: java/engine/org/apache/derby/impl/sql/compile/UnaryOperatorNode.java
   ===================================================================
   --- java/engine/org/apache/derby/impl/sql/compile/UnaryOperatorNode.java	(revision 911671)
   +++ java/engine/org/apache/derby/impl/sql/compile/UnaryOperatorNode.java	(working copy)
   @@ -198,7 +198,7 @@
    	 *
    	 * @return	The operator of this unary operator.
    	 */
   -	String getOperatorString()
   +	public String getOperatorString()
    	{
    		return operator;
    	}
   @@ -267,6 +267,16 @@
    	}
   
    	/**
   +	 * Get the methodName
   +	 *
   +	 * @return String The methodName.
   +	 */
   +	public String getMethodName()
   +	{
   +		return methodName;
   +	}
   +
   +	/**
    	 * Get the parameter operand of this unary operator.
    	 * For the example below, for abs unary operator node, we want to get ?
    	 * select * from t1 where -? = max_cni(abs(-?), sqrt(+?))
   Index: java/engine/org/apache/derby/impl/sql/compile/BinaryListOperatorNode.java
   ===================================================================
   --- java/engine/org/apache/derby/impl/sql/compile/BinaryListOperatorNode.java	(revision 911671)
   +++ java/engine/org/apache/derby/impl/sql/compile/BinaryListOperatorNode.java	(working copy)
   @@ -95,6 +95,26 @@
    	}
   
    	/**
   +	 * Get the operator
   +	 *
   +	 * @return String The operator.
   +	 */
   +	public String getOperatorString()
   +	{
   +		return operator;
   +	}
   +
   +	/**
   +	 * Get the methodName
   +	 *
   +	 * @return String The methodName.
   +	 */
   +	public String getMethodName()
   +	{
   +		return methodName;
   +	}
   +
   +	/**
    	 * Prints the sub-nodes of this object.  See QueryTreeNode.java for
    	 * how tree printing is supposed to work.
    	 *
   

3. Build Derby:

   alias javac='/System/Library/Frameworks/JavaVM.framework/Versions/1.6/Commands/javac -target 1.5'
   export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/1.6/Home/
   ant -quiet clobber ; ant -quiet buildsource ; ant -quiet buildjars

4. Archive $CFROOT/lib/derby*.jar to a cool, dry place.

5. Stop ColdFusion.

6. Copy $DERBY/jars/sane/derby*.jar to $CFROOT/lib.

7. Restart ColdFusion.

8. Execute template:

   <cfscript>
   createObject("java", "java.lang.System").
   	setProperty("derby.debug.true", "StopAfterParsing");
   driver = createObject("java", "org.apache.derby.jdbc.EmbeddedDriver").init();
   props = createObject("java","java.util.Properties");
   conn = driver.connect("jdbc:derby:memory:dummy;create=true",props);
   sql = "SELECT a, b FROM foo, bar";
   try { conn.prepareStatement(sql); }
   catch(org.apache.derby.impl.jdbc.EmbedSQLException e) {
   	if(e.sqlState neq "42Z55") rethrow();
   }
   qt = conn.getContextManager().
   	getContext("LanguageConnectionContext").
   	getLastQueryTree();
   select = qt.getResultSetNode();
   from = select.getFromList();
   tableCount = from.size();
   for(tableNum = 0; tableNum lt tableCount; tableNum++) {
   	baseTable = from.getOptimizable(tableNum);
   	tableName = baseTable.getTableName().toString();
   	writeDump(tableName);
   }
   // bytes = createObject("java", "java.io.ByteArrayOutputStream").init();
   // createObject("java", "org.apache.derby.iapi.services.sanity.SanityManager").
   	SET_DEBUG_STREAM(createObject("java", "java.io.PrintWriter").init(bytes));
   // qt.treePrint();
   // writeDump(bytes.toString());
   try { driver.connect("jdbc:derby:memory:dummy;shutdown=true", props); }
   catch(any e) { }
   </cfscript>

Anybody wanna rewrite QoQ?