Genres

So I've been using XBMC to listen to music more.  With that I've become interested in party mode and smart playlists.  My year data is pretty good in my tags, but my genres have always been terrible.  I used to classify everything as either hardcore, heavy metal, or hard rock.  But abandoned that long long ago.  Recently I found the last.fm API.  I really like APIs, especially on user generated data.  The genre data was ideal for tagging my mp3s.  As an example of what you can get from the last.fm API:
http://ws.audioscrobbler.com/2.0/?method=artist.getinfo&artist=Manowar&api_key=b25b959554ed76058ac220b7b2e0a026

And that is just band data, they have a bunch of different requests for different sets of data.  Concerning genres they have them for either band, album, or song.  Originally, I figured song would be too specific and band too broad.  Album seemed like the best choice.  However, once I began looking at the album tags they seemed too arbitrary.  I convinced myself that just doing per band genres would be best.  While the general info API has genre data, they have a different API call just for genres:
http://ws.audioscrobbler.com/2.0/?method=artist.gettoptags&artist=horse%20the%20band&api_key=b25b959554ed76058ac220b7b2e0a026

That lists all the genres a band has, with the count, sorted from most popular to least.  It was ideal for what I was doing.  I knew from an earlier adventure with the last.fm API that my usual method for pulling data from the web, i.e., lynx, wouldn't work here.  Lynx attempted to parse the XML, while I wanted the raw unformatted XML data.  That meant going with wget, which I rarely use.  While there are no shortage of wget binaries compiled for Windows I found one that I liked and I recomend:
http://users.ugent.be/~bpuype/wget/

Once you have wget you need to put the executable either in the same directory as you are trying to call it from or in your Windows root (C:\WINDOWS\system32) so it can be called from any directory.  Once it's there downloading a webpage is done with this command:
wget "http://ws.audioscrobbler.com/2.0/?method=artist.gettoptags&artist=Yes&api_key=b25b959554ed76058ac220b7b2e0a026" -O tags.xml

It was then a simple matter of making a perl script to find band names (based on directory names), download genre data with wget and then add that data to the tags via some sort of as yet unknown CLI tagging program.

It was around this time that the problems began to creep up.  Whenever I have to interface with the Windows command line from a perl script it's a nightmare.  The problem is there are nurmerous differences in how Windows and *nix handle things.  A good example would be backslash \ vs forward slash / for directories.  While that may not seem like much it is just one of many subtle differences that need to be accounted for.  Combine this with the fact that Windows isn't consistent if you enter commands directly on the CLI (or via system call in perl) or via a batch file, and it makes working out the interface between perl and Windows the longest part of the project.

If you ever find yourself doing something like this, I recommend having perl create a batch file and then run that.  This is nice because you can easily test the batch file by manually creating it.  Once you find the correct format to do what you want, it is easy to get perl to output your data formatted into a batch file.

In this case though, using a batch file added another difficulty.  I needed to escape certain characters for URLs (& to %26).  While *nix uses % to indicate a hex value, batch files use it for variables.  Thus all the percent signs needed to be escaped themselves to %%.  Getting data to output just right to work for the requests was a very tedious process.  However it was blown away by actually trying to write the data to the tags.

As you probably already know tag data in mp3s is a mess.  For mp3s there are 3 independent tagging formats, ID3v1, ID3v2, and APEv2.  ID3v1 is more or less worthless and deprecated.  It has a pretty short limit on the data length.  Despite the name ID3v2 isn't a second version of ID3v1, but rather a completely different format.  It has subversions ID3v2.2, ID3v2.3, and ID3v2.4.  Despite 2.4 coming out in 2000 some music players don't support it.  Lastly, APEv2 is my preferred format.  There is an APEv1, but it is rare enough that it can be ignored.  All these tag types can have conflicting data.  Which your music player displays will depend on what order of preference it has. 

For some naive reason I assumed finding a way to add tags via a CLI would be easy.  It was not.  I use Mp3tag for all my tagging needs, and it works very well.  For some reason it does not have a CLI.  I began searching for a simple tagging program with a CLI.  I found a few, and settled on one that seemed to work well.
http://www.synthetic-soul.co.uk/tag/
Despite its claims I couldn't get it to write anything besides APE tags.  Getting the tags written correctly was a very long process.  It seems if there wasn't already an APE tag it would create one and fill in all the fields using the ID3v1 data.  That meant that everything was cut off.  The solution was to use Mp3tag to ensure that everything already had APE tags, so that the tag CLI would just overwrite the genre field.  I'd then use Mp3tag to overwrite the ID3 tags with the APE data.

Also I had to do ogg/m4a/wma manually (as a side note the fact that I still can't get good copies of the stuff I have in m4a and wma is the bane of my existence).

The next issue was how to separate the different genres in the tags.  It turns out ID3v2.3 uses a semicolon ; while ID3v2.4 uses a null \0.  I preferred the null, since it was more logical, and foobar2000 formatted things much more nicely when using it.  A new interesting problem developed when I decided to use nulls though.  It is impossible to pass a null to a program as a parameter.  Strings are null terminated.  So when a program sees a null that is the end of the string.  So it was impossible to pass the genres with nulls to my CLI tagger.  Now it is possible to get around this if the program used some method to escape nulls.  For example it could convert any \0 it sees into a null.  You'd think a program made to write tags, which use nulls as deliminators would at least address nulls, but it didn't.

Luckily it turns out that spiting a field by a deliminator is a pretty common feature.  Both Mp3tag and Foobar2000 could do it.  The solution then was to separate the genres with a semicolon and then split them with Mp3tag.  In retrospect it may have been easier to simply write the tag data myself with perl.  I may try to do just that for future use.

After solving those two problems wrapping the whole thing up in a perl was easy.  The final program scans directories (non recursively) and treats all directory names as band names.  It then pulls the genre data and creates a single batch file that will use tag CLI to write the tag data to every file in each directory.  Before writing the data you should review the batch file and make sure all the tags make sense.

The script is on my site at the end of the perl section.
http://daleswanson.org/programs.htm#Perl