94 Comments

Summary:

I love iTunes, and I love that it provides me with convenient access to all my music. But I’m beginning to get frustrated with some of the interesting DB issues present in the way iTunes was developed.

I love iTunes, and I love that it provides me with convenient access to all my music. But I’m beginning to get frustrated with some of the interesting DB issues present in the way iTunes was developed.

I have a large library of music and audiobooks running at just under 31,000 items using 205GB of storage. I have everything on here from Tomita to Eminem and Janet Jackson to Rufus Wainwright. All of the music is (currently) stored on a single volume on the main file server (which itself runs OS X Server).

iTunes is not, and never was, a music manager – it’s a player/organizer, but there are some aspects of the database that iTunes uses that make it frustrating to use.

Database size and updates

One of the things about iTunes I love is the ability to identify songs you haven’t played recently. But for this to work, the database has to be updated each time you play a song. And that’s my current gripe.

You see, to update the database with the play count and ‘last time played’ information iTunes has to save the in-memory version of the database to disk. The iTunes database is stored in the iTunes library file in the Music folder in your home folder. It’s usually many megabytes – there’s a lot of information in there – and it obviously increases in size significantly as the size of your music library incrases. For me, the iTunes database file is just shy of 70MB in size.

So, when updating the information for one song (something iTunes does each time you play a song), iTunes has to save a file that is 70MB in size. On my PowerBook, where I store the main version of my iTunes library, it takes 3-4 seconds, and sometimes 10 seconds, for iTunes to load, modify and resave this file. For those precious seconds, iTunes locks up – it still plays music, but the interface to iTunes freezes completely until the DB update is complete. Sometimes, it even locks up the rest of the OS – annoying if I happen to be typing and then have to wait a few seconds while the system sorts itself out.

Even worse, iTunes also generates a XML file version of the database each time it updates the song playcount too. For my library that means an additional 52MB, plus the overhead for generating the XML content in the first place.

To summarize, each time I play a song – any song – iTunes has to write out 122MB of data.

It begs the question of why on earth Apple chose to go with a single database for this – I can’t be unusual in using iTunes to manage a vast library of music. I can also foresee that as we use iTunes to access videos the problems are likely to increase as we put more and more pressure on iTunes to store the information.

Using a monolithic single file is just slowing the application down and it really wouldn’t be hard to split the file up into manageable chunks, or to use one of the embedded database libraries to access a more structured and easier to update version of the information. At a push, I don’t think it unreasonable to at least be allowed the option of using MySQL (an included component of OS X) to store the data, although I realize for many that would be extreme. There’s no reason thought that iTunes couldn’t support both a file and SQL db model.

Now I wouldn’t mind the use of a single file, if it wasn’t for the fact that it caused such a hiccup while using the application. I don’t want iTunes, or the system on which I’m running it, to lock up for a few seconds while it manipulates a huge file for the sake of updating a few bytes.

It just leaves a bad taste in the mouth.

Database storage

The other issue with the monolithic file style is that backups identify a change in that file as a requirement to backup. That means I get 70MB of backup each day just for listening to my music. That’s not backing up the music I listen to, just the file that holds the database that iTunes uses to store information about what I’ve listened.

To put it into perspective, for all my other material (the articles, books, code, applications, screenshots and diagrams) that I might create in a day, the average amount to backup is just a few megs.

So, just listening to content causes larger quantities of material to be backed up than the amount I generate. :(

XML

I love the fact that I can export my database and playlists into XML. It provides a great way for me to take my music and update an SQL database with the information. It also means I can easily share the data with other people.

So why does iTunes create an XML file every time it also saves the database?

Yes, I know the XML file is a backup, but why create it for every update to the database? Why not just each time you quit iTunes?

Those operations (sharing the DB data through XML) are steps I would elect to do when I needed to do them, not done automatically each time I update the database. Why isn’t the generation of an XML version of the database an operation I can switch off in the preferences?

99% of the time – particularly when simply playing music – I don’t need the XML file regenerated. For some users, they never need the XML file generated.

iTunes, however, thinks differently.

Using a database that links to music

One of the other oddities of the iTunes database is the way it refers to and finds the music you want to play. The server holding my music has changed IP address a couple of times in the years I’ve been using it, although the name of the machine and the name of the volume (imaginatively, Music) on which the files reside has not changed in that time.

If I forget to mount the shared Music volume before I start to play a tune, iTunes can lock up while it works out where the file is. For music added recently, recent versions of iTunes have fixed the ability to automatically mount the volume where the music is located. No problems there.

If the music is old though, then iTunes starts looking for the IP address (not the name) of the server on which the volume used to reside. Now I can, kind of, understand why it does this, but why does iTunes wait around for a minute expecting this machine to answer?

Ignoring this annoying aspect, if iTunes retains this specific information about where the file is located, how is it able to find the file when I mount a volume with the same name? If iTunes wants to be so specific about where the file is located, why does it ignore that information when it first looks for the file?

Since iTunes does find the music at /Volumes/Music/… why does it even bother recording the rest of the information? It obviously doesn’t need it to play the file.

Of course, there’s no way of block updating the information so that it always looks at the new location. Nor does iTunes intelligently update it’s database when it plays the file and discovers it on a different server (but same apparent volume) than that registered in the database.

Oh no. Instead, iTunes makes you wait a minute while iTunes times out looking for a server it shouldn’t be accessing any more anyway.

Summing up

None of these problems are show stoppers. They are, however, all annoying and show some annoying lack of thought that we are normally not subjected to by Apple engineers.

In some cases we’re talking about adding a option in the preferences and a simple check during the save process – that would eliminate the need to save the XML each time the DB was updated and save both disk space and CPU cycles most users just dont need.

In others, all that’s needed is a minor rethink on the way the database and the information it stores are handled. Splitting out the DB by the first letter of the album name would go a long way to alleviating the monolithic file problem.

Despite this, I still love iTunes and, frustrating though these issues are they are not going to stop me from using it.

By Martin MC Brown

You're subscribed! If you like, you can update your settings

Related stories

  1. This is just one of the many things I hate about iTunes. For me Audion was always the perfect solution but it stopped wanting to run for longer than 15 minutes at Mac OS X 10.2 for me.

    I have an iBook and my music is stored on an external hard drive but I keep a subset of my music on my internal hard drive. I change this subset quite often and have to delete my iTunes database files and then re-add all the music otherwise it likes to do weird stuff. Audion had a great Linked Playlist feature to deal with stuff like this. It would watch a folder and update the playlist whenever changes were made to the folder.

    And then there is the large collection at home. If I use iTunes I also have to use a companion application to switch between the two libraries. I got sick and tired of it and just started using MacAmp Lite X. I might be losing some features but I also lost tons of headaches. Also the CPU load is way lower than with iTunes. It was a heavy app to begin with but it continues to become more bloated with each release.

    Share
  2. Whoa, you just pointed out why iTunes likes to freeze up at the beginning of songs – it’s not the beginning of songs, it’s the end of songs!

    Share
  3. i also wonder whether having a single huge database file that holds all your music information increases the risks of said file to get corrupted should it ever sit on bad hard drive sectors.

    Would journaling mitigate this issue?

    Share
  4. Thanks so much for this article. I use iTunes to manage just shy of 50,000 songs, and I have all of these same issues. At least with them voiced here, perhaps someone at apple will get some impetus to change of these things.

    Share
  5. It also occurs to me that there may very-well be a healthy market for Music power-users (which i’m not). Is it time for industrious developers to come-up with a “Pro” music management utility? Perhaps, as Twist mentioned, something like Audion
    .

    Share
  6. Chris:

    Yes, a power or disk failure could potentially corrupt the file.

    Journaling will not help – journaling only journals filesystem metadata changes (ie, directory/file names and block usage) it doesn’t journal any data.

    Journaling is there to ensure the consistency of the file system data, not, as many people think, the data itself.

    Share
  7. You QUIT iTunes???????

    The only part of the disk that is journaled is the filesystem metadata — not the files

    Share
  8. If the file gets corrupted, but you still have all of the music in the folders the way iTunes organizes them, simply delete those xml and database-related files, and drag the music folder into iTunes. It will rebuild the whole database, but you will lose the play counts, etc. (of course, if it’s corrupted already, that’s the least of your worries)

    I am a third-party developer who makes use of the XML file, and I would rather that Apple not change the way it works. It’s really easy for me to work with.

    I have no issues in my everyday use of iTunes. Maybe you simply have too much music. I think your complaints are unwarranted. I don’t know anyone who has that much music. Maybe if you think it’s necessary to have that much music on your computer, you should consider Audion. iTunes wasn’t intended to handle that much data. They never thought that anyone would be sticking that many songs into the app. Granted, they might still make some changes in the next version. They tend to change the way the XML data is structured in major point releases. Of course, I’m basing this on my study of the XML, and not the standard data file, which is undocumented.

    Share
  9. Jason: I understand completely why they generate XML, and as I state in the article I use that XML myself, what I can’t understand is why the generation of the XML file isn’t an option that you can switch off if you don’t want it. Most iTunes users have no idea what it is, let alone why iTunes generates the file in the first place.

    As to having a large library, I’m sorry, I don’t but that argument. Apple computers have been used by music professionals for years and I don’t believe for a second that musicians don’t use iTunes to organize their music. They will have significantly larger collections than I do.

    As a developer, I would never design a system to use such a performance limiting database method, however big or small I expected a users music collection to be – it just seems sloppy.

    It seems though that the same approach is used in Aperture for metadata, judging by the reports I’ve seen of slow metadata editing, but Apple have stated they are going to fix that.

    Hopefully theu will fix iTunes soon too.

    Share
  10. Someone out there was telling me that Apple in the past had made iTunes save the XML on quit, after changes, based on the data file, and that they may do that again in the future. That would probably help with the speed issue.
    I’m still suprised that the music library file is that big. After all, it’s merely a simple little text file… or binary or something.

    Share

Comments have been disabled for this post