Cumulative lifetime play counts

A rambling, self-indulgent, inconsequential post about habits, statistics, speculations, accumulation and missing data.

::

I can’t help but be disappointed that I can’t see lifetime stats for my music listening habits. In these days of play count-tracking programs like iTunes and websites like Last.fm, it’s easy to get caught up in the musical trends of your life. It’s especially interesting when you look at the numbers and discover that you perhaps don’t like a certain style of music as much as you thought you did or you find that you listen to a particular band much more than you would have guessed.

The problem is that your revelations are only going to be as good as the data you’ve collected. I’ve been a “serious music listener” for about 16 years, yet Last.fm has only been tracking my habits for three and my iTunes library only goes back six. I have ten years worth of listening that I will never have any way to quantify simply because the data was never collected.

Missing data, of course, skews results and in this case, snapshots of my habits are skewed in favor of recent years, especially when looking a cumulative lifetime stats. Using data from my library as it stands today, I put together a graph of my most popular years in music. I’ve been for the most part, a contemporary music listener, so the vast majority of my library contains music released from 1993-2008, adding new releases each year.

I calculated the total number of play counts received by all songs in my library that were released in a given year. Here’s the result:

itunes play counts by year

This graph shows the distribution of all my play counts generated since July of 2002 (when iTunes began recording them). We see a peak in 2001 and a general downward slope since.

My explanation for the shape of the graph is that, as years come and go and a music library grows, newer music receives more attention than older music. Familiar tunes give way to new acquisitions and explorations. However, those old tunes never entirely go away; they continue to co-exist with the new ones. As the years pile up, each one’s presence is diluted among the rest and it becomes and increasingly uphill struggle to for the songs of a new year to reach parity with those of the past.

So in this particular graph, I attribute the 2001 peak to the simple coincidence that the songs from 2001-early 2003 were in high rotation at the time that iTunes started tracking play stats. As a result, the initial rate of change for those songs was quite high. And even though the rate at which those songs get played has decreased (exponentially) over time, the songs from other years still have to compete with them for attention, so we find a general trend decreasing cumulative play counts.

average play count by year
Average Play Count by Year

Further evidence of this idea can be seen in the average play count for the songs of each year. There’s a bump in the 2003-2004 area, reflecting the idea that older songs tend to accumulate more play counts over time.

I can’t help but wonder what that play count graph would look like if iTunes had been released in the early 1990s? How much cumulative lifetime play would we see throughout the years?

Of course, there’s no way to figure that out. That information is trapped in the fog of memory, stored in transitory listenings of cassette and compact disc. But while that individual play counts may be lost forever, it might not be impossible to make a decent educated cumulative guess.

I’ll start with the premise that from the years 1993-2001, I averaged a mere 10 songs per day between school bus rides, studying, hanging out, commuting and partying from early high school, through college and my entry into the workforce. That’s probably a conservative estimate, considering the general lengths of my bus rides and commutes. Heck, I’ve managed to generate nearly as many plays in the past 6 months, and I’ve lately been slacking on my music listening in favor of podcasts and audiobooks. But 10 is a good number, so I’ll stick with it.

So, at 10 songs per day, that’s 3650 plays per year. Consider the state of my collection in those early years. Throughout high school and into college, I managed to add records to my library at an average rate of one per week. If iTunes had been around at the time, play counts by now would be heavily concentrated in those early additions, with the highest concentrations being in the earliest records I bought.

By the end of the first year, my estimated 3650 plays would be spread among a mere 500ish songs, an average of 7.3 per songs. By the end of the next year, another 3650 plays would be spread out among about 1000 songs, 3.6 per song. Except that I expect that drop off in older songs to be exponential, not linear.

After some more conjecture and guess work, I extrapolated the accumulation of play counts over the years. After some number-crunching, I had a graph that looks like this:

cumulative play counts by year adjusted

The blue line is the same as above, showing the cumulative distribution of play counts by year of release in my iTunes library. The green line represents what the graph would look like if my estimated historical plays were added to the existing totals.

What does this totally unscientific, made up graph tell me? Basically what I already suspected: that I’d have to stop listening to my older tunes altogether and for a long time if I ever wanted current tunes to “catch up.” Of course, in the time it would take to do that, future tunes would be at a deficit. So really, while it’s a somewhat nice visualization, in reality it will have no bearing on my future plans.

Standard Deviation of the Years in my iTunes Library

I spent part of the past weekend doing some basic statistical analysis of my iTunes Library. I’ve been collecting music for 16ish years now, so I decided to see what kind of historical trends I could find.

One task I assigned myself was to look at the variety of the time span of the releases in my collection. Now I don’t have to do any fancy calculations to tell you that the vast majority of the songs in my library date to the same 16 year period that I’ve been collecting for. Indeed, if you line up all the songs in my library in chronological order by release date, the Median year is 1998. That is to say that half the music in my library was released before or during 1998 and the other half was released during or after that year.

The next step I took was to look at the variety of the release years for each calendar year that I’ve been collecting. I did that by segmenting my library by each year since 1993 using iTunes’ Date Added field, then calculating the standard deviation of the Year field for every song on that list. The lower the result, the more “consistent” that year’s additions were. The higher the number, the greater the eclecticism in that year’s acquisitions.

The results are plotted in this graph:

The green line is the standard deviation for my library as a whole.

In the 90s, I was pretty much an “alternative rock” junkie, so the span of years is pretty narrow overall. But see the bump from 2000-2002? That was late college and my hipster days, when I really had all the time in the world to haunt record shops, variety stores and Usenet groups in attempts to explore the most obscure nonsense. I mean, Morton Subotnik and film scores to Godzilla movies. That kind of nonsense.

It’s cool though, I also discovered Can and Neu! during that same time.

Ratatat – LP3: Expansive and diverse sounds

In the past, I’ve taken issue with the tendency in some circles to lump Ratatat’s music in with that of the 8-bit crowd. I can understand the temptation, what with the band’s programmed, electronic beats, screaming guitars and ample keyboarding. But while their tones may sometimes sound similar to those produced by the Nintendo Entertainment System, their origins are much more organic.

So it surprised me to see that Ratatat appears to be overtly embracing the 8-bit sound while simultaneously diversifiying its non-electronic sound on its latest record, the straight-forwardly titled LP3. This record is a virtual homage to the keyboard. Indeed, the album cover features three of them. The effect is that just about any sound that can be produced by playing the keys finds its way onto this record somewhere. Indeed, one of the lead tracks, Mirando, mixes the bright and clean upper register of a grand piano with the laser beam-like sounds of an 8-bit system near its crescendo.

Don’t fret though, the duo haven’t thrown their guitars away. In fact, Ratatat seems to be well on their way to finding world peace and ultimate truth, the wailing guitar, Wyld Stallyns way. But even there, the stringed instruments shows some surprising variety. Again, the cacophonous Mirando mixes Ratatat’s thrashing riffs and slide guitars with the interjection of a banjo.

Other standout tracks include the disc’s opener, Shiller, which spends most of its time as a contemplative, baroque-style dirge before exploding into a high-flying spaced-out waltz. From there, LP3 hits overdrive with Falcon Jab further demonstrating the band’s new-found commitment to diversity. The guitars talk Peter Frampton style, the percussion is accented by shakers, and the keys of a harpsichord and baby grand trade expressions.

Mi Viejo has a distinct world-music flare, like a caravan moving up and down over the crests of sand dunes. Likewise, Mumtaz Khan shows a distinct Middle Eastern flavor, like what you might expect to find in a Turkish nightclub. Meanwhile Shempi, another highlight, is a wurlitzer-powered merry-go-round spinning through hyperspace. Gypsy Threat takes on the atmosphere of a Scooby Doo chase through an abandoned carnival.

Of the thirteen songs presented here, there’s only one that could arguably be referred to as a “typical” Ratatat song. With its mid-tempo beats and harpsichord melodies, Dura would almost feel at home as the backing track for one of Ratatat’s infamous remixes if it weren’t such a compelling track on its own.

With three albums under their belt, Ratatat has consistently shown themselves to be on the top of their game. But that game keeps expanding, with each successive album adding a new layers of complexity and textures to the band’s modus operandi. LP3 shows that whatever sights they set for themselves, they’ll reach them with gusto.

::

Note, the Amazon MP3 store offers a exclusive bonus track: Shempi [E*Rock Remix].

::

Mirando video:

iTunes Tip: Back-date the songs in your library

I’ve mentioned before that one of my standard library organization procedures is to back-date the “Date Added” field for all the songs in my iTunes library. That is, if I originally received an album for my birthday in 1999, I make sure the Date Added field in my library is my birthday, 1999. Same goes for every CD I’ve bought or mp3 I’ve downloaded.

Unfortunately, Apple for whatever reason, has decided that the Date Added field should not be user-modifiable. You can’t change it yourself, either manually or via AppleScript. And honestly, I’m tempted to think of that behavior as a bug/product defect. In this digital age, where at some point each and every iTunes user *will* have to rebuild or replace their library after some sort of data catastrophe, it seems like an obvious feature to be able to reconstruct one’s musical history chronologically. Why should users have to settle for the post-reconstruction dates for albums they’ve actually owned for years?

Well, there’s a bit of a workaround, but it is a tedious one. So make sure you regularly backup your iTunes Library file so that you don’t have to do it all over again in the event of a hard drive crash. I use my .mac/Mobile Me account to upload my library file to my iDisk every night at midnight.

How To

The secret is that iTunes relies on your computer’s system clock to assign the Date Added to songs in the library. So back-dating is as “simple” as changing your computer’s clock, dragging your music files into iTunes, then resetting the clock to the current time.

If you have hundreds of albums to do this with, the procedure can get quickly tiresome. Unfortunately, there is no way to automate it. Plus, if you are trying to fix songs that are already in your library, you have to remove them, change the system date, then re-add them. In those cases, make sure you note the play counts and star ratings, because you’ll have to re-enter those manually. Like I said, tedious.

But all that work is worth it when, in the span of five seconds, you conjure up a Smart Playlist called Best Music from High School:

Date added is in the range 8/15/1993 - 5/15/1997
My Rating is 5 Stars

That is truly awesome.

::

One warning though:

If you are using Mac OS X 10.5 (Leopard) and you use iCal alarms, be sure to disable them in Preferences before setting your clock back. I found this out the hard way when I was suddenly flooded by couple hundred notifications for events that had already passed. It seems that iCal travels back in time with you, then when you return to the present, it feels the need to update you on all the stuff you missed.