Cumulative lifetime play counts

A rambling, self-indulgent, inconsequential post about habits, statistics, speculations, accumulation and missing data.

::

I can’t help but be disappointed that I can’t see lifetime stats for my music listening habits. In these days of play count-tracking programs like iTunes and websites like Last.fm, it’s easy to get caught up in the musical trends of your life. It’s especially interesting when you look at the numbers and discover that you perhaps don’t like a certain style of music as much as you thought you did or you find that you listen to a particular band much more than you would have guessed.

The problem is that your revelations are only going to be as good as the data you’ve collected. I’ve been a “serious music listener” for about 16 years, yet Last.fm has only been tracking my habits for three and my iTunes library only goes back six. I have ten years worth of listening that I will never have any way to quantify simply because the data was never collected.

Missing data, of course, skews results and in this case, snapshots of my habits are skewed in favor of recent years, especially when looking a cumulative lifetime stats. Using data from my library as it stands today, I put together a graph of my most popular years in music. I’ve been for the most part, a contemporary music listener, so the vast majority of my library contains music released from 1993-2008, adding new releases each year.

I calculated the total number of play counts received by all songs in my library that were released in a given year. Here’s the result:

itunes play counts by year

This graph shows the distribution of all my play counts generated since July of 2002 (when iTunes began recording them). We see a peak in 2001 and a general downward slope since.

My explanation for the shape of the graph is that, as years come and go and a music library grows, newer music receives more attention than older music. Familiar tunes give way to new acquisitions and explorations. However, those old tunes never entirely go away; they continue to co-exist with the new ones. As the years pile up, each one’s presence is diluted among the rest and it becomes and increasingly uphill struggle to for the songs of a new year to reach parity with those of the past.

So in this particular graph, I attribute the 2001 peak to the simple coincidence that the songs from 2001-early 2003 were in high rotation at the time that iTunes started tracking play stats. As a result, the initial rate of change for those songs was quite high. And even though the rate at which those songs get played has decreased (exponentially) over time, the songs from other years still have to compete with them for attention, so we find a general trend decreasing cumulative play counts.

average play count by year
Average Play Count by Year

Further evidence of this idea can be seen in the average play count for the songs of each year. There’s a bump in the 2003-2004 area, reflecting the idea that older songs tend to accumulate more play counts over time.

I can’t help but wonder what that play count graph would look like if iTunes had been released in the early 1990s? How much cumulative lifetime play would we see throughout the years?

Of course, there’s no way to figure that out. That information is trapped in the fog of memory, stored in transitory listenings of cassette and compact disc. But while that individual play counts may be lost forever, it might not be impossible to make a decent educated cumulative guess.

I’ll start with the premise that from the years 1993-2001, I averaged a mere 10 songs per day between school bus rides, studying, hanging out, commuting and partying from early high school, through college and my entry into the workforce. That’s probably a conservative estimate, considering the general lengths of my bus rides and commutes. Heck, I’ve managed to generate nearly as many plays in the past 6 months, and I’ve lately been slacking on my music listening in favor of podcasts and audiobooks. But 10 is a good number, so I’ll stick with it.

So, at 10 songs per day, that’s 3650 plays per year. Consider the state of my collection in those early years. Throughout high school and into college, I managed to add records to my library at an average rate of one per week. If iTunes had been around at the time, play counts by now would be heavily concentrated in those early additions, with the highest concentrations being in the earliest records I bought.

By the end of the first year, my estimated 3650 plays would be spread among a mere 500ish songs, an average of 7.3 per songs. By the end of the next year, another 3650 plays would be spread out among about 1000 songs, 3.6 per song. Except that I expect that drop off in older songs to be exponential, not linear.

After some more conjecture and guess work, I extrapolated the accumulation of play counts over the years. After some number-crunching, I had a graph that looks like this:

cumulative play counts by year adjusted

The blue line is the same as above, showing the cumulative distribution of play counts by year of release in my iTunes library. The green line represents what the graph would look like if my estimated historical plays were added to the existing totals.

What does this totally unscientific, made up graph tell me? Basically what I already suspected: that I’d have to stop listening to my older tunes altogether and for a long time if I ever wanted current tunes to “catch up.” Of course, in the time it would take to do that, future tunes would be at a deficit. So really, while it’s a somewhat nice visualization, in reality it will have no bearing on my future plans.

Standard Deviation of the Years in my iTunes Library

I spent part of the past weekend doing some basic statistical analysis of my iTunes Library. I’ve been collecting music for 16ish years now, so I decided to see what kind of historical trends I could find.

One task I assigned myself was to look at the variety of the time span of the releases in my collection. Now I don’t have to do any fancy calculations to tell you that the vast majority of the songs in my library date to the same 16 year period that I’ve been collecting for. Indeed, if you line up all the songs in my library in chronological order by release date, the Median year is 1998. That is to say that half the music in my library was released before or during 1998 and the other half was released during or after that year.

The next step I took was to look at the variety of the release years for each calendar year that I’ve been collecting. I did that by segmenting my library by each year since 1993 using iTunes’ Date Added field, then calculating the standard deviation of the Year field for every song on that list. The lower the result, the more “consistent” that year’s additions were. The higher the number, the greater the eclecticism in that year’s acquisitions.

The results are plotted in this graph:

The green line is the standard deviation for my library as a whole.

In the 90s, I was pretty much an “alternative rock” junkie, so the span of years is pretty narrow overall. But see the bump from 2000-2002? That was late college and my hipster days, when I really had all the time in the world to haunt record shops, variety stores and Usenet groups in attempts to explore the most obscure nonsense. I mean, Morton Subotnik and film scores to Godzilla movies. That kind of nonsense.

It’s cool though, I also discovered Can and Neu! during that same time.

Ratatat – LP3: Expansive and diverse sounds

In the past, I’ve taken issue with the tendency in some circles to lump Ratatat’s music in with that of the 8-bit crowd. I can understand the temptation, what with the band’s programmed, electronic beats, screaming guitars and ample keyboarding. But while their tones may sometimes sound similar to those produced by the Nintendo Entertainment System, their origins are much more organic.

So it surprised me to see that Ratatat appears to be overtly embracing the 8-bit sound while simultaneously diversifiying its non-electronic sound on its latest record, the straight-forwardly titled LP3. This record is a virtual homage to the keyboard. Indeed, the album cover features three of them. The effect is that just about any sound that can be produced by playing the keys finds its way onto this record somewhere. Indeed, one of the lead tracks, Mirando, mixes the bright and clean upper register of a grand piano with the laser beam-like sounds of an 8-bit system near its crescendo.

Don’t fret though, the duo haven’t thrown their guitars away. In fact, Ratatat seems to be well on their way to finding world peace and ultimate truth, the wailing guitar, Wyld Stallyns way. But even there, the stringed instruments shows some surprising variety. Again, the cacophonous Mirando mixes Ratatat’s thrashing riffs and slide guitars with the interjection of a banjo.

Other standout tracks include the disc’s opener, Shiller, which spends most of its time as a contemplative, baroque-style dirge before exploding into a high-flying spaced-out waltz. From there, LP3 hits overdrive with Falcon Jab further demonstrating the band’s new-found commitment to diversity. The guitars talk Peter Frampton style, the percussion is accented by shakers, and the keys of a harpsichord and baby grand trade expressions.

Mi Viejo has a distinct world-music flare, like a caravan moving up and down over the crests of sand dunes. Likewise, Mumtaz Khan shows a distinct Middle Eastern flavor, like what you might expect to find in a Turkish nightclub. Meanwhile Shempi, another highlight, is a wurlitzer-powered merry-go-round spinning through hyperspace. Gypsy Threat takes on the atmosphere of a Scooby Doo chase through an abandoned carnival.

Of the thirteen songs presented here, there’s only one that could arguably be referred to as a “typical” Ratatat song. With its mid-tempo beats and harpsichord melodies, Dura would almost feel at home as the backing track for one of Ratatat’s infamous remixes if it weren’t such a compelling track on its own.

With three albums under their belt, Ratatat has consistently shown themselves to be on the top of their game. But that game keeps expanding, with each successive album adding a new layers of complexity and textures to the band’s modus operandi. LP3 shows that whatever sights they set for themselves, they’ll reach them with gusto.

::

Note, the Amazon MP3 store offers a exclusive bonus track: Shempi [E*Rock Remix].

::

Mirando video:

iTunes Tip: Back-date the songs in your library

I’ve mentioned before that one of my standard library organization procedures is to back-date the “Date Added” field for all the songs in my iTunes library. That is, if I originally received an album for my birthday in 1999, I make sure the Date Added field in my library is my birthday, 1999. Same goes for every CD I’ve bought or mp3 I’ve downloaded.

Unfortunately, Apple for whatever reason, has decided that the Date Added field should not be user-modifiable. You can’t change it yourself, either manually or via AppleScript. And honestly, I’m tempted to think of that behavior as a bug/product defect. In this digital age, where at some point each and every iTunes user *will* have to rebuild or replace their library after some sort of data catastrophe, it seems like an obvious feature to be able to reconstruct one’s musical history chronologically. Why should users have to settle for the post-reconstruction dates for albums they’ve actually owned for years?

Well, there’s a bit of a workaround, but it is a tedious one. So make sure you regularly backup your iTunes Library file so that you don’t have to do it all over again in the event of a hard drive crash. I use my .mac/Mobile Me account to upload my library file to my iDisk every night at midnight.

How To

The secret is that iTunes relies on your computer’s system clock to assign the Date Added to songs in the library. So back-dating is as “simple” as changing your computer’s clock, dragging your music files into iTunes, then resetting the clock to the current time.

If you have hundreds of albums to do this with, the procedure can get quickly tiresome. Unfortunately, there is no way to automate it. Plus, if you are trying to fix songs that are already in your library, you have to remove them, change the system date, then re-add them. In those cases, make sure you note the play counts and star ratings, because you’ll have to re-enter those manually. Like I said, tedious.

But all that work is worth it when, in the span of five seconds, you conjure up a Smart Playlist called Best Music from High School:

Date added is in the range 8/15/1993 - 5/15/1997
My Rating is 5 Stars

That is truly awesome.

::

One warning though:

If you are using Mac OS X 10.5 (Leopard) and you use iCal alarms, be sure to disable them in Preferences before setting your clock back. I found this out the hard way when I was suddenly flooded by couple hundred notifications for events that had already passed. It seems that iCal travels back in time with you, then when you return to the present, it feels the need to update you on all the stuff you missed.

The Breeders – Live in Atlanta

the breeders rock out live in atlanta

I just pulled in from midtown Atlanta, The Loft, specifically, where The Breeders just put on a hell of a show. I’ve been a big Breeders fan since the Last Splash days (14 years ago). Yet due to the band’s somewhat sporadic history, tonight was my first opportunity to see them play live. And what a great performance they put on.

The band was in good form and in good spirits, clearly having a good time.

This was the last show of the tour (or as Kim Deal said at one point “the last tour of the show”). Maybe it was just road exhaustion, but the band was positively loopy. Or maybe everyone was just in a good mood. Jokes were told, funny faces were made and anecdotes were shared. At one point, Kim introduced the band’s third guitar player Cheryl, mentioning that she was from Florida and asking who else in the audience was from there as well. In response to a number of “whooops” that erupted from the crowd, Kim said “We’re from Dayton Ohio, so we got you there” and merriment ensued.

The show itself covered the gamut of Deal-related material, including songs from The Amps side project. In fact, the show opened with Tipp City, which happens to be one of my favorite Amps songs. Last Splash got some good treatment, with renditions of New Year, No Aloha, I Just Wanna Get Along, Divine Hammer, a sublime version of Drivin on 9 and of course Cannonball. A house-collapsing performance of Saints closed the first encore.

The Breeders latest album, Mountain Battles, was also well represented. Bang On, We’re Gonna Rise, Here No More, Night of Joy, German Studies, Walk it Off and No Way all made appearances.

Noticeably absent though was much of the material from Title TK. Huffer was brilliant, but other than that I don’t remember the band playing any other song from that album. Similar story with the songs from The Breeders’ debut album, Pod. Only a handful from there as well: Iris, Happiness is a Warm Gun (Beatles cover) and one of my most favoritest songs ever Fortunately Gone, which closed out the second encore, the show and the tour.

A perfecting end note for a wonderful experience.

Given just how great a band The Breeders are, the sporadicness of its output can be quite frustrating. But going to bed now on a natural musical buzz, I can say that after seeing the band perform live, I have accomplished a life goal that’s been with me for nearly one and half decades.

I leave you with Bang On, from Mountain Battles, performed Live in Atlanta, June 13, 2008:


Nine Inch Nails – The Slip: Familiar Territory

nine inch nails: halo 27: the slip

I’m beginning to think that being stuck in a soul-crushing contract with a major corporate record label was a good thing for Trent Reznor. Sure, his new-found freedom is allowing him to push new boundaries in promotion, product, distribution and sales. But maybe all the frustration and suffering that came with having to deal with “record industry types” was just the fuel he needed to produce his compelling brand of angst-n-rage.

About a month ago, nine inch nails continued on its untethered course and released Halo 27: The Slip as a completely free, no-strings-attached download. True to the example set with Ghosts, the album is available in multiple formats, from the lowly mp3 to ridiculously high quality 24bit/96khz wav. Also, like Ghosts, each track has individualized album art. Unlike Ghosts, since the songs on The Slip actually have words, lyrics have thoughtfully been embedded in the files as well.

And while I’m glad to see the evolution of the online model, a part of me wishes Trent had spent a little more time on the music itself.

The Slip clocks in at a standard 43 minutes and largely follows With Teeth’s straightforward rock stylings. And it’s good, but it seems to lack that spark of greatness that I generally associate with nails’ releases. After listening to it several times now, there’s nothing on The Slip that strikes me as exceptional.

Which is not to say that I haven’t enjoyed it, it’s just I’ve heard a lot of this before. 999,999, the opening track, is Pinion for the new millennium, while 1,000,000 echoes the catchy poprock sensibility of The Hand That Feeds. Demon Seed, perhaps the catchiest song on the album, feels a lot like a rehash of Starfuckers, Inc.

All great artists are entitled to recycle their ideas occasionally. Let’s hope Trent has gotten it out of his system this time around. If not, someone get him a record contract to be pissed off about.

You can grab The Slip for yourself at theslip.nin.com.

Smart Playlist Ideas: Master List and Newest Tunes

With more than 16,000 songs to manage, there is no more essential a tool in my library than iTunes’ Smart Playlists. From building simple playlists for listening to creating complex queries for examination, Smart Playlists turn what would be a tedious burden into a trivial task. At the moment, I have more than 50 of them slicing, organizing and corralling my expansive collection of tunes into an easily navigable, self-sustaining ecosystem of music.

It seems a shame to keep all those playlists to myself when they could be benefiting other iTunes users, helping them find new ways to organize and listen to their libraries. On this first of a new tunequest segment, I’ll share some of the criteria for playlists that I’ve developed to help manage my library.

This first installment is a two-for. We’ll start with the foundation of my listening habits: the master tunequest list.

The master tunequest list was one of the earliest Smart Playlists I created. Its job is to act as a filter on the main iTunes library and determine which files are eligible for inclusion in other Smart Playlists. The premise is that only properly tagged music without any playback glitches should be included in subsequent lists.

Podcasts, audiobooks, iTunes U courses, videos and other files that are not strictly musical should be excluded from the standard rotation. But how to do it?

master tunequest smart playlist selectors

This is the actual criteria for my master list. There are multiple ways to create one, you just have to tell iTunes what to exclude. Here’s a brief description of the selections I’ve made:

Date Added is not 1/3/02.

I had a major hard drive crash on 1/2/02 which wiped out an early version of my Library. When I restored it from back up the next day, I discovered that the id3 tags for 5 years worth of mp3s had only been made on the library, not the back ups. I took the crash as an opportunity to re-evaluate my songs and make sure that all my files were “up to code” with proper tags and acceptable bitrates.

When Smart Playlists were introduced later that year, I didn’t want songs that I hadn’t checked going into my rotation. With the Date Added for all 7500 songs (my library size at the time) set to 1/3/02, I was easily able to exclude those songs that were pending evaluation. After evaluation, I re-imported my songs with the appropriate Date Added and they were automatically re-included in the master list. Today, about 200 rather obscure songs remain that I haven’t had the wherewithal to track down, so excluded they sit.

Date Added is a powerful tool for segmenting your library based on time period. You can set it to before, after or between dates to isolate just those songs, like a “Songs of Summer 2005” playlist (Date Added is in the range 6/1/05 and 9/1/05).

My Rating is not 1 Star

Rating a song 1 star is my arbitrary way of taking a song out of circulation. If I notice a song has glitches or that its tags have errors, I’ll mark it as 1 star until such time as I can fix it.

Podcast is false

Keeps podcasts out.

Playlist is not SpokenAudio

I have several playlists of just spoken audio that isn’t an iTunes Audiobook: iTunes U courses, comedy albums and other spoken word pieces. These playlists are kept in a sidebar folder called “SpokenAudio,” which iTunes treats as a single unified playlist for the purposes of Smart selecting.

You can create some complex hierarchies and conditional listening schemes using nested folders and playlists.

Kind does not contain video

Keeps all video content off the list. Movies, TV shows and video podcasts are not welcome here.

Playlist is not Audiobooks

Keeps files from iTunes’ Audiobooks sidebar from mixing with music. iTunes offers similar selectors for Movies and TV Shows as another way to exclude video content.

Genre is not Podcast

Another method to exclude podcasts from everyday listening.

Playlist is not xmas

I have a playlist dedicated to Christmas and other holiday tunes. This selector keeps it out of the way for ~330 days of the year. I remove it on or around Thanksgiving and replace at after New Year’s.

::

Now that we’ve cordoned off our healthy files, we can slice and sub-slice it to fit as many different listening schemes as we have whims. This is a relatively recent playlist I’ve been using to handle new music.

Newest Tunes

Some music falls through the cracks around here. Some albums get overshadowed and as time marches on, they don’t get the attention they deserve, receiving only cursory glances before being supplanted by newer music. This playlist is meant to allow all new acquisitions to have an full opportunity for listening.

It takes 4 parts:

Playlist is master list

The master list ensures that only “safe” music is eligible for inclusion.

Play Count is less than 4

I generally feel that 3 plays per song is enough to consider a new album adequately vetted. You can adjust it to suit your tastes.

Limit to 150 songs selected by Most Recently Added

This limiter means that the 150 most recently added songs that have been played 0-3 times (and are on the master list) will be included in the playlist. When one song on the list reaches 4 plays, it disappears from the list and is replaced by an older song that meets the criteria. When new songs are added to the library, they automatically appear on this playlist, pushing off older songs.

Since I implemented this playlist, I’ve been able to keep a handle on the inflow of new music into my library. Enjoy.

Nirvana: In Bloom [unmixed]

Was doing some web-sleuthing this afternoon and ran across this video excerpt from the Classic Albums documentary series episode about Nirvana’s Nevermind. The short clip features producer Butch Vig demonstrating the individual mixing tracks from In Bloom, commenting on its various parts and pieces, from the awesome groove of the isolated drums and bass (which they got on the first take), to Dave’s harmonizing on the chorus.

I was actually quite surprised to learn about Dave’s singing contributions. I’ve been listening to In Bloom for 17 years now and I had no idea that Dave Grohl had performed any vocals for the song. I can clearly hear it now that its been pointed out, but for nearly two decades I thought it was just studio effects. Amazingly, after all this time, I’m still learning Nevermind’s tricks.

In Bloom has always been a favorite of mine, so I particularly enjoyed this fascinating look at its skeleton.

In Bloom deconstructed:

Here’s a Google Video of what appears to be the entire documentary. It starts with a similar deconstruction of Drain You, perhaps the most complex song on Nevermind (squeaky toys and five overlayed guitar tracks!), then continues with anecdotes by Dave, Krist and Butch recounting how the record was made.

hat tip to MixMyMusic.net

Cat Power – Jukebox

Jukebox is the second record in Cat Power’s career that mostly features her renditions of songs written by “noted elder statesmen” of the rock/pop music pantheon, including Billie Holiday, Bob Dylan, Janis Joplin, Joni Mitchell, and James Brown, among others. Not being all that familiar with large portions of the source material, I can’t say how well it compares to the originals, but as a stand-alone work, Jukebox is largely a sultry success.

The Good
Roughly an hour’s worth of inventive reinterpretations and intriguing recastings that stand on their own. Cat Power is smooth and confident, presenting and album full of class and soul. Her soft, rough-textured voice, ever wispy, commands a presence, despite sounding conjured from the ether. New York, New York, perhaps the most well-known song on Jukebox opens the album with forged bombastic drums that never let up before ending as abruptly as they began. From there, her take on Ramblin’ (Wo)Man one-ups Hank Williams, reaching like a spectre from the ghost town in your brain.

Also of note, Metal Heart, a piano-powered Cat Power original continues the space-filled and resonant balladry found on 2006’s The Greatest. And the somber, bittersweet guitar strains of Silver Stallion provide the album’s highlight.

The not-as-good
Front-loading. The first four tracks are delightfully splendid, attention-grabbing standouts. Beyond that, while pleasant, Cat Power’s style is so singular that the album struggles to make a lasting impression. Which is all the more excuse to spend more time with it.

Ramblin’ (Wo)Man AOL Session performance. The sound isn’t synced for some reason, but the song works.

Further Reading:
PopMatters
DayTrotter