statistics

Cumulative lifetime play counts

A rambling, self-indulgent, inconsequential post about habits, statistics, speculations, accumulation and missing data.

I can’t help but be disappointed that I can’t see lifetime stats for my music listening habits. In these days of play count-tracking programs like iTunes and websites like Last.fm, it’s easy to get caught up in the musical trends of your life. It’s especially interesting when you look at the numbers and discover that you perhaps don’t like a certain style of music as much as you thought you did or you find that you listen to a particular band much more than you would have guessed.

The problem is that your revelations are only going to be as good as the data you’ve collected. I’ve been a “serious music listener” for about 16 years, yet Last.fm has only been tracking my habits for three and my iTunes library only goes back six. I have ten years worth of listening that I will never have any way to quantify simply because the data was never collected.

Missing data, of course, skews results and in this case, snapshots of my habits are skewed in favor of recent years, especially when looking a cumulative lifetime stats. Using data from my library as it stands today, I put together a graph of my most popular years in music. I’ve been for the most part, a contemporary music listener, so the vast majority of my library contains music released from 1993-2008, adding new releases each year.

I calculated the total number of play counts received by all songs in my library that were released in a given year. Here’s the result:

itunes play counts by year

This graph shows the distribution of all my play counts generated since July of 2002 (when iTunes began recording them). We see a peak in 2001 and a general downward slope since.

My explanation for the shape of the graph is that, as years come and go and a music library grows, newer music receives more attention than older music. Familiar tunes give way to new acquisitions and explorations. However, those old tunes never entirely go away; they continue to co-exist with the new ones. As the years pile up, each one’s presence is diluted among the rest and it becomes and increasingly uphill struggle to for the songs of a new year to reach parity with those of the past.

So in this particular graph, I attribute the 2001 peak to the simple coincidence that the songs from 2001-early 2003 were in high rotation at the time that iTunes started tracking play stats. As a result, the initial rate of change for those songs was quite high. And even though the rate at which those songs get played has decreased (exponentially) over time, the songs from other years still have to compete with them for attention, so we find a general trend decreasing cumulative play counts.

Average Play Count by Year

Further evidence of this idea can be seen in the average play count for the songs of each year. There’s a bump in the 2003-2004 area, reflecting the idea that older songs tend to accumulate more play counts over time.

I can’t help but wonder what that play count graph would look like if iTunes had been released in the early 1990s? How much cumulative lifetime play would we see throughout the years?

Of course, there’s no way to figure that out. That information is trapped in the fog of memory, stored in transitory listenings of cassette and compact disc. But while that individual play counts may be lost forever, it might not be impossible to make a decent educated cumulative guess.

I’ll start with the premise that from the years 1993-2001, I averaged a mere 10 songs per day between school bus rides, studying, hanging out, commuting and partying from early high school, through college and my entry into the workforce. That’s probably a conservative estimate, considering the general lengths of my bus rides and commutes. Heck, I’ve managed to generate nearly as many plays in the past 6 months, and I’ve lately been slacking on my music listening in favor of podcasts and audiobooks. But 10 is a good number, so I’ll stick with it.

So, at 10 songs per day, that’s 3650 plays per year. Consider the state of my collection in those early years. Throughout high school and into college, I managed to add records to my library at an average rate of one per week. If iTunes had been around at the time, play counts by now would be heavily concentrated in those early additions, with the highest concentrations being in the earliest records I bought.

By the end of the first year, my estimated 3650 plays would be spread among a mere 500ish songs, an average of 7.3 per songs. By the end of the next year, another 3650 plays would be spread out among about 1000 songs, 3.6 per song. Except that I expect that drop off in older songs to be exponential, not linear.

After some more conjecture and guess work, I extrapolated the accumulation of play counts over the years. After some number-crunching, I had a graph that looks like this:

cumulative play counts by year adjusted

The blue line is the same as above, showing the cumulative distribution of play counts by year of release in my iTunes library. The green line represents what the graph would look like if my estimated historical plays were added to the existing totals.

What does this totally unscientific, made up graph tell me? Basically what I already suspected: that I’d have to stop listening to my older tunes altogether and for a long time if I ever wanted current tunes to “catch up.” Of course, in the time it would take to do that, future tunes would be at a deficit. So really, while it’s a somewhat nice visualization, in reality it will have no bearing on my future plans.

Standard Deviation of the Years in my iTunes Library

I spent part of the past weekend doing some basic statistical analysis of my iTunes Library. I’ve been collecting music for 16ish years now, so I decided to see what kind of historical trends I could find.

One task I assigned myself was to look at the variety of the time span of the releases in my collection. Now I don’t have to do any fancy calculations to tell you that the vast majority of the songs in my library date to the same 16 year period that I’ve been collecting for. Indeed, if you line up all the songs in my library in chronological order by release date, the Median year is 1998. That is to say that half the music in my library was released before or during 1998 and the other half was released during or after that year.

The next step I took was to look at the variety of the release years for each calendar year that I’ve been collecting. I did that by segmenting my library by each year since 1993 using iTunes’ Date Added field, then calculating the standard deviation of the Year field for every song on that list. The lower the result, the more “consistent” that year’s additions were. The higher the number, the greater the eclecticism in that year’s acquisitions.

The results are plotted in this graph:

The green line is the standard deviation for my library as a whole.

In the 90s, I was pretty much an “alternative rock” junkie, so the span of years is pretty narrow overall. But see the bump from 2000-2002? That was late college and my hipster days, when I really had all the time in the world to haunt record shops, variety stores and Usenet groups in attempts to explore the most obscure nonsense. I mean, Morton Subotnik and film scores to Godzilla movies. That kind of nonsense.

It’s cool though, I also discovered Can and Neu! during that same time.

Impact report update

My original High Impact formula had a fundamental flaw, which I think I may have fixed.

I spent the last post talking about the albums that made the biggest “impact” on me during 2007, but what exactly does that mean? Over the summer, I came up with the general concept, which basically defined impact as the average number of times any particular song from an album or artist in my iTunes library had been played, showing who has received the most attention relative to their presence in my library. Add together the play counts of all songs by an artist or on an album, then divide by the number of songs. There’s the impact score.

While this method produced some interesting results, it suffers from a substantial deficiency: it grossly inflates the relative impact of artists and albums that have a low number of songs. Artists or albums with a lot of songs have to be played a lot more in order to keep up. I noted this in my original post on the subject:

In cases where an artist has a low number of songs, each play count is “worth more” in relation to other artists. A single play by an artist with only one song nets that artist a full “point,” whereas as single play by an artist with 20 songs would only gain .05 points.

The solution I devised at the time was to set a threshold for inclusion, eg. artists must have more than 5 songs in my library to be ranked.

I never really liked that workaround because the threshold was arbitrary and it excluded artists who should really have been able to be ranked. So I spent a bit time recently tweaking the formula and I think found something that works to my satisfaction:

Total Play Counts [squared] divided by the number of songs.

This is essentially the same as the original formula, except that the total play counts an artist or album has received is multiplied by itself. The effect of this is to give the impact of a single play more weight as the number of songs increases. My thinking goes like this: albums with a lot of songs should rightly have a larger impact vs those which have fewer, even when the average play count is the same.

Suppose we have an album with 10 songs on it and an EP that has 4. Each song on both albums has been played twice.

Using the original formula:

Album: total play count (10 * 2) / number of songs (10) = 2 EP: total play count (4 * 2) / number of songs (4) = 2

Eight (8) plays gives the EP the same impact as the album that has received twenty (20).

Now the new formula:

Album: total play count squared (10 * 2)(10 * 2) / 10 = 40 EP: total play count squared (4 * 2)(4 * 2) / 4 = 16

Even though both recordings have been listened to the same number of times, the album’s larger footprint leaves a greater impact score than the EP.

The best analogy I can think of is mass vs speed. More songs equal greater “mass.” More plays equal greater “speed.” Just as lighter objects have to travel faster to hit with the same amount of force as heavier objects, an artist with a lighter presence in my library has to be played more times to have the same impact as an artist with a lot songs. A ping pong ball must travel at higher speeds to equal the same force as a baseball.

This table below shows the formula in action.

Rank	Artist	# of Songs	Plays	AVG	Impact
61	The Cardigans	92	255	2.77	706.794
62	To Rococo Rot	34	155	4.56	706.618
63	Tomoyasu Hotei	1	26	26	676.00
64	Molotov vs Dub Pistols	1	26	26	676.00
65	National Skyline	17	107	6.29	673.47

The two singles by Tomoyasu Hotei and Molotov vs Dub Pistols happen to be in the top 20 most played songs in my library (out of ~16000). That showing places each of them relatively high on the list, but not overwhelmingly so, considering the differences between the various AVG play counts. That’s an equitable result I’m pretty happy with.

iTunes Report: High Impact Artists

I’ve spent the past couple days playing around with Alex King’s iTunes Stats program. It’s written in PHP/MYSQL and requires a web server to run. With the MAMP one-click server running on my PowerBook, I had little trouble installing the program (though I did have to substantially increase the PHP timeout setting so it could handle my large library).

iTunes Stats reads XML files, one can load an entire iTunes library or export playlists for more selective examinations.

The program also comes with a number of built-in reports, such as Most Played Albums, Most Played Artists. Top Rated Albums/Artists and the option to weight by number of rated songs and play counts. The engine supports adding custom reports, so if you are motivated enough, you can create new methods of analyzing your music.

My PHPMYSQL-fu is not very strong virtually non-existent. Still, I was able to cobble together my own metric: Artist Impact.

An Artist’s Impact is measured by the total number of play counts they have received divided by the number of songs that artist has in the library. For example, Artist A has 20 songs in the library and those 20 songs have been played a total of 100 times. Thus, 100/20 = 5. That’s Artist A’s Impact. Basically, it tells you, the avergage number of times a song by that artist has been played.

This formula is a way to compensate for the bias present when some artists have significantly more tracks than others. In my case, I have 329 songs by film composer Jerry Goldsmith. He has an inherent advantage over say, The Breeders, who have 63 songs in my library. By virtue of being so prolific, Jerry is naturally going to have more play counts.

After playing around with it a bit, I’ve made some interesting observations. Firstly, the top 21 most impactful artists in my library, with one exception, have only one or two songs. In cases where an artist has a low number of songs, each play count is “worth more” in relation to other artists. A single play by an artist with only one song nets that artist a full “point,” whereas as single play by an artist with 20 songs would only gain .05 points.

Fortunately, I’m able to set a threshold for display, keeping outliers from being counted. Here’s my top ten High Impact Artists. The artist must have at least five songs in my library. Additionally, this list only takes into account “popular music” (excludes classical, live shows and film/tv scores).

	Artist	# of Songs	Plays	Impact (avg plays / song)
1	Jet	7	73	10.4286
2	Rilo Kiley	55	512	9.3091
3	Glitter Mini 9	7	65	9.2857
4	National Skyline	10	92	9.2000
5	The Strokes	47	426	9.0638
6	String Theory	5	38	7.6000
7	Mercury Rev	34	257	7.5588
8	mouse on mars	154	1116	7.2468
9	Cex	37	264	7.1351
10	Bran Van 3000	34	228	6.7059

I’m very surprised to see Jet in the number one spot. That’s because my affinity for the band has waned to virtually nil. Still, it’s hard to argue with the dent Are You Gonna Be My Girl? made in my listening habits back in ’03. The ironic part is that overall, I didn’t care for Get Born, so I ended up deleting five of the thirteen songs on the album. Those weaker songs aren’t there to dilute Jet’s Impact by lowering the ratio of play counts to songs, so the band’s number seems artificially inflated.

Overall, I’m having quite the enjoyable time playing with iTunes Stats, even though it’s a little rough around the edges. I’m working on a couple of new reports and have even figured out how to get it to tell me which albums are missing ratings and how many ratings need to be completed.

Fun times ahead!

Playgrounds: Fun and interesting applications of Last.fm’s technology

The vast array of listening information available at Last.fm probably had a great deal to do with CBS’s decision to purchase the company. Though I’m wary of the deal, I’ve not lost all hope for the site. The Audioscrobbler technology behind it is some pretty fascinating stuff and the data it collects is open and available be analyzed, interpreted, shared and displayed in a lot of diverse applications.

Hopefully, now that CBS’s hand is in the cookie jar, this aspect of the service won’t change. As long as the data is accessible, here’s a number of cool things that can be harvested from Last.fm.

LastGraph

My waveform for 2007, through the beginning of June.

Lee Byron’s work on Last.fm data visualization made a fairly large splash on the net recently. The multi-colored waveforms showed undulating music tastes as artists’ popularity expands and contracts over time. It’s fascinating stuff.

And of course, after a moment of exclaiming "cool!" and "pretty!" the question on everyone’s mind was "How do I get one for myself?" Since Byron’s page was more of a demonstration and proof-of-concept, there was no way for someone to enter their username and get a graph of their own listening habits, leaving many visitors disgruntled.

Enter LastGraph, which does what all those disgruntled users were requesting, for whatever username you want. Results are offered in PDF and SVG formats, which are vector based, so you can zoom very close to see small-scale changes in data. The only thing that’s missing is the ability to track an individual artist within the ebb and flow of your listening. Specifically, I’d like to hover over a line and see that artist’s trends highlighted. That’s not going to happen with a PDF though. Oh well.

~~The site is running kinda bare-bones right now and there is a queue system in place. You may have to wait several hours before your PDF is ready to download. So be patient. It’s worth it.~~ The site’s performance has much improved since it launched.

Also note: the PDFs produced by the site do not render in Mac OS X’s Preview app, so be sure to view them in Acrobat.

Musicmapper’s Last.fm in Time

This chart shows my listening habits during the past 121 weeks (roughly the beginning of March 2005). Click to see larger.

Musicmapper’s Last.fm in Time generates a single graphic that displays a variety of data. The bar graphs in the background represents the total of each weeks play counts. Your top 50 artists are displayed, in rank order, on the right. The line graphs show how each of the top 50 have grown over time.

This can be useful for determining trends in your tastes and habits. In my case, before the 52 week mark, I see a lot flat-lined activity, especially among my top ten, that suddenly takes off. Also, I notice that Susuma Yakota, who I had never heard of before January this year, is in my top 50 and that he got there rather quickly. There is a very steep curve for him starting 23 weeks ago.

Tuneglue relationship explorer

Click for full size.

Tuneglue creates a web of related bands and artists. Start with one artist or band, expand the results to find similar artists or bands, then do the same to those. With four or five clicks, you’ll have a large interconnected web of new bands to explore based on similarities and relationships to your tastes. It’s a neat visual metaphor of musical interest and a good jumping off point for new music recommendations. The lack of sound samples limits its usefulness as an exploratiom tool, though the map is still fun to play with.

One killer app of the site, however, is missing. I talk of course, about a "six degrees" linker. It would be very cool to input two artists and see how many jumps are necessary to connect to two. For example, it takes four jumps to connect Mogwai to the Strokes (Mogwai » Radiohead » The Beatles » The White Stripes » The Strokes, according to Tuneglue). I figured that out on my own, but it would be nice of the site to do it for me.

Last.fm tools by Anthony Liekens

cloud of recommendations

This site features a number of Last.fm related tools. My favorite is the artist recommendation cloud, which generates a number of suggestions for musical exploration based on your top artists. Higher recommendations appear at a larger type size. Recommendations can be based on stats from your overall list, the past 12, 6 or 3 months or the past week.

Also be sure to check out your eclectic rating. I scored an 80 out of 100.

How compatible are your tastes with a radio station?

sekrit

Last.fm user profile bbc6music is, you guessed it, created by the songs BBC Radio 6 (6music) plays on air. Though not every song that the station broadcast gets uploaded to Last.fm, the user profile still manages to add about 100 play counts per day. As of August 2011, the station has an accumulated track count of nearly 380,000. The most played artist is David Bowie.

Mainstream-o-Meter

mainstreamnes

Finally, there’s the Mainstream-o-Meter, which compares your top stats with the overall most played artists site-wide. Each of your most-listened-to artists are given a weighted score which is then used to calculate your overall "mainstreamness."

Last.fm is certainly a vast treasure trove of information, so hop to it and get exploring.

In search of a definitive album rating formula

When it comes to my iTunes library, I’m a regular statistics nut. Sure, my library exists primarily for my own enjoyment, but it contains so much organically-compiled data about my habits and tastes that I can’t help but want to take a look at it and find out what the data says about my interests.

But for a while now, I’ve struggled to quantify, tabulate and analyze the overall sense of my library. Which of my albums albums are truly the greatest? Which artists, when the sum of their parts are combined, are really my favorites? And by how much? I want numbers.

None of the iTunes stats options available at the moment give me the type of results that I want. The Album Ranking AppleScript provides a simple average that skews toward albums with fewer tracks. SuperAnalyzer provides a top 10 list that is skewed toward albums with more tracks.

Most iTunes stats tools simply provide averages or totals of play counts and/or star ratings. Averages, while somewhat useful, can be misleading. An album could have a handful of awesome songs and a bunch of filler and still rank as well as and album that’s consistently good, but without much breakout material.

And that can be frustrating to me, because, in terms of album or artist worth, I tend to value the ones with consistent performance.

Take, for example, my recent run-down of Air’s discography, specifically the albums 10000 Hz Legend and The Virgin Suicides. After many years of listening, my artistic impression is that Virgin Suicides is ever so slightly the better of the two. The songs on Legend vary from excellent to clunkers. Suicides is overall pretty good, with only one exceptional track. However, averaging my ratings shows that Suicides is a 3.85 while Legend rates as an even 4.

So, to reward albums that don’t veer wildly around the quality wheel, I’ve developed my own album rating formula that takes into account the consistency of all the star ratings on a given album.

The Formula

album rating = (mean of all songs + median of all songs) - standard deviation of the set

The mean sums up the whole of the album. The median shows the state of the album at its core. The standard deviation indicates the variety of the individual ratings. The result is a number on a scale of 1 to 10. (Alternately, divide that number by 2 to return the result to a 5-star scale).

Let’s take a look at the formula in action. Suppose we have two albums with twelve songs each. The first is generally excellent, but varies in quality. The second is good stuff throughout.

	Ex. 1	Ex. 2
	5	4
	4	4
	5	4
	2	4
	4	4
	5	4
	5	4
	2	4
	5	4
	3	4
	5	4
	3	4
Mean	4	4
Median	4.5	4

total	8.5	8

STDEV	1.21	0

Score	7.29	8

This table shows the individual star ratings for the two theoretical albums, as well as all the statistical data, as calculated by Excel. As you can see, both albums average score is the same (4) and Ex 1 even has a higher median than Ex 2. But, because the quality of Ex 1’s songs vary a great deal, its standard deviation is substantial, so much so that its album rating becomes 7.29 (or 3.645 on a 5-star scale) when my formula is applied. Ex 2’s score suffers no penalty and its score remains 8 (4). In this case, the standard deviation awarded Ex 2 a bonus for being of uniform quality.

Let’s take a real world example, the two Air albums I mentioned above.

	10 kHz Legend	Virgin Suicides
	4	4
	5	4
	4	4
	5	3
	5	3
	4	4
	3	5
	4	4
	3	4
	3	4
	4	4
		4
		3
Mean	4	3.84
Median	4	4

total	8	7.84

STDEV	0.77	0.55

Score	7.23	7.29

When the formula is applied to my ratings for each, the scores for 10000 Hz Legend and The Virgin Suicides become 7.23 (3.62) and 7.29 (3.65), respectively. So factoring in the standard deviation results in a score that more closely reflect my thoughts of those two albums.

So what does this mean? I’m not sure exactly. In practice, I could whip up some listy goodness and see which albums are truly my favorites. A comprehensive analysis would be cool. I’d love to see the distribution of my album ratings. However, that would require more programming skills than I have. Though that could be a good project to help me learn.

Out of curiosity though, I have picked 10 albums, just to see how they rate. One provision, of course, is that every song on an album must have a rating before the album score can be calculated. These ratings are on a 5-star scale.

	AVG	My Score
Radiohead – OK Computer	4.5	4.41
Air [french band] – Moon Safari	4.5	4.39
Nirvana – Nevermind	4.5	4.24
Mouse on Mars – Radical Connector	4.33	4.23
Ratatat – Ratatat	4.45	3.97
Nine Inch Nails – With Teeth	4.31	3.77
The Strokes – Is this it?	4.09	3.7
LCD Soundsystem – LCD Soundsystem	4	3.68
Basement Jaxx – Remedy	3.73	3.51
Prefuse 73 – One Word Extinguisher	3.82	3.47
Weezer – Make Believe	3.58	3.21

This is by no means a top 10 list, but it is interesting to see where things ended up. It’s also interesting to see how minor fluctuations in star ratings can change the final score. For instance, if that Ratatat album had one more 5 star song in place of a 4 star song, its median number would become 5 and its album score would jump to 4.51. Lower a 5 star to a 4 star and the score only drops slightly to 3.93. I don’t know if this is a flaw in the formula or a reward for albums that have a lot of good songs.

Problems and issues

Small data sets. These are troublesome in all statistical circumstances and this formula is no different. Albums with only one song will, by definition, not have a mean, median or standard deviation, and that kills the formula with a divide-by-zero error. Also, because the formula uses the average rating as a component, albums with a low number of songs will tend to skew one way or the other.

In my library, Boards of Canada’s EP In A Beautiful Place Out In The Country has four fantastic songs and ranks at 4.63, higher than anything on that list above. As a release, I’d say that’s accurate, but I’m sure it doesn’t surpass OK Computer. I would be interested to see a chart of how the album score changes as the number of tracks on an album increases.

Additionally, I haven’t figured out a way to rank partial albums, i.e. albums where I either don’t own all the songs or albums where I’ve deleted songs I didn’t like. For now, I’m just excluding them altogether.

Still, I’m fairly pleased with the results I’ve been getting as I run various albums through the formula. It’s working for me and my own song rating system, but I’m curious to see how it works with someone else’s.

Fortunately, Webomatica has posted his song-by-song ratings for The Beatles’ Sgt. Pepper’s Lonely Hearts Club Band. Using his numbers, the average for the album is 4.38, while my formula renders a 4.28. I’d say that’s a consistently good album.

Here’s a Microsoft Excel file you can download. Plug in your star ratings to find the album score. AlbumScore.zip

iTunesRegistry is now open source

itunesregistry

The ultimate in iTunes statistics and library analysis is once again live and on faster/better hardware. Yes, my music-listening friends, the iTunesRegistry is back and better than ever.

Faster processing, more graphs, interesting facts. The works.

It’s open source to boot. The site runs on PHP+MYSQL and now you too can grab the code, muck with it and enhance it on your own. Mac users can probably run it locally using their built-in versions of Apache.

If you’ve not seen the site before, head on over and create an account. Then upload your iTunes XML and let the statistics begin!