library

Smart Playlist Idea: Eldest Tunes (that need attention)

This is a fun little playlist.

In my last Smart Playlist example, I showed how to create a list of the most recently added songs that had not reached a certain play count. Today’s list takes the opposite tack: what are the oldest songs in the library that haven’t been played a certain number of times. I call it “Eldest Tunes” and it is a great way to give attention to songs in your library that have, for whatever reason, found themselves neglected. I’ve found it to be particularly good for reminding yourself just how much you loved those songs of yesteryear.

Before I continue though, I have to point out that if you started your music collection before July 2002, then this playlist works best if you’ve back-dated your library’s Date Added field.

Setting the playlist up is actually ridiculously easy. Here’s a screen shot of mine:

Playlist is tunequest

Songs must be in my master playlist. If you haven’t set up a master playlist, then you can leave this criterion out.

Play count is less than 4

This number is arbitrary and can be whatever you want. In my case, I want to find songs that have been played 0-3 times. When a song reaches 4 play counts, it can no longer appear on this list.

Last Played is not in the last 3 months

This is a recycling mechanism. If, like in my example above, you’ve set your threshold to 4 and then listen to a song that only has 1 play count, the count increases to 2 and the songs stays on the playlist.

When I first put this playlist together, I found that I was listening to the same songs over and over again for that reason. It was taking multiple listens to rise above the 4 threshold that I had set. So I added this rule that says once a song has been played it is “embargoed” for 3 months, giving other songs an opportunity to be heard.

Finally, the last two criteria:

This is the actual engine of the playlist. Limit your playlist to the number of songs you want, but make sure you select least recently added. That least recent part is why it’s important to have accurate info in your Date Added fields. iTunes uses the date in that field to determine which songs will go on this playlist and in what order.

In my case, the oldest songs that currently appear on my Eldest Tunes playlist are from Primus’ Pork Soda and Blind Melon’s debut. I originally bought them over the summer 1993 (backdated in iTunes), but I’ve hardly listened to them in the past few years, so their play counts are low. But now that I’ve seen that they’ve been under-appreciated for so long, I can take steps to give them them proper consideration.

As for your playlists, I wish you happy listening!

Cumulative lifetime play counts

A rambling, self-indulgent, inconsequential post about habits, statistics, speculations, accumulation and missing data.

I can’t help but be disappointed that I can’t see lifetime stats for my music listening habits. In these days of play count-tracking programs like iTunes and websites like Last.fm, it’s easy to get caught up in the musical trends of your life. It’s especially interesting when you look at the numbers and discover that you perhaps don’t like a certain style of music as much as you thought you did or you find that you listen to a particular band much more than you would have guessed.

The problem is that your revelations are only going to be as good as the data you’ve collected. I’ve been a “serious music listener” for about 16 years, yet Last.fm has only been tracking my habits for three and my iTunes library only goes back six. I have ten years worth of listening that I will never have any way to quantify simply because the data was never collected.

Missing data, of course, skews results and in this case, snapshots of my habits are skewed in favor of recent years, especially when looking a cumulative lifetime stats. Using data from my library as it stands today, I put together a graph of my most popular years in music. I’ve been for the most part, a contemporary music listener, so the vast majority of my library contains music released from 1993-2008, adding new releases each year.

I calculated the total number of play counts received by all songs in my library that were released in a given year. Here’s the result:

itunes play counts by year

This graph shows the distribution of all my play counts generated since July of 2002 (when iTunes began recording them). We see a peak in 2001 and a general downward slope since.

My explanation for the shape of the graph is that, as years come and go and a music library grows, newer music receives more attention than older music. Familiar tunes give way to new acquisitions and explorations. However, those old tunes never entirely go away; they continue to co-exist with the new ones. As the years pile up, each one’s presence is diluted among the rest and it becomes and increasingly uphill struggle to for the songs of a new year to reach parity with those of the past.

So in this particular graph, I attribute the 2001 peak to the simple coincidence that the songs from 2001-early 2003 were in high rotation at the time that iTunes started tracking play stats. As a result, the initial rate of change for those songs was quite high. And even though the rate at which those songs get played has decreased (exponentially) over time, the songs from other years still have to compete with them for attention, so we find a general trend decreasing cumulative play counts.

Average Play Count by Year

Further evidence of this idea can be seen in the average play count for the songs of each year. There’s a bump in the 2003-2004 area, reflecting the idea that older songs tend to accumulate more play counts over time.

I can’t help but wonder what that play count graph would look like if iTunes had been released in the early 1990s? How much cumulative lifetime play would we see throughout the years?

Of course, there’s no way to figure that out. That information is trapped in the fog of memory, stored in transitory listenings of cassette and compact disc. But while that individual play counts may be lost forever, it might not be impossible to make a decent educated cumulative guess.

I’ll start with the premise that from the years 1993-2001, I averaged a mere 10 songs per day between school bus rides, studying, hanging out, commuting and partying from early high school, through college and my entry into the workforce. That’s probably a conservative estimate, considering the general lengths of my bus rides and commutes. Heck, I’ve managed to generate nearly as many plays in the past 6 months, and I’ve lately been slacking on my music listening in favor of podcasts and audiobooks. But 10 is a good number, so I’ll stick with it.

So, at 10 songs per day, that’s 3650 plays per year. Consider the state of my collection in those early years. Throughout high school and into college, I managed to add records to my library at an average rate of one per week. If iTunes had been around at the time, play counts by now would be heavily concentrated in those early additions, with the highest concentrations being in the earliest records I bought.

By the end of the first year, my estimated 3650 plays would be spread among a mere 500ish songs, an average of 7.3 per songs. By the end of the next year, another 3650 plays would be spread out among about 1000 songs, 3.6 per song. Except that I expect that drop off in older songs to be exponential, not linear.

After some more conjecture and guess work, I extrapolated the accumulation of play counts over the years. After some number-crunching, I had a graph that looks like this:

cumulative play counts by year adjusted

The blue line is the same as above, showing the cumulative distribution of play counts by year of release in my iTunes library. The green line represents what the graph would look like if my estimated historical plays were added to the existing totals.

What does this totally unscientific, made up graph tell me? Basically what I already suspected: that I’d have to stop listening to my older tunes altogether and for a long time if I ever wanted current tunes to “catch up.” Of course, in the time it would take to do that, future tunes would be at a deficit. So really, while it’s a somewhat nice visualization, in reality it will have no bearing on my future plans.

iTunes Tip: Back-date the songs in your library

I’ve mentioned before that one of my standard library organization procedures is to back-date the “Date Added” field for all the songs in my iTunes library. That is, if I originally received an album for my birthday in 1999, I make sure the Date Added field in my library is my birthday, 1999. Same goes for every CD I’ve bought or mp3 I’ve downloaded.

Unfortunately, Apple for whatever reason, has decided that the Date Added field should not be user-modifiable. You can’t change it yourself, either manually or via AppleScript. And honestly, I’m tempted to think of that behavior as a bug/product defect. In this digital age, where at some point each and every iTunes user *will* have to rebuild or replace their library after some sort of data catastrophe, it seems like an obvious feature to be able to reconstruct one’s musical history chronologically. Why should users have to settle for the post-reconstruction dates for albums they’ve actually owned for years?

Well, there’s a bit of a workaround, but it is a tedious one. So make sure you regularly backup your iTunes Library file so that you don’t have to do it all over again in the event of a hard drive crash. I use my .mac/Mobile Me account to upload my library file to my iDisk every night at midnight.

How To

The secret is that iTunes relies on your computer’s system clock to assign the Date Added to songs in the library. So back-dating is as “simple” as changing your computer’s clock, dragging your music files into iTunes, then resetting the clock to the current time.

If you have hundreds of albums to do this with, the procedure can get quickly tiresome. Unfortunately, there is no way to automate it. Plus, if you are trying to fix songs that are already in your library, you have to remove them, change the system date, then re-add them. In those cases, make sure you note the play counts and star ratings, because you’ll have to re-enter those manually. Like I said, tedious.

But all that work is worth it when, in the span of five seconds, you conjure up a Smart Playlist called Best Music from High School:

Date added is in the range 8/15/1993 - 5/15/1997 My Rating is 5 Stars

That is truly awesome.

One warning though:

If you are using Mac OS X 10.5 (Leopard) and you use iCal alarms, be sure to disable them in Preferences before setting your clock back. I found this out the hard way when I was suddenly flooded by couple hundred notifications for events that had already passed. It seems that iCal travels back in time with you, then when you return to the present, it feels the need to update you on all the stuff you missed.

Smart Playlist Ideas: Master List and Newest Tunes

With more than 16,000 songs to manage, there is no more essential a tool in my library than iTunes’ Smart Playlists. From building simple playlists for listening to creating complex queries for examination, Smart Playlists turn what would be a tedious burden into a trivial task. At the moment, I have more than 50 of them slicing, organizing and corralling my expansive collection of tunes into an easily navigable, self-sustaining ecosystem of music.

It seems a shame to keep all those playlists to myself when they could be benefiting other iTunes users, helping them find new ways to organize and listen to their libraries. On this first of a new tunequest segment, I’ll share some of the criteria for playlists that I’ve developed to help manage my library.

This first installment is a two-for. We’ll start with the foundation of my listening habits: the master tunequest list.

The master tunequest list was one of the earliest Smart Playlists I created. Its job is to act as a filter on the main iTunes library and determine which files are eligible for inclusion in other Smart Playlists. The premise is that only properly tagged music without any playback glitches should be included in subsequent lists.

Podcasts, audiobooks, iTunes U courses, videos and other files that are not strictly musical should be excluded from the standard rotation. But how to do it?

master tunequest smart playlist selectors

This is the actual criteria for my master list. There are multiple ways to create one, you just have to tell iTunes what to exclude. Here’s a brief description of the selections I’ve made:

Date Added is not 1/3/02.

I had a major hard drive crash on 1/2/02 which wiped out an early version of my Library. When I restored it from back up the next day, I discovered that the id3 tags for 5 years worth of mp3s had only been made on the library, not the back ups. I took the crash as an opportunity to re-evaluate my songs and make sure that all my files were “up to code” with proper tags and acceptable bitrates.

When Smart Playlists were introduced later that year, I didn’t want songs that I hadn’t checked going into my rotation. With the Date Added for all 7500 songs (my library size at the time) set to 1/3/02, I was easily able to exclude those songs that were pending evaluation. After evaluation, I re-imported my songs with the appropriate Date Added and they were automatically re-included in the master list. Today, about 200 rather obscure songs remain that I haven’t had the wherewithal to track down, so excluded they sit.

Date Added is a powerful tool for segmenting your library based on time period. You can set it to before, after or between dates to isolate just those songs, like a “Songs of Summer 2005” playlist (Date Added is in the range 6/1/05 and 9/1/05).

My Rating is not 1 Star

Rating a song 1 star is my arbitrary way of taking a song out of circulation. If I notice a song has glitches or that its tags have errors, I’ll mark it as 1 star until such time as I can fix it.

Podcast is false

Keeps podcasts out.

Playlist is not SpokenAudio

I have several playlists of just spoken audio that isn’t an iTunes Audiobook: iTunes U courses, comedy albums and other spoken word pieces. These playlists are kept in a sidebar folder called “SpokenAudio,” which iTunes treats as a single unified playlist for the purposes of Smart selecting.

You can create some complex hierarchies and conditional listening schemes using nested folders and playlists.

Kind does not contain video

Keeps all video content off the list. Movies, TV shows and video podcasts are not welcome here.

Playlist is not Audiobooks

Keeps files from iTunes’ Audiobooks sidebar from mixing with music. iTunes offers similar selectors for Movies and TV Shows as another way to exclude video content.

Genre is not Podcast

Another method to exclude podcasts from everyday listening.

Playlist is not xmas

I have a playlist dedicated to Christmas and other holiday tunes. This selector keeps it out of the way for ~330 days of the year. I remove it on or around Thanksgiving and replace at after New Year’s.

Now that we’ve cordoned off our healthy files, we can slice and sub-slice it to fit as many different listening schemes as we have whims. This is a relatively recent playlist I’ve been using to handle new music.

Newest Tunes

Some music falls through the cracks around here. Some albums get overshadowed and as time marches on, they don’t get the attention they deserve, receiving only cursory glances before being supplanted by newer music. This playlist is meant to allow all new acquisitions to have an full opportunity for listening.

It takes 4 parts:

Playlist is master list

The master list ensures that only “safe” music is eligible for inclusion.

Play Count is less than 4

I generally feel that 3 plays per song is enough to consider a new album adequately vetted. You can adjust it to suit your tastes.

Limit to 150 songs selected by Most Recently Added

This limiter means that the 150 most recently added songs that have been played 0-3 times (and are on the master list) will be included in the playlist. When one song on the list reaches 4 plays, it disappears from the list and is replaced by an older song that meets the criteria. When new songs are added to the library, they automatically appear on this playlist, pushing off older songs.

Since I implemented this playlist, I’ve been able to keep a handle on the inflow of new music into my library. Enjoy.

Impact report update

My original High Impact formula had a fundamental flaw, which I think I may have fixed.

I spent the last post talking about the albums that made the biggest “impact” on me during 2007, but what exactly does that mean? Over the summer, I came up with the general concept, which basically defined impact as the average number of times any particular song from an album or artist in my iTunes library had been played, showing who has received the most attention relative to their presence in my library. Add together the play counts of all songs by an artist or on an album, then divide by the number of songs. There’s the impact score.

While this method produced some interesting results, it suffers from a substantial deficiency: it grossly inflates the relative impact of artists and albums that have a low number of songs. Artists or albums with a lot of songs have to be played a lot more in order to keep up. I noted this in my original post on the subject:

In cases where an artist has a low number of songs, each play count is “worth more” in relation to other artists. A single play by an artist with only one song nets that artist a full “point,” whereas as single play by an artist with 20 songs would only gain .05 points.

The solution I devised at the time was to set a threshold for inclusion, eg. artists must have more than 5 songs in my library to be ranked.

I never really liked that workaround because the threshold was arbitrary and it excluded artists who should really have been able to be ranked. So I spent a bit time recently tweaking the formula and I think found something that works to my satisfaction:

Total Play Counts [squared] divided by the number of songs.

This is essentially the same as the original formula, except that the total play counts an artist or album has received is multiplied by itself. The effect of this is to give the impact of a single play more weight as the number of songs increases. My thinking goes like this: albums with a lot of songs should rightly have a larger impact vs those which have fewer, even when the average play count is the same.

Suppose we have an album with 10 songs on it and an EP that has 4. Each song on both albums has been played twice.

Using the original formula:

Album: total play count (10 * 2) / number of songs (10) = 2 EP: total play count (4 * 2) / number of songs (4) = 2

Eight (8) plays gives the EP the same impact as the album that has received twenty (20).

Now the new formula:

Album: total play count squared (10 * 2)(10 * 2) / 10 = 40 EP: total play count squared (4 * 2)(4 * 2) / 4 = 16

Even though both recordings have been listened to the same number of times, the album’s larger footprint leaves a greater impact score than the EP.

The best analogy I can think of is mass vs speed. More songs equal greater “mass.” More plays equal greater “speed.” Just as lighter objects have to travel faster to hit with the same amount of force as heavier objects, an artist with a lighter presence in my library has to be played more times to have the same impact as an artist with a lot songs. A ping pong ball must travel at higher speeds to equal the same force as a baseball.

This table below shows the formula in action.

Rank	Artist	# of Songs	Plays	AVG	Impact
61	The Cardigans	92	255	2.77	706.794
62	To Rococo Rot	34	155	4.56	706.618
63	Tomoyasu Hotei	1	26	26	676.00
64	Molotov vs Dub Pistols	1	26	26	676.00
65	National Skyline	17	107	6.29	673.47

The two singles by Tomoyasu Hotei and Molotov vs Dub Pistols happen to be in the top 20 most played songs in my library (out of ~16000). That showing places each of them relatively high on the list, but not overwhelmingly so, considering the differences between the various AVG play counts. That’s an equitable result I’m pretty happy with.

iTunes Report: High Impact Artists

I’ve spent the past couple days playing around with Alex King’s iTunes Stats program. It’s written in PHP/MYSQL and requires a web server to run. With the MAMP one-click server running on my PowerBook, I had little trouble installing the program (though I did have to substantially increase the PHP timeout setting so it could handle my large library).

iTunes Stats reads XML files, one can load an entire iTunes library or export playlists for more selective examinations.

The program also comes with a number of built-in reports, such as Most Played Albums, Most Played Artists. Top Rated Albums/Artists and the option to weight by number of rated songs and play counts. The engine supports adding custom reports, so if you are motivated enough, you can create new methods of analyzing your music.

My PHPMYSQL-fu is not very strong virtually non-existent. Still, I was able to cobble together my own metric: Artist Impact.

An Artist’s Impact is measured by the total number of play counts they have received divided by the number of songs that artist has in the library. For example, Artist A has 20 songs in the library and those 20 songs have been played a total of 100 times. Thus, 100/20 = 5. That’s Artist A’s Impact. Basically, it tells you, the avergage number of times a song by that artist has been played.

This formula is a way to compensate for the bias present when some artists have significantly more tracks than others. In my case, I have 329 songs by film composer Jerry Goldsmith. He has an inherent advantage over say, The Breeders, who have 63 songs in my library. By virtue of being so prolific, Jerry is naturally going to have more play counts.

After playing around with it a bit, I’ve made some interesting observations. Firstly, the top 21 most impactful artists in my library, with one exception, have only one or two songs. In cases where an artist has a low number of songs, each play count is “worth more” in relation to other artists. A single play by an artist with only one song nets that artist a full “point,” whereas as single play by an artist with 20 songs would only gain .05 points.

Fortunately, I’m able to set a threshold for display, keeping outliers from being counted. Here’s my top ten High Impact Artists. The artist must have at least five songs in my library. Additionally, this list only takes into account “popular music” (excludes classical, live shows and film/tv scores).

	Artist	# of Songs	Plays	Impact (avg plays / song)
1	Jet	7	73	10.4286
2	Rilo Kiley	55	512	9.3091
3	Glitter Mini 9	7	65	9.2857
4	National Skyline	10	92	9.2000
5	The Strokes	47	426	9.0638
6	String Theory	5	38	7.6000
7	Mercury Rev	34	257	7.5588
8	mouse on mars	154	1116	7.2468
9	Cex	37	264	7.1351
10	Bran Van 3000	34	228	6.7059

I’m very surprised to see Jet in the number one spot. That’s because my affinity for the band has waned to virtually nil. Still, it’s hard to argue with the dent Are You Gonna Be My Girl? made in my listening habits back in ’03. The ironic part is that overall, I didn’t care for Get Born, so I ended up deleting five of the thirteen songs on the album. Those weaker songs aren’t there to dilute Jet’s Impact by lowering the ratio of play counts to songs, so the band’s number seems artificially inflated.

Overall, I’m having quite the enjoyable time playing with iTunes Stats, even though it’s a little rough around the edges. I’m working on a couple of new reports and have even figured out how to get it to tell me which albums are missing ratings and how many ratings need to be completed.

Fun times ahead!

8 Ways to Improve the iPod (and could be done with a firmware update)

The iPod is supposed to be “iTunes to go” but as the little music player has advanced over the years, it still lags behind in some relatively basic features, features that have been a part of the desktop program for some time. iTunes’ capabilities seem to be constantly improved and refined; its portable counterpart’s behavior has remained relative unchanged, even as it has gained photo and video support.

Forget touchscreens and Bluetooth, FLAC and DivX; here, I present a list of the iPod’s more troublesome foibles, all of which could be overcome with a firmware update, making it an even better music player.

Toggle display of the Composer tag

This is something I’ve wanted since Apple added the Composer field to iTunes five years ago: A display of the composer when listening to classical music. The 5G iPods have more than enough screen real estate to accommodate an extra line of text. It makes no sense that after all this time and after adding a way to browse and select by composer, Apple still doesn’t allow a way to view it while playing. Classical music aficionados have to either do without or devise elaborate tagging systems to see who the composer of a piece is.

Of course, not everyone has need for composer display. There certainly are people who don’t appreciate Prokofiev. Also, the field is often populated with junk from Gracenote/CBBD. A simple toggle in the iPod settings would fix that. Those of us who want to see the composer can turn it on and those who don’t can leave it off.

no composer visible
At a glance, there’s no telling who the composer is. One hack, though, would be to embed the composer name in the album artwork.

Support for the Album Artist field

iTunes 7 introduced a new data field to the song info dialogue box: Album Artist. Apple says it’s for assigning a primary artist to an album with multiple artists. It signifies a way to separate the artists producing the work from the artists performing it.

It’s a great idea for classical works that have a featured soloist in addition to the orchestra or when one artist is a featured guest on someone else’s song, eg, William Shatner featuring Henry Rollins. In this case, William Shatner is the primary artist and would be to sole “Album Artist” while “William Shatner featuring Henry Rollins” are the performing artists.

The tag works well in iTunes, keeping song listing nicely and tidily organized. The iPod, however, still separates “William Shatner” from “William Shatner featuring Henry Rollins,” leading to a cluttered interface that is difficult to use. Most of my music listening is done via iPod, so Album Artist remains under-utilized.

Album Artist would be a very useful tag. It would even solve my dilemma for tagging remix/dj albums. But without iPod support, the tag is DOA.

two shatners
Despite having the same Album Artist, these listings are still displayed by regular Artist.

Full Support for Sort fields. (accomplished)

UPDATE 3/19/08: Firmware version 1.3 for the Fifth Generation iPod adds support adds support for Sort Album and Sort Composer.

Other options recently introduced into iTunes but not into the iPod are customizable Sort Fields, which let you control how iTunes alphabetizes your artist and album listings.

By default, the iPod is smart enough to ignore “A,” “An” and “The” at the beginning of artist names. The Chemical Brothers are sorted with the C’s, for example. Starting with iTunes 7.1, you can customize the Sort name for Artists, Albums, Songs, Album Artists, Composers and TV Shows.

If you want Fiona Apple to appear with the A’s rather than the F’s, just set the Sort Artist to “Apple, Fiona” and you’ll soon see Fiona next to Aphex Twin.

It’s pretty cool, but…… on the iPod, it only works with Artists. You can customize all the albums and composers in your library and Gustav Mahler will still be chillin’ with the G’s and The Colour and The Shape will still be sorted with the T’s.

the thes
The “thes” like to hang out together in album view.

Browsable playlists

Music libraries get larger every day it seems. And the iPod’s hard drive does its best to keep up. At 80 GB, the device can hold a month or so of continuous music. For myself and others with large libraries, it’s effortless to create Smart Playlists that contain hundreds or thousands of songs based on specific criteria. Navigating those playlists can be nearly impossible as they show naught but a long list of song titles.

In my library, creating a Smart Playlist of Ambient music from between 1990 to 2000 returns 305 songs from 44 albums by 11 artists. Viewing the playlist on my iPod is a jumble of songs. I would love the option to sort and browse the artists and albums in a playlist.

Perhaps, when you select a playlist, the iPod displays an entry at the top of the song list: “Browse this playlist.”

Full-screen album art

When in full screen mode, I want the iPod to display album art as large as it can, no margins, no scaling. Just like when browsing photos, I want the image to take up the entire screen. This, the iPod can already sort of do…… if you plug it into an iPod HiFi, Apple’s own speaker system. I would like it to be standard. For more, read this recent rant.

Bonus Wishlist

I’m not annoyed by these missing features, but if they were real, I’d find them useful:

iPod Party Shuffle

A more limited version of iTunes’ Party Shuffle. When you’re shuffling, this would let you see a handful of upcoming songs. You could skip ones you don’t want to hear.

Profiles/Pre-sets

My listening preferences are different depending on whether I’m at work, in the car, at the gym, or moseying around the house. At the gym, I like to shuffle by song while at work I like to shuffle by album. When listening to ear buds, I like to use the bass booster EQ, but the bass response in my car is a little heavy, so I like to turn on the bass reducer.

It would be convenient to save different settings configurations for easy switching.

Grouping behavior that makes sense

“Grouping” is the red-headed stepchild of ID3 fields. No one *really* knows what it’s for or how to use it. Ostensibly, it’s for creating “groups” or subsets of related songs within an album. But it wasn’t until iTunes 7 that you could do anything with it (you can shuffle by Grouping).

It seems to me that an effective behavior for songs with the same Grouping to be “always keep these songs together.” For example, Mouse on Mars’ Varcharz has one song, One Day Not Today, that is broken into 12 tracks. Give all 12 tracks the same Grouping, “One Day Not Today” and the iPod would know to start at the first track and play through all of them sequentially, even when shuffling.

Hopefully, one day, these wishes will come true. I still love my iPod, but I’m looking for reasons to love it more.