Crunching the Math On iTunes 276
markmcb writes "OmniNerd has posted an interesting article about the statistical math behind iTunes. The author makes some interesting observations concerning the same song playing twice in a row during party shuffle play, the impact that star ratings have on playback, and comparisons with plain old random play (star ratings not considered)." From the article: "To test the option's preference for 5-stars, I created a short playlist of six songs: one from each different star rating and a song left un-rated. The songs were from the same genre and artist and were changed to be only one second in duration. After resetting the play count to zero, I hit play and left my desk for the weekend. To satisfy a little more curiosity, I ran the same songs once more on a different weekend without selecting the option to play higher rated songs more often. Monday morning the play counts were as shown in Table 1."
Re:iTunes is a monopoly (Score:2, Informative)
Re:I am not sure I see what he sees (Score:5, Informative)
Underlying formula (Score:5, Informative)
From their results, I'd venture a guess as to the underlying algorithm:
Each song is given a number of points equal to (rating + 1). Then the probability of the song being played is (song rating)/(total points).
Or, to put more succinctly:
prob(song) = (rating)/(n + sum(i=1..n)(rating(i)))That yields probabilities in the given test case of:
5 star - .285 .238 .190 .143 .095 .048
4 star -
3 star -
2 star -
1 star -
0 star -
Which is reasonably close to what the author found. Heck, if I were implementing that feature, it's what I'd try first...
Some calculations errors in my opinion.. (Score:5, Informative)
A way to calculate the odds that 2% will be played in the next 50 songs doesn't work 50* (2/100) = 100% as the author does, and neither 25*(2*100) = 50% is correct.
The correct calculations are: 1-(98/100)^50 = 63% and 1-(98/100)^25 = 39%.
This way you calculate the odds a song will be played at least once in the next 50 or 25 songs.
If you want to calculate the odds the song will be played exactly once in the next 50 or 25 songs:
50 * (2/100) * ((98/100)^49) = 37% or 25 * (2/100) * ((98/100)^24) = 31%.
I guess that's all..
From the article...trick of the mind (Score:4, Informative)
"Many claim to still see patterns as iTunes rambles through their music collection, but the majority of these patterns are simply multiple songs from the same artist. Think of it this way: If you have 2000 songs and 40 of them are from the same artist, there is always a 2% chance of hearing them next with random play. So right after one of their songs finishes, odds almost guarantee they will be played again within the next 50 songs and show a 50% chance they will play again within the next 25 songs. It's simply the mind's tendency to find a pattern that makes you think iTunes has a preference."
Re:Interesting (Score:3, Informative)
I can't really say how well this works in practice, or which programs support it, because I don't use the feature myself. However, I suspect it would work better than an explicit rating system, much like bayesian spam filters work better than explicit ones.
Re:Ok... (Score:2, Informative)
Like most random numbers generated by computer they aren't really random. They're pseudo-random, generated by a deterministic algorithm and having the appearance of randomness. The less processing power you dedicate to the algorithm, the less random the numbers will appear on inspection. The iTune statistics can only be as good as the algorithm they used to generate their (pseudo-)random numbers.
You can find true random numbers here [random.org] and also some more information on pseudo-random and random numbers.
Re:Interesting (Score:2, Informative)
Re:Modal Music (Score:3, Informative)
4 Classic Rock songs
3 Blues Songs
3 Folk songs
4 Female Rockers
3 Grunge
Not entirely true, and it depends on the station (as you stated). Some stations make it a point *not* to put songs with female lead singers together; *not* to put songs from the same R&B/Dance/whatever genre together; *not* to put songs from solo artists next to each other. And so on. And don't forget issues with playing more than one old song after another.
Again, it all depends on the station and format and what they're trying to do at the time, but as a general rule of thumb, there's a lot more analysis and thought that goes into creating a radio playlist than what iTunes automatically does, or by building "blocks" of music (Twofer Tuesdays, Rocktober, and Deep Cut Thursdays aside). It's all about how to best reach the desired target audience for the advertisers.
In other words, I can't imagine any station that's trying to make money that would rely on solely using an iTunes algorithm (current or proposed) for scheduling music.
Besides, what would the stations do with all the music scheduling software they already use?
Re:Why Assume a Bell Curve? (Score:3, Informative)
When you look at data, particularly from modest numbers of samples, it seldom fits the bell curve nicely. Often it is multimodal.
However, the Guassian (bell curve) distribution has one important property: it is the distribution that (given a number of stipulations like having a finite integral over any interval, and there being no set uppler or lower limit to the variable) has the highest entropy. It is, in a sense, the most random of random variable distributions.
So, the reason you tend to fall back on the Gaussian distribution as an assumption is that it is the distribution that, should you assume it, means you are makign the fewest possible assumptions. Any other distribution chosen must be chosen with a reason.
Now, some judgement has to be exercised still. If I am choosing a distribution to represent a throw of a die, I'll choose the Uniform distribution as being the one that assumes the least. That is because it fails to meet one of the criteria for choosing the Gaussian: there IS a lower limit(1) and an upper limit (6) to the variable. Superficially, this would seem to resemble the iTunes rating system of no rating = 0 through best = 5 (I think). However, note that the rating is actually just a truncated record of an underlying, continuous variable: how much you like or dislike a song. There's no actual limit on that. Sometimes, it is better in situations like this to assume log-normal (geometric) distributions. In a sense we're talking about artifacts of our scale specification rather than anything fundamental in the underlying situation.
In any case the fact that the user has selected the songs has no actual bearing on the usefulness of Gaussian assumption. I'm dredging this up from a course I took many years ago at MIT called "Stochastic Processes" taught by Giancarlo Rota (18.313 I think it was). If I had administered an IQ test to my fellow students, they would have scored very high indeed. Not only were they selected by the MIT admission process, but they were mostly people (unlike me) who were pursuing degrees in mathematics. But still, the rules apply: there is for practical purposes no upper or lower limit on the IQ of a 18.313 student; but sicne there was a finite supply of 'em, the integral of the distribution function is bound to be finite.
Of course, since we know there are at least two populations in the class (math majors and assorted dilletentes and mathematical hangers-on like me), you probably could get more precise results by modeling the class that way. But if you're going with a single mean and std dev to describe the population, Guassian's still the best way.
By the way, if anybody else out there remembers GC Rota, I just googled him and found out that he passed away in 1999 at the young age of 67. A damnable shame too, he was a great mathematician and, if possible, an even greater teacher. Here is a quote:
Re:iTunes is a monopoly (Score:2, Informative)
Re:Maybe we should think different (Score:2, Informative)
1 star - bad
2 stars - background music
3 stars - good
4 and 5 stars mean good and very good but also that they'll be included in several smart playlists. so a 4 star isn't necessarily better than a 3 star but I want it on my ipod etc.
Re:Database server that can handle the load? (Score:2, Informative)
The DB server can keep up now, and the protocol is stable. The radio even works fairly well, but during peak hours it sometimes has trouble connecting (I've only ever used it at my Grandmother's house, and her cable net connection is kind of shitty so it may have been that).
A year ago last.fm/as weren't really usable. Now they are fairly solid. It's worth another shot.