Music Listeners Test 128kbps vs. 256kbps AAC

notthatwillsmith writes "Maximum PC did double-blind testing with ten listeners in order to determine whether or not normal people could discern the quality difference between the new 256kbps iTunes Plus files and the old, DRM-laden 128kbps tracks. But wait, there's more! To add an extra twist, they also tested Apple's default iPod earbuds vs. an expensive pair of Shure buds to see how much of an impact earbud quality had on the detection rate."

The results...

[Kagura (843695)]

Apple's iTunes store--in partnership with EMI--is now hawking DRM-free music at twice the bit rate of its standard fare (256Kb/s vs. 128Kb/s) and charging a $0.30-per-track premium for it. We're all for DRM-free music, but 256Kb/s still seems like a pretty low bit rate--especially when you're using a lossy codec.

So we decided to test a random sample of our colleagues to see if they could detect any audible difference between a song ripped from a CD and encoded in Apple's lossy AAC format at 128K/s, and the same song ripped and encoded in lossy AAC at 256Kb/s.

Our 10 test subjects range in age from 23 to 56. Seven of the 10 are male. Eight are editors by trade; two art directors. Four participants have musical backgrounds (defined as having played an instrument and/or sung in a band). We asked each participant to provide us with a CD containing a track they considered themselves to be intimately familiar with. We used iTunes to rip the tracks and copied them to a fifth-generation 30GB iPod. We were hoping participants would choose a diverse collection of music, and they did: Classical, jazz, electronica, alternative, straight-ahead rock, and pop were all represented; in fact country was the only style not in the mix. (See the chart at the end of the story for details.)

We hypothesized that no one would be able to discern the difference using the inexpensive earbuds (MSRP: $29) that Apple provides with its product, so we also acquired a set of high-end Shure SE420 earphones (MSRP: $400). We were confident that the better phones would make the task much easier, since they would reveal more flaws in the songs encoded at lower bit rates.

METHODOLOGY

We asked each participant to listen with the Apple buds first and to choose between Track A, Track B, or to express no preference. We then tested using the SE420's and asked the participant to choose between Track C, Track D, or to express no preference. The tests were administered double-blind, meaning that neither the test subject nor the person conducting the test knew which tracks were encoded at which bit rates.

The biggest surprise of the test actually disproved our hypothesis: Eight of the 10 participants expressed a preference for the higher-bit rate songs while listening with the Apple buds, compared to only six who picked the higher-quality track while listening to the Shure's. Several of the test subjects went so far as to tell they felt more confident expressing a preference while listening to the Apple buds. We theorize that the Apple buds were less capable of reproducing high frequencies and that this weakness amplified the listeners' perception of aliasing in the compressed audio signal. But that's just a theory.
LEAVE IT TO THE OLD FOGEYS

Age also factored differently than we expected. Our hearing tends to deteriorate as we get older, but all three of our subjects who are over 40 years old (and the oldest listener in the next-oldest bracket) correctly identified the higher bit-rate tracks using both the Apple and the Shure earphones. Three of the four subjects aged between 31 and 40 correctly identified the higher bit-rate tracks with the Apple earbuds, but only two were successful with the Shures. Two of three under-30 subjects picked the higher-quality tracks with the Apples, but only one of them made the right choice with the Shures. All four musicians picked the higher-quality track while listening to the Apples, and three of the four were correct with the Shures.

Despite being less able to detect the bit rate of the songs while listening to the Shure SE420 earphones, eight of 10 subjects expressed a preference for them over the Apple buds. Several people commented on the Shure's ability to block extraneous noise. While listening to the SE420s, one person remarked "Wow, I'd forgotten that wood-block sound was even in this song." Another said "The difference between the Shure earphones and the Apple earbuds was more significant than the difference between the song encoded at 128Kb/s and the one recorded

Re:The results...

[TheRaven64 (641858)]

It would be crazy to pay that premium if you're going to buy the entire album.

DRM'd and DRM-free albums cost the same. There is no reason to buy the DRM, if you are buying a whole album.

In the end, Apple's move doesn't change our opinion that the best way to acquire digital music remains buying the CD:

They tested music ripped from CD and encoded by iTunes. That makes this test irrelevant to the music to the iTunes store, since that music comes from the original masters (higher quality than the CD) and is encoded using customised settings (per-album or per-song), while iTunes uses some fairly general settings.

On my own, completely unscientific, tests, the 256Kb/s tracks are noticeably better. I upgraded a couple of albums yesterday and discovered I could hear the lyrics clearly in a few places where they had been obscured by instrumentals in one of them. The difference is only noticeable if you are specifically listening for it though; I wouldn't be able to tell you the bitrate in a blind listening (hearing them one after the other I probably could).

Having the songs DRM-free is definitely worth it though. I stopped buying music from iTMS when I started owning multiple portable devices that could play back AAC, but not Apple DRM.

Re:The results...

[omeomi (675045)]

That makes this test irrelevant to the music to the iTunes store, since that music comes from the original masters (higher quality than the CD)

Do you have any actual evidence that iTunes tracks are encoded from master tracks that are higher quality than CD (i.e. greater than 44.1kHz/16bit)? I have a hunch they're encoded from the same 44.1kHz/16bit file that you'd get if you ripped the CD yourself...In fact, I know they've done exactly this in at least once case, my own album...but I'm not signed to a major label, so it's possible things are different, but I doubt it...

Re:The results...

[node 3 (115640)]

When Jobs introduced the music store, he stated that this is exactly the case. It's not universal, but for some (many? a few? most? I have no idea) they went back to the original masters and used those to for the iTunes music store.

Re:The results...

[dangitman (862676)]

We're all for DRM-free music, but 256Kb/s still seems like a pretty low bit rate--especially when you're using a lossy codec.

Are they on crack? 256 Kbps is quite a high bitrate for a lossy CODEC. Their wording is also really bizarre. A low bitrate would be worse for a lossless track, because an uncompressed or lossless track, by definition, should have a much higher bitrate than a track compressed with a lossy CODEC.

Do they even know what they are talking about?

Re:The results...

[tackledingleberry (1109949)]

The MPEG community uses a MUSHRA test* to judge the quality of new codecs and to decide on bitrates etc. If there are n-codecs under test than the subject can switch A-B style between n+2 different versions of the same piece of music. These are the n-codecs and a reference or lossless version. He does not know which is which. He can also switch to one which he knows is the reference track (so the reference track is in there twice, labelled in one case and not labelled in the other). The task is to rate (0-100) each of the unknown tracks based on how similar it is to the reference track. One important thing to remember about the task is that the subject must rate similarity, rather than 'quality' or anything else. A certain codec could, for instance, add a load of warm bass to a piece of music making it more pleasurable (maybe) to listen to, but decreasing its similarity to the reference piece. The idea is that the subject should be able to pick the reference track from the unknowns (giving it a score of 100) and then rate all of the other unknowns in terms of similarity to the reference. The codec with the highest score wins. This type of test would be carried out for each of a number of pieces of music, with a lot of listeners.

* sorry, I've no good link- it's in ITU-R BS.1534-1 "Method for the subjective assessment of intermediate quality level of coding systems".

Re:The results...

[Just Some Guy (3352)]

We theorize that the Apple buds were less capable of reproducing high frequencies and that this weakness amplified the listeners' perception of aliasing in the compressed audio signal. But that's just a theory.

Can anyone explain this to me? I know what aliasing is; basically it's when your top frequencies hit the Nyquist limit and kind of bounce back downward (how's that for scientific?), and I know what it sounds like. However, the last time I checked, you'd remove aliasing by cutting high frequencies out of the final analog wave with a lowpass filter. Unless something's radically changed since then, wouldn't the presumably lower-response Apple buds actually show less aliasing that the expensive ones that can better reproduce the higher (and unwanted) frequencies?

Or have I been trolled into reasoning with audiophiles? If that's the case, let me know so I can pack up and go home.

Re:The results...

[callmevinny (1101147)]

This is a population proportion test.

Is the quality level distinguishable such that the
proportion of people detecting it is greater than
a coin toss (p = 0.5)?

The hypothesis:
Null : p = 0.5 The quality is not distinguishable
Alternative : p != 0.5 The quality is distinguishable

This is, arguably, a two-tailed test. We wish to see if the
null hypothesis is rejected.

The test has a requirement that:

np >= 5 and
n(1-p) >= 5

p = 0.5
n = sample size = 10

In both cases np = 10 x 0.5 = 5 so we barely make it.
and have an approximately normal distribution.

p_bar = sample proportion = 0.6 (in the one case)

sigma_p_bar = sqrt(p(1-p)/n) = 0.158

95% confidence interval: alpha = 0.05, two-tailed means
use alpha/2 = 0.025 as rejection region on both ends of the
normal distribution.

z_0.025 = 1.96

Right-tail rejection value:
p_bar_alpha/2 = p + z_0.025 x sigma_p_bar = 0.5 + 1.96 x 0.158
p_bar_alpha/2 = 0.809

Left-tail:
p_bar_alpha/2 = p - z_0.025 x sigma_p_bar = 0.5 + 1.96 x 0.158
p_bar_alpha/2 = 0.190

Decision rule:
If p_bar is greater than 0.809 or less than 0.19 we can
reject the null hypothesis and declare distinguishable
quality.

Since p_bar = 0.6 the null hypothesis is not rejected and
their is no statistical evidence that the quality was
distinguishable.

For p_bar = 0.8 (the second sample with the shures)
the null hypothesis is also not rejected. Just barely though.

The problem is the sample size is just too small to try
and prove anything with any statistical validity.

Although, I suspect the article was written more as a
case to generate ad revenue and perhaps push shure
headphones.

Re:The results...

[Proudrooster (580120)]

True, the only time you will generally notice the difference is if the track has a crowd clapping or drumkit (hi-hat) cymbals. At 128k I think cymbals sound horrible and undefined. At 192k I start not to be less annoyed.

Classical makes it evident

[WidescreenFreak (830043)]

Actually, I notice a huge difference between 128K and 192K when listening to classical music. For music that doesn't contain the brashness of percussion or brass instruments, the distortion at lower encoding levels is fairly good; however, brass instruments (including brass cymbals) in particular are unbearably distorted when 128K is used but come across rather cleanly when >192K is used. I've finally accepted that a variable rate between 224K and 320K is where I need to encode my tracks in order to make them as close to the original CDs as I can tolerate without using the actual CDs.

Re:The results...

[by Anonymous Coward]

You will have to pry my Ampex from my cold dead fingers.

Frankly, I'd rather they bury your Ampex with you.

Re:The results...

[bursch-X (458146)]

The harmonics in brass instroments and symbols

Is this higher maths?
.
.
.
.
Aah, you mean cymbals?

Re:The results...

[LokiSnake (795582)]

Noise canceling ear buds are never good and definitely not worth the money to begin with. The Shure product is actually noise isolating, therefore allowing you to play music at a lower volume, and be able to hear even more details. Also, noise isolating ear buds tend to also block out more noise than noise canceling ones do, at least in terms of the decibel rating.

Personally, I prefer a set of good earphones (without noise canceling, mind you, perhaps a good set of Grados) for those times at home, and in noisy environments, nothing beats a pair of decent in ear noise isolating ear buds. They are essentially ear plugs with embedded speakers, absolutely amazing products. Check out a pair of Shures or Etymotics, definitely won't disappoint.

Synopsis

[sc0p3 (972992)]

8/10 Picked High Bit Rate with Apple Headphones
6/10 Picked High Bit Rate with Shure Headphones


100% certainty that 10 people sample-set is too little for a Yes-No experiement.

Re:Synopsis

[no_opinion (148098)]

Not only that, but audio professionals typically do codec and compression tests using an ABX test.

http://en.wikipedia.org/wiki/ABX_test [wikipedia.org]

This would have been more interesting if they had used a statistically valid sample size and not only compared 128 to 256, but also to lossless.

Re:Synopsis

[Ohreally_factor (593551)]

The new standard for research methodology: finding 10 people at the corner starbucks, asking them to help you for an "article" you're writing.

Oh ,and while we're at, let's throw another variable into the mix! That'll make it even more scientifical! (And that's not even getting into any other variables that slipped in thru carelessness.)

Frankly, I wouldn't trust these MPC bozos to tell me if it was raining while I was urinating on their backs.

Reading the FA was a waste

[emptybody (12341)]

that article doesnt provide enough data to make any conclusions.
maybe they should go back to statistics 101

Humidity??

[TopSpin (753)]

Clearly these test are inadequate, or at least they haven't disclosed enough information on the testing conditions. As any true audiophile knows, headphone performance is strongly affected by atmospheric conditions; I'll bet that if they had bothered to maintain proper water vapor saturation levels in the test facility the complete the inadequacy of the ear buds would have been obvious to everyone involved, because sensory receptors (hair cells) in the human ear only achieve full sensitively under controlled conditions.

No doubt they also failed to account for magnetic field alignment; the flaws of low bit rate reproductions are much easier to perceive when the listener is not aligned with Earth's natural axial vectors. The solenoidal force lines ruin the high band pass attenuation of any digital audio and will make both low and high bit rate reproductions equally poor, so naturally there wasn't a strong correlation among the test subjects.

Idiots.

</sarcasm>

Phew - saved by slashdot!

[rueger (210566)]

Judging by the comments from the six people who actually got to read the article I'm glad it got slashdotted before I wasted my time on it.

Wait a sec...

[cciRRus (889392)]

Maximum PC did double-blind testing with ten listeners in order to determine whether or not normal people could discern the quality difference between the new 256kbps iTunes Plus files and the old, DRM-laden 128kbps tracks.

Shouldn't that be a "double-deaf" test?

Better for albums

[AlpineR (32307)]

The big difference that the 256 Kb/s + DRM-free option makes for me is that now I'll buy albums from iTunes Store. Previously I would use iTunes to buy one to three tracks if there was some artist I liked but didn't want a whole album from. But usually I order the CDs online for $8 to $14, rip them to AAC at 192 Kb/s, and put the disc away to collect dust on my overflowing CD rack. Now I can get higher quality cheaper and faster.

Yes, ideally I would rip all my music to a lossless format. And ideally everything would be available on SACD at 2822 KHz rather than 44.1 KHz CDs. But that's just not practical with my 500+ album collection. It'd fill up my laptop's hard drive real quick and allow me to put only a fraction onto my iPod.

I'm also disappointed that the article only tested the tracks on iPods with earbuds. Most of my listening is on a decent stereo system fed from my laptop. Ripping is about convenience, not portability. I only use my iPod when riding the Metro or an airplane. With all the outside noise the bitrate doesn't matter.

And being DRM-free isn't just a matter of idealism. I get frustrated when I go to burn an MP3 CD for my car and discover that one of the tracks I selected is DRMed. Sure there are ways to get around it, but it's just not worth the bother.

AlpineR

No Country music

[flyingfsck (986395)]

Both kinds of music were missing: They had neither Country nor Western music in the test.

What is America coming to?

Re:I don't have to read this article...

[Compholio (770966)]

So, the detection rate dramatically improved when people had listening devices that were more balanced and more accurately presented the audio.

You should RTFA then b/c that's not what they found.

Re:Cost and quality

[FightCopyright (1098295)]

Yeah, well I used to have a gf who claimed to be a Scientologist, and she gave over $40k to the church. She states that some alien is responsible for blowing up volcanos that created humans, but you know what... the bitch is just wrong.

Re:Cost and quality

[adminstring (608310)]

I agree with your statement that audiophiles don't use "professional" equipment, but I disagree with your statement that studio monitors will give you the sound that the recording engineer intended. This is because, as you imply, there is a distinct difference between accurate speakers and good-sounding speakers, and recording studios use accurate speakers, while consumers, even audiophiles, are better off with good-sounding speakers.

If you working in a recording studio, you want accuracy at all costs. You must hear everything distinctly, because you need to make important decisions based on what you hear. If "it sounds great" is all you are getting from your speakers, you won't make those tough decisions (more cymbals, different reverb, more compression on the vocals, or whatever.) You'll just leave it alone and it won't be as good as it could be. However, those extremely accurate speakers that are perfect for recording studio use are NOT pleasant for casual listening. Everything is too crisp and sharp, and they will tend to make you want a break from all that detail.

When I'm working on a mix in the studio, I want everything in very crisp detail so I can make judgments; when I'm listening to the final product, I want the music to "hang together" and present itself to me as a coherent whole. There are other differences between studio monitors and "normal" speakers (for example, consistency of frequency response) but this relatively subjective factor of detailed sound vs. coherent sound is one of the more important ones I have experienced.

The recording engineer did not intend for you to listen to the music on studio monitors. Studio monitors are a tool with a specific use, and that use is not everyday listening. The attributes of a good studio monitor just don't match up with the attributes of a good audiophile speaker. This is why audiophiles buy certain kinds of speakers, and recording engineers buy other kinds. I've been lucky enough to own both kinds of speakers, and I've tried using them for the wrong purpose with less-than-stellar results. Mixes made on good-sounding speakers are inconsistent on other speakers, and music played through accurate speakers isn't as pleasant to the ear.