Music Listeners Test 128kbps vs. 256kbps AAC

notthatwillsmith writes "Maximum PC did double-blind testing with ten listeners in order to determine whether or not normal people could discern the quality difference between the new 256kbps iTunes Plus files and the old, DRM-laden 128kbps tracks. But wait, there's more! To add an extra twist, they also tested Apple's default iPod earbuds vs. an expensive pair of Shure buds to see how much of an impact earbud quality had on the detection rate."

The results...

Apple's iTunes store--in partnership with EMI--is now hawking DRM-free music at twice the bit rate of its standard fare (256Kb/s vs. 128Kb/s) and charging a $0.30-per-track premium for it. We're all for DRM-free music, but 256Kb/s still seems like a pretty low bit rate--especially when you're using a lossy codec.

So we decided to test a random sample of our colleagues to see if they could detect any audible difference between a song ripped from a CD and encoded in Apple's lossy AAC format at 128K/s, and the same song ripped and encoded in lossy AAC at 256Kb/s.

Our 10 test subjects range in age from 23 to 56. Seven of the 10 are male. Eight are editors by trade; two art directors. Four participants have musical backgrounds (defined as having played an instrument and/or sung in a band). We asked each participant to provide us with a CD containing a track they considered themselves to be intimately familiar with. We used iTunes to rip the tracks and copied them to a fifth-generation 30GB iPod. We were hoping participants would choose a diverse collection of music, and they did: Classical, jazz, electronica, alternative, straight-ahead rock, and pop were all represented; in fact country was the only style not in the mix. (See the chart at the end of the story for details.)

We hypothesized that no one would be able to discern the difference using the inexpensive earbuds (MSRP: $29) that Apple provides with its product, so we also acquired a set of high-end Shure SE420 earphones (MSRP: $400). We were confident that the better phones would make the task much easier, since they would reveal more flaws in the songs encoded at lower bit rates.

METHODOLOGY

We asked each participant to listen with the Apple buds first and to choose between Track A, Track B, or to express no preference. We then tested using the SE420's and asked the participant to choose between Track C, Track D, or to express no preference. The tests were administered double-blind, meaning that neither the test subject nor the person conducting the test knew which tracks were encoded at which bit rates.

The biggest surprise of the test actually disproved our hypothesis: Eight of the 10 participants expressed a preference for the higher-bit rate songs while listening with the Apple buds, compared to only six who picked the higher-quality track while listening to the Shure's. Several of the test subjects went so far as to tell they felt more confident expressing a preference while listening to the Apple buds. We theorize that the Apple buds were less capable of reproducing high frequencies and that this weakness amplified the listeners' perception of aliasing in the compressed audio signal. But that's just a theory.
LEAVE IT TO THE OLD FOGEYS

Age also factored differently than we expected. Our hearing tends to deteriorate as we get older, but all three of our subjects who are over 40 years old (and the oldest listener in the next-oldest bracket) correctly identified the higher bit-rate tracks using both the Apple and the Shure earphones. Three of the four subjects aged between 31 and 40 correctly identified the higher bit-rate tracks with the Apple earbuds, but only two were successful with the Shures. Two of three under-30 subjects picked the higher-quality tracks with the Apples, but only one of them made the right choice with the Shures. All four musicians picked the higher-quality track while listening to the Apples, and three of the four were correct with the Shures.

Despite being less able to detect the bit rate of the songs while listening to the Shure SE420 earphones, eight of 10 subjects expressed a preference for them over the Apple buds. Several people commented on the Shure's ability to block extraneous noise. While listening to the SE420s, one person remarked "Wow, I'd forgotten that wood-block sound was even in this song." Another said "The difference between the Shure earphones and the Apple earbuds was more significant than the difference between the song encoded at 128Kb/s and the one recorded

Re:The results...

It would be crazy to pay that premium if you're going to buy the entire album.

DRM'd and DRM-free albums cost the same. There is no reason to buy the DRM, if you are buying a whole album.

In the end, Apple's move doesn't change our opinion that the best way to acquire digital music remains buying the CD:

They tested music ripped from CD and encoded by iTunes. That makes this test irrelevant to the music to the iTunes store, since that music comes from the original masters (higher quality than the CD) and is encoded using customised settings (per-album or per-song), while iTunes uses some fairly general settings.

On my own, completely unscientific, tests, the 256Kb/s tracks are noticeably better. I upgraded a couple of albums yesterday and discovered I could hear the lyrics clearly in a few places where they had been obscured by instrumentals in one of them. The difference is only noticeable if you are specifically listening for it though; I wouldn't be able to tell you the bitrate in a blind listening (hearing them one after the other I probably could).

Having the songs DRM-free is definitely worth it though. I stopped buying music from iTMS when I started owning multiple portable devices that could play back AAC, but not Apple DRM.

Re:The results...

They tested music ripped from CD and encoded by iTunes. That makes this test irrelevant to the music to the iTunes store, since that music comes from the original masters (higher quality than the CD) and is encoded using customised settings (per-album or per-song), while iTunes uses some fairly general settings.

So then, it seems that there would be an even more noticeable difference between 128Kb/s and 256Kb/s. Which means if using this lower quality 128Kb/s track, the research showed that the difference in quality isn't worth an extra 30 cents, then doesn't it still hold true that a higher quality 128Kb/s track purchased from iTunes would be even closer in quality to the 256Kb/s track, and still not worth the extra 30 cents?

If ripping a CD to iTunes at 128Kb/s creates a lower quality track than purchasing a 128Kb/s track from the iTunes Store, then I think ripping from a CD to iTunes actually adds more weight to the argument that the 256Kb/s tracks are not worth an extra 30 cents.

Re:The results...

I wouldnt make any conclusions. There were only 10 people tested.

Re:The results...

That makes this test irrelevant to the music to the iTunes store, since that music comes from the original masters (higher quality than the CD)

Do you have any actual evidence that iTunes tracks are encoded from master tracks that are higher quality than CD (i.e. greater than 44.1kHz/16bit)? I have a hunch they're encoded from the same 44.1kHz/16bit file that you'd get if you ripped the CD yourself...In fact, I know they've done exactly this in at least once case, my own album...but I'm not signed to a major label, so it's possible things are different, but I doubt it...

Re:The results...

When Jobs introduced the music store, he stated that this is exactly the case. It's not universal, but for some (many? a few? most? I have no idea) they went back to the original masters and used those to for the iTunes music store.

Not quite...

Nearly all music is recorded and processed at 48kHz. The Red Book standard unfortunately went with 44.1 (for some esoteric reason having to do with syncing with an analog video standard or something back in the 80s). So, there's at least a down-conversion from 48 to 44.1, which isn't the end of the world but you lose some fidelity in the process since its really hard to do that "right" (and its only been recently that people have stopped using langrangian techniques and used truncated sinc functions or polyphase filters to do a decent job without it taking 50 forevers)

Re:Not quite...

The Red Book standard unfortunately went with 44.1 (for some esoteric reason having to do with syncing with an analog video standard or something back in the 80s).

Huh, you're right...

http://www.cs.columbia.edu/~hgs/audio/44.1.html [columbia.edu]

I always assumed that 44.1kHz was chosen because they took the necessary (Nyquist) sample rate to be able to record up to 20kHz (40kHz), and added a bit for good measure. There's always been that rumor that the time length of a CD was chosen to be able to fit Beethoven's Ninth Symphony, so I always figured they knew they wanted 16 bit, and a length of about 74 minutes, and just picked the >40kHz sampling rate that would get them there with that fancy new "CD" technology that was being developed. I'm happy to know that we're all using 44.1kHz for an even stupider reason ;-).

Re:The results...

no you're quite correct. the gp seems to think that they break out the 1\2" masters to re-encode for iTunes. They don't. Much music from the last 15 years was mastered at 48 or 44.1 digitally and there isn't even a higher quality master to be had.

- an audio pro

Re:The results...

"I could hear the lyrics clearly in a few places"

Hmm, I guess that is only because you don't have low skin effect monster cables on your earbuds... ;)

Anyhoo, if you can hear a difference, then you haven't gone to enough heavy metal shows.

Re:The results...

We're all for DRM-free music, but 256Kb/s still seems like a pretty low bit rate--especially when you're using a lossy codec.

Are they on crack? 256 Kbps is quite a high bitrate for a lossy CODEC. Their wording is also really bizarre. A low bitrate would be worse for a lossless track, because an uncompressed or lossless track, by definition, should have a much higher bitrate than a track compressed with a lossy CODEC.

Do they even know what they are talking about?

Re:The results...

Many similar tests have proven that most humans have trouble detecting any change in audio quality above 160->192 Kbps or in mp3s. A quick web search will show that even "audiophiles" really can't discern the difference. 128 has a clear "tinny" quality that disappears as the bit rate goes up. Based on this, I believe that 256 tracks as compared with the original cds would never be accurately identified. Clearly this should have been a part of this test. The idea that "lossy" means "audible" has not been proven in any real world tests.

     

Re:The results...

True, the only time you will generally notice the difference is if the track has a crowd clapping or drumkit (hi-hat) cymbals. At 128k I think cymbals sound horrible and undefined. At 192k I start not to be less annoyed.

Classical makes it evident

Actually, I notice a huge difference between 128K and 192K when listening to classical music. For music that doesn't contain the brashness of percussion or brass instruments, the distortion at lower encoding levels is fairly good; however, brass instruments (including brass cymbals) in particular are unbearably distorted when 128K is used but come across rather cleanly when >192K is used. I've finally accepted that a variable rate between 224K and 320K is where I need to encode my tracks in order to make them as close to the original CDs as I can tolerate without using the actual CDs.

Re:Classical makes it evident

Classical music usually has a wide dynamic range whilst most of the rest doesn't. The audio engineer working on a pop track runs everything through an audio volume-level compressor, bringing every sound to more or less the same volume level. In classical music it is quite normal to play certain things at the level of a whisper.

This means that most of popular music never uses the digital bits representing these low-volume whispers but confines itself to loud shouts and blaring synths, so a lot of the 'bandwidth' on a CD is wasted because of it. Classical music on the other hand uses most of the available bandwidth thanks to the sane use of audio level compressors. When this wideband signal is to have its data compressed then it requires a lot more storage space than the popular music would.

Parse error

At 192k I start not to be less annoyed.

I don't think that means what you think it means.

Re:Parse error

As the bit-rate rises, he becomes less annoyed. At 192k, he starts to notice less of a difference, and so starts to become less annoyed.

Yes, but the original poster said that at 192k, he starts not to become less annoyed.

(I'm too literal-minded to let it pass just because no one else is reading this far down. Actually, I'm only here because I was looking for the inevitable Princess Bride reference...)

Re:The results...

The MPEG community uses a MUSHRA test* to judge the quality of new codecs and to decide on bitrates etc. If there are n-codecs under test than the subject can switch A-B style between n+2 different versions of the same piece of music. These are the n-codecs and a reference or lossless version. He does not know which is which. He can also switch to one which he knows is the reference track (so the reference track is in there twice, labelled in one case and not labelled in the other). The task is to rate (0-100) each of the unknown tracks based on how similar it is to the reference track. One important thing to remember about the task is that the subject must rate similarity, rather than 'quality' or anything else. A certain codec could, for instance, add a load of warm bass to a piece of music making it more pleasurable (maybe) to listen to, but decreasing its similarity to the reference piece. The idea is that the subject should be able to pick the reference track from the unknowns (giving it a score of 100) and then rate all of the other unknowns in terms of similarity to the reference. The codec with the highest score wins. This type of test would be carried out for each of a number of pieces of music, with a lot of listeners.

* sorry, I've no good link- it's in ITU-R BS.1534-1 "Method for the subjective assessment of intermediate quality level of coding systems".

Re:The results...

We theorize that the Apple buds were less capable of reproducing high frequencies and that this weakness amplified the listeners' perception of aliasing in the compressed audio signal. But that's just a theory.

Can anyone explain this to me? I know what aliasing is; basically it's when your top frequencies hit the Nyquist limit and kind of bounce back downward (how's that for scientific?), and I know what it sounds like. However, the last time I checked, you'd remove aliasing by cutting high frequencies out of the final analog wave with a lowpass filter. Unless something's radically changed since then, wouldn't the presumably lower-response Apple buds actually show less aliasing that the expensive ones that can better reproduce the higher (and unwanted) frequencies?

Or have I been trolled into reasoning with audiophiles? If that's the case, let me know so I can pack up and go home.

Re:The results...

I can explain this to you, but it will probably easier to use an analogy to get the point across.

We know that a listening device (in this case earphones) has a certain frequency response, and can introduce noise into the source. Some listening devices produce less noise, and have more accurate frequency responses. In terms of simple examples, think: (Speaker > Landline > Mobile > Tin-can phone) (I know, the phones have sound systems behind them that affect the sound, but you get my point.).

Well, you know what? This is also true of encoding audio in a lossy format. So, instead of thinking of the anti-aliasing, imagine that we are encoding into another format. In the case of the apple phones, think of the transitions as (Source -> 128k AAC -> 192k MP3 (The apple phones)) versus (Source -> 256k AAC -> 192k MP3 (The apple phones)). Since additional noise is being introduced into the system, it should be pretty obvious which comes from the higher quality source. If we imagine the Shure headphones as having a perfect response, it will be (Source -> 128k AAC -> FLAC) versus (Source -> 256k AAC -> FLAC). There is no additional noise added, so you have to discern entirely based on the difference between the two AAC files.

To get back to the issue of aliasing, aliasing is what happens when a signal of one frequency gets recorded in a medium without enough precision to record that frequency. The nyquist limit says that for any frequency, you need twice that frequency in recordings to be able to capture the frequency (so a 5Khz sound can be heard on a 10Khz recording) but that assumes that the recording is in phase with the sound, and so it's a little more complicated than that. In any case, you can think of aliasing as the "beat" between two different frequencies. For example, if you listen to a sound at 3000 Hz and one at 3100 Hz at the same time, you will hear a 100 Hz "beat" that is the difference between the two. However, if you listen to the 3000 Hz frequency, and then the 3100 Hz Frequency, you might not be able to tell the difference between the two. It's only when playing the two sounds together that you hear the beat (just like you won't notice aliasing unless you actually record it into another format.)

Re:The results...

There's so many factors involved with these things that it is very hard to make a judgement. A well organised test would specifically select songs that do not compress well with lossy codecs. It is conceivable that music with a fairly even PSD (power spectral density) would not compress as well as one with a PSD that focusses more on certain areas, since the amount of information stored would have to be spread across a greater range. Hence the higher bitrate should sound better because more detail is preserved. Think speech quality (telephone, AM radio) vs CD quality, it sounds like the original, but the detail is all missing. That's what that extra bitrate adds back in. 128K is acceptable to the majority of people out there. Some people are more sensitive, I know people who work in professional audio and they can't stand 128K, personally, the vast majority of times I can't tell. I generally use OGG at around 160Kbit, and when an mp3 gets played I do get a sense that it is not quite the same, but it's not conclusive - it could just be the encoder used.

The headphones do make a difference. I used the stock headphones with my portable music player. Dropped them in/on/off something and broke them and got a set of Sennheiser ear buds. They do not cost $400. The interesting thing is I perceived the same effect as the people in the test: A reduction in bass 'kick' but a clearer response. There is definitely a lot to be said for good quality listening equipment, but in that arena, proper over the ear headphones are the only way to go. They aren't that practical though. The standard ear buds don't have the high frequency response and clarity you can get from slightly more expensive ones. Spending as much on your ear buds as on the player itself seems a little excessive though. You could probably get a larger size player, decent headphones, and use FLAC and get better quality than 256K mp3 through a set of very expensive ear buds. Also, you are going to be even more upset when they end up in your beer or something.

Finally, spotting mp3 artefacts is a strange thing. I'd never noticed any (at 128K) until someone pointed out the sound to me (usually it's cymbals). From then on, it became much clearer, and now I notice it a lot more (again it's mostly cymbals). Some songs are more susceptible than others, again I guess it is related to the make-up of the music.

Essentially I have come to the conclusion that: OGG sounds better than MP3 (although some of the audio professionals I know think the oposite), ear buds can only go so far and break - not worth spending a fortune, but worth spending a little, and that if you _really_ want to hear stuff at the finest detail, you should invest in some good over the ear headphones. It's a different experience: the noise occlusion, crisp, clear sound, and defined and powerful bass. The main thing you notice is that strong bass does not corrupt the higher frequencies, giving a very different overall feel of the sound, one that is, in my opinion, quite unique.

Re:The results...

This is a population proportion test.

Is the quality level distinguishable such that the
proportion of people detecting it is greater than
a coin toss (p = 0.5)?

The hypothesis:
Null : p = 0.5 The quality is not distinguishable
Alternative : p != 0.5 The quality is distinguishable

This is, arguably, a two-tailed test. We wish to see if the
null hypothesis is rejected.

The test has a requirement that:

np >= 5 and
n(1-p) >= 5

p = 0.5
n = sample size = 10

In both cases np = 10 x 0.5 = 5 so we barely make it.
and have an approximately normal distribution.

p_bar = sample proportion = 0.6 (in the one case)

sigma_p_bar = sqrt(p(1-p)/n) = 0.158

95% confidence interval: alpha = 0.05, two-tailed means
use alpha/2 = 0.025 as rejection region on both ends of the
normal distribution.

z_0.025 = 1.96

Right-tail rejection value:
p_bar_alpha/2 = p + z_0.025 x sigma_p_bar = 0.5 + 1.96 x 0.158
p_bar_alpha/2 = 0.809

Left-tail:
p_bar_alpha/2 = p - z_0.025 x sigma_p_bar = 0.5 + 1.96 x 0.158
p_bar_alpha/2 = 0.190

Decision rule:
If p_bar is greater than 0.809 or less than 0.19 we can
reject the null hypothesis and declare distinguishable
quality.

Since p_bar = 0.6 the null hypothesis is not rejected and
their is no statistical evidence that the quality was
distinguishable.

For p_bar = 0.8 (the second sample with the shures)
the null hypothesis is also not rejected. Just barely though.

The problem is the sample size is just too small to try
and prove anything with any statistical validity.

Although, I suspect the article was written more as a
case to generate ad revenue and perhaps push shure
headphones.

Re:The results...

You will have to pry my Ampex from my cold dead fingers.

Frankly, I'd rather they bury your Ampex with you.

Re:The results...

The harmonics in brass instroments and symbols

Is this higher maths?
.
.
.
.
Aah, you mean cymbals?

Re:The results...

The open reel tape used in the studio was recorded at ether 15 or 30 ips.

And you had to either pretty wealthy to use virgin tape or hope the previous recordings would be properly wiped. It's an analog medium with the main advantage that overdriving the inputs gives a nice effect ("warmth") - compared to early digital boxes who just clipped and truncated instead of dithered. Every time you have to play or record tape, it degrades a little bit; surely you know of the multitracking in Bohemian Rhapsody that went on and on until the tape was nearly transparent

Furthermore, vinyl is lowpass filtered at 16khz anyway. Gone are the harmonics. The higher fidelity is in the first few playings; after that, the medium degrades. What use is it to have something that'll play properly 10-20 times?

good CRO2 tape and a quality recording and playback deck and you really couldn't tell the difference between live and tape.

Live sound is always a compromise; always an unpredictable venue, crowd, and response (and in the worst case a clueless mixing engineer or band member who decides that eleven is just not enough for his guitar); soundchecks just can't fix this.

There is absolutely nothing wrong with digital. The whole 24/96 deal is a godsend because it means much more headroom. Having it in digital format means that you can play and record without ghosts from the past, without degradation. This caused some engineers to add noise afterwards to get rid of the sterility - but what they call sterility is simply unheard-of silence that couldn't be had previously. Engineers back in the day would've killed to have the possibilities we have now.

As for sounding plastic, I think you're confusing the medium with the mixing. Are you familiar with the term "loudness wars"?

Re:no, YOUR methodology is flawed

So, you would be happy with my codec, which replaces hip-hop with Bach played on original instruments? That sounds best to me...

Re:The results...

Noise canceling ear buds are never good and definitely not worth the money to begin with. The Shure product is actually noise isolating, therefore allowing you to play music at a lower volume, and be able to hear even more details. Also, noise isolating ear buds tend to also block out more noise than noise canceling ones do, at least in terms of the decibel rating.

Personally, I prefer a set of good earphones (without noise canceling, mind you, perhaps a good set of Grados) for those times at home, and in noisy environments, nothing beats a pair of decent in ear noise isolating ear buds. They are essentially ear plugs with embedded speakers, absolutely amazing products. Check out a pair of Shures or Etymotics, definitely won't disappoint.

Re:The results...

It's a matter of personal taste, but I was given a pair of very expensive noise-blocking earbuds, and I *hate* them. Firstly, to block the noise, you have to jam them into your ears till it hurts. And then, the "sound-stage" is moved to directly between the earbuds, so the orchestra sounds like it is inside my head(*). Ugh. I tend to prefer lightweight in-ear headphones with a folding headband for travel (much more comfortable), and proper fullsize headphones (not necessarily especially expensive) for non mobile listening. On aircraft, I've given up on classical music completely.

(*)If interested in this effect, try playing with sox, and the "earwax" plugin. Some samples are on the web too.

Synopsis

8/10 Picked High Bit Rate with Apple Headphones
6/10 Picked High Bit Rate with Shure Headphones


100% certainty that 10 people sample-set is too little for a Yes-No experiement.

Re:Synopsis

Not only that, but audio professionals typically do codec and compression tests using an ABX test.

http://en.wikipedia.org/wiki/ABX_test [wikipedia.org]

This would have been more interesting if they had used a statistically valid sample size and not only compared 128 to 256, but also to lossless.

Re:Synopsis

The new standard for research methodology: finding 10 people at the corner starbucks, asking them to help you for an "article" you're writing.

Oh ,and while we're at, let's throw another variable into the mix! That'll make it even more scientifical! (And that's not even getting into any other variables that slipped in thru carelessness.)

Frankly, I wouldn't trust these MPC bozos to tell me if it was raining while I was urinating on their backs.

Re:Synopsis

The new standard for research methodology: finding 10 people at the corner starbucks, asking them to help you for an "article" you're writing.

This is what the Internet has reduced us to: it does not matter if it is correct, so long as it is delivered quickly.

Of course..

Test confirms the generally known (but debatable) points:
1. Not many can detect the improvement of higher kbps
2. Expensive earbuds are way better than the default ones.

But what do you do with this fanboi? "One of the two people who expressed a preference for Apple's product told us "It seemed like I got better kick from the bass."" I hope he was completely deaf.

It's fairly easy to detect the differences

I can tell the difference between 192 and 128 kbps. It's all in the treble which sounds less bright at 128 kbps. It's very easy to detect lower bit rates if you concentrate on the treble.

Reading the FA was a waste

that article doesnt provide enough data to make any conclusions.
maybe they should go back to statistics 101

Not worth it?

FTFA

we just don't think DRM-free tracks alone are worth paying an extra 30 cents a track for..

Have fun buying your album again to play it on your cell phone's MP3 player.

Cost and quality

"Eight of the 10 participants expressed a preference for the higher-bit rate songs while listening with the Apple buds, compared to only six who picked the higher-quality track while listening to the Shure's."

I don't buy this. I have a friend who claims to be an audiophile - and he is - with sound equipment worth well over $40,000. He states that the more expensive and professional your gear is, the easier it is to spot low quality music.

So the article contradicts with his statement