Slashdot Log In
Music Listeners Test 128kbps vs. 256kbps AAC
Posted by
CowboyNeal
on Thu May 31, 2007 08:22 PM
from the perfect-pitches dept.
from the perfect-pitches dept.
notthatwillsmith writes "Maximum PC did double-blind testing with ten listeners in order to determine whether or not normal people could discern the quality difference between the new 256kbps iTunes Plus files and the old, DRM-laden 128kbps tracks. But wait, there's more! To add an extra twist, they also tested Apple's default iPod earbuds vs. an expensive pair of Shure buds to see how much of an impact earbud quality had on the detection rate."
Related Stories
[+]
News: After 4 Years, HydrogenAudio Opens New 128kbps Listening Test 267 comments
kwanbis writes "After more than four years, a new MP3@128kbps listening test is finally open at HydrogenAudio.org! The featured encoders are: LAME 3.97, LAME 3.98.2, iTunes 8.0.1.11, Fraunhofer IIS mp3surround CL v1.5, and Helix v5.1 2005.08.09. The low anchor is l3enc 0.99a. The purpose of this test is to find out which popular MP3 VBR encoder outputs the best quality on bitrates around 128 kbps. All encoders experienced major or minor updates that should improve audio quality or encoding speed, and we have a totally new encoder on board. Note that you do not have to test all samples — it is a great help even if you test one or two. The test is scheduled to end on November 22nd, 2008."
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
The results... (Score:5, Informative)
So we decided to test a random sample of our colleagues to see if they could detect any audible difference between a song ripped from a CD and encoded in Apple's lossy AAC format at 128K/s, and the same song ripped and encoded in lossy AAC at 256Kb/s.
Our 10 test subjects range in age from 23 to 56. Seven of the 10 are male. Eight are editors by trade; two art directors. Four participants have musical backgrounds (defined as having played an instrument and/or sung in a band). We asked each participant to provide us with a CD containing a track they considered themselves to be intimately familiar with. We used iTunes to rip the tracks and copied them to a fifth-generation 30GB iPod. We were hoping participants would choose a diverse collection of music, and they did: Classical, jazz, electronica, alternative, straight-ahead rock, and pop were all represented; in fact country was the only style not in the mix. (See the chart at the end of the story for details.)
We hypothesized that no one would be able to discern the difference using the inexpensive earbuds (MSRP: $29) that Apple provides with its product, so we also acquired a set of high-end Shure SE420 earphones (MSRP: $400). We were confident that the better phones would make the task much easier, since they would reveal more flaws in the songs encoded at lower bit rates.
METHODOLOGY
We asked each participant to listen with the Apple buds first and to choose between Track A, Track B, or to express no preference. We then tested using the SE420's and asked the participant to choose between Track C, Track D, or to express no preference. The tests were administered double-blind, meaning that neither the test subject nor the person conducting the test knew which tracks were encoded at which bit rates.
The biggest surprise of the test actually disproved our hypothesis: Eight of the 10 participants expressed a preference for the higher-bit rate songs while listening with the Apple buds, compared to only six who picked the higher-quality track while listening to the Shure's. Several of the test subjects went so far as to tell they felt more confident expressing a preference while listening to the Apple buds. We theorize that the Apple buds were less capable of reproducing high frequencies and that this weakness amplified the listeners' perception of aliasing in the compressed audio signal. But that's just a theory.
LEAVE IT TO THE OLD FOGEYS
Age also factored differently than we expected. Our hearing tends to deteriorate as we get older, but all three of our subjects who are over 40 years old (and the oldest listener in the next-oldest bracket) correctly identified the higher bit-rate tracks using both the Apple and the Shure earphones. Three of the four subjects aged between 31 and 40 correctly identified the higher bit-rate tracks with the Apple earbuds, but only two were successful with the Shures. Two of three under-30 subjects picked the higher-quality tracks with the Apples, but only one of them made the right choice with the Shures. All four musicians picked the higher-quality track while listening to the Apples, and three of the four were correct with the Shures.
Despite being less able to detect the bit rate of the songs while listening to the Shure SE420 earphones, eight of 10 subjects expressed a preference for them over the Apple buds. Several people commented on the Shure's ability to block extraneous noise. While listening to the SE420s, one person remarked "Wow, I'd forgotten that wood-block sound was even in this song." Another said "The difference between the Shure earphones and the Apple earbuds was more significant than the difference between the song encoded at 128Kb/s and the one recorded
Re:The results... (Score:5, Informative)
On my own, completely unscientific, tests, the 256Kb/s tracks are noticeably better. I upgraded a couple of albums yesterday and discovered I could hear the lyrics clearly in a few places where they had been obscured by instrumentals in one of them. The difference is only noticeable if you are specifically listening for it though; I wouldn't be able to tell you the bitrate in a blind listening (hearing them one after the other I probably could).
Having the songs DRM-free is definitely worth it though. I stopped buying music from iTMS when I started owning multiple portable devices that could play back AAC, but not Apple DRM.
Parent
Re:The results... (Score:5, Informative)
Do you have any actual evidence that iTunes tracks are encoded from master tracks that are higher quality than CD (i.e. greater than 44.1kHz/16bit)? I have a hunch they're encoded from the same 44.1kHz/16bit file that you'd get if you ripped the CD yourself...In fact, I know they've done exactly this in at least once case, my own album...but I'm not signed to a major label, so it's possible things are different, but I doubt it...
Parent
Re:The results... (Score:5, Informative)
Parent
Re:The results... (Score:5, Insightful)
Are they on crack? 256 Kbps is quite a high bitrate for a lossy CODEC. Their wording is also really bizarre. A low bitrate would be worse for a lossless track, because an uncompressed or lossless track, by definition, should have a much higher bitrate than a track compressed with a lossy CODEC.
Do they even know what they are talking about?
Parent
Re:The results... (Score:5, Interesting)
* sorry, I've no good link- it's in ITU-R BS.1534-1 "Method for the subjective assessment of intermediate quality level of coding systems".
Parent
Re:The results... (Score:5, Interesting)
Can anyone explain this to me? I know what aliasing is; basically it's when your top frequencies hit the Nyquist limit and kind of bounce back downward (how's that for scientific?), and I know what it sounds like. However, the last time I checked, you'd remove aliasing by cutting high frequencies out of the final analog wave with a lowpass filter. Unless something's radically changed since then, wouldn't the presumably lower-response Apple buds actually show less aliasing that the expensive ones that can better reproduce the higher (and unwanted) frequencies?
Or have I been trolled into reasoning with audiophiles? If that's the case, let me know so I can pack up and go home.
Parent
Re:The results... (Score:5, Informative)
Is the quality level distinguishable such that the
proportion of people detecting it is greater than
a coin toss (p = 0.5)?
The hypothesis:
Null : p = 0.5 The quality is not distinguishable
Alternative : p != 0.5 The quality is distinguishable
This is, arguably, a two-tailed test. We wish to see if the
null hypothesis is rejected.
The test has a requirement that:
np >= 5 and
n(1-p) >= 5
p = 0.5
n = sample size = 10
In both cases np = 10 x 0.5 = 5 so we barely make it.
and have an approximately normal distribution.
p_bar = sample proportion = 0.6 (in the one case)
sigma_p_bar = sqrt(p(1-p)/n) = 0.158
95% confidence interval: alpha = 0.05, two-tailed means
use alpha/2 = 0.025 as rejection region on both ends of the
normal distribution.
z_0.025 = 1.96
Right-tail rejection value:
p_bar_alpha/2 = p + z_0.025 x sigma_p_bar = 0.5 + 1.96 x 0.158
p_bar_alpha/2 = 0.809
Left-tail:
p_bar_alpha/2 = p - z_0.025 x sigma_p_bar = 0.5 + 1.96 x 0.158
p_bar_alpha/2 = 0.190
Decision rule:
If p_bar is greater than 0.809 or less than 0.19 we can
reject the null hypothesis and declare distinguishable
quality.
Since p_bar = 0.6 the null hypothesis is not rejected and
their is no statistical evidence that the quality was
distinguishable.
For p_bar = 0.8 (the second sample with the shures)
the null hypothesis is also not rejected. Just barely though.
The problem is the sample size is just too small to try
and prove anything with any statistical validity.
Although, I suspect the article was written more as a
case to generate ad revenue and perhaps push shure
headphones.
Parent
Re:The results... (Score:5, Interesting)
Parent
Classical makes it evident (Score:5, Interesting)
Parent
Re:The results... (Score:5, Funny)
Parent
Re:The results... (Score:5, Funny)
Is this higher maths?
.
.
.
.
Aah, you mean cymbals?
Parent
Re:The results... (Score:5, Informative)
Personally, I prefer a set of good earphones (without noise canceling, mind you, perhaps a good set of Grados) for those times at home, and in noisy environments, nothing beats a pair of decent in ear noise isolating ear buds. They are essentially ear plugs with embedded speakers, absolutely amazing products. Check out a pair of Shures or Etymotics, definitely won't disappoint.
Parent
Synopsis (Score:5, Informative)
6/10 Picked High Bit Rate with Shure Headphones
100% certainty that 10 people sample-set is too little for a Yes-No experiement.
Re:Synopsis (Score:5, Interesting)
http://en.wikipedia.org/wiki/ABX_test [wikipedia.org]
This would have been more interesting if they had used a statistically valid sample size and not only compared 128 to 256, but also to lossless.
Parent
Re:Synopsis (Score:5, Insightful)
Oh
Frankly, I wouldn't trust these MPC bozos to tell me if it was raining while I was urinating on their backs.
Parent
Reading the FA was a waste (Score:5, Informative)
maybe they should go back to statistics 101
Humidity?? (Score:5, Funny)
No doubt they also failed to account for magnetic field alignment; the flaws of low bit rate reproductions are much easier to perceive when the listener is not aligned with Earth's natural axial vectors. The solenoidal force lines ruin the high band pass attenuation of any digital audio and will make both low and high bit rate reproductions equally poor, so naturally there wasn't a strong correlation among the test subjects.
Idiots.
</sarcasm>
Phew - saved by slashdot! (Score:5, Funny)
Wait a sec... (Score:5, Funny)
Better for albums (Score:5, Interesting)
Yes, ideally I would rip all my music to a lossless format. And ideally everything would be available on SACD at 2822 KHz rather than 44.1 KHz CDs. But that's just not practical with my 500+ album collection. It'd fill up my laptop's hard drive real quick and allow me to put only a fraction onto my iPod.
I'm also disappointed that the article only tested the tracks on iPods with earbuds. Most of my listening is on a decent stereo system fed from my laptop. Ripping is about convenience, not portability. I only use my iPod when riding the Metro or an airplane. With all the outside noise the bitrate doesn't matter.
And being DRM-free isn't just a matter of idealism. I get frustrated when I go to burn an MP3 CD for my car and discover that one of the tracks I selected is DRMed. Sure there are ways to get around it, but it's just not worth the bother.
AlpineR
No Country music (Score:5, Funny)
What is America coming to?
Re:I don't have to read this article... (Score:5, Informative)
Parent
Re:Cost and quality (Score:5, Funny)
Parent
Re:Cost and quality (Score:5, Informative)
If you working in a recording studio, you want accuracy at all costs. You must hear everything distinctly, because you need to make important decisions based on what you hear. If "it sounds great" is all you are getting from your speakers, you won't make those tough decisions (more cymbals, different reverb, more compression on the vocals, or whatever.) You'll just leave it alone and it won't be as good as it could be. However, those extremely accurate speakers that are perfect for recording studio use are NOT pleasant for casual listening. Everything is too crisp and sharp, and they will tend to make you want a break from all that detail.
When I'm working on a mix in the studio, I want everything in very crisp detail so I can make judgments; when I'm listening to the final product, I want the music to "hang together" and present itself to me as a coherent whole. There are other differences between studio monitors and "normal" speakers (for example, consistency of frequency response) but this relatively subjective factor of detailed sound vs. coherent sound is one of the more important ones I have experienced.
The recording engineer did not intend for you to listen to the music on studio monitors. Studio monitors are a tool with a specific use, and that use is not everyday listening. The attributes of a good studio monitor just don't match up with the attributes of a good audiophile speaker. This is why audiophiles buy certain kinds of speakers, and recording engineers buy other kinds. I've been lucky enough to own both kinds of speakers, and I've tried using them for the wrong purpose with less-than-stellar results. Mixes made on good-sounding speakers are inconsistent on other speakers, and music played through accurate speakers isn't as pleasant to the ear.
Parent