Music Listeners Test 128kbps vs. 256kbps AAC 428
notthatwillsmith writes "Maximum PC did double-blind testing with ten listeners in order to determine whether or not normal people could discern the quality difference between the new 256kbps iTunes Plus files and the old, DRM-laden 128kbps tracks. But wait, there's more! To add an extra twist, they also tested Apple's default iPod earbuds vs. an expensive pair of Shure buds to see how much of an impact earbud quality had on the detection rate."
The results... (Score:5, Informative)
So we decided to test a random sample of our colleagues to see if they could detect any audible difference between a song ripped from a CD and encoded in Apple's lossy AAC format at 128K/s, and the same song ripped and encoded in lossy AAC at 256Kb/s.
Our 10 test subjects range in age from 23 to 56. Seven of the 10 are male. Eight are editors by trade; two art directors. Four participants have musical backgrounds (defined as having played an instrument and/or sung in a band). We asked each participant to provide us with a CD containing a track they considered themselves to be intimately familiar with. We used iTunes to rip the tracks and copied them to a fifth-generation 30GB iPod. We were hoping participants would choose a diverse collection of music, and they did: Classical, jazz, electronica, alternative, straight-ahead rock, and pop were all represented; in fact country was the only style not in the mix. (See the chart at the end of the story for details.)
We hypothesized that no one would be able to discern the difference using the inexpensive earbuds (MSRP: $29) that Apple provides with its product, so we also acquired a set of high-end Shure SE420 earphones (MSRP: $400). We were confident that the better phones would make the task much easier, since they would reveal more flaws in the songs encoded at lower bit rates.
METHODOLOGY
We asked each participant to listen with the Apple buds first and to choose between Track A, Track B, or to express no preference. We then tested using the SE420's and asked the participant to choose between Track C, Track D, or to express no preference. The tests were administered double-blind, meaning that neither the test subject nor the person conducting the test knew which tracks were encoded at which bit rates.
The biggest surprise of the test actually disproved our hypothesis: Eight of the 10 participants expressed a preference for the higher-bit rate songs while listening with the Apple buds, compared to only six who picked the higher-quality track while listening to the Shure's. Several of the test subjects went so far as to tell they felt more confident expressing a preference while listening to the Apple buds. We theorize that the Apple buds were less capable of reproducing high frequencies and that this weakness amplified the listeners' perception of aliasing in the compressed audio signal. But that's just a theory.
LEAVE IT TO THE OLD FOGEYS
Age also factored differently than we expected. Our hearing tends to deteriorate as we get older, but all three of our subjects who are over 40 years old (and the oldest listener in the next-oldest bracket) correctly identified the higher bit-rate tracks using both the Apple and the Shure earphones. Three of the four subjects aged between 31 and 40 correctly identified the higher bit-rate tracks with the Apple earbuds, but only two were successful with the Shures. Two of three under-30 subjects picked the higher-quality tracks with the Apples, but only one of them made the right choice with the Shures. All four musicians picked the higher-quality track while listening to the Apples, and three of the four were correct with the Shures.
Despite being less able to detect the bit rate of the songs while listening to the Shure SE420 earphones, eight of 10 subjects expressed a preference for them over the Apple buds. Several people commented on the Shure's ability to block extraneous noise. While listening to the SE420s, one person remarked "Wow, I'd forgotten that wood-block sound was even in this song." Another said "The difference between the Shure earphones and the Apple earbuds was more significant than the difference between the song encoded at 128Kb/s and the one recorded
Synopsis (Score:5, Informative)
6/10 Picked High Bit Rate with Shure Headphones
100% certainty that 10 people sample-set is too little for a Yes-No experiement.
Re:I don't have to read this article... (Score:5, Informative)
Reading the FA was a waste (Score:5, Informative)
maybe they should go back to statistics 101
Re:The results... (Score:5, Informative)
On my own, completely unscientific, tests, the 256Kb/s tracks are noticeably better. I upgraded a couple of albums yesterday and discovered I could hear the lyrics clearly in a few places where they had been obscured by instrumentals in one of them. The difference is only noticeable if you are specifically listening for it though; I wouldn't be able to tell you the bitrate in a blind listening (hearing them one after the other I probably could).
Having the songs DRM-free is definitely worth it though. I stopped buying music from iTMS when I started owning multiple portable devices that could play back AAC, but not Apple DRM.
Re:The results... (Score:4, Informative)
The headphones do make a difference. I used the stock headphones with my portable music player. Dropped them in/on/off something and broke them and got a set of Sennheiser ear buds. They do not cost $400. The interesting thing is I perceived the same effect as the people in the test: A reduction in bass 'kick' but a clearer response. There is definitely a lot to be said for good quality listening equipment, but in that arena, proper over the ear headphones are the only way to go. They aren't that practical though. The standard ear buds don't have the high frequency response and clarity you can get from slightly more expensive ones. Spending as much on your ear buds as on the player itself seems a little excessive though. You could probably get a larger size player, decent headphones, and use FLAC and get better quality than 256K mp3 through a set of very expensive ear buds. Also, you are going to be even more upset when they end up in your beer or something.
Finally, spotting mp3 artefacts is a strange thing. I'd never noticed any (at 128K) until someone pointed out the sound to me (usually it's cymbals). From then on, it became much clearer, and now I notice it a lot more (again it's mostly cymbals). Some songs are more susceptible than others, again I guess it is related to the make-up of the music.
Essentially I have come to the conclusion that: OGG sounds better than MP3 (although some of the audio professionals I know think the oposite), ear buds can only go so far and break - not worth spending a fortune, but worth spending a little, and that if you _really_ want to hear stuff at the finest detail, you should invest in some good over the ear headphones. It's a different experience: the noise occlusion, crisp, clear sound, and defined and powerful bass. The main thing you notice is that strong bass does not corrupt the higher frequencies, giving a very different overall feel of the sound, one that is, in my opinion, quite unique.
Re:Cost and quality (Score:5, Informative)
If you working in a recording studio, you want accuracy at all costs. You must hear everything distinctly, because you need to make important decisions based on what you hear. If "it sounds great" is all you are getting from your speakers, you won't make those tough decisions (more cymbals, different reverb, more compression on the vocals, or whatever.) You'll just leave it alone and it won't be as good as it could be. However, those extremely accurate speakers that are perfect for recording studio use are NOT pleasant for casual listening. Everything is too crisp and sharp, and they will tend to make you want a break from all that detail.
When I'm working on a mix in the studio, I want everything in very crisp detail so I can make judgments; when I'm listening to the final product, I want the music to "hang together" and present itself to me as a coherent whole. There are other differences between studio monitors and "normal" speakers (for example, consistency of frequency response) but this relatively subjective factor of detailed sound vs. coherent sound is one of the more important ones I have experienced.
The recording engineer did not intend for you to listen to the music on studio monitors. Studio monitors are a tool with a specific use, and that use is not everyday listening. The attributes of a good studio monitor just don't match up with the attributes of a good audiophile speaker. This is why audiophiles buy certain kinds of speakers, and recording engineers buy other kinds. I've been lucky enough to own both kinds of speakers, and I've tried using them for the wrong purpose with less-than-stellar results. Mixes made on good-sounding speakers are inconsistent on other speakers, and music played through accurate speakers isn't as pleasant to the ear.
Re:The results... (Score:5, Informative)
Personally, I prefer a set of good earphones (without noise canceling, mind you, perhaps a good set of Grados) for those times at home, and in noisy environments, nothing beats a pair of decent in ear noise isolating ear buds. They are essentially ear plugs with embedded speakers, absolutely amazing products. Check out a pair of Shures or Etymotics, definitely won't disappoint.
Re:The results... (Score:5, Informative)
Is the quality level distinguishable such that the
proportion of people detecting it is greater than
a coin toss (p = 0.5)?
The hypothesis:
Null : p = 0.5 The quality is not distinguishable
Alternative : p != 0.5 The quality is distinguishable
This is, arguably, a two-tailed test. We wish to see if the
null hypothesis is rejected.
The test has a requirement that:
np >= 5 and
n(1-p) >= 5
p = 0.5
n = sample size = 10
In both cases np = 10 x 0.5 = 5 so we barely make it.
and have an approximately normal distribution.
p_bar = sample proportion = 0.6 (in the one case)
sigma_p_bar = sqrt(p(1-p)/n) = 0.158
95% confidence interval: alpha = 0.05, two-tailed means
use alpha/2 = 0.025 as rejection region on both ends of the
normal distribution.
z_0.025 = 1.96
Right-tail rejection value:
p_bar_alpha/2 = p + z_0.025 x sigma_p_bar = 0.5 + 1.96 x 0.158
p_bar_alpha/2 = 0.809
Left-tail:
p_bar_alpha/2 = p - z_0.025 x sigma_p_bar = 0.5 + 1.96 x 0.158
p_bar_alpha/2 = 0.190
Decision rule:
If p_bar is greater than 0.809 or less than 0.19 we can
reject the null hypothesis and declare distinguishable
quality.
Since p_bar = 0.6 the null hypothesis is not rejected and
their is no statistical evidence that the quality was
distinguishable.
For p_bar = 0.8 (the second sample with the shures)
the null hypothesis is also not rejected. Just barely though.
The problem is the sample size is just too small to try
and prove anything with any statistical validity.
Although, I suspect the article was written more as a
case to generate ad revenue and perhaps push shure
headphones.
Re:The results... (Score:5, Informative)
Do you have any actual evidence that iTunes tracks are encoded from master tracks that are higher quality than CD (i.e. greater than 44.1kHz/16bit)? I have a hunch they're encoded from the same 44.1kHz/16bit file that you'd get if you ripped the CD yourself...In fact, I know they've done exactly this in at least once case, my own album...but I'm not signed to a major label, so it's possible things are different, but I doubt it...
Re:The results... (Score:5, Informative)
hydrogenaudio.org - Better source of information (Score:4, Informative)
Re:The results... (Score:1, Informative)
BINOM(8,10;0.5)=0.04. Small enough to conclude that the listeners could tell the difference with the Apple headphones.
Similarly, BINOM(6,10;0.5)=0.20. Too large to conclude that the listeners could tell the difference with the Shure headphones.
However, you are correct that the sample size and sampling methodology is not adequate to draw any real conclusions.
A vs. B (Score:2, Informative)
Couple things they missed: (Score:3, Informative)
Second, I remember there was a comment on Slashdot awhile back, before they actually came out with these, and I want to confirm... Apparently, CDs are recorded at a certain physical bitrate/frequency, and there are digital masters which are at a higher rate... it's late, so I'm not entirely coherent, but think of it as somewhat equivalent to resolution of a DVD (quality of video is proportional to resolution (HD vs normal) and bitrate). The point was that 256k may actually sound better than a CD, since it comes from a better source than a CD.
If so, this whole test is BS, since they did not do a comparison of CD vs 128k (either iTunes-DRM'd or custom-ripped) vs 256k (un-DRM'd, from the iTunes store). Specifically, I'd want to hear 256k vs CD. But at the same time, I don't know if any iPod, or specifically the one they are using, would be able to handle the higher resolution. If not, you'd have to specifically check your soundcard, too.
And finally, again vaguely remembering this from a Slashdot comment (so correct me if I'm wrong), but there was some comment about "The 30c may seem small, but imagine buying a whole album at these prices..." And I seem to remember that a full album is always $9.99. Still high compared to, for example, the minimum you'd pay for a FLAC-encoded album at MagnaTune, but if you're buying a whole album (and if that's accurate), you may as well just opt for un-DRM'd -- especially if it sounds better than a CD (which would probably cost more anyway.)
But then, of course, I'd like to hear a much bigger study, with more rigorous controls in place. 10 people is just not enough, no matter how you set it up.
And personally, if I had any money to spend on music, I'd be buying un-DRM'd stuff. But probably not from iTunes -- not till there's a web interface (or at least an API) that doesn't require me to download their client software. After all, if I'm buying a single file, the point of the client software is to implement the DRM, and if I buy the un-DRM'd version... Not that it shouldn't also work in the iTunes client, but it'd be nice for it to work natively in AmaroK, or just in a browser.
So they don't think it's worth 30 cents? (Score:4, Informative)
First off, I've yet to see a lossless formula that WORKS. And by works I mean is easily convertible into mp3/aac so I can use it on a portable player I already own, able to be used. I've seen APE and FLAC, both are too much hassle, and the APE files I got were in japanese. Here's a little fact, Ape doesn't necessarily know how to correctly encode Japanese into ID3, end result? Buffer overflow, bad data. Oh and if they work? They are larger than mp3s and AAC. Lossless codec means all the data has to remain, trust me, that's not a good thing when coupled with all the other little hassles it has.
Second. It'd be crazy to spend 99 cents just to license your files so that you can only use as Apple approves. Paying money to crack the music so I can use it as I want is illegal according to them so why am I paying the money to get locked into their plan. However DRM free music is easily worth 1 dollar and 30 cents because it's mine (It AAC but I can live with that). I don't have to ask permission to use it in another player, I don't have to ask permission to convert it to a data format I choose. Personally I'm fine with 192 for most recordings, I'm not an audiophile, I'm just a listener. If you want the highest grade data, or are an audiophile you'll be buying CDs or fully lossless data, you're not going to fuck with iTunes anyways.
Btw their other idea is to get rid of the apple iBuds and get quality recievers. Hint, This is what got the less interchangable results? I don't exactly see why getting a "higher quality" headset would be desirable if it creates less of a difference instead of more of a difference between two bit rates. Higher quality means I should hear everything. If you are asking people "can you hear the difference" they already should be listening as hard as they can. The theory they try to explain it with doesn't make much sense. They are telling us 30 cents doesn't make a difference but they are trying to sell us on dropping 400 bucks on noise reducing headsets you can get for around 100 if you're clever. Hell they are EARBUDS!!! So far I've notice two things about earbuds. They are uncomfortable, and they are worthless compared to my headphones. If you're talking about noise reducing earbuds just be smart buy a good set of headphones.
Overall a throw away article, I'm still only going to buy DRM less music (I expect you out there to do the same, I'm assuming 30 cents won't kill you, but that's your choice) and hope to soon if Apple ever put the DRMless music out, and had the music I listen to (so far not really). I'm assuming you all are STILL buying music like you are going to. The only mind's this article changes is the cavemen hiding under the rock who still scream "ahhh cds bad", and he's still trying to figure out our compooters, so showing him the internet might not be smart just yet.
Re:The results... (Score:5, Informative)
- an audio pro
Re:The results... (Score:5, Informative)
I can explain this to you, but it will probably easier to use an analogy to get the point across.
We know that a listening device (in this case earphones) has a certain frequency response, and can introduce noise into the source. Some listening devices produce less noise, and have more accurate frequency responses. In terms of simple examples, think: (Speaker > Landline > Mobile > Tin-can phone) (I know, the phones have sound systems behind them that affect the sound, but you get my point.).
Well, you know what? This is also true of encoding audio in a lossy format. So, instead of thinking of the anti-aliasing, imagine that we are encoding into another format. In the case of the apple phones, think of the transitions as (Source -> 128k AAC -> 192k MP3 (The apple phones)) versus (Source -> 256k AAC -> 192k MP3 (The apple phones)). Since additional noise is being introduced into the system, it should be pretty obvious which comes from the higher quality source. If we imagine the Shure headphones as having a perfect response, it will be (Source -> 128k AAC -> FLAC) versus (Source -> 256k AAC -> FLAC). There is no additional noise added, so you have to discern entirely based on the difference between the two AAC files.
To get back to the issue of aliasing, aliasing is what happens when a signal of one frequency gets recorded in a medium without enough precision to record that frequency. The nyquist limit says that for any frequency, you need twice that frequency in recordings to be able to capture the frequency (so a 5Khz sound can be heard on a 10Khz recording) but that assumes that the recording is in phase with the sound, and so it's a little more complicated than that. In any case, you can think of aliasing as the "beat" between two different frequencies. For example, if you listen to a sound at 3000 Hz and one at 3100 Hz at the same time, you will hear a 100 Hz "beat" that is the difference between the two. However, if you listen to the 3000 Hz frequency, and then the 3100 Hz Frequency, you might not be able to tell the difference between the two. It's only when playing the two sounds together that you hear the beat (just like you won't notice aliasing unless you actually record it into another format.)
Re:The results... (Score:3, Informative)
Re:The results... (Score:3, Informative)
I have The Best Of Charles Mingus on a Compatible Stereo Cassette from the early 80's (ATLANTIC CS 1555) and it sounds amazing. The folks responsible for mastering this cassette saturated every magnetic particle with information, producing one of the best sounding recordings I've ever heard. I recorded it as a 24-bit AIF for archive purposes and imported it into iTunes as a 256k VBR AAC and it sounds pretty close to the original output from the tape deck.
I'd suggest capturing any cassette not recorded with Dolby B at a higher bit than you would a CD to get the saturation feel they have.
Re:AAC a standard? (Score:3, Informative)
All my files on my Nokia 9300 Symbian phone are AAC even including ringtones embedded in device ROM. The other, "real" phone of mine, not anything close to smart (SE K700i) has everything in aac too. In fact thanks to AAC's better compression, I can use its 46mb flash memory for music.
"(Advanced Audio Coding) An audio compression technology that is part of the MPEG-2 and MPEG-4 standards."
http://www.answers.com/aac [answers.com]
As long as MPEG committee builds the standards, AAC is a standard. All players can't support AAC because the companies making them can't afford or won't get the AAC license. Not because it is only an Apple standard. Zune has AAC because it is iPod competitor and MS also have money to afford that extra format.
Whole confusion comes from the container format of mpeg-4 is built on the quicktime standard I guess. Now there are people who thinks mpeg4/h264 is Apple only format while their new HD-Satellite receiver probably has it in a chip.
The propetioary standards are the ones you mentioned. WMA (if we ignore their propaganda) and ATRAC3 which Sony really missed a huge chance by not opening it.
Re:Classical makes it evident (Score:5, Informative)
This means that most of popular music never uses the digital bits representing these low-volume whispers but confines itself to loud shouts and blaring synths, so a lot of the 'bandwidth' on a CD is wasted because of it. Classical music on the other hand uses most of the available bandwidth thanks to the sane use of audio level compressors. When this wideband signal is to have its data compressed then it requires a lot more storage space than the popular music would.
Re:The results... (Score:3, Informative)
There's an informative piece about bit rates here:
http://www.3daudioinc.com/3daudio_hi-res.html [3daudioinc.com]
The big problem with digital is the bandwidth not the bit rate.
Re:The results... (Score:3, Informative)
Lossless codecs aren't lossy codecs that just haven't been cranked down enough. The fundamental difference is that lossy encoding is happy throwing away parts of the input that it thinks you won't miss. But take the example of a sine wave at a constant frequency. A CBR MP3 will dutifully store that wave in 128kbps glory. An ideal lossy compressor will write "play a sine wave at 2KHz for 1:30" into a few bytes and be done. In that admittedly contrived situation, a lossless compressor could have much better compression.
I know what you meant and you're right in general, but those aren't universal truths - just what happens to be correct in most common cases.
Re:The results... (Score:3, Informative)
While listening at 24bit 48kHz is certainly just a bit better, *recording* at 24bit is certainly "highly superior" to recording at 16 bit. The larger dynamic range means that one can record at a much lower level into the computer, and not have to worry about clipping on the high end or quantization error/noise on the low end...just as a matter of convenience, 24bit recording is vastly better than 16bit
In mastering the 16/24bit question is largely irrelavant unless the source has wide dynamic range (like classical not pop/rock/dance)
The mastering process is always done at high bit and sample rates. After that, however, assuming the conversion to 44.1kHz/16bit was done properly with a good lowpass filter and a good dithering algorithm, the audible difference between the original master and the 44.1kHz/16bit master is very slight, even with classical music.
Re:The results... (Score:1, Informative)
This is digital folks. The CD's are digital copies of the master... so it really doesn't matter which get used, except for PR reasons.
Comment removed (Score:2, Informative)
Re:Classical makes it evident (Score:1, Informative)
When recorded digitally (in mono) if the samples are too infrequent the net effect of the string's high-pitch and quiet trem on the wave will end up sounding like there were no strings playing and the orchestra was out of tune, or the trem will sound like a sustained note.
If we look at the output of the sampling on an oscilloscope, the result will have the same basic shape, and if you stood far away with the two oscilloscope pictures next to each other you'd say they were the same. Increasing the bitrate will increase the "focus" (for lack of a better word) of sampled oscilloscope image.