Please create an account to participate in the Slashdot moderation system

 


Forgot your password?
Close
typodupeerror
×
Media (Apple) Businesses Media Music Apple

Latest AAC Encoder Comparison Results 173

Posted by Hemos from the learn-more-about-it dept.
bullitB writes "For fans of the world wide patent conspiracy's latest audio format, the latest double blind AAC encoder comparison test results are in. If nothing else, this suggests much of the complaints regarding the iTunes Music Store's lossyness might be unfounded."
This discussion has been archived. No new comments can be posted.

Latest AAC Encoder Comparison Results More

Latest AAC Encoder Comparison Results

Comments Filter:

  • Discussion (Score:5, Informative)

    by doofusclam ( 528746 ) <slash@seanyseansean.com> on Monday March 01, 2004 @08:25AM (#8428234) Homepage
    You can discuss this test with the author and others at http://www.hydrogenaudio.org

    • A warning to potential HA posters (Score:5, Informative)

      by waaka! ( 681130 ) on Monday March 01, 2004 @04:58PM (#8434084)
      For good reasons, the posters on Hydrogenaudio don't take kindly to people making unfounded assertions about which codecs are better, so if you're going to argue with them, think twice and ABX [hydrogenaudio.org] first. You will be, after all, arguing with many audio developers, e.g. people who make contributions to LAME, people who've tuned the Vorbis encoder, and a surprising number of people who work for Ahead (makers of Nero, of course).

  • go AAC (Score:5, Interesting)

    by sleepypants ( 599905 ) on Monday March 01, 2004 @08:27AM (#8428242)
    Glad to see development on AAC's sound quality...especially on the free side with the vast improvement of the previously terrible quality of FAAC. More 'useful' (although it would stir the pot a bit more) would be a comparison with the latest MP3 encoders. To stay within the AAC bubble in comparisons won't encourage people to convert (or to stay away).

  • Lossy is lossy (Score:5, Interesting)

    by Anonymous Coward on Monday March 01, 2004 @08:32AM (#8428262)

    When will people realise that half the trouble with a lossy format is transcoding? Sure, AAC may sound high-quality when it's in its original format, but when you transcode it to MP3 for your MP3 player, the quality turns to shit. This is inevitably the case when dealing with lossy formats, and why I'd rather buy CDs and rip them to FLAC [sf.net].

    • Why would you go from AAC to MP3? Of course that's going to degrade the quality.

      Oh, don't have an iPod to play your AACs? Do you have a CD player? Because you can burn them.

      (Anybody burned them and then ripped them to mp3 from the CD? I can imagine that would be less lossy than straight transcoding.)

      • Why would burning AAC's to CD and then ripping to MP3 be any less lossy than straight transcoding?

        Considering that all transcoding does is render the AAC data into a waveform and then translate the waveform into MP3, what difference would storing the waveform on some intermediate medium make?

      • Re:Lossy is lossy (Score:5, Insightful)

        by gl4ss ( 559668 ) on Monday March 01, 2004 @09:35AM (#8428596) Homepage Journal
        *(Anybody burned them and then ripped them to mp3 from the CD? I can imagine that would be less lossy than straight transcoding.)*

        if every device was perfect it would be bitwise identical. and what in the world led you to believe otherwise? if you ran it through analog form in some point you might get 'smoother' sound or something but that's just it and self deceit.

        that being said, if you buy music for an mp3 player, buy it in mp3. or rip it yourself from a cd, or just get high enough bitrate it doesn't matter for your golden ears if you code it from one format to another.

        better yet buy from some independends that are willing to provide both formats. or fuck, just encode good old amiga mods.

      • Re:Lossy is lossy (Score:4, Insightful)

        by fok ( 449027 ) on Monday March 01, 2004 @10:06AM (#8428821) Homepage
        Why would you go from AAC to MP3?

        To load it in your mp3 player?
        • Why would you have an mp3 player that's not an iPod?
          • Pricing the way it is, I didn't even consider an iPod when looking for a HDD based player. I decided on the Neuros (www.neurosaudio.com), and bought one Friday. I'll post a quick list of its advantages as compared to the 20GB iPod:

            -Half the price, at $200
            -Ogg support
            -Open source firmware and software, including good Linux support.
            -Removable HDD 'back packs', you can buy additional 20GB storage for a reasonable ~$100
            -Built in FM radio reciever and broadcaster (very cool)
            -Hardware MP3 enocding, you can rec

            • Re:Lossy is lossy (Score:3, Insightful)

              by Snowspinner ( 627098 ) *
              Yeah, I looked at many of those players.

              Then I bought an iPod, because it was actually well designed - it had an interface that actually lends itself to playing music. I want an mp3 player that I can change the song on while driving - that is, one that's fast to use, and is forgiving of being put down halfway through a menu and picked up again five minutes later because the traffic got bad.

              And, frankly, though I could carry an mp3 player larger than an iPod around, if I don't have to, I'm just as happy no
          • Why would you have an mp3 player that's not an iPod?

            Because I have a car. Until Apple makes an indash iPod I'm not going to be using AACs anytime soon. I'll stick with my kenwood indash mp3 player.
          • Cause I bought it before the iPod was available. Any other questions?
    • Comment removed based on user account deletion

  • Unfortunately... (Score:5, Informative)

    by lotsofno ( 733224 ) on Monday March 01, 2004 @09:29AM (#8428549)
    Winamp 5.02's encoder (which got a lot of help from hydrogenaudio's own Menno, a FAAD AAC-decoder developer/Ahead MPEG4 developer) wasn't included in the listening test because of a bug they found before testing [hydrogenaudio.org].

    Too bad, too. I would've loved to have seen how it compared.

  • Error bars (Score:5, Insightful)

    by thesp ( 307649 ) on Monday March 01, 2004 @09:33AM (#8428575)
    As a physicist, I'd just like to draw everyone's attention to the error bars on these charts. For the majority of the tests, it's possible to draw a horizontal line through the 95% confidence intervals of nearly all the points.

    Hence, the conclusions declaring clear winners/losers in these cases are invalid. If 99% confidence intervals were used (which gives a better statistical test), I feel that no clear winners or losers would be drawn.

    Be careful with these sort of studies - even though the author has used confidence intervals, he has failed to use them to infer the proper conclusions.

    That said, it's awfully nice to see error bars on this sort of website. Simple data points give such a false sense of precision, I find...

    • Re:Error bars (Score:5, Informative)

      by patman600 ( 669121 ) on Monday March 01, 2004 @12:06PM (#8430357)
      As a high school senior in a Statistics AP class currently studying confidence intervals and hypothesis testing, I think you are missing something. At the beginning, he clearly states your point: "One codec can be said to rated better than another codec with 95% confidence if the bottom of its line segment is at or above the top of the competing codec's line segment." The author gives which one is in first place, but announces at the beginning the requirements for a clear winner. And the author seems to me to be requiring at least half an interval of difference to even say that, much of the time he says they are tied.
      • >The author gives which one is in first place, but announces at
        >the beginning the requirements for a clear winner.

        The more comparisons we do, the greater the chance that we made an error in one of our comparisons (you can model this with a binomial distribution fairly trivially). Here, in each graph, there are 10 possible comparisons. The odds of us having a false positive in one of those graphs is around 40.1%. Across multiple studies, like this one, you can imagine the odds that one of our com
        • " The odds of us having a false positive in one of those graphs is around 40.1%. "

          That should read "The odds of having a false positive among the tests on any one of those graphs is around 40.1%"

    • Good point. You also have to very be careful about doing multiple comparisons like this without some form of correction (such as Bonferroni)--otherwise your 95% confidence interval gets progressively weaker for each test you add since the probability that you made a Type I error *somewhere* becomes very high very quickly.

    • Uhm, the author actually specifically states that itunes is NOT a statistical winner.

  • Perspective (Score:5, Insightful)

    by UnknowingFool ( 672806 ) on Monday March 01, 2004 @11:46AM (#8430046)
    Not to quash anybody's personal opinion of AAC, there are a few things that we should note.

    1) Most people can't tell the difference between formats that are similiar in performance.
    2) Some people actually can tell the difference.
    3) Some people are just posers who can't tell the difference but say they can.
    4) Lastly, most people don't really care as long as it is convenient to use either format.

    • Sounds good to me (Score:5, Informative)

      by kherr ( 602366 ) <kevin&puppethead,com> on Monday March 01, 2004 @11:57AM (#8430219) Homepage
      I had all of my CDs ripped to 192kbps MP3. When iTunes came out with AAC, I did a bunch of rip testing. I ripped from Donald Fagen's The Nightfly in a bunch of formats and bitrates. I found, for my personal preference, that 128kbps AAC was at least as good as 192kbps MP3, if not better. So I reripped all of my CDs to 128kbps AAC and got more songs onto my 5GB iPod. Now I'm on to ripping all of my old vinyl to AIFF, eventually to end up as AAC. Huzzah!

      • Re:Sounds good to me (Score:5, Interesting)

        by MikeXpop ( 614167 ) <mike&redcrowbar,com> on Monday March 01, 2004 @03:10PM (#8432958) Journal
        I found,
        for my personal preference, that 128kbps AAC was at least as good as 192kbps MP3, if not better.

        This statement irks me to no end. Your wording is better than most others though as you used for your preference.

        The thing to remember is that MP3 and AAC are different encodings. Comparing AAC to MP3 (to OGG to WMA...) is not like comparing MP3 algorithms. AAC will throw out different sounds that MP3 will keep, and vice versa. For example, a symbol crash sounds a lot better on an MP3 than it does on a similarly encoded AAC (I use LAME MP3s and iTunes AACs, they might sound different on others). However, vocals are clearer on AAC than MP3. I find overall AAC is superior to MP3, and that's what I have my songs as. However, saying a 192 AAC == 128 MP3 is a bit faulty. Both have their strengths and both have their weaknesses.

      • Re:Sounds good to me (Score:3, Interesting)

        by hondo77 ( 324058 )

        . So I reripped all of my CDs to 128kbps AAC and got more songs onto my 5GB iPod.

        Sounds like what I did. Then I played my iPod through my home stereo. Yikes! Now I'm re-ripping to 256 MP3. Like you said, though, it's personal preference and what you're using the files for and everybody's different.

        • Yikes! now why are you using 256kbps mp3s when VBR mp3s using the latest version of LAME and --preset standard average around 190kbps and sound better? And don't say compatibility - damn near everything will play them just fine.

  • AAC vs. AAC not the issue (Score:4, Interesting)

    by carbona ( 119666 ) on Monday March 01, 2004 @11:56AM (#8430201)
    Most of the complaints I've heard registered about the iTunes Music Store 128 Kbps format isn't that it sounds like crap compared to other AAC implementations. The major complaint is that it not only falls far behind Apple's claim that it sounds indistinguishable from the original lossless CD, but it also fails to sound even as good as MP3 with a decent encoder like LAME using --alt-preset standard or OGG at medium quality.

    I understand Apple trying to keep filesizes to a minumum, but in these days of 3.0 Mbps DSL links to people's apartments and storage prices at absolutely mind-boggling low price points, their logic is becoming less and less understandable with each passing month.

    AAC actually sounds like a well-developed and efficient lossy format but let's up the bitrate a bit especially when the price of a physical CD with all the artwork and liner notes along with lossless tracks and the ability to rip them to a lossy format for portable use is only a few dollars more, and in some cases the same price, than an album on the iTMS.
    • I want to second this thought. I won't buy songs off iTunes for this reason: the sound quality really stinks. This, combined with the DRM and lack of support on portable players, means that I still buy CD's so I can rip at a reasonable bitrate.
    • Storage space is not the issue - it's bandwidth on the server end. It might only be a small increase in file size for the consumer, but Apple's bills soon add up with that extra size.

      I know they use Akami, land of the infinite bandwidth, but that doesn't mean it's free.

  • Do you ask a car mechanic... (Score:5, Insightful)

    by SnowDog74 ( 745848 ) on Monday March 01, 2004 @12:02PM (#8430289)
    For the chemical formula for internal combustion? No. Likewise, I wouldn't go to the local audiophile shop to ask them about audio engineering-related issues. After being accosted by about ten salespeople in ten minutes at a local audio store that sells everything from Sony ES, to Krell, Wadia, Sunfire and the like... I caught a sales rep in a bold-faced lie. I was looking for a receiver without many bells and whistles, and he tried steering me towards Denon. When I asked why Denon is "better", he replied, "Because they focus solely on making audio components unlike Sony." I chuckled and asked him to explain to me the funadamental difference between the sample & hold buffers on a Sony DAC vs. a Denon DAC... Naturally, he had no clue what I was talking about. The "double-blind" survey is somewhat misleading... but that being said, it's clearly not measuring which format is superlative... it's only measuring people's perceptions. And people were pretty much even on those various formats. The study in question just shows that people cannot consistently tell the difference between AAC formats. Now, I've read articles in audiophile magazines that insisted that SACD (Super Audio CD) was brilliant in comparison to CD. And every one of those articles was a load of crap. Fundamentally, even the most "discriminating" audiophiles cannot tell the difference between 16-bit, 44.1kHz PCM (Pulse Code Modulation - e.g. AIFF, WAV, in the computer world) and the 1-bit, 2.7GHz DSD bitstream of SACD... nevermind the minute differences betweeen various AAC-enabled codecs. Hell, I would challenge anyone to be able to tell the difference between 16-bit PCM and MPEG-4 AAC. The AES (Audio Engineering Society) has stated that MPEG-4 AAC is perceptibly indistinguishable from uncompressed 16-bit, dual channel PCM (e.g. CD-DA spec audio).and I would wager any experienced audio engineer's pair of ears (my own included) against any consumer "audiophile" any day of the week. My advice to the idle rich? Don't buy the $45,000 pair of speakers... instead buy yourself better hearing and some common sense. My personal preference? MPEG-4 AAC. As a content creator intensely familiar with a variety of media standards including AES, NTSC, ISO, ITU-R/CCIR, etc. I believe MPEG-4 w/AAC (not Quicktime MPEG-4, mind you, but straight MPEG-4) is the superlative format for compressed audiovisual media. However, for critical listening, only uncompressed audio is the way to go. The general rule of thumb is that higher bitrates are preferable over higher sampling frequencies. Frequency response roll-off is what you want to avoid. But in order to support the higher bitrates, you need a D/A Converter (DAC, Digital-to-Analog Converter) with an effective sample-hold buffer that can crunch the necessary data to make an accurate conversion of the digital source. That being said, I'm going to begin digitally remastering my own compositions soon... and go straight from the 24-bit master to a 24-bit multichannel DVD-Audio format. Why? Even an audiophile deafened by the sound of their money burning a hole in their wallet can actually tell the difference between my 24-bit master recordings and the dithered 16-bit CD audio.
    • Typo... replace "2.7GHz" with "2.7MHz".
    • Nice to read this here ;)

      But... about AES statement: What hearing conditions and sound material do they use for their tests?

      And seems like you possibly know: what is the MAX dynamic range of compressed (AAC, MP3, whatever) sound? WTF, not only that, but I can't find ANY (s/n, distortions, intermodulations, etc) parameters of this conversion. I really can't found that, trying hard. And I have no interest/time(to recall) in calculating by myself, so, if you have something on this, please, share info/link.
    • The "double-blind" survey is somewhat misleading... but that being said, it's clearly not measuring which format is superlative... it's only measuring people's perceptions.

      And here I was thinking that the whole point of lossy audio compression was about throwing away information that people could not hear. :)

      The format that throws away the least audible information (as determined, in fact, by "measuring people's perceptions"), other things (encoder complexity, file size, etc) being equal, is the winner.

    • Re:Do you ask a car mechanic... (Score:4, Insightful)

      by valkraider ( 611225 ) on Monday March 01, 2004 @07:56PM (#8435540) Journal
      All the "formats" sound the same when played through my fairing speakers on my motorcycle - at 75mph...

      However, when I am in my car playing them off my iPod->TapeDeckConvertor, well - all the formats still sound the same.

      At home on my expensive stereo, I can easily tell the difference between CD and AAC. The AAC is the one playing off my iPod, and the CD is stored in a corrigated box in the basement...
  • I would like to see a graph or two comparing AAC, MP3, WMF, and OGG.
  • I have to admit to being puzzled as to why people spend so much time and energy trying to determine the relative merits of lossy formats.

    I find the music I've downloaded from iTMS perfectly acceptable; ditto the music I hear on my car's factory-equipment FM receiver. That doesn't mean I can't tell the difference between them and better sound.

    Actually, I've been transferring my LP's to CD... and recently I've been converting the CD's to .mp3 format with iTunes. The first recording I compressed, using some

    • Its all about the file size (Score:4, Insightful)

      by acomj ( 20611 ) on Monday March 01, 2004 @01:54PM (#8432009) Homepage
      People agonize because they want the best quality / size ratio. for example if aac encodes at 128 Kbps as well as mp3 at 192 Kbps you can fit more tunes on you harddrive in the same space and be happy. This isn't critical on computers with they're large hard drives, but for portables] players it matters. People also don't want to rip all there CDs multiple times.

      In someways your right, people should pick a bit rate/format that works for them and not worry about it. but this is slashdot..

    • Actually, you bring up an interesting issue.

      Many people will use vinyl simply for the reason that it does sound better than a CD. While the sound from a vinyl may not be as accurate as a CD, most people will find that they rather enjoy the distortion caused by the vinyl when compared.

      News flash, folks. Many people perfer vinyl to CD... if you have a good record, expensive player, you'll notice that vinyl does sound a lot better as long as those two conditions are met. Of course, the beauty of a CD is
      • Lossy codecs don't discard "unnecessary" sounds, they discard inaudible sounds. If you can hear a scratch or 'pop' in the original, then it's going to be in the lossy version as well. This is in theory of course. In practice, even the best perceptual codec throws away some things that the human ear will miss.
    • You can get a bigger sound quality improvement by switching from iTunes to LAME for your MP3s. Unfortunately, iTunes has a very poor MP3 encoder.

  • Don't use ANOVA here (Score:4, Insightful)

    by fscmj ( 757942 ) on Monday March 01, 2004 @01:24PM (#8431461)
    In this analysis he presents the results of an ANOVA (Analysis of Variance). This reminds me of the saying: "If your only tool is a hammer, every problem looks like a nail". The tool the author is using is ANOVA and so he is trying to force the data into that form. Hovever, the assumptions of any ANOVA is that the data is independent, normally distributed, and has constant variance. The response ranges from 1.0 to 5.0. I haven't taken the test myself and do not know if users were allowed to select non-integer values or not but even if they were we can see from the graphs he does present that most of the responses were near the upper bound of 5.0. This type of response is clearly not normally distributed. ANOVA is faily resistant to departures from normality but one would need to fully explore the degree of the departure before placing any weight to the confidence intervals presented. My gut feeling here is that it is highly skewed and will present confidence intervals smaller than what they should be (data is forced to be artifically close due to the upper bound and having so many people report values close to that upper bound). The data can probably be viewed as independent but it must be recognized that this is an assumption. Constant variance departures may be a problem as higher responses are less variable than middle responses (due to the upper bound again). It would probably be much better to use a non-parametric test alternative to ANOVA such as a Kruskal-Wallis. Scope of inference: This is not a random sample from any population and as such cannot be interpreted to represent anything more than the perceptions of the respondants themselves. -chris

    • Re:Don't use ANOVA here (Score:2, Insightful)

      by ff123 ( 514860 )
      The intervals in the rating scale are 0.1 steps, which is close enough for argument's sake.

      And ANOVA is a robust method as you've commented, so it's probably reasonable to assume normality. In any case, the raw data is available for any stats weenies to play with, and there are a couple of more conservative methods besides the Fisher LSD readily available to try, if anyone has an uncontrollable urge.

      I point you to:

      http://ff123.net/friedman/stats.html [ff123.net]

      where you can run a non-parametric analysis of the

  • the comments (Score:3, Interesting)

    by eclectic4 ( 665330 ) on Monday March 01, 2004 @01:50PM (#8431939)
    "Easy Listening

    Results: iTunes wins, with Nero closely behind it, more or less tied. Faac is tied with Nero, and Compaact! and Real are tied just below Faac."

    "House (Electronic/Techno)

    Results: iTunes and Nero tied at first place, Real and Compaact!tied with Nero, and Faac tied to Real and Compaact!."

    They were like that. Did we really need a play by play? Did he think we wouldn't be able to "decipher" these "complex" graphs?

    ...
    • They were like that. Did we really need a play by play? Did he think we wouldn't be able to "decipher" these "complex" graphs?

      Blind people listen to music, and would probably be very interested in his results (many of them having more acute hearing than we sighted folk). However, they would be unable to decipher or even perceive the graphs. I'm not sure how well their screen readers would deal with "faac" or "compaact!" though.

  • Minor Detail (Score:4, Informative)

    by tm2b ( 42473 ) on Monday March 01, 2004 @05:26PM (#8434305) Journal
    Just a minor detail to mention here. Dolby licenses [vialicensing.com] two different versions of their AAC codes. iTunes, when encoding for end users, uses the Dolby Consumer codec (affordably licensed by Apple in Quicktime 6). The Itunes Music Store uses the Dolby Professional codec (which would not be affordable to license in iTunes). Thus, AACs coming out of the iTunes Music Store have a higher quality at the same compression rate than the same songs you rip and convert on your own copy of iTunes.

    • Re:Minor Detail (Score:2, Interesting)

      by bullitB ( 447519 )
      This is unfortunately not really true. Just to quote your source,

      What is the difference between a "professional" encoder or decoder product, and a "consumer" encoder or decoder product?

      A professional product is purchased for commercial (i.e., revenue generating) purposes. A consumer product is purchased or made available for non-revenue generating purposes. Examples of professional products include broadcast encoders, and high-end audio or audio/video workstation applications. Professional products are t

  • Objectively comparing formats (Score:3, Insightful)

    by Experiment 626 ( 698257 ) on Monday March 01, 2004 @06:10PM (#8434704)

    These kinds of tests comparing codecs always seem to be something involving playing two versions of the song and asking someone which in their opinion sounded better. Isn't there a more quantitative way to measure the effect of the lossy compression? For instance:

    1. Start with the digitized CD recording
    2. Make a copy of it
    3. Compress the copy with the codec to be analyzed
    4. Do a lossless uncompression of the MP3 or whatever it is back to CD-resolution
    5. At each sample point (44k per second) on the resulting track, compare the 16-bit sample value of the compressed-and-back version to the control version.
    6. Sum the absolute value of the differences across all sample points.

    In other words, whichever codec introduces the least error into the track in a closed loop encode and decode test did the best job of faithfully reproducing the original signal. No subjective human testing required. You might have to tweak it a bit (say, sum the squares of the error or something) but would an approach like this work to settle the codec debates, or is there a fundamental flaw in this technique?

    • Unfortunately useless with lossy audio codecs. The moment a frequency, or range is dropped the wave form is significantly different than when it started, even if that frequency (say 22Khz) can't be heard in the music.

      You could do a fourier analysis and get the frequencies / phase, but how do you visually compare these when they could be significantly different but sound virtually identical. (and which one is more pleasing and less distracting)

      I'm waiting for audiophiles to start comparing and critiquing l

  • Whoa there! Look at the error bars... (Score:3, Informative)

    by Equuleus42 ( 723 ) on Monday March 01, 2004 @11:08PM (#8436971) Homepage
    I have looked at these sorts of test results before, and used to take them as the truth. That was before I started taking a Design of Experiments grad class, and have some evidence to the contrary. For what it's worth, I'm the top performer in the class right now.

    The issue I have is with the error bars. These are the vertical lines above and below the mean of each encoder. Like the beginning of the report says, "One codec can be said to rated better than another codec with 95% confidence if the bottom of its line segment is at or above the top of the competing codec's line segment." This is very much true for these sorts of statistical tests -- if the error bars overlap, that indicates that the means of the two groups are statistically identical. One could always adjust their confidence interval to a lower percentage, but 95% is quite often the standard.

    Note how many of the plots in this test have overlapping error bars. In the first plot, for example, all of the encoders tested have overlapping error bars. The results drawn from this plot should be that no encoder was measurably different than any other encoder -- not that iTunes won, like the results say. (Note: I own a Powerbook G4, and am typing this post on it right now, and I love Apple. I just don't like bad statistics, that's all)

    The results given in many of the plots are based strictly on the means of the samples, and not the error bars, which are actually more important in this case. Do not trust them. Interpreting the plots with the logic stated at the beginning of the article is the only statistically sound method (that I know of). I hope this sheds some more light on these tests...
  • All this fussing over miniscule differences in quality is plain stupid. I'm yet to hear a stereo system that could make me think that the performers were actually in the room with me - or even come close (at least for reproductions of acoustic material - electronically generated sounds can of course be reproduced convincingly). This is like arguing over which of two different lumps of shit is actually the better gourmet meal.

Slashdot Top Deals

As you will see, I told them, in no uncertain terms, to see Figure one. -- Dave "First Strike" Pare

Close