Kim Lajoie's blog

Audio perception and ABX testing

by Kim Lajoie on June 26, 2012

Rob Schlette:

[I]t’s not uncommon for people to be asking the question, “can you really hear the difference?” This is very good news for music and music lovers.

Rob then goes on to describe a particularly thorough method for conducting audio tests.

But it doesn’t matter.

If you have to squint to hear the difference, the difference doesn’t matter. We’re talking about comparing two different signal chains and the audio difference between them is so slight that you need an ABX test to tell them apart. If that’s the case, the minuscule difference will be dwarfed by other practical considerations. You know, real-life factors like cost, ease-of-use, ability to impress people, availability, etc.

MP3 encoding is a great example. Low-resolution (below 128kbps) MP3 encoding is easy to hear. You don’t need a controlled test to know when you’re down there. But above a certain point (192kbps for me), MP3 is *almost* indistinguishable from uncompressed audio. At that point, it doesn’t matter how close it is, because the fact that it takes a fraction of the time to transfer online is far more important.

In fact, MP3 encoding is a great example because if you’re encoding your audio with a  lossy format, you already don’t care about perfect reproduction.

Amp sims are another example. They sound just fine. People doing extensive comparative testing are wasting their time (unless they’re developing amp sims themselves!). Amp sims suck. They don’t suck because they sound terrible. They don’t (they used to, but thankfully we’re past that point). They sound great. Amp sims suck because they don’t give you the experience of playing in front of a kicking amp (also, iso booths suck for the same reason!). A better performing experience will give you a better performance. And that is what matters.

-Kim.

 

13 thoughts on “Audio perception and ABX testing

  1. Rob Schlette says:

    Your concern about the imperceptibility of the difference between two things is exactly the sort of thing that ABX (and other similar methods of testing) exposes. Taking a few moments to be more quantitative in our decision making can free us from all of the silly minutia that you are (rightly) dismissing.

  2. Kim Lajoie says:

    Hi Rob, thanks for dropping by.

    Maybe we’re in agreement? I’m not sure.

    What I’m trying to say is ‘if you have to ABX-test it to hear the difference, then the difference is not worth worrying about.’

    -Kim.

  3. Kim Lajoie says:

    Another interesting perspective on the irrelevance of ABX tests:

    http://blog.szynalski.com/2009/07/27/should-we-care-about-abx-test-results/

    “The relevance of ABX tests to the lives of music lovers is questionable. Neither does the absence of audible differences imply equal quality, nor does the presence of audible differences imply that the compressed version is inferior. Rather than being the argument to end all debate, the results of ABX tests are just one data point and the relative strengths of various audio formats may well be put in a new light by further research.”

    -Kim.

  4. Rob Schlette says:

    ABX testing rarely reveals small differences between two audio source. Rather, it generally proves their irrelevance. ABX is just one of several accepted ways of actually proving what you’re says about small, nuanced differences not mattering. The difference is the attempt at objectivity.

    Here’s a classic piece on ABX from 1990 via the Boston chapter of the AES:
    http://bostonaudiosociety.org/bas_speaker/abx_testing.htm

  5. Jordan says:

    This is a great post, I love your blog for a lot of the no-nonsense attitude abouth things, and I agree. If you have to go through meticulous measures to check, it doesnt even matter.

  6. Kim Lajoie says:

    Nice article – thanks!

    Interesting comment about objectivity. I’d say ABX testing is a middle-ground (in objectivity) between an intuitive judgement (which can miss some details) and a technical analysis (which can consider too many details).

    -Kim.

  7. Kim Lajoie says:

    Cheers – glad you like it!

    -Kim.

  8. Frank Nitsch says:

    Hi Kim,

    I like your comment on Rob’s article. I tend to find such topics very interesting and are tempted to do some related testing. However you got it right: if the differences are so small that you need to use such techniques to find out, if there really is a difference, you could/should better use that time for something more productive. 😉
    I made some listening test with bitrates for MP3 compression and found the same number to be sufficient to hear no difference (maybe thorough ABX testing would show? 😉 ). So I used variable bitrate encoding ranging from 128 kbps to 192 kbps with the highest VBR quality setting. Todays recommendation when using Lame is to use the few presets (V#), but choosing V2, which should come out as ~192 kbps, results in files mostly having 220 kbps or more. This is not necessary in my opinion. The V3 preset results in an average bitrate, which is too low for my taste (far below 192 kbps). So I just thought about using a CBR with 192 kbps at the highest quality setting (-q 0) and stereo mode. I would like to hear what encoding settings you prefer and why. Suggestions are welcome. 🙂

    Thanx for sharing your thoughts via this blog. They are very useful, because they often don’t deal with the technical details but with focusing on the big picture and how to actually make more music.

    Take care

    Frank

  9. Kim Lajoie says:

    Hi Frank,

    I almost always use 192kbps CBR. I’ve written a bit about it here: http://blog.kimlajoie.com/more-things-you-cant-hear/

    It sounds like your own listening tests have come up consistent with mine!

    -Kim.

  10. Kim Lajoie says:

    Thanks Justin! Great blog of yours too!

    -Kim.

  11. Audiofanzine says:

    Hi! We’ve written a short guide on how to conduct abx testing here: http://en.audiofanzine.com/getting-started/editorial/articles/the-real-blind-test.html

    and cool blog 🙂

  12. Gabriel says:

    Hi Kim,
    I get your point, and I quite agree with it. I happen to be conducting a survey on CD vs AAC 256k through blind testing (http://cdvsmp3.wordpress.com). My motivation for doing this is that I get sick of people talking nonsense about lossy formats, and I wanted to offer them a tool to get more objective. But unfortunately those who need it the most are the ones that would never take my test… 90% of my huge music library is mp3 192k CBR, but I am just about to switch to Apple lossless because it allows me to convert to lower bit rates on the fly with good quality depending on the device I am transferring the music to. I am just about to explain it in a coming post.

Leave a Reply

Your email address will not be published. Required fields are marked *