Testing audio codecs

A few days ago, while bringing over my Nakamichi cassette deck for revision, I had a discussion with the technician about lossy audio compression. If you would be able to hear the compression and and how to set up a proper test.

Inspired by this discussion, and curious about the results, I decided to follow up on this idea and implemented the test a few days later.

In each test two versions of the same music are played. One version has been compressed using a lossy audio codec, the other version is the original. Except for compression, both versions are identical.

The goal is to find out if you can hear any difference between the two versions. It's a simple yes-or-no question.

Each test compares a single compressed audio file against the original. There are many separate test; one for each codec and bitrate you want to test. You can choose the music for the test yourself.

During the test the version you hear will be changed every few seconds. Originally the idea was to interactively switch between versions, but that would make it difficult to use anything other than a computer. And this way you can use your portable audio player, smart amplifier or anything that accepts a .wav file.

To create the different versions of the music needed for the tests I wrote a small script. It wraps around FFmpeg and deals with all compression and decompression parameters. The script then creates output files with alternating sections of original and compressed audio. You can play the resulting audio file anywhere you like.

To make this test available to as many people as possible, I wanted to offer an online version as well. With a bit of creative programming, the same script can be run both from command-line and from CGI. The only dependencies are Python3 and FFmpeg.

Not all encoders are created equally. Different encoders can result in differences in sound quality, even for the same format and bitrate. So test results are not directly comparable between different encoders.

As MP3 encoder libLAME is used, which is widely considered to be the best. Most other tests seem to use a constant bitrate (CBR), while variable bitrate (VBR) usually yields a higher audio quality. For this test both are available.

As Opus encoder the reference implementation libopus is used. Sound quality should be more or less comparable with MP3 at 60% of the bitrate. Only VBR is available for this test.

The results will also vary depending on source material. So try different songs and different genres.

A lot has been written about testing and comparing codecs. A good example is this HydrogenAudio thread.

I would have liked to add additional codecs such as ATRAC (Sony MiniDisc) and PASC (Philips DCC), but those are difficult to integrate with this test as only decoders are easily available.

To get a reference I have searched the internet for suggested MP3 bitrates. Most websites tell you that the absolute lowest MP3 bitrate you should consider is 128k constant bitrate. A bitrate of 192k should be good enough for normal people, but audiophiles should use at least 256k, preferable 320k. I wonder why none of the sites mentioned variable bitrate or differences between encoders.

My personal test results were not quite what I expected. To me MP3 with a constant bitrate of 128k sounds identical to the original. So does Opus at 96k. So despite my assumptions, apparently my hearing is just about average.

Don't forget to try the codec tester yourself.

The codec tester in this article is developed using free and open source software where possible. Code is written using Vim and runs on Python. Audio is compressed with LAME and Opus, using FFmpeg.

In turn this code and article are released into the public domain. You can find all relevant files in the downloads directory of this article.