How We Test Earbuds and Headphones
Here at GearLab, we conduct some of the most advanced testing of consumer audio products in the world. We've assembled a first-rate audio testing suite with the help of industry leader Listen, Inc. Listen's products are often found in the quality control and research and development labs of top audio production companies.
We've leveraged the same technology to bring accurate and comparable quantitative data directly to you. Listen, Inc.'s CEO Steve Temme, an audio engineer with decades of experience at the forefront of his field with multiple academic papers to his name, has a word for the system we use, “world-class”.
The GearLab Audio Testing System
Our audio test suite includes a Brüel & Kjaer Type 5128 Head Simulator that transfers high-quality recordings to a computer running SoundCheck software via a professional-quality audio interface.
Let's break that down. The Brüel & Kjaer Type 5128 Head Simulator is a precisely engineered and anatomically correct head and torso mannequin that simulates the acoustic properties of the human form. Highly sensitive microphones in its realistically crafted ears allow us to make accurate, real-world recordings of the headphones and earbuds we test. A top-notch speaker in its mouth replicates human speech, allowing us to test voice commands and call quality. We've spent a lot of time with our Type 5128, and have grown fond of its endless work ethic and trusted, objective ear. We've dubbed it Darwin.
While it records the performance of each pair of earbuds and headphones, it doesn't have the hardware or software to process them. So, we use an audio interface to pass its recordings to a computer running Listen, Inc.'s SoundCheck audio test and measurement software.
It's the SoundCheck software that does all the clever analysis, measuring a wide range of audio characteristics such as frequency response, noise cancellation, distortion, and call quality. The software also compares the performance of each set of earbuds and headphones to our idealized Gear Lab Target Response Curve. That covers the basics; now let's dive into the details.
Frequency Response
Frequency response is charted as a curve, with frequencies listed on the X-axis in Hertz (Hz) and sound intensity, measured in decibels (dB), on the y-axis. It measures the relative volume of all audible frequencies to find, for instance, if low bass tones are balanced by brightening, high treble notes or if one or the other is overly emphasized. A set of headphones or earbuds with an elevated response, or peak, in the lower frequencies will have a louder, more powerful bass sound. A peak in the middle frequencies will emphasize sounds like vocals and guitar.
Take a look at the frequency response chart below. The horizontal axis spans from low, bass frequencies on the left to high, treble frequencies on the right. The vertical axis represents relative sound intensity. When you see a curve above the horizontal line at zero, the earbuds emphasize sounds in that frequency, making them relatively louder. If the curve dips below that line, sounds at that frequency sound relatively softer.
The chart above is an example of a nicely equalized frequency curve. Notice how the blue curve closely matches the red target response curve? Accurate frequency response curves like that translate to an earbud with a higher sound quality. To provide a counterexample, the chart below represents a pair of earbuds with less-than-stellar audio quality. The bass and mid-levels are overemphasized, with much more deviation in the high range.
The Harman Curve
You may be asking yourself, is there an ideal frequency response curve for headphones and earbuds? Researchers have been laboring for years to answer that question. Harman International, a parent company of many leading audio brands, created one of the first idealized frequency response curves. It helps companies build headphones that work for most people and helps other organizations test their effectiveness. It's called the Harman curve, and about 60% of people find it to provide the best overall listening experience.
The House Curve
The Harman Curve is not perfect. It was created using older equipment, and it isn't specific to earbuds, which have unique acoustic properties since they nestle into your ear instead of resting over it. To honor our state-of-the-art audio testing suite and our expert tester's well-tuned ears, we created our own GearLab Target Response Curve, aka the House Curve. It emphasizes the bass and treble frequencies a touch more than the Harman Curve. We think it sounds better and hope you agree.
How We Measure Sound Quality
To test the sound quality of each set of earbuds and headphones, our team of audio experts listen closely and critically to a curated Spotify playlist using each earbud's baseline settings. We then rate each earbud independently on a range of characteristics, including their soundscape, balance, gain, and distortion in each frequency range — bass, mid-range, and treble. We also take copious notes about which earbuds sound better and why, citing specific, time-stamped examples from the playlist.
After these subjective tests, we bring the Bruel and Kjaer 5128-B head simulator, Listen, Inc., and SoundCheck in to provide objective rigor while testing each pair's frequency response curves, noise cancellation skills, call quality, and distortion.
How We Measure Frequency Response
To measure the unique frequency response curves of each set of earbuds or headphones, we play a known sound file through each pair to the microphones in the ears of the Bruel and Kjaer 5128-B. It records the sounds it “hears,” and a Listen, Inc. audio interface transfers them to a computer running SoundCheck. The sophisticated software measures and displays the differences between the original source sound file and how each set of speakers reproduces it. Those differences represent the baseline equalization, or EQ settings for the model, and how well they optimize your listening experience.
We then compare each frequency response curve to both the Harman Curve and our very own House Curve. We find that the frequency response curves that most closely match our House Curve correlate well with our expert listener's preferences. These earbuds receive a higher score. Your tastes may differ from ours, but our ratings serve as a reference point to help you find the right headphones or earbuds for your needs.
Speakers and the Harman Curve
It's important to note that the Harman Curve was created using over-the-ear headphones and does not translate well to free-standing speakers. Our brains interpret sounds played directly into our ears much differently than when sounds emanate from a speaker in an open room. In fact, the frequency response curve is often meant to help headphones sound more like a room-based speaker system.
User Experience
We also tested what it's like to use these earbuds — including how comfortable they are, how securely they settle into your ear, and how easy it is to adjust their settings. These tests help us decipher which earbuds are best suited for exercise and which are meant for relaxing.
How We Measure Comfort
To test the comfort of each set, we had a number of male and female testers with various ear sizes wear them. First we noted whether or not the earbuds come with a selection of ear tip sizes, which allow you to customize their fit and offer you a lot of control over their comfort. Then we asked each tester to take note of whether they could easily wear the earbuds in for an hour or more at a time. We also took note of additional features, like the shape of the ear tips, whether or not they are able to rotate to shift their position, and the feel of the ear tip material. All of these factors affect their comfort.
How We Measure Stability
Nothing is more irritating than expensive earbuds that fly out of your ears at every turn, or feel like they're going to. To test the stability of these earbuds, our testers wore them to work out, run, and even subjected them to the biggest stress test of all, headstands. If they can stay in your ears when you're upside down, you're probably good to go. If they fell out, they earned a lower score.
How We Measure Customization
We also tested how well each set of earbuds worked with different types of technology and how easy it is to access their various features. Popular features include ear fit tests to make sure you're getting a good seal, location features to help you find lost buds, and associated apps that adjust your settings.
It's Better to Match Platforms
Not to spoil the surprise, but we found that Android earbuds do not play nice with Apple products, and vice versa.
We tried out each feature and noted how well it worked in comparison to other options in our testing. To help you understand what it's like to navigate each app, we created videos that walk you through the settings. These give you an accurate feel of how difficult it is to, say, adjust the relative volume of different frequencies, like the bass or treble notes, a process known as equalization (EQ).
How We Measure Connectivity
Ideally, each set of earbuds would offer easy-breezy device pairing, followed by an uninterrupted listening experience. However, that is not always the case. To measure the reliability of connection in each model, we take note of any issues we had while pairing to various devices, as well as if there were any random breaks in audio or inconsistencies in sound quality. Environmental factors like the age and operating system of your device, whether you have the latest firmware update, proximity to other devices, etc., can all affect the connection between your earbuds and phone or laptop. To augment our hands-on testing of connectivity, we also analyzed the frequency of 1-star Amazon user reviews that cite a connectivity issue as one of the key critiques. We used the statistical ratio of negative connectivity feedback relative to the total number of Amazon user reviews as a measure of connectivity issue dissatisfaction. Keep in mind that every product will have some negative feedback, as Bluetooth technology is not perfect. Our statistical connectivity complaint analysis helps us gauge the relative severity and frequency of connectivity issues in an apples-to-apples statistical manner. Our overall scoring of connectivity is based on the combination of our statistical analysis of Amazon user reviews and our direct observations of connectivity issues from our extensive hands-on testing with multiple devices.
Noise Cancellation
To judge how well each pair of earbuds and headphones block outside sounds to let you concentrate on your audio entertainment, we also test their noise cancellation abilities. Some of the earbuds in the test offer active noise cancellation (ANC), which they achieve by analyzing incoming sound waves and cancelling them by emitting an inverse wave. All of them offer some level of passive noise cancellation by physically blocking sound waves from entering your ear.
Fit Matters
If you're using earbuds, the best passive cancellation depends on an effective seal in your ear canal. To optimize it, make sure you try out the various ear tips that often come with a set of earbuds. Keep going until you get a good fit.
First, we use these earbuds in the real world to experience how well they block and cancel noise in day-to-day life. We test them back to back, taking copious notes and rating them comparatively. Then we turn to the audio lab for some objective data.
To test both active and passive noise cancellation, we pipe pink noise into a room and have the Bruel and Kjaer 5128-B record it. We start by playing the background sound without the earbuds in place. Then we add the earbuds, hit play on the pink noise, and the Bruel and Kjaer 5128-B records how much sound each set blocks out physically. If they offer ANC, we turn it on and run the test again to see how much more sound they can actively scrub out.
We use SoundCheck to calculate and graph the passive and active noise attenuation of each set and then add them together to calculate the total attenuation for each speaker set. If a pair of earbuds or headphones doesn't offer ANC, we only test their passive noise attenuation. This process shows us exactly how many decibels the earbuds or headphones can block or cancel in each frequency range.
We also took note of which earbuds came with a transparency mode that can be turned on and off. Often activated by pressing or tapping your earbuds, transparency mode allows you to quickly bypass noise cancelling mode when you need to hear a colleague's question or an announcement at the airport.
What Is Transparency Mode?
Transparency mode uses the earbud's microphones to bypass the passive noise isolation at your behest. Some earbuds offer additional awareness modes, some of which use artificial intelligence to decide when to let ambient sounds through based on your environment
Battery Life
To test the battery life of each set of earbuds, we took them from a full charge down to 0% battery life, playing music in a continuous loop the whole time. We kept their volume levels consistent, turning each to about 99 decibels. Then, we compared how long they lasted to their manufacturers' claims.
How We Measure Call Quality
The magic of earbuds is that they allow you to exist in your own little world of sound. But they also connect you to friends in an instant with a phone call. To compare how well each set transmits your voice over the airwaves, we tested how well these earbuds isolate your voice from background noise. This is technically known as voice isolation.
To test it, we use a high tech speaker inside of our Bruel and Kjaer 5128-B simulator head, which is located where a human's mouth would be (as seen in the photo below).
We use it to broadcast a script while blasting noise in the background. Then, we rated each earbud set by how well their mics isolated the voices. Our human testers judged the quality of these clips. Some earbuds did pretty well in these conditions. Others did not.
We varied the volume of the background track — playing voices talking in a quiet room, mid-level office noises, and downright blaring subway sounds. The highest levels were a true stress test, with concert-level decibels that would keep you from hearing a friend standing right beside you. That was the point.