Introducing the DXOMARK Battery protocol

Welcome to DXOMARK’s Battery testing protocol! If you’re reading this, you already know how much you rely on your smartphone in your day-to-day life. How long a battery charge lasts can have a direct impact on whether or not you can complete a business transaction, whether or not you can use your phone’s camera to capture a special moment with family and friends, whether or not your phone alarm goes off to wake you up on time, and so on. Literally billions of people all over the world similarly rely on their smartphones every day.

Battery technology has advanced over the years, with more power and longevity packed into smaller and smaller forms. But when choosing a smartphone, buyers discover that often the only battery information available is its capacity (in milliamp hours, or mAh), which while important, is not the only information they need to have. Most potential buyers want to know if the phone charges quickly, and especially how long a charge lasts, and an mAh number will not tell them that — but we at DXOMARK can.

We measure autonomy (how long a charge lasts or “battery life”), of course, but also charging (how long it takes to charge), and efficiency (how much power the phone consumes in charging or discharging). What makes the DXOMARK Battery protocol unique is that our engineers use state-of-the-art equipment to run specially-designed measurements of battery performance based on real-life use cases — in other words, we base our reproducible scientific tests on many of the most common ways people tend to use their phones.

For one thing, how long a charge lasts depends on several factors, including the phone’s hardware and software; the kinds of features it has; whether it runs processes in the background when actively used and/or when in idle mode; and of course, how much and in what ways people use their phones. The challenge to manufacturers is to find the right balance between high-end features and battery life: it is not an easy task to respond to consumers’ expectations for the best and brightest displays, along with (for example) providing 5G and other power-consuming connectivity, better and better camera and video functionality, while also ensuring that the phone’s charge will last sufficiently long so as not to require too-frequent charging.

Just as with our other test protocols, our focus is on the end-user experience. So our job at DXOMARK is to give users like you a clear idea of how long the phone you choose will last as you go about your daily activities, and how long it will take to charge it. Finally, for all but our calibrated use case tests (explained further below), we test all devices using their default settings right out of the box.

The DXOMARK Battery team and lab

To achieve our goals, DXOMARK has put together a dedicated team of engineers and experts whose sole responsibility is to test battery performance in the latest smartphones. In addition to precision measuring devices such as oscilloscopes and probes, the company has invested in a state-of-the-art set of touch robots and created an exclusive lab space dedicated to conducting Battery protocol tests. This space includes a special Faraday cage into which our team sets up smartphones for robot-based touch-testing while shielding the phones from external WiFi, phone network, and other wireless signals that can affect power consumption. (The only connectivity that can occur is through our own controlled network inside the cage itself.)

Our measurements are reliable and repeatable, and follow a specific set of testing activities that our engineers conduct both in the controlled environment of the Faraday cage as well as in the outdoors, with all devices tested under identical conditions. We test all devices using the same protocol so that you can compare devices with confidence in the objectivity of our test results.

Let’s take a closer look now at the three attributes that comprise the DXOMARK Battery protocol — autonomy, charging, and efficiency.

Autonomy

How long will a battery charge last if you’re surfing the web? If you’re shooting a video? If you’re streaming a movie or listening to a podcast you’ve previously loaded on your phone? As we said at the beginning, knowing what a phone’s battery capacity is will not tell us everything we need to know about its autonomy, as the different situations just mentioned have different power requirements. A phone that automatically searches for a network or WiFi connection will use more power than a phone with that feature turned off. (And so on.)

Our engineers have devised a comprehensive series of tests that cover many different smartphone uses that impact battery life. We have divided our Autonomy category into three sub-categories, each comprising many different tests: stationary, on the go, and calibrated use case. Let’s dive into a few specifics about these tests whose aggregated results make up our Autonomy sub-score.

Stationary

The robots inside our Faraday cage perform our stationary tests, which consist of a series of actions that mimic how people use their smartphones inside their homes and/or offices on a daily basis — what we refer to as our “typical usage scenario.” A simulated day starts at 7 a.m. and runs until 11 p.m. Between those “waking hours,” each robot uses the smartphone under test to make calls, to check social networks, to play music and videos and games, to browse the web, and so on, for a total of 4 hours of active use, which is the global daily average. (Just as people do not constantly use their phones during waking hours, our protocol builds in breaks during the 16-hour testing period.) At the end of the day, between 11 p.m. and 7 a.m., the robots “sleep” and the smartphones pass the night in their default idle modes, just like in real life.

The same daily routine begins again the next day at 7 a.m., and this routine continues for each device until it runs out of battery power.

On the go tests

Our protocol takes into account the fact that power consumption can be affected when people are on the go, as their phones search for connectivity, use GPS/location services, employ fitness trackers (etc.). In our on the go testing, an engineer takes 4 or 5 devices outside for about three hours while following an itinerary of several miles — walking, taking the bus, and riding the subway. While out, the engineer systematically makes calls, checks social apps, takes photos and videos, uses GPS navigation, and streams music according to a precisely-defined testing scenario.

Calibrated use case tests

Finally, we bring the smartphones back into the Faraday cage and have our robots perform several hours of a single activity. These activities include gaming, video streaming (cellular and wifi), music streaming, calling, and so on. Furthermore, all devices are calibrated in the same way for these tests so they can be evaluated on the same basis — for example, display brightness is always set to 200 cd/m2, phone speakers are set to 60 dB (measured by a sound pressure meter), all phones are connected to the same network, and so on. The graph below shows the autonomy per use case you can expect on a full battery charge. Unsurprisingly, the more energy an activity requires, the shorter the amount of battery autonomy you will have.

Charging tests

We at DXOMARK view charging as fully part of the overall battery experience; after all, most of us have experienced to some extent the stress of realizing — often too late — that our phone has run out of power or will die at any moment. When this happens, knowing how fast you can charge becomes your main concern. Our Charging score is based on two sub-scores, full charge and quick boost.

Our principal full charge test evaluates how much power and time it takes for the phone to charge from 0 to 80% and then from 80 to 100%… and then from when the battery indicator says 100% to when the phone is really and truly fully charged. (Here’s something you might not know: just because your phone’s display shows the battery is at 100% does not mean the battery is fully charged. We measure the time it takes to actually achieve a completely full charge.)

 

The scenario on which our quick boost testing is based is likely familiar to travelers in particular: sometimes you have only a few minutes to charge your smartphone, so we measure how much power you can gain by plugging in your smartphone for five (yes, 5!) minutes. Depending on how much power your phone has left, sometimes a quick five-minute boost will bring you three more hours of active use, and sometimes only one hour. 

		Samsung Galaxy M51	Motorola Moto G9 Power	Huawei P40 Lite
Autonomy boost (hh:mm)	20%	2:12	1:51	5:13
	40%	2:26	2:00	5:04
	60%	2:06	2:03	4:02
	80%	2:05	2:06	2:15
Percentage boost	20%	4.1 %	4 %	15.2 %
	40%	4.5 %	4.3 %	14.8 %
	60%	3.9 %	4.4 %	11.7 %
	80%	3.9 %	4.6 %	6.6 %
Energy consumed	20%	1405 mWh	1452 mWh	3142 mWh
	40%	1553 mWh	1561 mWh	3055 mWh
	60%	1342 mWh	1601 mWh	2425 mWh
	80%	1331 mWh	1641 mWh	1355 mWh

Quick boosts can also be very important to serious gamers: you’ve been playing for a long time and your battery is almost empty, but your game still isn’t over. Further, your phone is very hot when you plug it into your charger. We check the impact of that kind of gaming experience on how fast the smartphone recovers battery power. In short, we take the phones we test to the limit because people often push their phones to the limit, and it’s important to know how our phones and chargers perform in these kinds of situations. 

Efficiency

Our engineers calculate our DXOMARK power efficiency score by weighting and combining two sub-scores, charge up and discharge. Both sub-scores are based on our analysis of the data we obtain during robot-based testing, on the go testing, charging evaluation, and calibrated use cases. Our charge up sub-score indicates the overall efficiency of a full charge (that is, how much energy you need to fill up the battery vs the energy capacity of the battery). We also test the specific efficiency of the travel adapter when it comes to transferring power from an outlet to your phone, as well as for residual consumption (when a charge is finished and the smartphone is still attached to the charger, and after the smartphone is disconnected from the charger).

Discharge is the ratio of battery capacity divided by autonomy: for example, though it’s a good thing for a 6000 mAh battery to provide two days of autonomy, it is even better if a smartphone with a 4000 mAh battery can provide the same experience and same level of autonomy as the phone with the larger battery, because it indicates that the latter smartphone has been optimized for greater efficiency.

Scores, rankings, and reviews

Our scoring system is based on the aggregated results for Autonomy, Charging, and Efficiency, and requires approximately 6.5 days of testing. In practice, that means 150 hours of charges and discharges, totaling 70 measurements (of which 30 are related to autonomy). We recombine these autonomy measurements with battery capacity to produce battery consumption information (for a total of 100 parameters).

We weight and score the cumulative measurements from each category’s respective sub-attributes, and then we use these weighted sub-scores to compute each of the three attribute scores (and from those three scores, the overall Battery score). Just as with our image quality and other test protocols, DXOMARK uses an open-scale scoring system, which means as the quality of batteries improves, the scores will also increase. This not only allows us to rank contemporary models, but will also allow us to rank and compare future devices with older smartphones.

After the scores are in, we rank each device according to how well it performed both globally (that is, among all smartphones we have tested) and within its specific price-related segment. These segments range from basic and affordable phones (“essential”) to ultra-premium devices. Although all smartphones are listed in the global ranking, it makes sense to give consumers the opportunity to compare the performance of devices at roughly similar prices, rather than simply comparing an economy model with a high-end flagship phone.

Finally, based on our comprehensive Battery test reports (which we make available to industry players), DXOMARK publishes reviews of our most important findings, including the three separate attribute scores, so that consumers like you can have this information available when looking to buy your next smartphone. Although we compute the overall Battery score from these three scores, of course, it’s important to note that our providing these three attribute scores allows people to choose a phone that most closely matches their own needs and the way they use their phones.

(For further information, see “How DXOMARK tests the smartphone battery experience.”)