The update includes four new doorbells in addition to last year’s models, all from major actors on the North American market: the new Google Nest Doorbell (wired, 2nd gen), the Arlo Wired Video Doorbell, the Ring Doorbell Pro 2, and the Wyze Video Doorbell Pro. With this latest generation of cameras, we can see real progress in terms of image quality. In general, the new devices perform better in our tests, and as a result rank higher than the 2021 models.

From left: Google Nest Doorbell (wired, 2nd gen), Ring Video Doorbell Pro 2,  Arlo Wired Video Doorbell, Wyze Video Doorbell Pro

For doorbell cameras, the image-quality requirements center on being able to identify the person who is at the door at all times. This means first having an accurate target exposure on faces in bright sunny conditions but also in strongly backlit situations and even at night, which is increasingly more difficult. Recognizing people also means having enough details on their faces. This comes with its own challenges, given the relatively low resolutions of doorbell cameras and the compression necessary to work with the cloud.

We put the eight doorbells through elements of our Camera protocol that was adapted for these devices, testing them in several lighting conditions ranging from day to night, with various dynamics and distances, to evaluate their performance. Here are their respective specifications:

In our first benchmark a year ago, the wired Google Nest Hello outperformed the other three doorbells we had tested at the time and Google continues to lead the pack with its latest Nest Doorbell (wired, 2nd gen). The battery-powered Wyze Video Doorbell Pro and the wired Ring Video Doorbell Pro 2 followed, pushing the models tested last year further down the ranking.

We also look at the breakdown of the scores and see how the doorbells performed specifically in daylight and night conditions. While the Google doorbell at the top of our ranking proved to be a well-rounded device with good performance in all conditions, that was not necessarily the case for the other doorbell cameras.

About DXOMARK Doorbell camera tests

Like for all DXOMARK test protocols, our doorbell evaluations take place in laboratories and in real-world situations using a variety of subjects. The scores rely on objective tests for which the results are calculated directly by measurement software on our laboratory setups, and on perceptual tests in which a sophisticated set of metrics allow a panel of image experts to compare aspects of image quality that require human judgment.

The following section gathers key elements of DXOMARK’s image quality tests and analyses for video doorbells. So let’s dive into the results of the protocol by looking at the specific use cases to better understand what sets the new Google Nest Doorbell (wired, 2nd gen) apart from the rest.

Daylight use cases

The Daylight use case is focused on the evaluation of the different image quality branches, mainly on portrait scenes, especially where the subjects are close to the camera. Among the image quality attributes, exposure and detail preservation are judged to be the most important, because they can influence the face detection and identification of subjects. The artifacts, like blocking and compression, can equally influence the overall image quality, leading to potential detection or identification failures. In the context of doorbells, in daylight conditions, accurate color rendering and white balance are not a requirement but a nice feature to have for the user experience. Finally, given the overall low preservation of details common to most doorbell cameras, noise is generally rarely an issue, so the weight of the noise evaluation is lower compared with the other image quality attributes.

Daylight use cases range from well-lit conditions on a sunny day, to a strong backlit situation, passing by a cloudy day. Doorbells must adapt to all these situations to be usable anytime. This chart shows the Daylight use case scores for all tested models.

The results for the Daylight use reflect the overall ranking, with the Google Nest doorbell (wired, 2nd gen) at the top of the list. Because daylight use cases are the most common ones, this is not necessarily a surprise.

We can also point out that battery-powered doorbells tend to be in the lower part of the ranking. This is likely the trade-off  between conserving battery power and the amount of heavy processing that is often necessary to produce a high-quality image. As a result, the wired doorbells tended to have better image quality because image processing didn’t have to come at the expense of power. The exception, however, was the battery-powered Wyze doorbell, which performed well enough to reach third place in the Daylight ranking.

Google Nest Doorbell (wired, 2nd gen): accurate target exposure on face and limited clipping in sky

Google Nest Doorbell (wired): slightly low target exposure on face, no clipping in the sky

Wyze Video Doorbell Pro: overexposed on face and background

Wyze Video Doorbell Pro: accurate target exposure on the face and acceptable level of detail even on background

Ring Video Doorbell 4: slightly low target exposure on the face and low level of detail

Google Nest Doorbell (battery): slightly low target exposure on the face and low level of detail

Night use cases

Night use case evaluation was focused on evaluating different image quality attributes of mainly portrait scenes, especially when the subjects were close to the camera. Such image quality attributes and their importance were similar to those for the daylight use cases with the exception of color. Most doorbells use infrared mode under a certain light level, which generally produces black and white images. Doorbells that provide a color mode at night, either by staying in visible mode longer or by recoloring the infrared feed, bring a significant advantage compared to cameras that remain in IR mode. In DXOMARK’s doorbell tests for night use cases, we used the same lab setups as for day use cases but with lower light levels. Also, the same real-life scenarios were performed but at night, after sunset.

Night use cases are more challenging for the small cameras in doorbells because the levels of light captured can be very low, resulting in images that contain noise and very few details. To work around those constraints, most doorbell cameras will switch to an infrared mode when light levels become very low, allowing for better exposure performances with less noise, but at the cost of color. This chart shows the Night use case scores of the test candidates.

The new Google Nest Doorbell (wired, 2nd gen) showed a strong performance in daylight situations, as well as in night situations, leaving the competitors far behind and taking the top spot in the Night ranking. The Ring Doorbell 4, which struggled a bit during daylight situations, showed a remarkable performance in night conditions, which allowed it to move close to the top of the Night ranking.

Google Nest Doorbell (wired, 2nd gen): accurate face exposure, good detail

Google Nest Doorbell (wired): accurate face exposure, lack of detail

Wyze Video Doorbell Pro: overexposure

Ring Video Doorbell 4: slight bright clipping on face but recognizable

Arlo Essential Video Doorbell Wire-free: face is strongly over-exposed and not recognizable

Google Nest Doorbell (wired): accurate target on face but low level of details person is hard to recognize

Let’s look deeper at the performance of the doorbells for the three main image quality attributes of exposure, texture, and artifacts.

Exposure

Most of the time, video doorbells will be used during the day to check on visitors or package deliveries. The key element in these situations is to be able to recognize and identify the person at the door, which requires a camera that provides good target exposure on the face.

In DXOMARK’s doorbell tests, exposure performances were measured in the lab on a setup including realistic mannequins and a light box, under controlled lighting conditions. Several conditions were reproduced, from bright light with low dynamic to low light with high dynamic up to EV4 difference. Results observed in the lab were backed-up by perceptual evaluations on real-life scenarios, where the doorbell cameras were placed outside, first in full light and then under the cover of an archway, to get more high dynamic conditions. In each case, a person approached the camera, so that we could evaluate the target exposure on a real person.

The Google Nest Doorbell (wired, 2nd gen) always delivered an accurate target exposure, coupled with a rather wide dynamic range, meaning that many details were preserved in the background. We tested this in the lab on our realistic mannequins and confirmed the behavior on a real scene, with both the doorbell and the subject well lit. In the same conditions, the previous Nest doorbells (both battery-powered or wired) tended to slightly underexpose the face, while doorbells like the Wyze Video Doorbell Pro, and the Arlo Doorbells to some extent, tended to over-expose the subjects.

*The graph shows the evolution of the lightness (measured in L*) with the level of lux, for multiple lighting conditions. The white area represents the region where the lightness is considered correct. Lightness is measured on the forehead of the left realistic mannequin (see setup example below).

Wyze Video Doorbell Pro, DuoHDR setup

Overexposed on the face and completely clipped in bright parts

Ring Doorbell Pro 2, Duo HDR setup

Accurate target exposure on face and completely clipped in bright parts

Google Nest Doorbell (battery), Duo HDR setup

Slightly under exposed on face and slight clipping in bright parts

Google Nest Doorbell (wired, 2nd gen): accurate target exposure on face and limited clipping in sky

Google Nest Doorbell (wired): slightly low target exposure on face, no clipping in the sky

Wyze Video Doorbell Pro: overexposed on face and background

Things generally get a bit more complicated when lighting conditions get more challenging. If the video doorbell is installed under a porch, for example, the dynamic range of the scene is increased because the camera is in the shadows and the scene is in the light. In those conditions, doorbells like the Google Nest Doorbell (wired, 2nd gen) or the Wyze start to struggle a bit, with target exposure barely high enough to recognize people. However, most of their competitors delivered even lower target exposure, making it nearly impossible to recognize who was at the door.

Google Nest Doorbell (wired, 2nd gen): slightly low target exposure on face, and low contrast; face is hard to fully recognize

Google Nest Doorbell (wired): slightly low target exposure on face

Wyze Video Doorbell Pro: very slightly underexposed but face is recognizable

During the night, most video doorbells have an infrared mode to help get better exposure when light levels are low. But even with that, good image quality is not guaranteed. In our real-life scenarios, we added an external lighting source to simulate a front door light. In those conditions, whether the new Nest activated its IR mode or not, target exposure was always accurate on the faces, allowing for easy recognition of the person. This was not the case for the competitors like the Wyze Video Doorbell Pro, below, which doesn’t switch to IR mode but has strong clipping on the face. On the Google Nest Doorbell (wired, 2nd gen), the background tended to be slightly underexposed, which meant some image elements were lost, but the background was not the main focus of the scene.

Google Nest Doorbell (wired, 2nd gen): IR mode is activated, accurate target on face, slightly low on background

Google Nest Doorbell (wired): IR mode is activated, accurate target

Wyze Video Doorbell Pro: overexposed

Google Nest Doorbell (wired, 2nd gen): accurate target on the face

Google Nest Doorbell (wired): accurate target

Wyze Video Doorbell Pro: overexposed

Texture

In DXOMARK’s Doorbell tests, texture performances were measured in the lab on a setup including realistic mannequins and a Colorchecker chart, under controlled lighting conditions. Several conditions were reproduced, from bright light to low light. Results observed in the lab were backed-up by perceptual evaluations on real-life scenarios, where the doorbell cameras were placed outside, first in full light and then under the cover of an archway, to get more high dynamic conditions. In each case, a person approached  the camera, so that we could evaluate the level of details rendered on a real person.

Good exposure is necessary to recognize people at the door, but a high level of image detail is necessary to positively identify people at the door. This is where the Nest doorbells outperformed the others, providing the highest levels of details when compared with the other doorbell cameras. In addition, a particularity of the Google Nest Doorbell (wired, 2nd gen) was that image details remained consistent in a given lighting condition, thanks in part to the nearly nonexistence of compression artifacts, which usually result in loss of details.

*This graph shows the evolution of facial details metric with respect to light conditions. Face detail metric is performed on the realistic mannequin face in the DXOMARK PortraitTimingColor set-up. The higher the metric the better the details preservation.

While most cameras maintained a generally consistent level of details through all lighting conditions, some, like the Google Nest Doorbell (wired, 2nd gen) or the Arlo Essential Video Doorbell Wire-free, lost some details quite significantly once they switched to infrared mode. You can see on the graph above that for the doorbells previously cited, this happened between 20 lux and 5 lux.

Arlo Wired Video Doorbell, 300 TL84

acceptable level of details

Arlo Wired Video Doorbell, 5Lux Tungsten

All details are lost

Ring Doorbell 4, 300Lux TL84

Acceptale level of details

Ring Doorbell 4, 5Lux Tungsten

Some details are lost

Nest Doorbell (wired, 2nd gen)

Acceptable level of detail

Nest Doorbell (wired)

Some loss of detail in the shadows

Wyze Video Doorbell Pro

Some loss of detail

Because of their placement, doorbell cameras can also be used to monitor a home’s front yard or driveway. The key test here is whether the camera is capable of providing a legible image of a car’s license plate. Our results showed that the Google Nest Doorbell (wired, 2nd gen) lacked the resolution to do this, and numbers and characters were barely distinguishable on cars that were parked about 4 meters from the camera. However, among the doorbells we tested, the Google Nest (wired) and the Ring Doorbell 4 were the only cameras that could provide a legible and clear-enough view of a car’s license plate.

Artifacts

In DXOMARK’s Doorbell tests, artifacts were evaluated on all tested scenes. The most common artifacts in video doorbells were blocking and compression artifacts due to video encoding and compression to be viewed on the cloud for example. Other common artifacts were color fringing in the corners of the image, due to the wide-angle lenses, ringing or even hue shift near saturation.

Most video doorbells suffered from video compression artifacts, but the Google Nest Doorbell (wired, 2nd gen) was generally free of them. These compression artifacts reduced the level of details in the image and could create an unpleasant frame reset effect when the compression level changes abruptly from one frame to another.

Distortion was also often visible on doorbell cameras. Nest doorbells generally were not too impacted by it thanks to their  different  field of view. The Google Nest Doorbell (wired, 2nd gen), in particular,  has a quite rare portrait format for video. This vertical field of view allows for a head-to-toe view of the person in front of the camera and avoids creating distortion on the sides. In contrast, the Wyze Video Doorbell Pro, for example, has a square video format, which results in a  very visible fish-eye effect.

While the Google Nest Doorbell (wired, 2nd gen)  avoided most of the common artifacts seen on the other doorbells, we noticed that the Nest showed some “ghosting,”  which appeared when people were close to the camera and moving. However, when people were standing still in front of the camera, the image was clear.

Steady improvements in image quality

Our testing showed many improvements in doorbell image quality from some of the major video doorbell brands. But the Google Nest Doorbell (wired, 2nd gen) was by far the best all-around performer. It did especially well for texture and exposure, making it the only one out of the eight cameras we tested that allowed for systematic and easy identification of faces in all light conditions, and earning itself the DXOMARK Gold label for Camera.

We’ll continue to test the latest video doorbells as they become available on the market and keep updating the rankings on an annual basis as we track the progress in image quality.

The post Video doorbells: 2022 ranking and comparisons appeared first on DXOMARK.

Strengthened by its expertise in camera image quality evaluation since 2003 (DSLRs, smartphones among others), DXOMARK is now extending its expertise to home surveillance cameras and smart doorbell cameras. The image quality of this new breed of connected devices is of particular interest to DXOMARK. Unlike smartphone or laptop cameras, where the focus is to produce images that are generally pleasing to the eye, the tiny sensors in security and doorbell cameras must focus on producing accurate facial details that would make it possible to identify the person in front of the camera.

DXOMARK has developed a new protocol that adapts its already stringent image-quality testing methods to evaluate doorbell cameras. It is the latest to join the company’s extensive family of protocols in smartphone camera, audio, display, and battery as well as video conferencing, wireless speakers, camera sensors, and lenses.
Currently focusing only on image quality, the setup is built based on the same unique combination of lab measurements and tests in real-life/natural scenes that have made DXOMARK the standard of image quality in the smartphone industry.

Test setup

Use cases are the basis of any DXOMARK protocol because they determine the types of lab setups, the real-life scenes to be shot, and the list of quality attributes to evaluate.

The scenes that we reproduce in the laboratory and outdoors are :
• An outdoor daylight scene to test the intrinsic quality of the camera (combination of lab and outdoor measurements)
• A backlit scene with a forefront in the shadow and a [glossary_exclude]background[/glossary_exclude] in the sun, to evaluate the HDR capability of the camera (combination of lab and outdoor measurements)
• A night scene, to either test the infrared (IR) mode or to test under a floodlight-helped color mode

DXOMARK lab setups (HDR and night vision) for security cameras and doorbell cameras

 DXOMARK natural scene setups (outdoor, HDR, night vision) for smart doorbell cameras

Field of view

The field of view is the cornerstone specification sheet of a doorbell or a security camera. Unlike most consumer cameras, security cameras claim their ability to cover an angle of up to 180°. However, in the specific context of doorbells, there isn’t an industry-level consensus yet on the right approach to field of view. Should the video format, for example, be square, or portrait?

Because of the diversity of the existing solutions, DXOMARK’s protocol tests each doorbell camera at its default field-of-view setting, respecting each manufacturer’s choices. However, we evaluate image quality attributes on every aspect of the image, including distortion artifacts. A doorbell model that claims to have a very wide viewing field but delivers a poor quality image would be impacted more in our evaluations than a device that had a more conservative field of view but guaranteed a higher-quality image — one that would allow recognition of the person in any part of the image.

The evaluation: Image quality attributes

We perform two kinds of evaluations on videos: objective and perceptual. Objective tests focus on testing the standards established by the industries, such as white balance and texture level, while perceptual evaluations are the quantification of qualitative parameters. Perceptual evaluation takes its roots in long-established scientific methods, also described in standards, like the International Telecommunications Union’s subjective video quality assessment methods for multimedia applications (ITU) ITU-T P.910.  Both objective measurements and perceptual evaluations are necessary to assess fully the image quality of a video, as they complement each other and lead to a more enriched assessment.

DXOMARK has developed its doorbell camera protocol around the most important image quality attributes for a security camera: Exposure, Details (texture and noise), and Artifacts, keeping in mind that the main purpose of a doorbell camera is to recognize the face of a person at the door, whether it is a friend or an intruder.

Exposure

Good exposure is crucial to properly identify a face. The DXOMARK doorbell testing protocol evaluates four parameters related to exposure.
• Exposure on the person, which is the amount of light on the individual that allows for their correct identification; when the identification is guaranteed, we then use our expertise from the consumer camera world to evaluate whether or not the representation is also pleasing to the viewer.
• The dynamic range of the camera, i.e., its ability to correctly render the dark areas of the scene as well as the bright areas. In particular, we test here the HDR performance of the doorbell camera.
• The contrast, which is the gradient of differences between dark and light areas on the video. Contrast is especially key in the context of HDR pictures, as a poorly balanced HDR setting can lead to unnatural images, which can in return make the identification of the person difficult for the viewer.
• Exposure adaptation, which is the ability of the camera to adjust, in real-time, the exposure when the lighting conditions suddenly change.

Details, Texture & Noise

Related to the need to identify the person at your door, the measurements related to  details, texture, and noise explore all the elements of the picture related to the clarity of the image:
• Although details can be roughly estimated with a resolution chart, they must also pass real-life scene tests, such as the ability to read the logo on a shirt or even a car license plate.
• Texture evaluates the way the camera depicts the details of facial characteristics such as a beard or skin, as well as surrounding areas such as grass and bushes. DXOMARK has developed a specific perceptual evaluation on realistic mannequins to complement the tests of natural scenes.
• Finally, noise assesses the graininess of the overall image. It is worth noting that here, again, the goal is to identify people, so having a low level of noise is not as important as having a high level of detail.

Artifacts

In this attribute evaluation, we chase mostly three types of artifacts: compression, color fringing, and distortion.
• Compression appears when the doorbell needs to transmit the video to a server, in real-time, forcing a rather high compression level. When the scene changes rapidly, the ISP sometimes fails at keeping up with the pixel count, and a phenomenon known as blocking appears. This is particularly true of battery-based systems.
• Color fringing is a type of chromatic aberration often caused by the failure of the camera at focusing all colors on one point; this is often seen at the edge of a subject, separating the foreground from the background.
• Distortion is very often present on wide-angle lenses of cameras. What matters in the context of doorbells and security cameras is whether the distortion could impede the possibility of identifying the person being filmed. The DXOMARK protocol measures not only the objective distortion but also the impact in real scenes, through its perceptual analysis methodology.

Conclusion

DXOMARK has put several doorbell cameras through its rigorous testing protocol. They cover all price points and regions. All the leaders of the market are represented, including Google Nest, Ring, and Arlo. The results are still being tallied, and we plan to publish the results very soon on dxomark.com!

The post A brief introduction to how we test doorbell cameras appeared first on DXOMARK.

The quality of the battery’s performance goes far beyond the battery’s size. How long a charge lasts depends on several factors, including the phone’s hardware and software; the kinds of features it has; whether it runs processes in the background, when actively used and/or when in idle mode; and of course, how much, when and in what ways people use their phones.

The challenge to manufacturers is to find the right balance between high-end features and battery life. It is not an easy task to respond to consumers’ expectations for the most powerful chipset, the best and brightest displays, along with (for example) providing 5G and other power-consuming connectivity, better and better camera and video functionality, while also ensuring that the phone’s charge will last sufficiently long so as not to require too-frequent charging.

Let’s begin with an in-depth description of the equipment we use, after which we’ll explain more about the specific test routines and use cases on which we base our scores.

Our testing tools

Faraday cage

The latest among DXOMARK’s laboratories is the one dedicated to battery testing, a major component of which is our Faraday cage. The walls, ceiling, and floor of this room is composed of 2 mm-thick steel panels, which serve to isolate smartphones from external networks and other disturbances that could affect measurement precision and repeatability. Inside the Faraday cage, a WiFi access point and a cellular repeater provide stable connectivity. An outdoor antenna located on the rooftop of our building receives a signal from the French network operator Orange, and our network repeater amplifies it to a pre-defined level and disseminates it inside the cage via the in-cage antenna (–100 dB for 4G, –80 dB for 3G, and –70 dB for 2G).

Robots in the Faraday cage

Touch robots

So far, we have an array of four touch-robots inside the Faraday cage that we use in two of our three major autonomy test routines, home/office, and calibrated [glossary_exclude]use case[/glossary_exclude]. In addition to their touch functions, which are programmed to use the same gestures as humans to interact with smartphones (scrolling, swiping, tapping), they are equipped with computer vision so they can recognize the various app icons, keyboards, and even react to intrusive popup messages. Further, each robot is equipped with a lighting system that reproduces lighting conditions at home, outside, and at the office. The intensity and the color temperature vary depending on the time of the day, and this forces the phone to adapt and adjust its brightness (which can have a significant impact on power consumption). Smart power sockets are installed near the robots to control phones’ charging level. For instance, they can simultaneously stop the charging of 4 devices fully charged to start a test at the same time, or even start charging a device at a specific battery level.

Robots are equipped with simple push-button actuators to wake up the screen before each test case, or from time to time just to mimic quick checks of notifications or of time during the day. Four of our robots work simultaneously and are controlled by a sequencer, which triggers all test routine events, the lighting system and the smart power sockets. We use a fifth robot to run a setup wizard before the test routine begins to verify that the other robots properly recognize each icon and that their gestures are adapted to the specific UI (icon design, gestures, screen size, layout) of the device under test.

We test phones using their default settings out of the box; the only thing we deactivate is 5G because our lab is not covered yet and devices supporting 5G connectivity would be negatively and unfairly impacted otherwise. (We will add 5G measurements to our protocol as soon as our lab has coverage.)

Oscilloscope

We use a Rohde & Schwartz RTE 10124 oscilloscope with current and voltage probes to measure both primary and secondary power over time. (Primary power is the energy taken from the wall outlet before it enters the charger; secondary power is the energy the charger delivers before it enters the smartphone.) To be able to measure the secondary power, we designed specific PCB cards (one USB-A and one USB-C) that allow plugging current and voltage probes without affecting the charging protocols.

Battery protocol tests

Autonomy

Home/Office

In our home/office typical usage scenario, the smartphones start from a full battery charge and our robots run through a precisely defined 24-hour cycle of activities and then repeat it until the phones’ batteries are completely depleted (that is, when the phones switch themselves off). The system monitors the battery gauges at every stage of the cycle to measure how much battery power has been consumed and what percentage of battery power is actually left.

Faraday cage

The touch robot

The 24-hour scenario includes 4 hours of screen-on time per day, which research indicates is the average time of use around the world*, and multiple types of activities: social and communications, music and video, and gaming, among others, using the latest versions of some of the most popular applications here in Europe, where our lab is located. (And speaking of our location, please note it is possible that some test results will vary if conducted elsewhere because of differences in network quality, available applications, and so on.)

The DXOMARK robots at work.

On the go

Mobile phones are, well, mobile, so we include an array of tests to see how smartphone batteries are affected when we are “on the go.” Just as for our stationary robotic testing, we set all phones to their default settings, but here turn on the location and turn off  WiFi and 5G. We bring along a reference phone (always the same one) to help us take into account fluctuations in weather, temperature, etc.

Our on the go tests include the kinds of activities people often do when commuting on public transport, such as making phone calls and browsing social apps; and we also test activities when traveling in a car (GPS navigation for example), and when on foot (streaming music, shooting photos and videos). We start each test case at different checkpoints along the predefined route and run it until the next checkpoint, where we measure its consumption before starting the next test.

Calibrated

For our calibrated [glossary_exclude]use case[glossary_exclude] tests conducted back in the Faraday cage, we have our robots repeat sequences of activities that belong to a particular [glossary_exclude]use case[/glossary_exclude]. Here are our current set of use cases:

• • Video streaming (in cellular & Wi-Fi)
  • Video playback
  • Music streaming
  • Gaming
  • Calls
  • Idle

“Calibrated” refers to the fact that we use the same settings and application parameters for each phone we test — for example, we set display brightness to 200 cd/m2; we measure sound coming from the phone’s speaker at a [glossary_exclude]distance[/glossary_exclude] of 20 cm; we set the phone speaker [glossary_exclude]volume[/glossary_exclude] level to 60 dB; we ensure that the ambient lighting conditions are the same. And then we measure how much power each of these activities consumes so that you will have an idea of how much time you will have to (for example) play an intense video game or how many hours of music you’ll be able to listen to.

The results of these three autonomy test categories will let you know how much battery life (in dd:hh:mm) you can expect from a given smartphone, including how much power it loses overnight when you’re not actively using it, and how much power specific kinds of activities consume. Going further, we’ve been able to devise 3 different autonomy profiles based on the results of our typical usage scenario and on-the-go test cases: light, moderate, and intense. In our estimation, light use means 2.5 hours of active use per day; moderate means 4 hours of active use; and intense means 7 hours of active use. These profile estimates are intended to give you a better idea of the kind of autonomy you can expect based on how much you use your phone.

Linearity

One other aspect of our Autonomy tests focuses on how accurate a smartphone’s battery power indicator or gauge is. It’s long been known that the battery percentages shown on the display user interface do not always accurately reflect the exact amount of power remaining in the battery. This can mean that two phones with the same battery size and whose gauges indicate 10% power remaining may run out of power at very different times.
To measure battery linearity, we have designed a [glossary_exclude]use case[/glossary_exclude] that drains a constant amount of power from the battery. After fully charging the battery, we play a video displaying a white screen with no sound in full-screen mode. The phones are set to airplane mode, fixed refresh rate, fixed screen resolution, and put at their maximum brightness.
We perform this measurement twice for each device. If the phone’s gauge shows 20% battery life remaining, but the actual power remaining is less than 20%, we deduct points from its score, because there is nothing more frustrating than seeing your precious last 20% battery percentage quickly collapse!

Charging

Our Charging score is based on two sub-scores, full charge and quick boost. When conducting these tests, we either use the official charger and cables provided with the phone or buy a recommended charger from the manufacturer’s official website.

Charging setup

Full charge

After we fully deplete the smartphone’s battery, we measure how much power and time it takes for the phone to charge from zero  to full charge, as indicated by our probes. We also indicate when 80% of a full charge is reached, as well as the time when the battery indicator says 100%. We deduct points depending on how much power is added to the charge after the smartphone gauge indicates 100%.

We also measure the primary power and the speed of wireless charging for those devices equipped with that feature.

Quick boost

In our quick boost tests, we measure the power gained from plugging in a smartphone for five minutes at various levels of the battery’s charge— 20%, 40%, and 60%, as how much charge the battery has left can make a significant difference how much power it takes on in that short time.

An engineer using the oscilloscope during a smartphone battery test.

In another test, we play Asphalt 9 from Gameloft for a [glossary_exclude]minimum[/glossary_exclude] of 20 minutes until the battery gauge indicates that 5% battery is left, and then we plug the phone to a charger to check on how much and how quickly power is drawn from the wall outlet. This helps us check the impact of intense gaming on the phone, as phones that are hot from heavy use take a charge differently than phones that are not.

Efficiency

Charge up

Our Efficiency score is partly based on measurements of a charge up — that is, the ability to transfer power from a power outlet to a smartphone, and how much residual power is consumed after the phone is fully charged and when detached from the charger, as measured with our probes and oscilloscope.

Let’s take the example of a 5000 mAh battery with a nominal voltage of 4V. We consider that the typical energy capacity of this battery is 20 Wh (Watt-hours = 5 Ah x 4V). In our Charging test plan, we measure the power drawn from the power outlet for a full charge. Let’s say we measure 25 Wh, which means the charge has an efficiency of 80% (battery has stored 20 Wh divided by the 25 Wh cost of energy).

We also calculate the travel adapter efficiency. It’s simply the ratio of the secondary power drawn (after the travel adapter, in Wh) to the primary power drawn (before the travel adapter, in Wh).

An engineer conducting and monitoring a smartphone battery test in the Faraday cage.

In our reference database, the charge efficiencies we measure range from 63% to 88.6%, and the travel adapter efficiencies range from 80% to 94%. When our tested smartphones are fully charged but still plugged into the charger, residual power consumption ranges from 90 mW to 850 mW; and when the smartphones are unplugged from the charger, but the charger is still plugged into the outlet, residual consumption ranges from 10 to 150 mW.

Discharge

We also calculate the Discharge efficiency, which is the ratio of battery capacity divided by the results from our stationary and calibrated [glossary_exclude]use case[/glossary_exclude] Autonomy sub-scores.

Why do we rate efficiency? While the impact of your smartphone on the electricity bill is negligible compared to heating or lighting, of course, if your smartphone is power efficient, a smaller battery will suffice (making your smartphone lighter and more compact). Good efficiency also demonstrates the quality of design and software robustness. In other words, an efficient device is better built.

Scoring

To briefly recap our scoring system (which we explained in more detail in our introductory article), we compute our overall Battery score from three sub-scores — Autonomy, Charging, and Efficiency. We calculate our Autonomy score from the results of three different kinds of tests: stationary, on the go, and calibrated use cases, along with battery linearity. Our Charging score takes into account full charge and quick boost results. And finally, our Efficiency score is based on charge up (the efficacity of the adapter) as well as discharge (the overall power consumption measured in our typical usage scenario and in our calibrated use cases).

We hope this article has given you greater insight into the equipment we use and the tests we undertake to produce our DXOMARK Battery reviews and rankings.

The post A closer look at how DXOMARK tests the smartphone battery experience appeared first on DXOMARK.

Test environments

Depending on the protocol, measurements may be done in different environments. While some recordings are purposely made in real-life settings – whether indoor or outdoor, a vast majority of our measurements are conducted under laboratory conditions for greater consistency.
Using a ring of speakers in an acoustically neutral room, our engineers can simulate any environment for recording purposes. Likewise, an acoustically treated room is dedicated to playback perceptual evaluation. Acoustic treatment of these rooms ensures a well-balanced frequency response.

As most objective measurements require very strict conditions, we test our devices in anechoic settings, thanks to a specially designed anechoic box that eliminates most of the sound reflections inside it. Measurements requiring larger setups are made within our custom-made anechoic room, which accommodates a wide range of protocols requiring minimized sound reflections, such as our audio zoom protocol.

DXOMARK’s anechoic chamber

Objective testing tools

The better part of audio testing relies on the ability to both convey and capture sound in a precise manner, hence the importance of using scientific-grade measurement microphones tuned with a sound-level calibrator, as well as carefully optimized loudspeakers to use as sound sources, crucial components of a controlled audio chain.

From left: An optimized Genelec 8010 speaker, a scientific-grade Earthworks Audio M23R microphone for measurement, a precision sound level calibrator (model CAL 200) for laboratory use, and a motorized rotary stage X-RSW60A-E03.

The device under test (DUT) can be secured with a clamp mounted on a stand, or a magnetic holder. In some cases, the DUT is mounted on a custom-made rotating stand, which rotation is automated via computer-controlled servo motors, allowing for precise 360° measurements.

Objective measurements are processed by our engineers using a set of software depending on the measurement type. Frequency responses, directivity and THD+N are processed by custom python libraries developed by our acoustic engineers, following state-of-the-art signal processing methods largely used by the audio industry. Other measurements, like volume measurement, use freeware tools like REW for their straightforward applications. Objective measurements, once processed, are scored using scoring algorithms developed internally and taking into account specific criteria depending on the measurement (flatness/dispersion of the frequency response, loudness values, percentage of [glossary_exclude]distortion[/glossary_exclude] by frequency bands, etc.), and carefully selected to best match the user experience and perceptual listening.

Perceptual testing tools

While objective measurements give us hints about how a smartphone may sound, nothing reflects the user experience better than a thorough perceptual evaluation – the human ear is an irreplaceable and complex tool that can provide unique, qualitative information no other tool could provide. Therefore, objective and perceptual complement and reinforce each other.

An engineer in one of DXOMARK’s audio laboratories monitoring the testing of a smartphone.

Keen ear accuracy is an essential skill of our experienced audio engineers, who further receive extensive training upon recruitment. DUTs are evaluated comparatively for playback against up to 5 other devices, as well as studio monitors calibrated as reference. Perceptual evaluations follow a strict protocol articulated around discretized evaluation items to ensure the precision of results, and they undergo a careful two-pass check involving different engineers, to eliminate bias.

Playback evaluations take place in an acoustically treated room. For the most part, devices are mounted on a semicircular arm on a stand, so that all comparison devices are equidistant from the engineer in charge of the evaluations.
Reflective panels are used to enhance spatial features of the devices and improve the quality of evaluations such as stereo wideness and localizability.

The laboratory set up for testing spatial playback capabilities with reflective panels.

Microphone evaluations are performed with studio headphones standardized for all our engineers, and they follow the same rigorous protocol as our playback evaluation.
All the smartphones previously tested in our labs can still be used as reference devices for perceptual evaluation, either for playback, for recording or both. This helps our database of evaluations and scores to stay consistent even after many years of testing.

Audio quality attributes

Audio quality attributes have been defined in accordance with the report issued by the International Telecommunication Union (ITU): ITU-R BS.2399: Methods for selecting and describing attributes and terms, in the preparation of subjective tests in a motion to standardize sound defining vocabulary, as illustrated by the wheel below. Dealing with perceptual evaluation means establishing a common understanding of the meaning of the words we use, and their definition.

An audio wheel showing the attributes and subattributes as well as the terms used for evaluation.

From the common descriptors, we can make out larger groups that constitute our main audio attributes. These attributes are subdivided into individual constituting qualities that we call sub-attributes.

Timbre

Timbre describes a device’s ability to render the correct frequency response according to the use case and users’ expectations, looking at bass, midrange, and treble frequencies, as well as the balance among them.
Good tonal balance typically consists of an even distribution of these frequencies according to the reference audio track or original material. Tonal balance is evaluated at different volumes depending on the use cases.
In addition, it is important to look for unwanted resonances and notches in each of the frequency regions as well as extensions for low- and high-end frequencies.

Dynamics

Dynamics covers a device’s ability to render loudness variations and to convey punch as well as clear attack and bass precision. Dynamics are the cornerstone of concepts such as groove, precision, punch, and many more. Musical elements such as snare drums, pizzicato, or piano notes, would sound blurry and imprecise with loose dynamics rendering, and it could hinder the listening experience. This is also the case with movies and games, where action segments could easily feel sloppy without proper dynamics rendering.
For a given sound, dynamics information is mostly carried by the envelope of the signal. Let’s take a look at a single bass line for instance: not only would the attack need to be clearly defined for notes to be distinct from each other, but sustain also needs to be rendered accurately for the original musical feeling to be conveyed
As part of dynamics, we also test the overall volume dependency, or in other words, how the attack, punch, and bass precision change based on the user volume step.
In addition, the Signal-to-Noise Ratio (SNR) is also assessed in microphone evaluation.

Spatial

Spatial describes a device’s ability to render a virtual sound scene truthful to reality.
It includes perceived wideness and depth of the sound scene as well as left/right Balance, Localizability of individual sources in a virtual sound field and their perceived distance.
As expected, monophonic playback in a smartphone is usually not a good sign for a good spatial rendition, if not for a good playback performance at all. But many impediments can hinder spatial features, such as inverted stereo rendering, or uneven stereo balance. Thankfully, these problems are less and less common. On the other hand, some sensitive details such as precise localizability or appreciable depth are much harder to fine tune, thus being recurring shortfalls in smartphone audio.
Spatial conveys the feeling of immersion and makes for a better experience whether in music or movies.
In the Recording protocol, capture directivity is also evaluated.

Volume

The volume attribute covers the perceived loudness whether in recording or playback, the consistency of volume steps on a smartphone, as well as the ability to render both quiet and loud sonic material without defect. This last item involves both perceptual evaluation and objective measurements.

Here Device A has a good volume consistency, with volume steps homogeneously distributed between its minimum and maximum values, with an almost consistent slope and no discontinuities or volume jumps. On the contrary, Device B has an inconsistent volume step distribution, with no precision in its low volume steps, enormous jumps in volume, and almost 5 identical volumes steps at its maximum volumes.

Artifacts

Artifact refers to any accidental or unwanted sound, resulting from a device’s design or its tuning. Artifacts can also be caused by user interaction with the device, such as changing the volume level, play/pausing, or simply handling the device – which is why we specifically assess a device’s handling of speakers and microphones occlusion. Lastly, artifacts may result from a device struggling to handle environmental constraints, such as wind noise in the recording use cases.
Artifacts are grouped into mainly two categories as they can be temporal (pumping, clicks…) or spectral ([glossary_exclude]distortion[/glossary_exclude], continuous noise, phasing…).

Background

The audio background attribute is specific to the Recording use cases, as it only focuses on the background of recorded content. Background covers some of the audio aspects mentioned above, such as tonal balance, directivity, and artifacts.

Audio protocol tests

DXOMARK’s Audio testing protocols are based on specific use cases that reflect the most common ways in which people use their phones: listening to music or podcasts, watching movies and videos, recording videos, selfie videos, concerts, or events, etc.  These use cases have been grouped into two protocols: Playback and Recording. Each use case covers the attributes and sub-attributes that are relevant to the evaluation.

Playback

According to a survey we conducted on 1,550 participants, movie/video viewing accounts for most of the smartphone speakers’ usage, followed by music/podcast listening, and then gaming. Our Playback protocol covers the evaluation of the following attributes: timbre, dynamics, spatial, volume, and artifacts.

Objective tests

Before our audio engineers delve into perceptual evaluations, any DUT undergoes a series of objective measurements in our labs. Regarding the playback protocol, these tests focus on volume, timbre, and artifacts.
Measurements are done within the anechoic box, which houses an array of calibrated microphones, a speaker, and an adjustable arm to magnetically attach a smartphone from either side. The interior of the box is lined with fiberglass wedges that cover the entire ceiling, floor, and walls, ensuring the dissipation of all energy from sound waves, thus strongly reducing reflections: only direct sound coming from the DUT is captured by the microphones.

The anechoic box set up with scientific-grade microphones and a speaker to test a smartphone’s audio capabilities.

A smartphone positioned in front of a speaker for testing in the anechoic box.

Objective tests are done using various synthetic signals (pink noise, white noise, swept sines, multi-tones) as well as musical content.
The table below summarizes the objective tests conducted for the Playback protocol.

Movies/Videos

As many users watch videos and movies with the integrated speakers of their phone, this use case has more weight in the playback part of our audio protocol. DXOMARK aims to provide a comprehensive perceptual evaluation focusing on how well the audio content from a movie or a video are rendered by the DUT’s internal speakers.
Tonal balance should be in line with the original material. Voice clarity is particularly important, but we also look at the overall richness of timbre, as well as the precision and impact of the low-end. Volume variations might be important in a movie or video, so we test the DUT’s handling of broad dynamic range, on the lookout for excessive compression.
Using our reflective panels, we assess the wideness of a device’s rendered stereo scene, as well as the localizability and depth of various audio-pictural elements.

Music

Smartphone audio has improved significantly over the past years, and surveys show that a surprisingly large number of users frequently listen to music on their phone’s internal speakers. With this in mind, our Music use case covers an expansive variety of genres.
Evaluation encompasses multiple aspects deemed to be relevant, such as the tonal balance’s truthfulness to the reference track, with proper repartition of bass, midrange, and treble. More often than not, smartphone playback tends to lack a bit of low and high-end frequencies, so we value the extra effort put towards a broader frequency response. We also pay close attention to bumps or notches in the spectrum, and we evaluate the consistency of tonal balance at different volumes.
Dynamics-related qualities such as attack, bass precision, or punch, are evaluated at different volumes as well. For instance, the presence of compression at maximum volume may hinder attack or bass precision, and punch may not be as good at low volume.
As with the Movies use case, evaluation encompasses spatial aspects such as wideness and depth of the rendered stereo scene, as well as localizability of instruments and voice. These sub-attributes are notably tested not only in landscape orientation but also in portrait and inverted landscape.
Maximum volume should be as loud as possible without excessive [glossary_exclude]distortion[/glossary_exclude] or compression. Smartphone volume not being loud enough is often commonplace. In the same manner, minimum volume should be quiet enough but still very intelligible.

Gaming

This use case answers to the growing use of smartphones for gaming. With chipsets and RAM performance skyrocketing, mobile games are becoming more and more performant, and so should smartphones regarding their audio capabilities.
DXOMARK’s audio gaming use case aims to evaluate the immersion audio provides to a game, meaning wideness and especially localizability should be on point. Impactful punch and good bass power are also essential.
These sub-attributes are evaluated at multiple volume steps, as the gaming experience must be optimal regardless of level. We look for potential timbre deterioration at maximum volume, as well as artifacts such as [glossary_exclude]distortion[/glossary_exclude] and compression.
We also test for speakers’ occlusion during gaming, as the sound coming from speakers might easily be blocked by a user’s hands during intense gaming sessions. This heavily depends on speaker placement on the phone, but also on the mechanical design of the speaker output holes.

Recording

Objective tests

Objective recording measurements focus on up to three attributes: timbre, volume and directivity. Frequency response is computed for the main camera app, and the default memo recorder app. The Max Loudness measurement consists in testing a device’s capabilities at handling very loud recording.
Timbre and Volume tests are conducted within the anechoic box using the speaker within it, while the Audio Zoom objective tests, requiring much more space, are performed within the anechoic chamber
The following table summarizes the tests conducted under the Recording protocol.

Simulated use cases

The simulated use cases are a series of recordings performed in an acoustically treated room using a ring of speakers. Using different combinations of pre-recorded background and voices, it is possible to recreate multiple scenarios relevant to common uses of a device’s microphones.
Simulating these environments allows for consistent recordings, in addition to easing the process of capturing a variety of situations.
The following table goes over the simulated use cases deemed most important:

The recorded simulated use cases are evaluated perceptually by our audio engineers, with user expectations in mind, including attenuation of voices out of the field of view and background noises, clear localization and perception of distance, wide and immersive stereo scene, faithful and natural tonal balance with intelligible speech, among other sub-attributes.

Indoor / Outdoor

The indoor/outdoor use cases are a complement to the previous simulated use cases, in that the recordings are done in situ and not in our labs. These tests focus on intelligibility, recording volume, and SNR. Recordings are done using a specially designed rig to hold up to 4 smartphones at once, and they are performed in either outdoors or indoors settings, with an announcer delivering Harvard sentences clearly at a set distance. The outdoor scenario features passing cars on a nearby road as well as some moderate wind, while the indoor scenario features a vacuum cleaner functioning in the background. For each scenario, the DUT is set up in three specific settings: landscape orientation + rear camera, portrait orientation + front camera, and portrait orientation + memo app.

Concert

As smartphones are commonly used to immortalize concerts and other events, this use case is designed to assess how well a device can handle the recording of music at a very loud volume.
Tests are performed within the anechoic box, where the DUT records a set of musical tracks at 115dBA. Each track features common grounds such as bass, drums, and vocals, but they offer significant differences in terms of genre, instrumentation, and mix.
Since the test conditions are intentionally extreme, one key issue addressed by the evaluation is of course the handling of artifacts, such as [glossary_exclude]distortion[/glossary_exclude], compression, pumping. Tonal balance is also in the limelight, with a special emphasis on musicality. Regarding dynamics, multiple elements are subjects to evaluation, such as overall punch, bass precision, and drums snappiness for instance.
This use case is also an opportunity to test a DUT’s audio zoom capability, by zooming on a specific element: the ability to successfully isolate a single element from the rest in the audio scene (including background noise) is most certainly a cutting-edge feature.

Occlusion

Depending on a phone’s construction, it is not uncommon to accidentally block one or more several microphones while recording. The aim of this use case is to assess how easily microphones can be blocked, and how the DUT’s audio processing handles it.
Recordings are done in landscape and portrait orientations each using front and rear cameras, as well as portrait and inverted portrait orientations when it comes to the memo app. Tests are performed with pre-established sets of hand positioning, while the engineer enunciates a series of sentences.
Perceptual evaluation focuses solely on the undesired effects potentially induced by hand misplacement, or more rarely improper DSP (Digital Signal Processing).

Wind noise

The presence of wind noise in a smartphone recording can be frustrating. Incorporating this use case into our audio protocol meets the increasing attention put towards the reduction of wind noise’s effect in recordings. Manufacturers can achieve these results with DSP, careful internal microphones placement, and usually a combination of both.
To attain consistency and precision in our measurements, the tests are performed in controlled conditions with a wind machine and a rotating smartphone stand, both automated via scripts. Four calibrated speakers are arranged around the rotating stand, so that the test speech is always diffused frontally in relation to the DUT: this way, the incidence of wind can be isolated as a factor. The wind machine is set consecutively at three gradual speeds, in addition to a reference step without wind. Recordings are conducted with three settings: landscape mode and main camera; portrait mode and selfie camera; portrait mode and memo app.

A smartphone undergoing a test for wind noise management.

The smartphone, right, is placed in front of the wind machine, and rotates.

The table below covers the parameters set for the measurement:

In addition to speech sequences, pink noise is used to measure wind rejection ratio. Other objective tests include the calculation of wind energy, and two-tracks correlation giving out reliable SNR values.

But objective measurements are just a small portion of our tests, which are mostly perceptual. Evaluation focuses essentially on intelligibility, with the help of a set of standardized evaluation rules. Artifacts are also considered during the evaluation.

Audio zoom

Audio zoom is a form of audio separation and filtering, which aims to isolate a sound source from its surroundings in accordance with the smartphone camera’s focal point and zoom level. This technology is becoming more and more prevalent in newer smartphones, and it is a notable feature in audio processing that can help manufacturers emerge from the competition.
You can read more about this technology here: dxomark.com/what-is-audio-zoom-for-smartphones
Audio Zoom recordings take place in the anechoic room, with the DUT in landscape orientation with main camera on. A pair of speakers are arranged in the corners of the room behind the smartphone as they emit background noise. One speaker directly in front at a distance of 3 meters, with a dummy head beneath it, handles playback of the main signal (speech, or music).

When zooming on the dummy head during recording, a smartphone with audio zoom capabilities is expected to isolate the main signal from the background more and more as zoom level increases. Using our automated rotating stand and a logarithmic swept sine, we measure the DUT’s directivity at three distinct levels of zoom, being: wide (x1), telephoto, and super telephoto. After that, if the DUT is proven to have audio zoom through objective measurements, we then perform a series of tests for each zoom level using two types of signals: speech and music. These recordings are subject to perceptual evaluation by our audio engineers.

Multiple sub-attributes are assessed during the evaluation: side rejection, which corresponds to the strength of the audio separation, volume consistency, which involves rating the correlation between zoom level and volume increment, but also tonal balance. Indeed, it is relevant to check the timbral integrity of the main signal after such processing is applied to it. While some audio zoom implementations are cut for speech, the handling of musical instruments is not always on point; not only can timbre deteriorate, but the DSP may also malfunction and even induce artifacts, which we also consider.

We hope this article has given you a more detailed idea about some of the scientific equipment and methods we use to test the most important characteristics of your smartphone audio.

The post A closer look at the DXOMARK Audio protocol appeared first on DXOMARK.

Before we head into the main topic, it’s important to remember that smartphone display performance isn’t just about display panel quality. Smartphones use programs or algorithms to control many display functions, and manufacturers choose which settings to use with those algorithms (a process known as “tuning”). Of course, some algorithms are more efficient than others, and the way an algorithm is implemented on a smartphone can make a big difference in performance, as in these examples:

• Software determines how smartphones [glossary_exclude]balance[/glossary_exclude] the trade-off between [glossary_exclude]frame rate[/glossary_exclude] and battery usage; depending on the apps used, some phones automatically adjust the [glossary_exclude]frame rate[glossary_exclude] to extend a battery charge (and thus autonomy). What this means is that a smartphone with a refresh rate of 120 Hz does not always refresh the screen at 120 Hz (for example).
• Many smartphones include an ambient light sensor, a photodetector that gauges surrounding lighting conditions; tuning determines how quickly and appropriately the auto-brightness feature responds to the input from the light sensor, in addition to how well the display adapts to the content being viewed.
• When people watch videos on their phones, motion interpolation algorithms generate frames in between “real” (existing) frames with the aim of making animations or moving actions appear smoother, and again, the battery vs. frame rate trade-off can have an impact here. (We will visit algorithms again in some of our articles about specific display attributes.)

DXOMARK conducts tests under many different (and sometimes changing) light conditions so as to recreate as closely as possible the real-world experiences of smartphone users, rather than just simply pitting display performance against “ideal” viewing conditions as defined in standards/norms.

Finally, as we head into our toolbox, just a short reminder that first, we test each and every display under the exact same conditions so as to ensure that our results are fair, scientifically rigorous, and repeatable. Second, apart from certain well-defined exceptions, we always test devices using their default settings. And third, DXOMARK measurements differ from those of other sites in that we not only include lab-based objective measurements, but perceptual measurements as well.

Objective testing tools

The images below show the array of tools our evaluation experts use when testing smartphone display performance. We use the tools below to measure color, [glossary_exclude]luminance[/glossary_exclude] (brightness), and contrast, and to count [glossary_exclude]frame[/glossary_exclude] drops:

Testing devices that DXOMARK uses to measure display quality, from left to right, spectroradiometer, video colorimeter, video colorimeter with conoscope, and compact camera.

We use these tools to measure reflectance, gloss, [glossary_exclude]flicker[/glossary_exclude], and illuminance:

Other testing devices, from left to right: spectrophotometer, glossmeter, flickermeter, and lux meter

We use the tools below to measure touch responsiveness, accuracy, and smoothness:

Ultra-high-speed camera and robot for measuring display touch attributes

We conduct many of our objective tests within the DXOMARK Display Bench, which we unveiled at launch; the Bench is a special testing chamber that facilitates testing automation and ensures that our engineers test all devices under the exact same conditions. It includes mounts for devices being tested and for testing tools (principally a spectroradiometer and a video colorimeter), computer-controlled LED lighting arrays to imitate all kinds of lighting types and brightness levels, and lux meters.

A device under test and video colorimeter inside the DXOMARK Display Bench

A device under test and spectroradiometer inside the DXOMARK Display Bench

In both photos showing the inside of the DXOMARK Display Bench above, you can see a device under test (DUT) mounted on the left, with the testing instrument on the right mounted on a rail; testing engineers use computer-controlled servo motors to move the instrument to various distances from the DUT. During testing, the Bench is first sealed against any external light sources, and an engineer controls the tests via computer.

In addition to the Display Bench, we have developed a fully computer-controlled Dome System, which serves to reproduce more intense outdoor lighting conditions. The shape of the dome allows the very intense light to be diffused so that it hits the smartphone’s screen from all directions, in the same way that we experience lighting conditions outdoors, for example, in the sun. But the dome’s ability to reach extreme levels of brightness permits us to really challenge the limits of a device’s screen capabilities.

The Dome System has been added for display testing

In the photo above, the DUT is attached to a rail within a chamber, with the screen facing the testing instrument. A lux meter sensor, which monitors the intensity of the light, is next to the DUT. The testing instrument, the Radiant imaging colorimeter, which is mounted on an external rail on the other side of the dome (not pictured), acquires contrast and brightness measurements  through a hole at the top of the dome as the DUT’s screen displays testing patterns for measurement.

Every element of the system–the DUT, the motors (or modules) controlling the light levels, and the instrument– is controlled by a computer.

Perceptual testing tools

One of the most important tools DXOMARK relies on for its perceptual testing is the human eye. Our perceptual tests confirm and complement our objective tests, in that we want to be sure that we can see in real life what the objective measurements are telling us. Further, objective tests measure only what they are strictly aiming to measure. Given the complexity of the software driving the display as well as the complexity of the human eye, perceptual tests are an essential ingredient in evaluating display quality.

Our Display protocol engineers receive careful and extensive training before conducting any perceptual tests, some of which involve closely evaluating multiple devices (a DUT and two or three comparison devices) against reference images displayed on a professional monitor. The color and brightness values of each carefully chosen image on the pro display have been precisely calibrated and measured. When making comparisons, engineers follow a very strict and scientifically sound protocol that requires conducting the test multiple times using different engineers each time so as to ensure against bias.

In addition to our most important perceptual tool (the human eye), our display engineers use a specially-designed smartphone apparatus for holding several smartphones at once, pro-level monitors, and lux meters.

Our engineers perform all perceptual evaluations by looking directly at the device’s display. We take photos only to use as illustrations, but never use them as a basis for any kind of test or evaluation.

Display protocol tests

The tables in each sub-section below cover all of the attributes that the DXOMARK Display protocol currently tests, and include information about the equipment we use, some of the testing conditions, and some of result parameters and definitions.

Readability

In our reviews, we regularly remind people that the most important consideration for end-users is how easily and comfortably they can read the display under different real-life conditions. DXOMARK uses its Display Bench and its dome system to recreate ambient light conditions ranging from total darkness to bright sunlight. Below is a sample graph of comparison data showing brightness/contrast measurements for three devices:

In the example above, you can see how the measured contrast in daylight conditions does not live up to claimed contrast values of 1:1,000,000 (or infinite), which are based on measurements taken in dark conditions (< 0.01 lux). Our measurements show what users experience — that is it hard to read our screens in sunlight.

Another test of display readability measures the homogeneity or uniformity of brightness output, as shown in the illustrative image below:

Brightness uniformity test

Photos for illustration only

False-color luminance map measurements

Color

From the end-user’s point of view, color fidelity — that is, having the display faithfully reproduce the exact same hues and shades that they see with their eyes — is second in importance only to readability.

We use a conoscope in the setup below to evaluate how color shifts when users view display content on axis versus when they look at content on a screen held off axis (tilted up to 70°).

Setup of conoscope-equipped video colorimeter (for illustrative purposes only; actual testing takes place at 0 lux).

We perform color fidelity measurements for different lighting conditions to see how well the device can handle color management under different ambient lighting conditions. Below is just one of our color fidelity results taken under a D65 illuminant at 1000 lux.

Color fidelity measurement made at 1000 lux in the DCI-P3 [glossary_exclude]color space[/glossary_exclude]

Most smartphone manufacturers include a feature we refer to as a blue light [glossary_exclude]filter[/glossary_exclude] (BLF); DXOMARK measures how effectively a phone reduces blue light, and what its impact is on overall display [glossary_exclude]color rendering[/glossary_exclude].

Video

A device may be good at handling still image content but fall down when handing video, and vice versa. DXOMARK tests displays using the device’s default video app. In the images below used to illustrate video test results, you can see that the device on the left has low brightness but is still visible; the image in the center has good brightness; and the device on the right is quite dark. As for color, the left-hand device shows good [glossary_exclude]color rendering[/glossary_exclude]; the middle device has a yellow cast; and the right-hand device is too blue.

Device has low brightness, but has good color

Device has good brightness but is slightly yellow

Device output is too dark and too blue

Photos  for illustration only

Motion

DXOMARK tests how well a display handles moving content. The left-hand image below shows our setup for testing [glossary_exclude]motion blur[/glossary_exclude], and the middle and right-hand images show sample test output.

We evaluate [glossary_exclude]motion blur[/glossary_exclude] perceptually by looking at frame duplications. The left-hand image below shows the setup we used to take the center and right-hand pictures below to illustrate what we evaluated perceptually. (We did not base any of our test results on these pictures.)

Setup for taking a photo of motion blur to use as an illustration

Illustration of device output showing duplications

Illustration of device output showing better control of blur

Photos for illustration only

Other motion phenomena we test for are stutter and [glossary_exclude]frame[/glossary_exclude] drops. The photo on the left shows our stutter/[glossary_exclude]frame[/glossary_exclude] drop testing setup; the GIF on the right illustrates the test video output of a white rectangle that is lit in successive frames.

Stutter and [glossary_exclude]frame[/glossary_exclude] drop testing setup

GIF illustrating video test for stutter and [glossary_exclude]frame[/glossary_exclude] drops

In the illustrative images below, a black or dark gray rectangle indicates a [glossary_exclude]frame[/glossary_exclude] drop (that is, when the display fails to display a frame properly before directly moving to the next one), and a white rectangle indicates stutter (that is, the display displays a frame twice before moving to the next one).

Illustration of a device showing almost no measured stutter or [glossary_exclude]frame[/glossary_exclude] drops

Illustration of a device showing a lot of measured stutter and [glossary_exclude]frame[/glossary_exclude] drops

Photos  for illustration only

Touch

As shown below, we have a set of high-end measuring instruments for performing touch analyses, including a robot that simulates human gestures (tap, zoom, and scroll) on a touchscreen with a precision of 0.05 mm at 1.5 m/s. In addition, we use a high-speed Phantom camera that records 1440 images per second for slow-motion capture of each frame on a smartphone display.

Touch robot

High-speed camera filming robot testing touch

In the video below, you can see a meaningful example of the results we obtain with our setup for touch response time measurement. In our gaming [glossary_exclude]use case[/glossary_exclude], the device on the left reacts three times faster than the device on the right, with response times of 3 ms and 10 ms, respectively.

In the example below showing smoothness, we observed during testing that the device on the left is not smooth and that the one on the right is very smooth when scrolling horizontally in a smartphone’s gallery app. The illustrations accurately depicts the phenomenon: on the left, we see a few sporadically spaced images, while there are many more evenly spaced images on the right. 

Illustration of test output indicative of choppy scrolling

Illustration of test output indicative of smooth scrolling

Photos  for illustration only

Artifacts

The DXOMARK protocol includes checking for performance, design, and image rendering flaws that can impact the end-user experience. For example, [glossary_exclude]flicker[/glossary_exclude] tests reveal that slow pulse-width modulation (PWM) can have an impact on visual comfort even for devices with a high refresh rate. (In the graph below, the first spike corresponds to the refresh rate, and the highest spike corresponds to the PWM.)

Rendering curved lines is a challenge for all displays; evaluating how well a device avoids the appearance of stair-stepping (known as aliasing) is part of our protocol.

All-purpose device showing significant aliasing

A gaming phone with better control of aliasing

An all-purpose phone with good control of aliasing

Photos for illustration only

We also use our spectrophotometer in SCI (Specular Component Included) mode to perform reflectance level measurements on smartphone displays that are turned off. Below are measurements that show the reflectance level for each 10 nm-spaced wavelength within the visible spectrum range (400 nm to 700 nm).

Conclusion

In our tests, we perform more than 400 measurements and 20 hours of perceptual analysis. These measurements are conducted in dark environments as well as in challenging indoor and outdoor lighting environments to imitate the end-user experience.

While having the latest high-quality panel on your smartphone is a good start toward reaching good display quality, it’s not enough. As we said earlier, display performance quality does not solely depend on hardware specifications, but also on the software and SW/battery strategy choices that manufacturers make to try to optimize end-user comfort across different use cases.

We hope this article has given you a more detailed idea about some of the scientific equipment and methods we use to test the most important characteristics of your smartphone display.

The post A closer look at the DXOMARK Display protocol appeared first on DXOMARK.

Testing and scoring

As with our DXOMARK Camera protocol, DXOMARK also tests front cameras using the default modes of firmware versions. In the process of testing a front camera, our engineers capture more than 1500 sample images and several hours of video of test charts and scenes in the DXOMARK image quality laboratory, as well as a variety of indoor and outdoor “real-life” scenes in and around DXOMARK headquarters near Paris. To test repeatability and consistency of results, we always take a series of images of the same test scene, rather than just individual shots.

While our front camera testing is very similar to our DXOMARK Camera protocol for smartphone main cameras, we have made some important modifications to take into account the way people primarily use front cameras. People capture selfies outdoors in bright light or indoors under many types of artificial lighting, and by definition show at least one human subject—the photographer herself or himself—in the image, which is why manufacturers should ideally optimize front cameras for portraiture at relatively short shooting distances and in a variety of lighting situations.

There are also image quality aspects for the front camera that are unique to video. Video stabilization is an important attribute that has a big impact on the overall quality of a video recording. The same is true for continuous focus, as an unstable autofocus can easily ruin an otherwise good clip. The most crucial difference between still image and video testing, however, is the addition of a temporal (time) dimension to most other image quality attributes. The tests for most static attributes that look only at a single frame of a video clip—for example, target exposure, dynamic range, or white balance—are pretty much identical to the equivalent tests for still images. However, a video clip is not a single static image but rather many frames that are recorded and played back in quick succession. We therefore also have to look at image quality from a temporal point of view—for example, how stable are video exposure and white balance in uniform light conditions? How fast and smooth are exposure or white balance transitions when light levels or sources change during recording?

It is also worth noting that for video testing we manually select the resolution setting and frame rate that provides the best video quality, while for stills we always test at the default resolution. For example, a smartphone front camera might offer a 4K video mode but use 1080p Full-HD by default. In these circumstances, we will select 4K resolution manually.

Subject distance and variation

Subject distance is very important in Selfie testing, but obviously, it can vary: some users like to capture close-up portraits of the face. In this kind of image, the attention tends to be focused on the subject and the rendering of the background is pretty much irrelevant. Another typical use case is a self-portrait taken at arm’s length. In this type of selfie, the subject is still the most prominent feature, but the background contains elements that the user wants to capture—for example, city sights or natural features in a landscape. Further, many users like to mount their smartphones on selfie-sticks in order to capture as much of the background scenery as possible, so the background should be just as well-exposed and sharp as the subject(s) in the foreground in the image.

To cover these most-typical front camera use cases, we perform our DXOMARK Selfie tests at three different subject distances:

• 30cm, close-up portrait
• 55cm, portrait
• 120cm, portrait with landscape (shot with selfie-stick)

Close-up portrait (30cm)

Portrait (55cm)

Portrait shot with selfie-stick (120cm)

It is a real challenge for most front cameras to produce good image quality at all three of these subject distances, mainly because focus, exposure, and other camera parameters have to be very well-balanced to achieve good results in such a wide range of shooting situations.

It’s not only the distance between camera and subjects that can vary in selfie photography, of course, given that the subjects themselves can vary in terms of both numbers and skin tones. This is why we create and evaluate test scenes in the studio and outdoors not only with individual subjects but also with groups of people. Shooting group selfies allows us to evaluate how a camera’s focus, white balance, and exposure systems deal with scenes that feature multiple human subjects in several focal planes and with a range of skin tones.

Outdoor selfie testing with a variation of skin tones

One of several studio setups for group selfie testing

Light conditions

As with our DXOMARK Camera  protocol, we perform all lab studio tests at light levels that range from the equivalent of dim candlelight to a very bright sunny day, as follows:

• Outdoor (Bright light, 1000–10000 lux)
• Indoor (typical indoor light conditions, 100–300 lux)
• Low light (typical light conditions in a bar or concert venue at night, 1–20 lux)

We undertake DxOMark Selfie testing under a wide range of light conditions.

We capture our real-life test samples at several outdoor locations on bright and sunny days, as well as inside the DXOMARK offices under typical artificial indoor lighting.

The Selfie score

We generate the DXOMARK Selfie scores from a large number of objective measurements obtained and calculated directly by the test equipment, and from perceptual tests for which we use a sophisticated set of metrics to allow a panel of image experts to compare various aspects of image quality requiring human judgment. We then use a sophisticated set of formulas and weighting systems to condense these objective and perceptual measurements into sub-scores, and then as a last step, we use the sub-scores to compute the Photo and Video scores and the Overall Selfie score.

The DXOMARK Selfie sub-scores

In this section, you can find more detailed information about all the image attributes we are testing and analyzing in order to compute our sub-scores. We’ll also show you some of the custom test equipment we have newly developed for the DXOMARK Selfie protocol in addition to a selection of real-life samples and graphs that we are using to visualize the results.

Exposure and Contrast

Like most of our scores, Exposure and Contrast for still images is computed from a mixture of objective and perceptual measurement. In video, the  exposure score looks at target exposure, contrast, and dynamic range from both a static and temporal point of view. When measuring exposure and contrast for selfies, we put a strong emphasis on the target exposure of the face, but we also look at the overall exposure of the image and the HDR capabilities of a camera—all of which can be important when the foreground of a scene is much brighter than the background, or vice versa. We also report if contrast is unusually high or low, but we don’t feed this information into the score, as contrast is mainly a matter of personal taste.

We have developed a dedicated weighting system that we use to compute the Exposure and Contrast sub-score from a large number of perceptual and objective measurements of images taken at different subject distances, based on the following image quality attributes:

• Face target exposure
• Face target exposure consistency across several faces in group selfies
• Overall target exposure
• Highlight clipping on skin tones
• Highlight and shadow detail in the background

When testing video,  we have to take temporal image quality attributes as well as static attributes into account. So for Exposure, we test for the following temporal attributes:

• Convergence time and smoothness
• Oscillation time
• Overshoot
• Stability

In our in-depth technical reports, we provide an overview of objective measurements (see graph below the left) and perceptual scores. We also prepare a variety of graphs to visualize objective measurements, such as the exposure graph below on the right. It shows target exposure at light levels from 1 to 1000 lux, and for a range of illuminants, including daylight and tungsten.

Exposure technical overview

Exposure across a range of illuminants

In addition to the lab tests, we evaluate Exposure and Contrast using our perceptual database of scenes covering different skin tones, subjects distances, lighting conditions and group shots. For example, the Eiffel Tower shot below on the left allows us to check exposure on the face versus the background and tells us a lot about the dynamic range of a camera as the background is brighter than the subject.

We also take sample shots indoors in low light. In the example below on the right, we use lateral lighting to see how well the camera’s exposure system deals with harsh contrasts on a variety of skin tones.

Eiffel Tower test scene

Low-light test scene with lateral lighting

In our perceptual analysis of group selfies, we check the exposure system’s ability to deal with multiple subjects and skin tones in the same scene. In the sample shot below on the left, some highlight and shadow clipping is visible on skin tones, but the camera manages a well-balanced exposure overall. The shot on the right is strongly overexposed, and the exposure system brightened the dark skin tones of the subject in the foreground too much.

Group selfie with good exposure

Overexposed group selfie

A wide dynamic range is important when trying to avoid highlight or shadow clipping on skin tones or with backgrounds that are much brighter than the image’s subject. In our backlit portrait scenes, we test how the exposure system deals with subjects of varying skin tones in front of a bright background. As usual, face exposure is the top priority, but we also value a good balance between foreground and background exposure. As you can see in the images below on the right, current front camera models deliver very different results in this scene.

Highlight clipping on skin tones

Backlit portrait with light and dark skin tones, results from different cameras

Color

When testing color in front camera images, we focus on color on the subject’s face and skin tones, but we also look at the overall color rendering, the white balance, and additional color metrics in the other parts of the frame. For the Color sub-score, we measure and analyze the following image quality attributes:

• White balance accuracy and repeatability
• Color rendering and repeatability
• Skin tone color rendering
• Color shading

We take objective color measurements using Gretag ColorChecker, Dead Leaves, and our own custom DXOMARK studio charts at different light levels and under different light sources. We produce measurement charts for color rendering, white balance accuracy, and color shading (among others), and have developed an ellipsoid scoring system for evaluating white balance and color saturation. This system takes into account the fact that good and acceptable color manifests on a continuum rather than having a single fixed value. Color tones within the small green ellipsoid (chart below on the right) are close enough to the target color to score maximum points; colors within the larger red ellipsoid are still acceptable, but score lower.

Color rendering for different light sources and levels

Example of ellipsoid scoring system measurement of white balance and color saturation

In our in-depth technical reports, we also provide a visual representation of the results, showing the color rendering of the camera (small squares) embedded in the target color (larger squares in the illustration below). This allows us to quickly identify problem areas and opportunities for improvements. As one would expect for our Selfie protocol, our main focus is on skin tone rendering (top-left color array below), but we examine and score the rendering of all tones.

Visual presentation of a camera’s color rendering

For our perceptual analysis of color, we use many of the same outdoor scenes that we also use to evaluate exposure and other metrics. However, we have also created several color-specific perceptual scenes, such as the indoor setups shown in the images below. The left scene allows us to see how the camera deals with different color light sources in low light. And as white balance systems can be confused by single-colored backgrounds, we designed the scene on the right for use with a range of different-colored backgrounds and artificial lighting.

Indoor color testing with colored light sources in the frame

Indoor color testing with specific background colors

Focus

For the Focus sub-score, we use a focus range chart with a portrait in the lab and a number of real-life scenes to measure and analyze the following image quality attributes:

• Focus range
• Depth of field
• Focus repeatability and shooting time (only for front cameras with autofocus systems)

The focus range tells us how good the camera is at focusing on the subject at different shooting distances. In our scoring, we put the most weight on the closer 30cm and 55cm shooting distances, as they represent the largest proportion of real-life use, but we also test at selfie-stick shooting distance (120cm). In addition, we test repeatability and shooting time for phone cameras with an autofocus system by taking multiple shots at the same subject distance and checking the consistency of the results.

In video testing, focus stability is the only temporal attribute we use in focus testing. Not many smartphones come with an autofocus system in the front camera. AF systems can help achieve better sharpness, but can also cause problems in terms of stability while recording.

Focus testing in the lab

Depth of field is the distance between the closest and farthest objects in a photo that appear acceptably sharp. Good depth of field is essential for images in which the background is an important element—for example, portraits with a tourist site behind the subject, or in group selfies, when we want good sharpness on all subjects, even if they are not in the same focal plane. Depth of field is less important for close-up portraits showing very little background, which is why we put more weight in our scoring on longer shooting distances.

As usual, we perform focus tests at different light levels, and compute scores from objective and perceptual measurements. In the laboratory, we use a focus chart with a portrait image that we capture at three different subject distances.

Focus chart with portrait in the lab

For our perceptual analysis, we capture individual and groups of subjects at different shooting distances, thus allowing us to check both focus range and depth of field in real-life images.

Real-life testing at different subject distances

A wide depth of field is useful for images with important elements in the background, and for group portraits with the subjects in different focal planes, such as in this example.

The face in the focal plane is well-focused and sharp (crop).

The faces towards the back of the scene are noticeably softer (crop).

Texture and Noise

For this sub-score, we analyze images objectively and perceptually to determine how well phone cameras render fine detail and textures, and how much noise is visible in different parts of the image. Texture and noise are strongly interlinked: strong noise reduction in image processing reduces visible noise, but also tends to decrease the level of detail in an image, so the two attributes should always looked at in conjunction. The best camera isn’t necessarily the one with the best texture score or the best noise score, but rather the best balance between the two.

Fully-automated texture testing with moving subjects and shaking platform

Control station

For the Texture and Noise sub-scores, we measure and analyze the following image quality attributes using a Dead Leaves chart and several custom-made lifelike mannequins in controlled lab conditions, as well as using real-life images:

• Texture acutance
• Visual noise

Technical overview for texture and noise

As usual, we undertake all tests at different light levels. In addition, we have designed texture tests for static (landscape) and moving (family) scenes. Moving scenes often pose problems for some cameras, especially in low light, as slow shutter speeds can cause blur on moving subjects or through camera shake (we use an automated hydraulic shaking platform in the lab to simulate the latter).

We measure texture acutance for static (landscape) and moving (family) scenes.

Both luminance and chromatic noise feed into the Noise score.

In addition to human subjects, we perform our perceptual analysis using a lifelike mannequin that was custom built for DXOMARK Image Labs. Perceptual analysis focuses on specific details on a face, such as the eyes, eyebrows, eyelashes, lips, and beard. Many device manufacturers choose to apply smoothing effects to the skin of selfie subjects; in the other direction, over-sharpened skin textures and too strong micro-contrast can result in an unnatural rendering of skin detail, so we keep a close eye on both those effects, and also check that skin is rendered in a consistent way across the face.

We use several lifelike mannequins for testing of detail and noise.

We perform perceptual texture analysis at different shooting distances and light levels.

Artifacts

For artifacts, we use a mixture of objective testing with MTF, Dot, and Grey charts in the studio, and perceptual analysis of real-life images. For our Selfie test protocol, we are looking for the same kinds of artifacts as for our DXOMARK Camera testing (including the ones in the list below), and also for any other unusual effects in our test images:

• Sharpness in the field (corner softness)
• Lens shading (vignetting)
• Lateral chromatic aberration
• Distortion
• Perspective distortion on faces (anamorphosis)
• Color fringing
• Color quantization
• Flare
• Ghosting

The Artifacts section in our technical reports provides an overview of all objective and perceptual measurements. Based on the severity and frequency of artifacts, we deduct penalty points from a perfect score of 100 to compute the overall Artifacts score.

Chromatic aberration

Unnatural rendering of skin texture in faces

Crop

Crop

Flash

For the Flash sub-score in Photo, we perform a subset of our full Selfie test protocol, looking at most of the image quality attributes that we evaluate for non-flash capture. Along with capturing real-life images in a dark indoor setting, we measure and analyze the following image quality attributes, using Gretag ColorChecker, Dead Leaves, and Gray charts in the lab:

• Target exposure (accuracy and repeatability)
• White balance (accuracy and repeatability)
• Color rendering
• Color shading
• Fall-off
• Texture and Noise
• Artifacts
• Red-eye effect

For these objective measurements in the lab, we shoot at a distance of 55cm at 0 and 5 lux light levels.

Flash white balance and skin tone color chart

Flash texture chart

For perceptual testing, we shoot with the front camera flash (LED or display flash), manually activated at a distance of 55cm, and at light levels of 0 and 5 lux. We then check the resulting images for exposure on the face, white balance, and rendering of skin color, as well as for noise on the skin and for details on specific elements of the face, such as eyebrows or beard.

Samsung Galaxy S9 Plus, flash shot at 0 lux

Apple iPhone X, flash shot at 0 lux

Bokeh

Some current smartphone front cameras are capable of simulating the bokeh and shallow depth of field of a DSLR camera. As on main cameras, some models use a secondary lens to estimate the depth of a scene; others rely on a single lens and use purely computational methods to analyze depth. We test bokeh simulation in a laboratory setup, looking at both the quality of the bokeh (depth of field and shape), as well as at the artifacts that are often introduced when isolating a subject.

For the Bokeh sub-score, we measure and analyze the following image quality attributes:

• Equivalent aperture (depth of field)
• Depth estimation artifact
• Shape of bokeh
• Repeatability
• Motion during capture
• Noise consistency

For Bokeh, we evaluate all images perceptually, using test scenes in the studio as well as indoors and outdoors. We designed our test scenes to replicate a number of light conditions and to help experts evaluate all the image attributes listed above.

Samsung Galaxy S9 Plus, standard front camera shot

Samsung Galaxy S9 Plus, front camera shot with simulated bokeh effect

Apple iPhone X, bokeh shot

Apple iPhone X, bokeh shot

Apple iPhone X, bokeh shot

Crop, depth estimation errors

Crop, foreground blurring

Crop, blur transition

Video Stabilization

Stabilization is just as important for front camera video as it is for main camera video. We test video stabilization by hand-holding the camera without motion and by shooting while walking at a subject distance of 30cm and at arms-length. For video calls–one of the most important use cases for front camera video—users typically hand-hold the device close to their faces (30 to 40 cm). For this kind of static video, the stabilization algorithms should counteract all hand-motion, but not react to any movements of the subject’s head.

For group selfie videos, we have to check that stabilization works well for all the faces in the frame. In the sample below, you can see that only the main subject’s head is well-stabilized, and that there is noticeable deformation on the other subjects in the video.

Walking with the camera is the most challenging use case for stabilization systems, as walking movement has a strong amplitude which requires much more heavy-handed stabilization than a static scene. Inefficient video stabilization in such cases can often result in intrusive deformation effects.

We hope you found this overview of our front camera still and video testing and evaluation useful. For an introduction to our new DXOMARK Selfie test protocol, an article on the evolution of the selfie, and even more information on how we test and evaluate front camera video quality, please click on the following links:

• Introducing the DXOMARK Selfie test protocol
• The evolution of the selfie

The post A closer look at the DXOMARK Selfie protocol for front cameras appeared first on DXOMARK.

This latest round of updates brings new elements to all protocols, but mainly to Camera and Audio, which both include new measurements to adjust to the evolution of both technology and device usage. In addition to the new measurements, the following improvements have been made:

• Increased transparency: For all scores, the high-level formula will be available to consult in the appropriate How we test sections on the DXOMARK website.
• Improved readability of the scores: We have modified the score scales in order to make the scores more comparable across protocols. For example, in the past, an overall score of 80 would have been low for Camera but comparatively high for Audio or Display. Scores have now been reformatted in order to ensure that they represent a similar performance level across all test protocols.
• Geometric averages are now used for all protocols bringing increased consistency: This measure is applied to all protocols and rewards a homogenous performance across all sub-tests of a protocol and penalizes devices that are very good in one key test area but perform poorly in another.

New for Camera

For Camera, we have added a range of new use case scores, providing neutral assessments that help consumers make smartphone purchasing decisions based on their device usage. These use case scores are computed from a subset of our measurements and cover Photo, Video, Bokeh and Preview. They are published next to the overall score, which they complement. We are updating our test protocols on a regular basis to reflect changes in user behavior and since our last release two years ago, some habits have changed and new trends have increased. This is reflected in this latest version of our Camera protocol, in which the impact of low light performance, the friends and family use case, zoom and close-up on the overall score have increased.

 Use case scores

Under the new protocol results, in addition to the usual attribute scores, results are presented based on use cases that were prioritized by the participants in our consumer studies. In our tests, we cover many more use cases, but we have selected the most important ones to be presented in detail within our test results:

• Friends & Family: This is a very typical use case and represents more than 50% of the photos and videos captured by the users who took part in the YouGov-DXOMARK survey undertaken in December 2021*. For this use case, smartphone photographers expect realistic and pleasant skin tones, capture at the right moment, natural contrast and face detail, as well as freezing of motion among other image quality characteristics. This sub-score is computed from all data collected for Photo, Video, Preview, and Bokeh tests.
• Low light: This is a very challenging use case and even the users of Ultra-Premium phones don’t find it easy to achieve great image quality in low light. According to the YouGov-DXOMARK survey, it is also a major source of dissatisfaction among users. Only 50% of users are satisfied with the quality of images and video clips that they capture in low light. Low-light images should deliver natural exposure and colors as well as a good trade-off between detail retention and image noise reduction. This trade-off is even more difficult to achieve when recording video footage than with still images. The camera should also be capable of freezing motion in the scene with only minimal motion blur.
  For version 5 of the DXOMARK Camera test protocol, we have expanded and enhanced the low-light analysis. Thanks to a dedicated focus group on the topic, we have been able to take consumer preference into account even more. A lot of effort went into defining what constitutes an acceptable low-light image result. By doing this, we have been able to improve all our lab metrics to match 2022 consumer expectations even better. In addition, Night is not an attribute under the Photo score anymore and has been integrated into the low-light score.
• Outdoor and Indoor: Even though low light is currently the most critical differentiator between smartphone cameras, all newly created test results will also feature scores for Indoor and Outdoor use cases.

New protocol and extension

Zoom score extension for Wide and Tele

Zoom performance is a key differentiator for smartphones in the Premium and Ultra-Premium segments ($600+), with most devices in these categories featuring dedicated tele zoom and ultra-wide cameras. Given that according to the YouGov-DXOMARK survey* 30% of users use the zoom function of their smartphones, it’s no surprise that manufacturers are putting a lot of work into improving their devices’ zoom capabilities. The good news for budget-conscious consumers is that the zoom technologies used in today’s top-end devices will no doubt trickle down into lower price segments over time.

A dedicated Zoom score was first introduced to the DXOMARK Camera test protocol with version 4. In addition to lab tests, the DXOMARK engineers perform in-depth perceptual evaluations across more than 35 real-life scenes, covering landscape, cityscape, architecture, interior design, portrait, and still life. Version 5 now brings the following enhancements and additions:

• New video zoom evaluation in the lab: DXOMARK is now evaluating video zoom quality from ultra-wide to tele
• Low light and very low light: Zoom tests are now also undertaken at a very low light level of 5 lux
• On-screen zoom buttons tested: We are now testing the image quality provided with the zoom buttons of the device’s camera app UI
• Enhanced lab testing: We have increased the number of zoom measurements in our perceptual scenes. They now include measurements (target exposure and white balance) and AI metrics (detail preservation and noise) on the DXOMARK Chart (DMC), repeatability across a series of shots (including defocusing of the camera after each shot), new video measurements (pinch-zooming before and during recording) as well as perceptual evaluation of zoom smoothness. Zoom smoothness is now also evaluated for two different pinch-zoom actions (from ultra-wide to 72mm and 144mm equivalent respectively).

Close-up

Close-up or macro photography of objects and details that are not visible to the human eye has been around as a photographic discipline for a long time, but in the past this type of photography typically required interchangeable lens cameras with dedicated macro lenses. More recently we are seeing more smartphone cameras with close-up functionality, and the fact that the feature is already used by 7% of users* indicates that interest in macro is growing, which is why close-up testing has been added to the DXOMARK Camera protocol.

Close-up testing is added as a bonus to the Photo score. This means a device can score a bonus if it comes with this feature, but no points will be deducted from the score of devices without a close-up feature. New close-up related tests include the following:

• Document scanning scene: An official document, such as a passport or national identity card is photographed in three different ways to cover most scanning scenarios. It is first photographed at full size and at a high level of magnification. The objective of this test is to check whether the camera can capture enough small detail in two ways, by moving the camera lens very close to the document, or by staying at the original distance and using the zoom in order to achieve maximum detail.
• Three additional close-up scenes: If the camera captures a sufficiently high level of detail in at least one of the two small detail document scanning tests described above, the DXOMARK testers shoot a total of three additional scenes that are subsequently evaluated for image quality attributes, with a focus on detail preservation. These additional tests focus on still-life macro photography.

Friends & Family use case: Measurements in HDR conditions and for moving subjects

Smartphone users take photographs of friends and family in all kinds of conditions, including difficult backlighting and other high-contrast conditions, with or without subjects in motion. With version 5 of the Camera test protocol, we have expanded our HDR camera testing to include a range of new measurements, including the following:

• Portrait HDR lab setup: The HDR portrait setup has been drastically improved by adding a new AI measurement for face detail as well as objective measurements for dynamic range and contrast. As usual, the objective measurements are complemented by perceptual evaluation. These new additions allow for a more exhaustive and complete evaluation of texture in portraits across various light conditions.
• Evaluation of scenes with motion: Freezing motion in a scene can be a difficult task for a camera in any light condition, but it is particularly tricky in high-contrast scenes where complex algorithms, such as HDR or motion detection, kick in. To better evaluate this kind of scene, we have added a motion blur measurement to our lab testing. Overall testing has been expanded to a total of 24 light conditions.

New Camera version 5 results

The new version of the Camera protocol does not completely reshuffle our current rankings. Camera V5 is simply a natural and more profound evolution of our Camera version 4 protocol. A smartphone camera that earned a high rank under V4 will still be well-ranked under the new protocol. But to reflect the latest in-depth additions to our testing, some adjustments had to be made, as seen here:

To give you a better idea of what to expect from the protocol updates described above, in this section we give you an overview of the test new test results for two popular devices: the Apple iPhone 13 Pro Max and the Samsung Galaxy S22 Ultra (Snapdragon). We also explain how the update affects the version 5 overall scores.

Apple iPhone 13 Pro Max

Close-up: The previous generation of the top-of-the-range iPhone is still one of the best devices we have for close-up photography to date. Image results are very well balanced, with pretty high levels of detail. Some artifacts are visible but all within acceptable limits.

Friends and family: It’s also one of the best performers in the friends and family use case. The new expanded analysis offers a lot of additional information, and our testers found the iPhone 13 Pro Max freezes motion nicely, controls low contrast issues well and delivers good skin tone rendering in all light conditions.

Low light: The iPhone does not do as well for the low light use case as it does for family and friends, blocking the Apple flagship from taking the top spot in the new Camera version 5 ranking.

Samsung Galaxy S22 Ultra (Snapdragon)

Close-up: The Samsung Galaxy S22 Ultra is a good device for this category. Both the Exynos and Snapdragon versions capture high-quality close-up images that are quite similar to the iPhone 13 Pro Max output, but with less detail.

Friends and family: This use case score is not as high as on some other top-tier phones. Our testers measured longer shutter delays in various  HDR conditions and in low light. This could be problematic for photographers who rely on instant capture.

Zoom: The new low light elements in our Zoom testing reveal interesting strong points of the Samsung. This is true for zooming into both directions, to ultra-wide and tele. In our testing, the S22 Ultra demonstrated great robustness and one of the most future-proof designs, earning it extra points for the Zoom and overall scores.

Image quality aspects making a stronger impact in Camera version 5

Contrast: It’s important for contrast to be well preserved, especially on portraits and backlit scenes. Consumer feedback studies on the topic show that consumers do not like low contrast on portrait shots. With the current evolution of the HDR technology, contrast becomes even more important as image quality is increasing overall. With version 5 of the DXOMARK Camera test protocol, we expand contrast evaluation with new measurements (local contrast indicator) and new perceptual analysis guidelines based on consumer feedback. As a result, the scores of some devices, for example, the Xiaomi Mi11 Ultra and 12S Ultra or the Huawei P50 Pro, will be slightly impacted by low contrast rendering.

These unnatural photos, above and below, will be penalized more

under the Camera V5 protocol.

Motion in the scene: Capturing the right instant and freezing motion in the scene are important capabilities in photography. Our updated testing shows that some devices are clearly better than others in this respect. For example, the Honor Magic4 Ultimate does very well in this respect under indoor lighting, while other phones, such as the Huawei P50 Pro or iPhone 13 Pro Max struggle more with detail rendering on moving subjects.

Low light and night “atmosphere”: Exposure is a crucial element of low-light photography. Our focus group results show that viewers do not like low light or night pictures that are overexposed and therefore do not transmit a low-light atmosphere. We have therefore updated our evaluation guidelines for target exposure in extreme low-light conditions. In testing, some devices deliver a bright target exposure in low light and very low light conditions, which goes against users’ preferences. These devices, for example, the Huawei P50 Pro, Xiaomi 12S Ultra and Apple iPhone 13 Pro Max, will lose some points with the updated scoring system.

New for Audio

Tests and measurements

The overall structure of the DXOMARK Audio score remains unchanged, with sub-scores for Playback and Recording. However, two new tests, Wind Noise and Audio Zoom, have been added to evaluate the performance of more advanced smartphone audio features.

Wind noise

Wind noise, which can ruin an otherwise perfectly fine audio recording, has been identified as the most frustrating issue in smartphone audio recording. A brand new setup in the DXOMARK Audio lab measures the impact of wind noise on various audio recordings, at a range of angles and wind speeds.

Within the Audio score structure, wind noise measurements can add a bonus or malus to the Friends & Family videos, Selfie videos and Memos use case scores. Within the test results that are published on the DXOMARK website, consumers will be able to compare wind noise track recordings from multiple devices.

Audio zoom

Audio zoom is a smartphone audio feature that is becoming more popular. It allows users to record sound that is coming from a specific direction and attenuates the incoming sounds from other directions. This is usually controlled dynamically, for example in conjunction with the camera’s tele zoom.

Audio zoom testing includes both objective and perceptual measurements and is undertaken on a brand new setup that allows for wide, tele, and super tele zooming. In the test results, the audio zoom measurements feed into the directivity measurements.

Custom audio tracks for precise evaluation

Most testers and reviewers of audio equipment use testing playlists to listen for specific audio cues. These custom-created sets of songs help highlight audio attributes, such as dynamics, distortion, tonal balance, low-end extension and spatial wideness, when played back on speakers or headphones. In 2020, the DXOMARK Audio team commissioned several musicians to compose custom tracks that are perfectly tailored for our wireless speaker testing, containing all the sound attributes required by the DXOMARK Audio protocol. The 10 tracks that were composed are now being used for the evaluation of the smartphone audio, allowing for even more precise and comprehensive testing.

 Audio version 2 results

Like for the other protocols, the new Audio score formula rewards devices with a balanced performance and stretches the scale for better differentiation between performance levels. The best-performing devices for audio zoom and wind noise receive bonus points.

In the updated audio ranking, the Black Shark 5 Pro remains the king in every way. It performs especially well in the new wind noise evaluation. Asus and Google devices move up in the ranking, thanks to overall good performance, so does the iPhone 13 Pro Max, thanks to a great audio zoom feature. The Samsung Galaxy S22 Ultra provides excellent wind noise reduction and therefore gains some spots in the ranking as well.

Refreshing the other DXOMARK test protocols

Even though the Camera and Audio test protocols are the ones that feature new or updated tests and other modifications, this latest update also affects all other DXOMARK smartphone test protocols — Selfie, Display, and Battery. For the first time, the general formulas for all protocols are now available on the DXOMARK website; score scales are now comparable between test protocols; and a new geometric score formula rewards a homogenous performance across all sub-tests. This is not where it ends, though. At DXOMARK, we will continue to optimize our testing methodologies for new technology developments and changes in user behavior, so stay tuned for future updates.

*YouGov RealTime survey conducted on behalf of DXOMARK from December 17 to 23, 2021, among 2,000 people per country, representative of the national population aged 18 and over (France, Great Britain, USA), the urban population for India and the online population for China, using the quota method.

The post What’s new in DXOMARK’s protocol updates appeared first on DXOMARK.

In this tech article, we tested the full duplex experience of a selection of laptops representative of the market: Will the Surface Pro 8 perform better than the MacBook Pro? Are commercial laptops aligned with the needs of telecommuters? Is it necessary to buy a premium laptop to have a good two-way conversation with friends and family? We answer these questions in the following paragraphs.

The challenge of full duplex audio

In a video mediated context, the audio channels must be handled either in half-duplex or full duplex mode. The first mode is the one of walkie talkies: one person talks at a time. The latter is like a conversation over the phone, during which one party can interrupt the other party before they are finished talking.

As two-way conversation is the heart and soul of videoconferences, DXOMARK’s “full duplex audio” tests evaluate how well the laptop handles both incoming and outgoing voices, along with background and ambient noises on both ends. What do participants hear when someone interrupts a person who is speaking? Is one voice suppressed, are both voices heard, or do both voices disappear? Do you hear an echo when you speak? In group settings, can the people furthest from the laptop microphone be heard clearly? How effective is noise cancelation? Is the software able to compensate for noisy environments?

Audio defects of full duplex mode

Some of the most frequent audio defects observed in a full duplex conversation are:

• leakage
• temporal artifacts
• spatial issues
• poor tonal balance

In this article we will focus on the two most critical ones: leakage and temporal artifacts.

Double-talk leakage

Full duplex audio is difficult to implement because the playback of the incoming voice channel may be sent again into the outcoming one, creating an audio loop and sometimes Larsen effect. Leakage happens when the audio played back on the laptop ends up in the captured audio and transmitted back. It is therefore necessary to implement ambient sounds filtering and echo cancellation on both sides of the conversation to make it possible to have a telephone-like performance.

Audio leakage can bring a person to hear themselves back in the middle of a two-way conversation.

Temporal artifacts

Temporal artifacts can be summarized as any changes in volume or dynamics over time that are unwanted and/or disturbing to the end user. As mentioned above, all laptops are tuned to filter out sounds below a certain loudness threshold to allow the “main signal” to get through without the interference of extraneous noise. Moreover, noise reduction is dependent on noise filtering algorithms. This leads to the apparition of Temporal artifacts.

Temporal artifact, fluctuations in volume

Temporal artifact, consistent volume levels

Temporal artifacts can lead to removing part of the sound of the voice of the person talking, a phenomenon known as gating.

Consequences of the defects

Echo cancellation

Hearing an echo of a participant is a direct consequence of the leakage defect. It can be amplified by factors outside of the device itself, such as the latency of the internet connection or the loss of packets during transmission. To stay at the device level, DXOMARK evaluates in priority the intrinsic leakage of the laptop under test, outside of the network aspect. The way manufacturers can reduce this issue is by implementing echo cancellation algorithms at the firmware level.

Gating due to ambient noise

Another big impediment to clear conversations is a phenomenon known as gating. It is a consequence of temporal artifacts, which occurs when devices are not tuned to properly differentiate between main signals and unwanted noise. Unwanted noise can be the sound of the keyboard of the laptop, some people talking in the background, or street sound coming by the window. Heavy gating can lead to simply suppress the voice of the speaker at the same time as what is perceived as noise.

The recent introduction of AI base noise reduction in the most advanced chipsets helps to improve this situation, sometimes removing gating completely; however, these algorithms are still limited to high-end solutions.

Gating due to voice priority

The double talk case, where both users speak at the same time, is difficult to manage correctly, and ends up often by one of the persons in the call taking “voice priority” over the other participant. Who has not experienced failing to make their viewpoint heard at a meeting, with many attendees, because another participant kept talking without stopping?

This phenomenon can lead to frustration, especially because we expect the same experience we are used to on a phone device. But in a conference call, the laptop manages multiple channels, sometimes over ten, while a phone only handles two channels. A very difficult algorithmic challenge for both DSP makers and laptop vendors.

Another person frustrated by double talk voice priority during an important videoconference meeting!

Our test setup

We developed a complete videoconference testing protocol, which covers video quality, audio playback and capture, as well as full-duplex audio. For the case of full duplex, 360° sounds recordings simulate meeting environments and a duplex scenario, in which the device plays content while recording. We also perform real-life tests in various venues, using common V/C software, such as Microsoft Teams, Google Meet, or Zoom.

One of our test setups in our listening studio reproduces a very difficult use case. We use recordings of Harvard Sentences, the industry standard for speech quality measurement ^[1]. The scenario consists of an input channel(what the microphone of the laptop must record) and an output channel (what the speakers of the laptop are playing). The input is composed of background voices and a main male speaker, whilst the output is the voice of a woman.

In a good videoconference experience, the male speaker hears the woman properly. A perfect device in full duplex will just record the voice of the man and ignore the voice of the woman. Moreover, the recording of the male speaker should reduce the volume of the background speakers, to allow the woman to clearly hear the conversation.

Text of the main user: “The tongs lay beside the ice pail. Sell your gift to a buyer at a good gain. Farmers came in to thresh the oat crop.”

DXOMARK duplex scenario allow to evaluate leakage, gating and voice priority issues.
The soundtrack above represents the full ambiance in front of the laptop tested.

Full duplex audio test results

In our initial round of testing, we evaluated 10 laptops from three segments ^[2] and a wide variety of CPUs ^[3].

DXOMARK ranking of a selection of laptops from the budget, commercial and premium segments.
The Acer Aspire 5 and the Asus ExpertBook show the best full duplex audio performance.

The Acer Aspire 5’s performance lands in the No.1 position in our evaluation: You do not hear yourself speaking, but gating occurs occasionally when two people speak at the same time. Our tests revealed that devices that present leakage often do not have gating, and vice versa.

The HP ProBook and the Lenovo IdeaPad 5 suffer from heavy gating and cut off the overall audio when the people on both sides of the call speak at the same time.

The Apple MacBook Pro suffers from slight duplex leakage but preserves well the vocal quality of the other person talking. This allows for easy filtering by the V/C platform afterwards. An example of bad leakage performance is the Huawei MateBook 14s, on which the woman’s voice is heavily distorted, making the voice separation difficult.

Results can also vary within the same brand: we were quite surprised to see that the IdeaPad 5 had one of the lowest duplex performances observed, while its Chromebook cousin, the IdeaPad Flex 5, figures at the fourth position. Another surprise is to see a commercial laptop, the HP ProBook 430 8G, perform poorly in duplex, given that one of the prime usages of commercial devices is to participate in online calls and meetings.

Another takeaway on duplex audio is that the price of the device is not always a good indicator of better performance. In our small but representative selection of devices, we can see that our top performer is a budget laptop, as well as the last, while premium laptops land in the middle.

Finally, remember that duplex audio is just one part of the whole audio experience, with the recording and the playback. While duplex is critical in the success of good conference calls, the ability of a laptop to have rich and powerful audio playback, as well as precise recording can sometimes make up for the flaws due to leakage or gating during duplex. Stay tuned for future DXOMARK tech articles on these subjects!

[1] “IEEE Recommended Practice for Speech Quality Measurements,” in IEEE No 297-1969 , vol., no., pp.1-24, 11 June 1969, https://doi.org/10.1109/IEEESTD.1969.7405210.

[2] To compare laptop videoconference performance in the context of their respecting audience, DXOMARK defined a segmentation system for the laptop industry, based on three dimensions: performance, design and price. Performance relates to the elements such as semiconductor components, screen technology, and storage. Design considers the originality of the curves of the device, materials and structural construction. Finally, the price refers to an evaluation of the entry point into the segment. We retained three segments for this analysis: consumer budget, mostly driven by price; commercial, which is mostly driven by design; premium, which is driven by both performance and design.

[3] Most of the Intel laptops we tested for videoconference run on 11th gen, except for the IdeaPad Flex 5 Chromebook, which uses a 10th gen, and the Asus ExpertBook, which is an older model with Intel core 8th gen. Finally, our model of Lenovo Ideapad 5 boasts an AMD Ryzen 5000, and the MacBook is the 2021 version with M1 Pro.

The post Videoconference full duplex audio: Do laptops meet expectations? appeared first on DXOMARK.

We’ve made a selection of phones that perform well not only under one or two of our protocols, but across the board — Camera, Audio, Display, Battery and Selfie. To offer something for every budget, we’ve also split things up by price segment.

Ultra Premium: Crème de la crème

With most phones in the Ultra Premium segment (launch price of $800 and up) being flagship devices of their respective brands, engineers and designers have to make fewer cost-related compromises than at lower price points and can often try to move the boundaries of what’s currently possible in terms of smartphone technology. Phones in the Ultra-Premium category tend to use high-end components all around, aiming for top performance in all areas. 

In this segment we have observed a lot of improvement over the past year or so and, as usual, the camera is one of the main areas that manufacturers focus on. With the latest generation of devices, we have seen significant improvements in image quality when using the tele zoom. Larger sensors and wider apertures in ultra-wide cameras mean better detail, lower noise, and a wider dynamic range at the widest end of the zoom range as well. As a result, images show better textures and more detail in the very brightest and darkest parts.

In addition, wider angles of view allow photographers to squeeze more of the scene into the frame, which is great for taking pictures of large groups in small space, for example. Most flagship models now offer 13mm equivalent lenses on their ultra-wide modules. Other testers also saw noticeable improvements compared to previous device generations in terms of night-shot image quality and HDR video, offering better textures and detail in the brightest and darkest parts of videos recorded in difficult high-contrast conditions. In addition, new in-house developed chipsets offer engineers more options to choose from to engineers.

We have seen significant improvements in image quality when using the tele zoom.

In terms of audio, most ultra-premium devices continue to provide a good user experience, but not much has changed when it comes to the quality of playback and recording. In the display category, we have seen improvements in brightness, but those do not always mean that displays are more readable. Most manufacturers also use LTPO (low-temperature polycrystalline oxide) panels in this class, which feature adaptive refresh rates. This way the device can optimize the refresh rate for the type of content that is being shown on the screen.

On the battery front, Ultra Premium phones loaded with demanding new features continue to struggle overall with autonomy. Heavy users will have to find a power outlet sooner or later. Otherwise, charging speed is where smartphone manufacturers currently battle it out in the Ultra Premium segment in terms of battery performance. Some of the latest phones in this class can be fully charged during a short coffee break.

For detailed device performance information, check out the protocol-specific test results on dxomark.com. Now, without further ado, here is DXOMARK’s list of the best Ultra Premium smartphones 2022 so far.

Apple iPhone 13 Pro Max

Released in September 2021, Apple’s flagship iPhone 13 Pro Max provides one of the best all-around smartphone experiences. The smooth interaction between the latest hardware and the software attests to the attention Apple pays to tuning and getting the most out of the device, no matter what the
use case.

This was particularly evident in the phone’s display performance, where it took the lead in our Display ranking because its screen was so readable  in all different kinds of use cases, whether you want to watch a movie outside in sun or indoors in the dark. With a dedicated brightness tuning for HDR, realistic highlights and incredible contrast, any content viewed on the display looks brilliant, making the iPhone an excellent choice for any type of content consumption.

The camera experience particularly shines in video, where the iPhone 13 Pro Max sets the standard with its default Dolby Vision HDR format video settings. Video exposure is very stable and the autofocus provides improved tracking and very smooth refocusing at the right moments. The outstanding overall camera performance is rounded out by excellent still images, with high-quality images in all shooting situations, especially for action shots, and improvements over the previous generation in terms of image detail and in backlit situations.

Unlike many of its power-hungry Ultra Premium competitors, the 13 Pro Max did also very well for battery. Despite using the smallest battery for a flagship device, the Apple managed to outperform all other phones in the segment in the perhaps most challenging area of all –  autonomy – thanks to its excellent efficiency. The device showed that its autonomy can adequately endure the demands of an ultra premium phone and that its autonomy is on-par with bigger batteries in phones that have less-demanding features.

Google Pixel 6 Pro

The Google Pixel 6 Pro, with its proprietary new Tensor chipset, does really quite well in most categories. Google made some huge strides in the camera department with the Pixel 6 Pro. To keep up with other devices in this segment, the Pixel 6 Pro is the first Pixel to have a triple camera set up, and the sensor in its primary camera is twice the size of its predecessor. The Pixel 6 Pro excels at taking portraits and capturing the nuances of all skin tone types. Its zoom range goes from an ultra-wide angle equivalent of 12.5mm to a tele of 102.6 mm, with excellent image detail even in long-range tele shots. Thanks to excellent video stabilization recorded footage is very stable.

And yet, it is with the Selfie camera that the Pixel 6 Pro really stands out, outperforming such heavy weights as the iPhone 13 Pro Max and the Samsung Galaxy S22 Ultra. Whether photos or videos, the Pixel 6 Pro’s front camera delivers high-quality images, with accurate skintones, exposure, and few artifacts. That’s bound to please any Instagrammer.

It’s actually also a good device for gaming, with a display that provides a smooth and accurate touch, as well as a decent response time, as well as audio playback in stereo. The positioning of the speakers means they are hard to block with your fingers when playing games. Overall the Pixel 6 Pro showed a balanced performance in audio, doing well in both playback and recording.

While taking all those pictures and playing games in your Pixel 6 Pro just keep in mind that battery is not the device’s strong point and keep a charger at hand. Autonomy is less than 2 days in moderate use and the charging experience is overall poor, especially when charging wirelessly.

Samsung Galaxy S22 Ultra (Exynos)

Known for loading its flagship Galaxy phones with top-of-the-line specs, such as the longest zoom or brightest display, the Galaxy S22 Ultra is a device of superlatives. But best specs don’t necessarily mean best scores.

The Galaxy S22 Ultra (Exynos) earns its place on this list mainly for its display performance, where it earned a top score not only for readability but also for video thanks to its faithful rendering of HDR10 content. Samsung has fine-tuned the tone mapping of its S22 Ultra display to the point that the device provides an experience that is similar to watching TV.

Combine that with the clear audio playback from the S22 Ultra’s stereo speakers, and you’ve got one powerful on-the-go entertainment system. In the camera department, the S22 Ultra flagship grade long-range zoom (with an equivalent focal length of up to 230 mm) produces images with quite good detail, giving mobile photographers plenty of options for capturing subjects at any distance. However, capturing the decisive moment can be a bit more tricky with the S22 Ultra because a fast autofocus is not one of the camera’s strong points.

Also be sure to pack a charger because the trade off for all the video display quality is that the battery doesn’t quite hold up, with autonomy for both the Snapdragon and Exynos versions at less than 2 days.

Premium: Top performance with some compromises

Premium segment devices (launch price of $600-799) differ from their “money-no-object” Ultra Premium counterparts in so far that designers might have to make compromises in some areas when picking hardware components. As a result, for example, camera specifications and performance in the Premium segment can vary a lot between devices, with cameras showing performance strengths and weaknesses in different areas. If you are looking to buy a Premium phone and the camera is a priority, it’s therefore important to find the device that is best suited to your specific photographic needs. In terms of recent technological developments, compared to the Ultra Premium segment, there seems to be generally less evolution. Many devices offer good zoom quality but are still some way off the flagship class in this respect.

Camera specifications and performance in the Premium segment can vary a lot between devices.

Unlike Ultra Premium phones, many Premium devices do not feature adaptive display refresh rates and maximum brightness of the displays tends to be lower as well. So to make up for the lower spec display hardware on Premium phones, display tuning has to be spot on in order to compete with Ultra Premium phones. Things work in a similar way for audio hardware and tuning.

For detailed device performance information, check out the protocol-specific test results on dxomark.com. Now, without further ado, here is DXOMARK’s list of the best Premium smartphones 2022 so far.

Apple iPhone 13

The Apple iPhone 13 shares most specs with the smaller iPhone 13 mini but comes with a larger Super Retina XDR display. In our tests the iPhone 13 was very balanced, delivering good to excellent results across the board.

Its display offers very good performance across all use cases and the best video playback experience in the Premium segment, with support for HDR10 and Dolby Vision and a focus on color and motion.

The excellent display is accompanied by very good and consistent audio playback but the iPhone 13 is also a great choice for users who want to record audio. Both front and rear cameras tested were among the best in the segment. The rear camera´s video performance is the very best you can find in this class, with a very wide dynamic range and beautiful colors, thanks to the Dolby Vision HDR format. When shooting still images the camera is also capable of rendering nice color, even in backlit scenes and other difficult light situations.

To enjoy the Apple´s great display, sound, and cameras all day long you need a good battery and the iPhone 13 does not disappoint in this respect either. Its battery is relatively small in terms of pure numbers but offers excellent discharge efficiency and therefore good autonomy, and wireless charging works very well, too.

Xiaomi 11T Pro

The Xiaomi 11T Pro follows on the Mi 10T Pro as Xiaomi´s contribution to the Premium price segment. In our testing it shone especially under the DXOMARK Camera and Battery test protocols.

The main camera comes with the best autofocus system in the Premium segment. Its zero-shutter-lag capabilities allow for very quick image capture when the shutter button is pressed. This means it’s easier to take an image of the decisive moment and of people in motion than with its rivals in the segment. In general still image camera performance is good in most shooting situations in bright light and video clips recorded on the 11T Pro in bright light or under typical indoor conditions offer good detail.

Battery anxiety is not something Xiaomi 11T Pro users need to worry about. The device offers an outstanding charging performance, achieving 80 percent in only 18 minutes and a full charge in 30 minutes, so even a short coffee break with the phone plugged in will give you many additional hours of use. The Xiaomi also has the best battery autonomy in its class, a whopping 56 hours with moderate use.

The 11T Pro is not among the very best Premium devices for Audio and Display but still performs on a level in these categories that will keep most users happy, making it overall an easy recommendation, especially for those who prioritize camera and battery.

Samsung Galaxy S22 (Exynos)

The Samsung Galaxy S22 (Exynos) is the junior model in the Korean company’s 2022 top-end S-series, and relies on the same Exynos 2200 chipset as the flagship S22 Ultra. However, it comes with a smaller and lower-resolution display than the Ultra, and doesn’t have the larger device’s S-Pen stylus or 10x tele lens.

The Samsung performs well across several of our test areas and our testers especially recommend it for its audio performance. Sound is clear in both playback and recorded audio files. Recording also offers great dynamics but on the downside playback can lack bass and sharpness.

The Galaxy also does well for Display where it is among the very best in the Premium segment. Appropriate brightness and faithful colors make the Samsung Galaxy S22 (Exynos) an ideal smartphone for viewing streaming content. The main camera does a good job as well and earns the S22 a place among the best in its class. In photo mode the Samsung captures very nice and vivid color, good exposures, and a wide dynamic range. The tele zoom works well at close and medium range and video clips offer excellent dynamic range in difficult high-contrast scenes.

The Galaxy S22 really only lags behind the Premium segment competition in the battery category, so isn’t ideal for power users who spend a lot of time away from charging points. 

High End: Great battery experience

In the High-End segment (launch price of $400 to $599), consumers can find a lot of performance at more affordable prices and in the battery categories, High-End phones often have advantages over their Ultra Premium and Premium counterparts. They often come with scaled-down (and therefore less battery-hungry) features which means they currently provide the best compromise when it comes to the overall battery experience.

As for cameras, our testers have not seen any significant improvements in the latest generation of High-End devices over the previous one, so purely from a camera point of view there is no need for an immediate upgrade if your current device is still fairly new. Generally High End devices often come with smaller image sensors than higher-priced models, or lack features like optical image stabilization or dedicated tele lenses. However, thanks to optimized tuning in the best devices they are capable of impressive image results in many shooting situations. This is particularly true for shooting with the primary camera and video recording but given the aforementioned hardware limitations, High-End devices mostly lag behind the more expensive models in terms of zoom image quality. The Google Pixel is somewhat of an exception to this general rule and really pretty good at all areas of imaging.

Thanks to optimized tuning, some High-End devices are capable of impressive  results.

For the displays, many manufacturers will make some compromises in terms of readability, and brightness is not usually up to the Premium or Ultra Premium levels. You won’t typically find any adaptive refresh rates in this segment either. In terms of audio hardware, some High-End devices only feature single mono speakers, which makes it hard to compete with dual-speaker setups. Still, the best High-End smartphones are capable of providing very pleasant overall display and audio experiences.

For detailed device performance information, check out the protocol-specific test results on dxomark.com. Now, without further ado, here is DXOMARK’s list of the best High-End smartphones in 2022 so far.

Google Pixel 6

For those who are after excellent camera, audio, and display performance without spending Ultra Premium money Google’s Pixel 6 device is an excellent pick. Its camera is the best in the High-End segment by quite a margin. It does an excellent job on portrait images and captures good detail at close and medium range tele settings, especially when considering that the Pixel 6 does not come with a dedicated tele camera.

The Pixel 6 clearly benefits from the tuning efforts that have gone into the Pixel 6 Pro and the overall image quality between the two models is very close. The main difference is the Pro model’s dedicated tele camera. If you can live without it, the vanilla Pixel is an excellent alternative.

If you are more of a gamer than a photographer, the Pixel 6 is a good choice as well, thanks to a display and sound capabilities that are both very suitable for the task. The screen’s 1100 nits max brightness is the highest in this segment (on par with the Samsung A52/A53) and the smooth and accurate touch allows for comfortable casual gaming. Audio playback is clear with good dynamics, and tonal balance is clear and faithful when recording sound, completing the Pixel’s excellent all-around performance for its class.

Battery performance is one of the few areas where the Google Pixel 6 does not shine, delivering an overall below-average performance for this class. 2 days of autonomy in moderate use should be good enough for most users but charging time was quite long, taking 1 hour and 3 minutes to reach 80% of full capacity.

Apple iPhone SE (2022)

Like the Pixel, the iPhone SE does well in most areas and is a great option for those who are invested in the Apple ecosystem but don’t need the bells and whistles of the more expensive iPhone models.In the camera tests it proved to be a strong and consistent performer all around. It lacks ultra-wide and tele cameras but the 12MP primary camera manages to capture images and videos with good detail. Thanks to a fast and accurate autofocus it’s easy to capture the decisive moment, with the focus on the right spot.

Unlike some rivals in the segment the SE comes with stereo speakers, which provide overall good audio playback, making the phone especially suitable gaming and watching movies. When recording audio it is particularly good at recording loud concerts, thanks to very little distortion. The device also comes with an excellent LCD display that offers smooth and accurate touch responsiveness which again makes it a good option for gaming, despite having a 60 Hz refresh rate.

Battery on the other hand is not a strong point. The battery has a very small capacity of 2018 mAh and offers only 1.5 days of autonomy in moderate usage. At 51 minutes to charge 80% of the battery capacity charging is slower than average as well.

Xiaomi 11T

The 11T is the middle offering in Xiaomi’s T line and a lower-priced alternative to the brand’s flagships, while still offering some impressive features. Our testers were especially impressed by the 11T’s Display and Battery performance. The 6.67-inch AMOLED display offers the best readability in the segment. The device is tuned to adapt readability and brightness to the environment and ensure smooth transitions. It also offers the best video viewing experience in its class.

The battery provides excellent autonomy for this segment (59 hours of moderate use) and recharging with 60 percent or less capacity remaining gets you an additional 8 hours of autonomy which should get you even through the toughest work days.

The Xiaomi 11T is no slouch in our other test categories either. It’s among the better phones for Audio in the High-End segment, thanks to a natural and satisfying tonal balance and good low-end and bass for a smartphone. It also performed well as a recording device across most use cases. The camera scores points with its excellent texture/noise trade-off in still images and nice colors in video.

Oppo Reno6 5G

The Reno6 5G‘s strong point is its battery performance. Charging efficiency is outstanding and charging times are very good. When under 50 percent capacity the device gains a whopping 10+ hours of autonomy with only a 5-minute charge, so even the quickest of coffee breaks next to a power outlet will get rid of your battery anxiety.

The device is also a decent camera option in the High-End segment but better alternatives are available if photo and video capture is a priority. It delivers a middle-of-the-pack performance for Display, where colors are faithful the outdoor image enhancement is unstable. With a single mono speaker, the Oppo isn’t your first option for audio playback but it can make a decent recording device where it offers great loudness and signal-to-noise, thanks to an efficient noise cancellation algorithm.

Oppo Find X5 Lite

Like its stablemate the Reno6 5G, the Oppo Find X5 Lite is the most affordable model in the Chinese manufacturer’s Find X5 series and is an interesting option in the High-End segment, especially when battery performance is high up the list of priorities. The device offers good charging times, reaching 80 percent in only 22 minutes, and excellent charging efficiency. In addition, 2.5 days of autonomy in moderate use is above-average for the segment.

The device also does reasonably well for camera, with well-exposed still images when the light conditions are not too challenging, and a wide dynamic range in video footage. However, both photo and video results drop in quality in more difficult and lower light conditions. In the audio category, the phone is held back by its single mono speaker, and while the display offers accurate touch and good readability in low light, our testers also noted a lack of brightness in indoor and outdoor conditions and color issues when watching HDR10 videos.

The post Best of smartphones [Summer 2022] appeared first on DXOMARK.