How DXOMARK tests display quality

Given that the touchscreen display is the way people interact with their smartphones — from looking at photos to watching videos, from dialing numbers to texting — it was only a matter of time before DXOMARK turned its attention to display performance quality. Well, that time has come. DXOMARK has installed a brand-new, dedicated state-of-the-art test laboratory and hired a team of experts who have been hard at work evaluating display performance since late spring.

Although our DXOMARK Display protocol includes the same kinds of measurements available on other sites — that is, performed on text and single images processed using the device’s gallery app — there are two very important and unique aspects of our new protocol that set it apart from the typical manufacturer’s quality assurance process and from other display quality review sites. The first is that DXOMARK tests displays under real-world conditions. Those manufacturers who calibrate smartphone displays as part of their end-of-line tests do so in a dark room (no ambient light), with the testing device pressed against the display.

In the real world, of course, users do not press their displays right up against their eyes, but hold them several centimeters/inches away, which means that ambient lighting has an impact on how easily people can see the contents on their display. Further, people look at their displays in all kinds of lighting conditions — dark night, indoors, in shade, in sunlight. So DXOMARK engineers evaluate display brightness, contrast, and color both objectively and perceptually by performing tests with the display exposed to different ambient lighting levels.

The second way in which the DXOMARK protocol is unique is that it is the first to test the dynamic attributes of a smartphone’s touchscreen display. This means two things: first, DXOMARK evaluates how well the display renders moving content such as videos and games — in other words, the content that is handled by the device’s default video (or browsing) app, which uses different algorithms to process brightness, color, and so on. Among other things, our engineers check for frame drops and stutter, problems with frame-to-frame image quality, motion blur, and for color and other rendering issues. Second, DXOMARK evaluates the display’s touchscreen functionality, testing its responsiveness, touch accuracy, and smoothness.

(Please note, by the way, that although we are premiering this new protocol by applying it first to smartphones, we plan on testing other kind of displays — tablets, monitors, televisions, etc.— in the future.)

A brand-new Display testing lab

In preparation for this new protocol, DXOMARK installed in its Paris headquarters a new state-of-the-art laboratory entirely dedicated to display testing, and equipped it with the latest testing instruments, including spectroradiometers, imaging colorimeters, conoscopes, high-end DSLRs and professional reference monitors, among other tools. Furthermore, DXOMARK recruited a team of expert scientists and engineers to perfect testing methodologies and to conduct tests.

But that’s not all. The Display team designed and built a brand-new testing chamber, called the Display Bench, to facilitate testing automation and to ensure that all devices are tested under the exact same conditions. The Display Bench includes adjustable mounts for testing tools and for the smartphone display being tested, LED lighting arrays to imitate all kinds of lighting types and brightness levels, and lux meters.

After the test device and the testing tool are attached to their respective mounts, the Display Bench is sealed against any external light sources, and then our experts use a computer console located outside the chamber to take all test shots, as well as to control the servomotors used to change the shooting distances, and to control the lighting arrays that recreate lighting conditions ranging from near-pitch darkness to bright daylight (above 30,000 lux).

What is a smartphone display?

Before we go further, it may be helpful to briefly describe how a typical display is put together. While LCD (“liquid crystal display”) smartphones are still in widespread use, many newer high-end smartphones have OLED (“organic light-emitting diode”) displays that are composed of multiple layers of different materials — glass with an underlying pixel array, a thin film of transistors (TFT) to drive each sub-pixel, a touch panel to enable touch control, and specialized coatings to reduce reflection and other problems. Both LCD and OLED screens are of course connected to their smartphone’s processor and battery.

A smartphone’s display is the feature that consumes most of the battery power available; a typical smartphone display consumes roughly 40% of a charge. The challenge for manufacturers has been to find a good balance between making the display visible in even the most challenging conditions while ensuring that the phone will not run out of power in too short a time interval.

Real-life use cases

Now let’s return to the DXOMARK protocol. But before we get into the specifics of the attributes we measure, it’s important to reiterate that all DXOMARK testing is based on the end-user experience. So when DXOMARK engineers were designing the new protocol, they thoroughly considered the ways in which and under what conditions people typically use their displays in real life. The end result of that study is an array of tests designed around eight specific (and largely self-explanatory) use cases:

• • Web browsing
  • Night reading
  • In-car navigation
  • Taking photos
  • Viewing photos
  • Gaming
  • Watching movies
  • Light transitions (that is, how quickly and how smoothly a display’s auto-brightness function responds to changing light conditions)

Attributes

What makes for a good end-user experience when using a display? Well, first and foremost is how easy it is to look at, how easy it is to see the content on it, whether it’s text or photos or videos. Brightness, contrast, and color fidelity all play a role here. When you look at your display in bright light, can you see the content, or are you simply holding a poor mirror in your hand? Is the color accurate — in other words, do the colors and shades of the red and yellow flowers shown on your display match the red and yellow colors of the flowers that your eyes are seeing in real life? Are there noticeable problems with flickering? When you watch a video or play a game on your phone, is the action you see jerky or smooth?

Interactivity is another big consideration (and not just for gamers). How responsive is the touchscreen? When you scroll and when you pinch to zoom in or out, does the display keep up with your commands? Do you end up where you want to be, or do you have to keep fiddling to get to the right place in a text or on a web page, or to obtain the right amount of magnification?

DXOMARK provides client camera and component manufacturers with highly technical and in-depth reports which include raw data and analyses of very specific aspects of display performance. In our reviews and articles prepared for the general public, however, we focus almost exclusively on the performance characteristics that impact the end-user experience in six different attribute categories: readability, color, video, motion, touch, and artifacts (flaws) — some or all of which are relevant to each use case mentioned above (and as shown in the chart below).

Readability

It bears repeating that the most important consideration for end-users is how easily and comfortably they can read their display under different real-life conditions. At the moment, quite honestly, all displays are hard to see in full outdoor sunlight to a greater or lesser degree. A smartphone display needs to be lit above 1000 nits (or candelas per square meter, cd/m2) to start seeing content under those conditions in a comfortable way, but that kind of output drains a battery very quickly. Average high brightness for many smartphones is often somewhere between 500 and 700 cd/m2 in bright light conditions.

… And this means that no matter the make and model of their smartphone (or nearly so), pretty much all users end up heading for the nearest patch of shade or using their hand to shield their display from direct sunlight. On the other end of the scale, some phone displays are unreadable when at their lowest brightness settings at night, or conversely, they are too bright for the dark ambient conditions (meaning that when users turn on their phone display, they immediately use the brightness slider to manually dim the display to a more comfortable viewing level).

The characteristics that comprise this attribute (and for which DXOMARK tests) include:

• • Brightness/contrast vs ambient lighting [widget]
  • Brightness vs average pixel level (APL)
  • Brightness vs time
  • Uniformity
  • Gamma vs ambient lighting: Gamma is the relationship between the numerical value of a pixel in an image file and the brightness of that pixel when viewed on a screen.
  • Blue light filter impact

Color

From an end-user’s point of view, what tends to be the most important consideration about color is how faithfully the hues on the display match what the user sees in real life. In addition to manufacturers’ design choices and tuning, different lighting conditions, reflectance, and angle viewing can significantly impact how well users perceive a display’s color rendering. DXOMARK tests within both the sRGB (standard red, blue, green) and the newer DCI-P3 (digital cinema intiatives – protocol 3) color spaces (also known as gamuts) for the following characteristics:

• • White color vs ambient light
  • Color fidelity vs ambient light
  • Color shift vs viewing angle
  • Uniformity
  • Blue light filter impact — the display’s blue light filter should not have so strong an effect as to significantly alter the display’s color profile.

Video

Unlike a smartphone’s default gallery app, which handles one image at a time, a smartphone video or browser app has to process many images or frames per second, which poses different kinds of challenges. The Display protocol tests for some of the same kinds of image quality attributes for video content as it does for still images — brightness, contrast, gamma, and color (including skin tone rendering).

Motion

How well does the display handle moving content? In practical terms, this means that DXOMARK tests for frame drops and stutter, motion blur, and video playback reactivity.

Touch

All end-users, not just gamers, care about how responsive, accurate, and smooth the touchscreen is. Among other considerations, our tests determine if the touchscreen avoid under- or overshoots when zooming, and if it scrolls quickly and smoothly — in other words, the DXOMARK protocol evaluates a display’s touch accuracy and smoothness.

Artifacts

An artifact is just another name for a flaw or problem that falls outside other attribute categories. DXOMARK has identified and devised ways of measuring the most common artifacts found in displays:

• • Notch: Manufacturers have to find a place to put front-facing features (for example, cameras and speakers). Rather than reduce the overall size of the screen by putting these components outside of the screen area, many models now include them right on the display. DXOMARK measures not just how much of the display area is taken up with the notch, but also how intrusive it is when it comes to viewing or interacting with content.
  • Reflectance: The lower the reflectance, the easier it is to see content on a display in high ambient light conditions.
  • Flicker: Most users’ eyes are not very sensitive to flicker, but flicker that matches or comes close to 30Hz (or any first multiples thereof) can be noticeable.
  • Aliasing: This phenomenon is seen as curved edges taking on a “stair-step” appearance.
  • Ghost touch: Does the phone inappropriately respond to your hand just resting on the perimeter of your display as you hold your phone? (Sometimes this means that the display cannot perceive your “real” touches.)
  • Judder: Jerky or uneven motion during playback; this can be particularly visible with videos at 24 fps when 3:2 pulldown is poorly managed.

Conclusion

Our reviews will largely follow the same format as for our other protocols (sensors, lenses, smartphone rear cameras, selfie cams, audio): An introduction, essential specifications, test summary, in-depth analyses per attribute (with occasional dives into sub-attributes), conclusion, and pro & con bullet points. We will also publish one or more articles that will provide much greater detail about various aspects of our Display protocol, including more information about our testing setup and the equipment we use to evaluate specific attributes, so stay tuned!