2017, the year of the new mobile protocol

Not only has 2017 been an exciting year for smartphone cameras, with all the big name manufacturers launching new products, embracing innovative technologies, and delivering a better-than-ever mobile imaging experience to consumers, it has also an extremely busy time for DxOMark, especially during the second half of the year. We launched a new website design and an updated version of our DxOMark Mobile test protocol that takes into account improved device performance and new technologies that weren’t around when we first started testing smartphone cameras in 2012. Among other new criteria, the new protocol incorporates zoom performance, bokeh quality, capture of moving scenes, and image quality in very-low-light conditions.

When we announced the new protocol in September, we also published the results of our retests of several iconic smartphone cameras that had performed very well in our tests using the original protocol — for example, the Apple iPhone 7, the Samsung Galaxy S6 Edge, and the Nokia 808 PureView. In this article, we want to give you an overview of all the smartphone camera reviews based on the new DxOMark Mobile test protocol that we have published since then.

Overall results

In the graph below you can see all smartphones that we have tested with the new DxOMark Mobile protocol. The ones marked in yellow are the older models that we’d previously tested using the original DxOMark Mobile, and which we retested for the launch of the update. As you can see, with a score of 98 points, Google’s Pixel 2 follows in the footsteps of its predecessor, which was last year’s top smartphone. Its overall best performance puts it at the top of our ranking this year. Apple’s new anniversary model iPhone X and the Huawei Mate 10 Pro with its Leica camera follow very closely behind the Pixel 2.

Other high-end models, such as the Samsung Galaxy Note 8 and the iPhone 8 Plus put in very good performances as well. With the Xiaomi Mi Note 3, a mid-range device came impressively close to considerably more expensive flagship models. Unsurprisingly, at the other end of the rankings we find entry-level smartphones that cost only a fraction of the latest Apple, Google, and Samsung flagships. That said, the Indian Lava Z25 and the Chinese Gionee S10L performed remarkably well, considering their price points.

Some of the older retested models were capable of holding their own as well. That’s not much of a surprise in the case of the HTC U11, which is still the  flagship and which occupied a top position in our old ranking just before the switch to the new protocol. However, it’s quite impressive that the Nokia 808 PureView, a device launched in 2012, is still able to compete with some current budget phones. Without a dual-camera setup or any computational imaging-based features, it is far from the level of current high-end models, however, despite the largest image sensor of all cameras in this comparison.

Photo performance

Looking at only still image performance, the situation is very similar to the overall score. However, some positions have been swapped and the podium is entirely occupied by dual-camera equipped phones that can produce better zoom quality and a more natural bokeh effect than the single-lens competition. That said, the scores between the leading devices are extremely close.

Exposure, color, and dynamic range

All of the high-end devices we tested with our updated protocol deliver good exposure, color, and contrast in pretty much all light conditions. Auto HDR modes are activated by default on all flagship models, allowing for the capture of good shadow and highlight detail in difficult high-contrast scenes. In the sample scene below, you can see how the Google Pixel 2 and the iPhone X are capable of recording a very wide dynamic range. The Samsung Galaxy Note 8 also does a good job, just avoiding highlight clipping in the brighter parts, but it captures noticeably less detail in the shadows, and shows stronger contrasts than the other two cameras.

At the other end of the budget scale, entry-level devices often don’t have the processing power to make Auto HDR a default option, or they simply rely on less-efficient HDR processing than flagship models. As a result, photographing high-contrast scenes is much more of a challenge, with a lot of detail lost in the brightest and darkest parts of the frame. In the sample below, both cameras fail to record almost any detail in the brightest areas. The Meizu Pro 7 Plus image shows a little more tonal range in the shadows than the Micromax, however.

Exposure in very low light is another area where spending more money on a device currently still gets you noticeably better results. Within our updated DxOMark Mobile protocol, we test exposure at light levels as low as 1 Lux. Thanks to fast apertures and multi-frame image stacking, most high-end models such as the iPhone X in the sample below on the left, are capable of producing slightly underexposed but usable images in such light conditions. Budget devices such as the Gionee S10L generally have to make do without those advanced features, and thus have much more difficulty recording good levels of detail in very low light.

Texture and noise

Detail and noise are two image quality parameters that are strongly interlinked, and improving one of them typically has a negative impact on the other. All of the high-end models in our testing are capable of recording images with detailed textures and low noise levels in all light situations. However, different manufacturers take different approaches to texture and noise, prioritizing one over the other. In our testing, the Google Pixel 2 achieved a class-leading texture score of 73, but a slightly lower noise score of 59. With a score of 65 for texture and 75 for noise, the Samsung Galaxy Note appears to take a reverse approach. Apple’s iPhone 8 Plus is the most balanced in this comparison, scoring 65 for texture and 69 for noise.

In the samples below, you can see what those numbers mean in terms of real-life image quality. In bright light (1000 Lux), the Pixel 2 delivers excellent detail with fine textures and precise edge definition. The Samsung image is noticeably softer when viewed at a 100% magnification, and the iPhone 8 Plus image output falls pretty much between the other two.

The advantages of Samsung’s low-noise approach become more obvious in low-light shooting, however. Compared to the Google and Apple devices, the Note 8 produces a noticeably smoother image with less luminance noise (which is particularly noticeable on the skin tones) at a light level of 5 Lux.

In comparison, budget-oriented smartphones, such as the Micromax Canvas Infinity (which scores 46 points for texture and 45 for noise), are capable of producing decent detail in bright light, but struggle as the light gets dimmer. A loss of detail and increased noise levels are very noticeable in the 20 and 5 Lux samples below, plus the camera has trouble capturing bright enough exposures at those low light levels.

Image artifacts

The imaging pipeline in a smartphone camera is a very complex construct comprising sensor, lens, and image processing. Image artifacts can be caused by any of these components. Thankfully, on most devices we have tested with the new protocol artifacts are well-controlled and can only be spotted when examining an image at full size. Below you can see a few examples of the more intrusive artifacts we have found during our testing. For example, the Google Pixel 2, like its predecessor, has a tendency to produce flare when shooting against a strong light source. We found an unusual blocking artifact in some outdoor images that we captured during our test of the Huawei Mate 10 Pro. You can spot it in the sample below in the blue sky to the left of the building.

While high-end smartphone cameras mostly offer good sharpness across the frame, more affordable devices frequently struggle with sharpness. The Meizu Pro 7 in particular suffered from inconsistent sharpness across the frame. Although many mobile devices in its class suffer a loss of edge sharpness, the Meizu can display a loss of sharpness in random parts of the picture, including in the center, which seriously affects image quality.

Autofocus performance

All high-end devices we have tested since our switch to the new test protocol come with on-sensor phase-detection autofocus and/or laser-assisted systems. As a result, the AF systems in almost all flagship devices worked swiftly and produced sharp and repeatable image results. Below you can see our test results for the Huawei Mate 10 Pro at 20 Lux. As you can see, even in these fairly low-light conditions, its autofocus system is capable of producing consistently good results at high speed. At 96 points, the Huawei snapped up our second-highest autofocus score per the new protocol, and was surpassed only by the Google Pixel 2 at 98.

At 73 points, the Meizu Pro 7 Plus was one of the weaker devices in this category. Shooting 30 consecutive frames in low light (20 Lux) in the lab, and defocusing each time, it was quick to fire after both a short or a long delay, but sharpness was inconsistent, with some shots slightly or completely out of focus, particularly after a short delay.

Zoom and bokeh

Zoom performance is one of the new components in the updated DxOMark Mobile test protocol. Until fairly recently, most smartphone cameras were limited to a simple digital zoom that had a very deleterious effect on image quality. But now some current flagship models come with new technologies for improved zoom performance. The Apple iPhone 8, for example, comes with a secondary tele-lens that doubles the focal reach of its main camera. The Huawei Mate 10 uses the high 20Mp resolution of its secondary monochrome sensor and computational imaging to capture a high-quality digital zoom image. In the comparison below, you can see that the Apple’s 2x zoom image shows better detail than the equivalent shot captured with the Huawei Mate 10 Pro. However, the latter’s system is still a noticeable improvement over a more conventional single-lens system and digital zoom such as that as used in the Google Pixel.

Bokeh quality is another new test criterion and again, an area in which dual-cam equipped devices generally have an advantage over single-lens systems, thanks to better depth-sensing capability. In the samples below, you can see how the iPhone X’s bokeh simulation in Portrait mode produces a natural and pleasant background blur effect, with a realistic shape to circular spectral highlights that’s similar to optical rendering. Slight depth estimation failures result in parts of the portrait subject being slightly blurred — although not as blurred as the background; and overall, the bokeh effect is pretty good, earning the iPhone X the best score of all tested devices in this category at 55 points. The Samsung Galaxy Note 8 achieved a lower score of 45 points, but image results are still decent, with a strong background blur effect, reasonably good subject masking, and excellent depth estimation.

It takes more than just two lenses to generate a natural-looking bokeh effect, however. This becomes obvious when looking at some of the results from more affordable devices, such as the Meizu Pro 7 Plus or the Gionee S10L. The blur intensity is very strong using the former’s bokeh simulation mode, which can make for a striking effect, but it’s not very natural. Further, the shape of the bokeh is pleasing in some areas, but depth estimation failures can result in visible artifacts around the subject. The latter’s bokeh results leave even more room for improvement: subject isolation is poor, and despite the camera’s face detection feature, even the portrait subject’s face is sometimes partly blurred. Background blur tends to be unnaturally strong, and the transition between sharp and blurred areas is very abrupt, generating quite unnatural-looking images overall.

Video performance

When looking at video performance in isolation, the picture is slightly different than the photo scores. The Google Pixel 2 has a 5-point lead at the top, and even its predecessor still occupies the third place in our rankings. The Samsung Galaxy Note 8 and iPhone X drop a few places, while Apple’s iPhone 8 models perform on a similar level as for stills. The same is true for the Huawei Mate 10 Pro, which achieves excellent results in both categories. Most high-end models we tested in recent months offer reliable autofocus and smooth exposure and white balance transition in changing light conditions. However, dynamic range and stabilization when walking or panning can still be a challenge.

Conclusion

It’s easy to see that current flagship smartphones are better than ever, and now that smartphone zoom performance has radically improved, there are even fewer reasons to carry a digital compact camera. It’ll still take some time before smartphones can capture DSLR-like image quality, but it is encouraging to see the massive steps that mobile imaging engineers have taken in terms of low light performance and bokeh quality in very little time.

There is still a considerable image quality gap between flagship devices and entry-level smartphones, however. The latter are generally capable of recording nice images in good light conditions, but struggle in low light and with advanced features such as zoom or bokeh simulation. So while smartphone imaging performance is good as never before, there are still a lot of areas of potential improvement across all device categories. We are looking forward to having a closer look at and testing what the manufacturers will surprise us with in 2018.