Photography expert debunks iPhone XS 'beautygate,' details Apple's software-driven camera ...

Posted:
in iPhone edited October 2018
Sebastiaan de With, the developer behind popular manual photography app Halide, has been putting the iPhone XS camera system through its paces. In a comprehensive look into Apple's latest shooters, de With provides an overview of computational photography, and explains how noise reduction technology might generate selfie photos that appear artificially enhanced.

iphone XS camera
iPhone XS and XS Max


In the extensive deep dive, de With outlines why the new iPhone's images look so vastly different to those taken by the iPhone X, even though much of the hardware was carried over from last year. Notably, the developer offers an explanation for an alleged skin-smoothing effect that made headlines over the weekend.

Computational photography is the future

Apple long ago realized that iPhone's future is not necessarily in its hardware. That is why the company generally shies away from publicizing "tech specs" such as processor speed or RAM, and instead focuses on real-world performance.

The same holds true for the camera. Instead of simply cramming more pixels into its cameras each year, Apple focuses on improving existing components and, importantly, the software that drives them. As noted by de With, pure physics is quickly becoming the main obstacle as to how far Apple can take its pocket-sized camera platform. To push the art further, the company is increasingly reliant on its software making chops.

Halide Camera app
Halide photo app


It is this software that has made such a substantial difference for iPhone XS.

SVP of worldwide marketing Phil Schiller during last month's iPhone unveiling discussed the lengths to which the Apple-designed image signal processor (ISP) goes when processing a photo.

With iPhone XS, the camera app starts taking photos as soon as you open the app, before a user even presses the shutter button. Once a photo is snapped, the handset gathers a series of separate images, from underexposed and overexposed frames to those captured at fast shutter speeds. An image selection process then chooses the best candidate frames and combines them together to create an incredibly high dynamic range photo while retaining detail. The process clearly has some unintended consequences.

Bring the noise

To back up all that computational photography, the iPhone XS' camera app needs to take a lot of photos, very fast.

In service of this goal, the new camera favors a quicker shutter speed and higher ISO. When the shutter speed is increased, less light is captured by the sensor. To compensate for the decrease in light, the camera app increases the ISO, which decides how sensitive the sensor is to light. So less light comes in, but the camera is more sensitive to light and is able to output a properly exposed image.

iPhone XS vs iPhone X
Sample RAW photos from iPhone X (left top and bottom) and iPhone XS RAW. | Source: Sebastiaan de With


However, as the ISO increases, the more noise starts to appear in the photo. That noise needs to be removed somehow, which is where Apple's Smart HDR and computational photography comes in to play.

More noise reduction leads to a slightly more smooth-looking image. This is part of the issue seen in the selfie-smoothing "problem" people have been reporting.

The second reason has to do with contrast.

Less local contrast

Local contrast is what most people recognize as sharpness in a photo.

Contrast Example
Source: Wikipedia


"Put simply, a dark or light outline adjacent to a contrasting light or a dark shape. That local contrast is what makes things look sharp," de With says.

When a sharpening effect is applied to a photo, no details are actually being added, but instead the light and dark edges are boosted, creating more contrast and thus the illusion that the image is now sharper.

Smart HDR on the iPhone XS and XS Max does a far better job at exposing an image, which decreases the local contrast and results in an image looking smoother than it otherwise would.

Cameras for the masses

The biggest issues with this method of photography arise when shooting RAW. De With hypothesizes that the iPhone XS still prioritizes shorter exposure times and higher ISO to get the best Smart HDR photo, even when shooting RAW -- which gets no benefit from Smart HDR. This leaves overexposed and sometimes blown out photos with details being clipped and unreclaimable.

RAW shots don't get noise reduction applied either, and with the added noise from the higher ISO, this makes for extremely poor looking images. This means when shooting RAW, you must shoot in manual and purposefully underexpose the image.


Sample RAW photos from iPhone X (left) and iPhone XS Max. | Source: Sebastiaan de With


Apple's apparent camera app decisions affect third-party apps -- such as Halide -- which is why de With says they have been working on an upcoming feature called Smart RAW. It uses a bit of their own computational magic to get more detail out of RAW photos while reducing noise. This new feature will be included in a forthcoming update to Halide.

While a lot of the analysis sounds critical, issues experienced at the hand of Apple's algorithms are so far outliers. Most selfies are taken in very unflattering light, and without the changes Apple has made, they would be poorly exposed and full of noise. Apple has likely erred on the side of caution by over-removing the noise and creating too-smooth images, but this can be pulled back.

More importantly, Apple has the ability to tinker with its firmware to solve the problems through subsequent updates if it so chooses.

As we saw in our recent iPhone XS vs X photo comparison, the iPhone XS and XS Max with Smart HDR have significantly improved photo taking capabilities, and the vast majority of users are already seeing the benefits.
theodore007bb-15wlym
«1

Comments

  • Reply 1 of 22
    saltyzipsaltyzip Posts: 193member
    The iPhone camera photos just seem to becoming more and more fake in my opinion. Reducing noise is one thing, but colours aren't even realistic either.

    This video demonstrates the issue perfectly


    Be interesting to see what dxomark make of the new IPhone's, as the premium android phones have outshone the iphone camera for a while now, iPhone X in 8th place.

    https://www.dxomark.com/category/mobile-reviews/
    theodore007claire1
  • Reply 2 of 22
    gradlygradly Posts: 11member
    saltyzip said:
    Be interesting to see what dxomark make of the new IPhone's, as the premium android phones have outshone the iphone camera for a while now, iPhone X in 8th place.
    https://www.dxomark.com/category/mobile-reviews/
    6th place actually as there are phones with similar marks
    jbdragonbb-15
  • Reply 3 of 22
    A better headline would be

    Photography expert explains iPhone XS 'beautygate,'

    I don't see how this explanation debunks the problem
  • Reply 4 of 22
    silvergold84silvergold84 Posts: 69unconfirmed, member
    I have the X Max and it take beautiful photos. It’s incredible to have a camera like that in a smartphone. The photos are like the eye see. Natural. On few other smartphones , like galaxy, you can see what filter is. The photos appear like fake. Much disturb on the background , especially if you see the photos on big screen like a tv. That do not happen with iPhone . Tremendly high quality and reality response from the photos.
    claire1
  • Reply 5 of 22
    silvergold84silvergold84 Posts: 69unconfirmed, member
    saltyzip said:
    The iPhone camera photos just seem to becoming more and more fake in my opinion. Reducing noise is one thing, but colours aren't even realistic either.

    This video demonstrates the issue perfectly


    Be interesting to see what dxomark make of the new IPhone's, as the premium android phones have outshone the iphone camera for a while now, iPhone X in 8th place.

    https://www.dxomark.com/category/mobile-reviews/
    Dxomark is a depliant of publicity for few android brands. Everybody know it. 
    claire1
  • Reply 6 of 22
    silvergold84silvergold84 Posts: 69unconfirmed, member
    A better headline would be

    Photography expert explains iPhone XS 'beautygate,'

    I don't see how this explanation debunks the problem
    I have the X Max and like many owners and professional photographers I’m surprised about the high quality. Yes , it’s almost incredible to have a camera like that in a smartphone. What you read often online is to don’t let you consider to buy this new iPhones. Look at what happened with the last iPhone X: “notch don’t like to the people, it will be dismissed, it don’t sell” ecc . What the reality was ? Everybody copied (in a bad way like android usually do) the notch , and the iPhone X is been the top seller. So fake news to make you less desiderabile the iPhones. 
  • Reply 7 of 22
    entropysentropys Posts: 1,648member
    A better headline would be

    Photography expert explains iPhone XS 'beautygate,'

    I don't see how this explanation debunks the problem
    I agree. I am sure for the vast majority of photos it isn’t a pr9blem. But every now and again the noise reduction overdoes it. The software needs a bit more tweaking
    bb-15
  • Reply 8 of 22
    saltyzip said:
    The iPhone camera photos just seem to becoming more and more fake in my opinion. Reducing noise is one thing, but colours aren't even realistic either.
    Even though some portraits taken with the XS cameras have ‘overly’ smoothed skin (I’ve seen only one photo so far from my XS like that...), there isn’t reason to generalise and say that iPhone’s take unrealistic photos. The consensus is that the XS cameras are much improved, except for this particular issue which will most likely be resolved.
    edited October 2018 StrangeDays
  • Reply 9 of 22
    melgrossmelgross Posts: 31,507member
    I would be very careful in describing a system by the worst possible circumstances. I know that a lot of these You Tubers try to do their worst. But most photography is not taken at the edges of what possible, nor should it be.

    the fact is that the iPhone takes great pictures at a higher percentage than any other smartphone, and has for a long time.

    when I take photographs, I try to mitigate the problems that may occur in difficult situations. That’s what pros, and knowledgeable amateurs, do. The difficulty with smartphone cameras is that because of the tiny cameras, what may not be an edge condition for a large camera, may be one for a smartphone. Each company deals with these problems differently. Samsung, for example, takes sharp pictures, but only when in high light conditions. And even then, looking at grass and foliage will show hopelessly smeary images there. Apple’s is better. Not quite as sharp in some areas, but not nearly as smeared in others. The Pixel can take some great pictures too, but it really blows out highlights, and even with the software fix, has some of the worst flare I’ve ever seen. But again, flare can be easily mitigated. Just don’t point your camera to the sun. Even $6,000 lenses can have a lot of flare.

    I'm seeing some incredibly good pictures from professionals on the web, taken with Apple’s new cameras, and that’s what matters to me, because it shows what they are capable of when used by people who know what they’re doing.
    edited October 2018 shatssie12StrangeDaysboltsfan17claire1muthuk_vanalingamwlym
  • Reply 10 of 22
    shatssie12shatssie12 Posts: 1unconfirmed, member
    It is one of the best cellphone cameras in the market right now, end of story. Beautygate or not, this phone takes great pictures. And why are people going nuts about not being able to see their wrinkles, pores, ingrown and/or stray hairs? Jeez! Leave the damn phone alone and let US the people who appreciate it just enjoy it. 
    As a photographer myself I see no fault. When I use my thousands of dollars camera I use the noise reducer to soften faces or places and people love that. I think what is pissing people off is the fact that they CAN'T CONTROL the feature. 
    People are so used to add filters to their faces and places that I imagine half the people complaining are pissed cause is no way to manipulate the way this phone take pictures. I suggest something, invest in some decent apps which do so and shut up about the "beautygate"!
    This is a very good article and appreciate the fact that FINALLY someone took the time to try to explain or "debunk" this trend.
    People REALLY have too much time in their hands to look for faults the phone may or may not have. You buy a NEW piece of technology, you have to deal with the issues of the NEW features. And Apple being the company it is, will fix them. They don't charge you a pretty penny for nothing. 
    I got a DOA phone, and they didn't even questioned me, they replaced it on the spot.
    Yes, I spend a lot of money on Apple because I am deeply invested but I don't regret it. I've had a couple of Samsungs break on me a day after the one year warranty expired and what they said to me was "Oh well, your warranty expired, too bad!" Not in those words but that's basically what I heard.
    This phone is great and probably next year's will be even better. That's what companies do and how they keep their customers happy.

  • Reply 11 of 22
    The great thing about computational & software based photography is that it can be tweaked and improved without having to buy new hardware. I am sure Apple is listening to all feedback. As the writer said, smartphone cameras are limited by physics. Theres only so much hardware you can cram into such a small space. I am sure much better hardware can & will be created in the future, but software can be improved infinitely. Apple has the edge here, cause unlike Apple, other companies don’t do both the hardware & software. With Apple they are both so perfectly in sync.

    As a photographer, i am always hoping for more manual control. I think smart HDR is great tho. And to those complaining, you can always go into settings and turn it off. I also think the computational DOF control in portrait mode is Amazing. Purists may scoff, but the background blur has really been improved. And it will keep on being improved. The fact that you can apply it live...or more incredibly after you take a photo blows my mind. I hope Apple keeps giving us manual controls. Like for shutter speed etc. in the meantime, there are other 3rd party photo apps that can provide those controls.

    What excites me as a photographer is that the “s” iphone updates are usually minor ones. I can’t help think what improvements to the camera and smartphone photography Apple will bring to next years iphone. Like a true depth camera system on the back cameras for even more precise DOF control. Until then, happy shooting!


    edited October 2018 shatssie12
  • Reply 12 of 22
    melgrossmelgross Posts: 31,507member
    Also, so that people will know, even companies such as Leica, with their multiple thousand dollar lenses, are correcting distortions and vignetting with in camera software. What else are they doing in there? We don’t know, but we do know that they have been having problems with image quality.

    so I really think that what Apple, and other smartphone companies are doing, even with these early teething problems, are significant advances. We just need to give it a bit of time, and not be so critical right now. The pluses are greater than the minuses, even now.
  • Reply 13 of 22
    bb-15bb-15 Posts: 252member
    entropys said:
    A better headline would be

    Photography expert explains iPhone XS 'beautygate,'

    I don't see how this explanation debunks the problem
    I agree. I am sure for the vast majority of photos it isn’t a pr9blem. But every now and again the noise reduction overdoes it. The software needs a bit more tweaking
    That’s how I see it. An iOS software update can adjust noise reduction. 
    And I appreciate AI interviewing someone who has expertise in photography to discuss what the Xs camera software is doing. 

    I am much less impressed by some people pulling in YouTube channels which make money by trashing Apple products. 
    YT channels which get views by bashing Apple are extremely biased in the selection of content and are worthless imo in comparing the performance of tech across different OEMs.  
    gilly017
  • Reply 14 of 22
    SpamSandwichSpamSandwich Posts: 31,008member
    Can't wait to see what kinds of movies Steven Soderbergh makes with an iPhone X[S] Max. His last low-budget movie was able to get rather high production values out of a lesser iPhone.
    edited October 2018 claire1
  • Reply 15 of 22
    netroxnetrox Posts: 719member
    I am actually surprised how nice the images look through Smart RAW files. They're far much more pleasing and more film-like than Apple's SmartHDR. 


  • Reply 16 of 22
    It is one of the best cellphone cameras in the market right now, end of story. Beautygate or not, this phone takes great pictures. And why are people going nuts about not being able to see their wrinkles, pores, ingrown and/or stray hairs? Jeez! Leave the damn phone alone and let US the people who appreciate it just enjoy it. 
    As a photographer myself I see no fault. When I use my thousands of dollars camera I use the noise reducer to soften faces or places and people love that. I think what is pissing people off is the fact that they CAN'T CONTROL the feature. 
    People are so used to add filters to their faces and places that I imagine half the people complaining are pissed cause is no way to manipulate the way this phone take pictures. I suggest something, invest in some decent apps which do so and shut up about the "beautygate"!
    This is a very good article and appreciate the fact that FINALLY someone took the time to try to explain or "debunk" this trend.
    People REALLY have too much time in their hands to look for faults the phone may or may not have. You buy a NEW piece of technology, you have to deal with the issues of the NEW features. And Apple being the company it is, will fix them. They don't charge you a pretty penny for nothing. 
    I got a DOA phone, and they didn't even questioned me, they replaced it on the spot.
    Yes, I spend a lot of money on Apple because I am deeply invested but I don't regret it. I've had a couple of Samsungs break on me a day after the one year warranty expired and what they said to me was "Oh well, your warranty expired, too bad!" Not in those words but that's basically what I heard.
    This phone is great and probably next year's will be even better. That's what companies do and how they keep their customers happy.

    Hey it’s an iPhone and for the next several news cycles from Forbes to Unboxing Therapy and other ‘YouTuber’ is going to be getting their two cents in. Same crap every year. Watch when Samsung comes out with their phone will be nothing but accolades and fireworks. Your point is well made. 
    claire1
  • Reply 17 of 22
    claire1claire1 Posts: 497unconfirmed, member
    saltyzip said:
    The iPhone camera photos just seem to becoming more and more fake in my opinion. Reducing noise is one thing, but colours aren't even realistic either.

    This video demonstrates the issue perfectly


    Be interesting to see what dxomark make of the new IPhone's, as the premium android phones have outshone the iphone camera for a while now, iPhone X in 8th place.

    https://www.dxomark.com/category/mobile-reviews/
    As soon as you reference a youtube moron over professionals, you lose.

    This unbox idiot is the same iKnockoff Knight who lied about iPhone XR's resolution.

    Anyone who references a youtuber as a source should be laughed at. lol

    "Beautygate".... this anti-Apple propaganda is getting worse every year.
  • Reply 18 of 22
    waltgwaltg Posts: 87member
    While I’m not a pro photographer, I’ve had my share of 35mm and other pieces of equipment, also being in the Apple realm,phone,iPad,iMac,stockholder,,, I DONT like what I see in these comparisons! The colors are NOT true, way too much “smoothing”! One of the selling features of the iPhone is the camera! To me at the moment, they have screwed it up,,, FIX IT!! Period...
  • Reply 19 of 22
    I’m re-posting here my recent comment in iMore.
    https://m.imore.com/beautygate

    Nice report, really. Beyond noise reduction and picking the sharpest buffered frame, imo it does not explains why skin tones and white balance change as soon as a face is detected during the capture. This happens either capturing photos from the front camera, the rear camera or even shooting a video. Just try to cover your face with your hand during a selfie and pay attention how your skin color changes. Such white balance and wierd skin tones even happen whether SmartHDR is active or not. It happens even if manual HDR is active or not. So iOS 12 is doing something in the background by default. And yes, iOS 12.1 still does. Looking at the pictures some time after, the orangy faces could look kind of normal, but it’s just not fine to me, far enough from real colors that causes kind of uncanny valley effect. Why should HDR vary white balance? Why specially noticeable in faces? This effect imo is contributing to the porcelain effect complained for some users.

    Furthermore I noticed erratic global white balance during camera preview that I didn’t observed in previous models, at least with 5s and 6s that I own, which resulted in not few photos during last days with evident magenta-biased coloration. Curiously I found the same issue referred by just a single post in the web during last weeks. [SPA] https://www.xataka.com/analisis/iphone-xr-analisis-caracteristicas-precio-especificaciones

    BTW I have the XR model.

    I hope Apple rectifies this wierd camera issues. To me it is simply unacceptable for the price of any of the X-family devices, to such an extent that I’m seriously thinking on going for an older model and save some bucks.

    Cheers

  • Reply 20 of 22
    kevin keekevin kee Posts: 1,001member
    valillon said:
    I’m re-posting here my recent comment in iMore.
    https://m.imore.com/beautygate

    Nice report, really. Beyond noise reduction and picking the sharpest buffered frame, imo it does not explains why skin tones and white balance change as soon as a face is detected during the capture. This happens either capturing photos from the front camera, the rear camera or even shooting a video. Just try to cover your face with your hand during a selfie and pay attention how your skin color changes. Such white balance and wierd skin tones even happen whether SmartHDR is active or not. It happens even if manual HDR is active or not. So iOS 12 is doing something in the background by default. And yes, iOS 12.1 still does. Looking at the pictures some time after, the orangy faces could look kind of normal, but it’s just not fine to me, far enough from real colors that causes kind of uncanny valley effect. Why should HDR vary white balance? Why specially noticeable in faces? This effect imo is contributing to the porcelain effect complained for some users.

    Furthermore I noticed erratic global white balance during camera preview that I didn’t observed in previous models, at least with 5s and 6s that I own, which resulted in not few photos during last days with evident magenta-biased coloration. Curiously I found the same issue referred by just a single post in the web during last weeks. [SPA] https://www.xataka.com/analisis/iphone-xr-analisis-caracteristicas-precio-especificaciones

    BTW I have the XR mode

    I hope Apple rectifies this wierd camera issues. To me it is simply unacceptable for the price of any of the X-family devices, to such an extent that I’m seriously thinking on going for an older model and save some bucks.

    Cheers

    Are you sure it's not just your screen because of truetone? Your iPhone feature sensors that measure the ambient light colour and brightness so it can correct white point and illumination based on your environmental lighting in order to render the right kinds of white under any conditions. Try turn it off and take the photo again.

    Further reading: https://www.cultofmac.com/571739/iphone-camera-white-balance/
    edited November 2018
Sign In or Register to comment.