How Apple's new Retina Flash -- 5 years in the making -- brightens up selfies on the iPhone 6s
While more than one rumor predicted that Apple would add a front-facing flash to its newest iPhone -- an increasingly common feature among high-end handsets --?nobody considered that the company would do so in a way that alleviated the need to build in extra hardware, despite evidence that Apple has been exploring such a solution for years.
Retina Flash is how Apple has chosen to refer to a new feature that, put simply, turns the display on an iPhone 6s or 6s Plus into a giant flash. If you're a Mac owner, you've probably seen the same kind of effect before when taking a webcam snap in Photo Booth.
Apple says that the iPhone's Retina Flash has several key differences from the Photo Booth incarnation, though.
To begin with, the iPhone's display goes to 11 when it's in Retina Flash mode. Apple touted a custom display driver during Wednesday's keynote that can jack the display's brightness up three times higher than it normally operates at.
Like the rear-facing True Tone flash, the Retina Flash also analyzes the ambient lighting situation --?utilizing a preflash, in this case -?before choosing a customized flash tone. Apple actually uses the True Tone name in reference to this effect as well, somewhat confusingly.
While Apple didn't reveal anything else about the internal secrets of the Retina Flash on Wednesday, a patent awarded to the company in 2012 sheds some more...light...on the subject.
First filed in 2010, the patent --?with the imaginative title "Image Capture Using Display Device As Light Source" --?describes several methods of analyzing the scene, configuring the flash, and finally capturing an image.
Apple contemplates using both a hardware light sensor and analysis of the pixel intensity from the camera feed to determine lightning conditions, then dynamically adjusting the duration, brightness, and color of the display to simulate multiple types of flash. It could be used as a photo flash to compensate for darkness, for instance, or as a fill flash to simply lighten shadows.
Prefacing the True Tone model, Apple also discusses ways of using the flash for color correction:
To combat this, Apple divides the flash time into three intervals --?known as rise time, sustain time, and fall time --?which describe the on and off cycle of the flash. Rise time is the time it takes the flash to reach peak intensity; sustain time is the period during which peak brightness is sustained; and fall time is the time it takes for the flash to turn itself off and ambient light levels to return to normal.
Image capture is then precisely timed to fall within the sustain time, but early enough that the camera does not have the chance to adjust to the flash. Apple also takes into account the camera's frame latency --?the time it takes an image to enter the camera lens, be processed, and then shown on the display --?to ensure the frame captured is the frame that's on screen at the time the user presses the shutter button.
All in all, what appears on the surface to be a simple feature -- flash the screen to make the room brighter -- seems to be another example of Apple's extraordinary attention to detail and drive for perfection.
Retina Flash is how Apple has chosen to refer to a new feature that, put simply, turns the display on an iPhone 6s or 6s Plus into a giant flash. If you're a Mac owner, you've probably seen the same kind of effect before when taking a webcam snap in Photo Booth.
Apple says that the iPhone's Retina Flash has several key differences from the Photo Booth incarnation, though.
To begin with, the iPhone's display goes to 11 when it's in Retina Flash mode. Apple touted a custom display driver during Wednesday's keynote that can jack the display's brightness up three times higher than it normally operates at.
Like the rear-facing True Tone flash, the Retina Flash also analyzes the ambient lighting situation --?utilizing a preflash, in this case -?before choosing a customized flash tone. Apple actually uses the True Tone name in reference to this effect as well, somewhat confusingly.
While Apple didn't reveal anything else about the internal secrets of the Retina Flash on Wednesday, a patent awarded to the company in 2012 sheds some more...light...on the subject.
First filed in 2010, the patent --?with the imaginative title "Image Capture Using Display Device As Light Source" --?describes several methods of analyzing the scene, configuring the flash, and finally capturing an image.
Apple contemplates using both a hardware light sensor and analysis of the pixel intensity from the camera feed to determine lightning conditions, then dynamically adjusting the duration, brightness, and color of the display to simulate multiple types of flash. It could be used as a photo flash to compensate for darkness, for instance, or as a fill flash to simply lighten shadows.
Prefacing the True Tone model, Apple also discusses ways of using the flash for color correction:
Another interesting method described in the patent is a way to essentially protect autofocus from itself. The company worries that some camera modules may accidentally adjust their sensitivity to the flash, making the resulting photos too dark.For example, the screen can be set to various shades of white or to another color (e.g., pink, yellow, etc.) which can affect skin tones. In some implementations, the color balance of the image can be determined, for example, by computing a histogram that represents the color distribution of the image to be captured. The color balance can be corrected by changing the color of the screen to a complementary color prior to flash. For example, if the scene is too blue, the screen color can be changed to a yellow or pink tint to compensate for the blue depending on the color balance the user is trying to capture.
To combat this, Apple divides the flash time into three intervals --?known as rise time, sustain time, and fall time --?which describe the on and off cycle of the flash. Rise time is the time it takes the flash to reach peak intensity; sustain time is the period during which peak brightness is sustained; and fall time is the time it takes for the flash to turn itself off and ambient light levels to return to normal.
Image capture is then precisely timed to fall within the sustain time, but early enough that the camera does not have the chance to adjust to the flash. Apple also takes into account the camera's frame latency --?the time it takes an image to enter the camera lens, be processed, and then shown on the display --?to ensure the frame captured is the frame that's on screen at the time the user presses the shutter button.
All in all, what appears on the surface to be a simple feature -- flash the screen to make the room brighter -- seems to be another example of Apple's extraordinary attention to detail and drive for perfection.
Comments
While seemingly insignificant in the grand scheme of things, this is exactly how Apple continues to innovate, setting them apart from the rest.
Brilliant. And so much better than other phones (like LG) which just turn the screen on full white (and don't even have the 3X overdrive ability).
I always wanted Apple to incorporate a tiny LED indicator for missed notifications, but they opted to use the flash or turn on the screen, both of which are much more power-hungry, especially now with a seemingly smaller capacity battery in the new generation iPhones. I mean, why light up 2,073,600 pixels when one diode would've done the same?
The battery issue is a non-issue. Those that take the dozens of selfies each day to run the battery dry deserve to be forever attached to a charger.
I don't take selfies, and I'm an incredibly handsome guy too! /s
Is it even a full second? I bet if measured it's probably a fraction of a second. And it doesn't need to be at the 3x brightness for the entire time, only when taking the photo. Which means it's probably at full brightness for a very short time (perhaps 1/20th of a second).
It's not going to have any effect on battery. Looking at the picture you just took, showing your friends, editing & cropping it and posting to Facebook are going to consume far more power from the screen than the brief flash to take the picture.
Considering that the screen draws the biggest power consumption of any component in a smartphone, I wonder how much power it takes when it cranks up the brightness and illuminates all the pixels.
I always wanted Apple to incorporate a tiny LED indicator for missed notifications, but they opted to use the flash or turn on the screen, both of which are much more power-hungry, especially now with a seemingly smaller capacity battery in the new generation iPhones. I mean, why light up 2,073,600 pixels when one diode would've done the same?
They did make one. It's called an Apple Watch. Of course you can also get yourself a Pebble that does the same thing.
A very well thought-out process. Changing the color of the screen to correct the color balance is a perfect example of attention-to-detail.
This is what they do for PhotoBooth, so why is it so "new" and innovative?
Considering that the screen draws the biggest power consumption of any component in a smartphone, I wonder how much power it takes when it cranks up the brightness and illuminates all the pixels.
Hard to say since I don't know the power efficiency of the screen in "flash" mode. But, lets assume its superlinear and it consumes 10x the energy during a flash. And, lets assume the flash takes 1 second. And, lets assume the screen is the only power draw in the phone. And, suppose your average use is to use the selfie flash once per hour. Then your power draw would be 3609/3600-1 or 0.25% greater. So if your phone currently lasts 14 hours on a charge, it will now last 13 hours 57 minutes and 54 seconds.
the flash is only for a second. wont drain much at all.
Is it even a full second? I bet if measured it's probably a fraction of a second. And it doesn't need to be at the 3x brightness for the entire time, only when taking the photo. Which means it's probably at full brightness for a very short time (perhaps 1/20th of a second).
It's not going to have any effect on battery. Looking at the picture you just took, showing your friends, editing & cropping it and posting to Facebook are going to consume far more power from the screen than the brief flash to take the picture.
Actually, with Live Photos, the picture is taken for at least 1 full second (half a second before and after the picture).
Yea, how does Retina Flash work with Live Photos?
If Live Photos take pictures a second before and a second after the actual picture, do they need three Retina flashes total for one picture.
No one? I was asking why they couldn't do something like this based on all the older "flashlight" apps that just turned the screen bright white and increased the brightness shortly after the first front facing camera came out.
Should have patented that and joined the legions of patent trolls. j/k those guys suck. LOL
To think that a Great Idea take 10 years to get from Mac OS X to the iphone. Almost exactly 10 years ago Apple introduced Apple Photo Booth. And Photo Booth turned the Screen White to help you get better Pictures up to this day. And we can expect that Apple Patented it bevor 2005. Well maybe we will get "True Tone" So that Funny Pictures get better /s
Actually, with Live Photos, the picture is taken for at least 1 full second (half a second before and after the picture).
I can understand how the camera can take some more photos after the shot, but how does it know to take some photos before you release the shutter? If the flash is required, wouldn't the portion of the photos before the actual shot have a different exposure? Very confusing. I hope they are not just recording for one second after the shutter release and then choosing the middle image in the sequence as your static shot. The shot I usually want is the one when I release the shutter not a photo delayed by a half second. Can someone clear this up for me?
This is a great little bit of technology that only Apple could of thought of. When I first heard the rumours that they would incorporate a front facing flash on the 6S, but then leaks of the 6S front panel came out showing no additional hole for a flash unit. I suspected that they would take this route.
In hindsight of course...when you think about it...using a smartphone's display/backlight as a selfie flash is such an obvious thing to do. If not so easy to engineer, apparently. Anyway kudos to Apple!
I can understand how the camera can take some more photos after the shot, but how does it know to take some photos before you release the shutter? If the flash is required, wouldn't the portion of the photos before the actual shot have a different exposure? Very confusing. I hope they are not just recording for one second after the shutter release and then choosing the middle image in the sequence as your static shot. The shot I usually want is the one when I release the shutter not a photo delayed by a half second. Can someone clear this up for me?
I haven't look at all at what Live Photo is...only the fragments of info people post about it. But my guess would be that the iPhone is constantly recording the last couple of seconds of data off the sensor and temporarily storing it in cache. Then when you hit the shutter it can go back in time to get the frame from half a second in the past (or whatever the timing is). Similar to how DVRs keep a running recording of the last 30 minutes of whatever TV programming is on. So when you get home 5 minutes after your favorite show has started you can go back and see the beginning of the show.
But my guess would be that the iPhone is constantly recording the last couple of seconds of data off the sensor and temporarily storing it in cache.
That was my initial thought as well, but it does seem incredibly inefficient constantly storing and deleting what could be a long duration of high MB images while you are waiting for the right shot, and it still doesn't answer the question of the flash. If it is recording continuously without the flash and then you trip the shutter and the flash goes on, it seems like the exposure might be completely different in some cases.