Apple patents Lytro-like refocusable camera suitable for iPhone
The U.S. Patent and Trademark Office on Tuesday awarded Apple a patent for a camera system that uses a microlens adaptor to enable refocusing of an image after the initial shot is taken, much like the Lytro light-field camera.
Source: USPTO
Apple's U.S. Patent No. 8,593,564 for a "Digital camera including refocusable imaging mode adaptor," describes a method in which a user can take a picture at a certain resolution and refocus the resulting image after the fact. Also noted in the patent is the inclusion of such a system in portable devices, like an iPhone.
The patent describes a plenoptic camera, otherwise known as a "light-field camera," that uses a microlens array in the form of an adaptor rather than a fixed and integrated component as seen on what is arguably the most famous consumer light field camera, the Lytro. Founded by Stanford University computer science graduate student Ren Ng, Lytro Inc. has drawn funding from heavy-hitting investors despite its limited hardware lineup. Late Apple cofounder Steve Jobs famously showed interest in the tech and reportedly invited Ng to his Palo Alto, Calif., home for a live demonstration of the camera before it went public.
In fact, Apple's patent cites Ng's invention as prior art, but notes improvements can be made in regard to picture quality and resolution.
Light field technology uses so-called microlenses disposed in front of a CMOS or CCD sensor, which together form a "ray sensor" capable of capturing intensity of light as a function of position and angle. The microlens array directs light in a specific and predictable manner based on the lens aperture. In the case of Apple's patent, each microlens corresponds to a fixed set of imaging pixels. After the light information has been recorded, software can be used to focus and refocus the corresponding image.
Unlike Lytro, Apple's design employs a movable adaptor situated between the lens element and the imaging sensor. The adaptor holds the microlens array, meaning the camera can operate in two separate modes: high-resolution non-refocusable and low-resolution refocuasable. Lytro's microlens component can also be moved closer and farther from the sensor, but cannot be completely removed from the light path.
Apple's patent describes the adaptor as a switchable optical module mounted to a bracket, on which a glass plate is also positioned for normal high-resolution photos.
Low-resolution refocusable mode with microlens array active.
When in low-resolution refocusing mode, as seen above, the optical module is positioned between the lens and the sensor, thus creating a ray sensor. The patent notes the sensor records light data from the microlens (positioning) and angle of incidence (direction). Spatial resolution is limited to that of the microlens array.
In high-resolution non-refocusing mode, seen below, the switchable optical module moves to a different position in which the glass plate is situated between the lens and imaging sensor. Light is redirected by the glass pane onto the sensor to create an image of the scene as would a normal camera. The glass has a thickness and refraction index that compensates for the focal plane adjustment made to accommodate the microlens array. This mode allows for the sensor's full resolution to be used.
High-resolution non-refocusable mode with microlens array inactive.
In some embodiments, the hardware includes smartphone-like features, like the ability to place phone calls and connect to a wireless network. Other forms of the patent include a system that can be used as an add-on accessory for any existing camera, an interesting thought given the standalone Lytro starts at $299.
It is unknown if Apple will employ the technology in a later iPhone model, though the solution would be a welcome addition for many users who have trouble focusing on small screens. Perhaps most interesting is the possibility of a light-field video camera. Refocusable movies would be a great addition to the iPhone, which lacks tactile controls for focusing on the fly.
Apple's plenoptic adaptor patent was first filed for in 2011 and credits John Norvold Border and Richard D. Young as its inventors.
Source: USPTO
Apple's U.S. Patent No. 8,593,564 for a "Digital camera including refocusable imaging mode adaptor," describes a method in which a user can take a picture at a certain resolution and refocus the resulting image after the fact. Also noted in the patent is the inclusion of such a system in portable devices, like an iPhone.
The patent describes a plenoptic camera, otherwise known as a "light-field camera," that uses a microlens array in the form of an adaptor rather than a fixed and integrated component as seen on what is arguably the most famous consumer light field camera, the Lytro. Founded by Stanford University computer science graduate student Ren Ng, Lytro Inc. has drawn funding from heavy-hitting investors despite its limited hardware lineup. Late Apple cofounder Steve Jobs famously showed interest in the tech and reportedly invited Ng to his Palo Alto, Calif., home for a live demonstration of the camera before it went public.
In fact, Apple's patent cites Ng's invention as prior art, but notes improvements can be made in regard to picture quality and resolution.
Light field technology uses so-called microlenses disposed in front of a CMOS or CCD sensor, which together form a "ray sensor" capable of capturing intensity of light as a function of position and angle. The microlens array directs light in a specific and predictable manner based on the lens aperture. In the case of Apple's patent, each microlens corresponds to a fixed set of imaging pixels. After the light information has been recorded, software can be used to focus and refocus the corresponding image.
Unlike Lytro, Apple's design employs a movable adaptor situated between the lens element and the imaging sensor. The adaptor holds the microlens array, meaning the camera can operate in two separate modes: high-resolution non-refocusable and low-resolution refocuasable. Lytro's microlens component can also be moved closer and farther from the sensor, but cannot be completely removed from the light path.
Apple's patent describes the adaptor as a switchable optical module mounted to a bracket, on which a glass plate is also positioned for normal high-resolution photos.
Low-resolution refocusable mode with microlens array active.
When in low-resolution refocusing mode, as seen above, the optical module is positioned between the lens and the sensor, thus creating a ray sensor. The patent notes the sensor records light data from the microlens (positioning) and angle of incidence (direction). Spatial resolution is limited to that of the microlens array.
In high-resolution non-refocusing mode, seen below, the switchable optical module moves to a different position in which the glass plate is situated between the lens and imaging sensor. Light is redirected by the glass pane onto the sensor to create an image of the scene as would a normal camera. The glass has a thickness and refraction index that compensates for the focal plane adjustment made to accommodate the microlens array. This mode allows for the sensor's full resolution to be used.
High-resolution non-refocusable mode with microlens array inactive.
In some embodiments, the hardware includes smartphone-like features, like the ability to place phone calls and connect to a wireless network. Other forms of the patent include a system that can be used as an add-on accessory for any existing camera, an interesting thought given the standalone Lytro starts at $299.
It is unknown if Apple will employ the technology in a later iPhone model, though the solution would be a welcome addition for many users who have trouble focusing on small screens. Perhaps most interesting is the possibility of a light-field video camera. Refocusable movies would be a great addition to the iPhone, which lacks tactile controls for focusing on the fly.
Apple's plenoptic adaptor patent was first filed for in 2011 and credits John Norvold Border and Richard D. Young as its inventors.
Comments
Apple "gets it". The camera on the iPhone is disrupting the entire photography industry. This is the next revolutionary (rather than evolutionary) change.
Apple doesn't appear to have either purchased or invested in Lytro. The only evidence of any interaction is Steve Jobs meeting with their CEO back in 2011.
There is nothing that the fixed focal length camera can do, that the "light field" does just as well
Fixed focal length cameras have much higher resolution due to the fact that they do not need to split the sensor. This might somehow change in the future, but at the moment that's how it works.
Kind of off topic, but it makes me wonder how many animals can see more depth than we can with single eyes, using tricks like this.
Some insects have multiple eyes, like a fly or a bee.
You got that right ...
... here we are all reading some of Samsung's and Google's R&D departments' research work ... err ... I mean Apple Insider ...
IT BEGINS. IT. BEGINS.
Here goes Apple, revolutionizing photography. Again. This will be huge.
Edit: Lytro already offer all of the above I now see ... OMG the future is now
AI was lit up with some great comments about all this back in 2011 (I assume).
Sensor sensitivity can conceivably be increased but, given the small sensor size in a cell phone, even the large-aperture diffraction limit for visible light of, at best, a couple of micron, limits raw sensor array size to around 10 MP. Reserving just a 10 x 10 array for each microlens to capture the directional light field data drops the spatial resolution by a factor of 100, leaving just a 0.1 MP image.
Right now it seems to me that sensor sensitivity improvements may instead lead to something resembling high-speed burst mode focus stacking, which leaves spatial resolution unchanged, as a far more practical solution to the same problem.
Oh ye of little faith
Actually what Apple will do IMHO is make an esoteric feature like this, an everyday feature people, as they always do, take for granted. How many folks using an iPhone even know what HDR is for example, yet use it everyday? That said I want this on a high end Canon
I think the parallax effect capable with a light field camera is a natural fit for the iPhone.
Being able to control the 3D view using the gyroscopes in iPhones and iPads would be magical. Apple loves magical.
Great picture of Walt ... I hope anti sea sickness pills are included ...
Ah you cracked the obviously hidden secret! That's how they did it! iOS7 obviously already has the play back ability ...