Apple patents Lytro-like refocusable camera suitable for iPhone

Posted:
in General Discussion edited August 2014
The U.S. Patent and Trademark Office on Tuesday awarded Apple a patent for a camera system that uses a microlens adaptor to enable refocusing of an image after the initial shot is taken, much like the Lytro light-field camera.

Plenoptic
Source: USPTO


Apple's U.S. Patent No. 8,593,564 for a "Digital camera including refocusable imaging mode adaptor," describes a method in which a user can take a picture at a certain resolution and refocus the resulting image after the fact. Also noted in the patent is the inclusion of such a system in portable devices, like an iPhone.

The patent describes a plenoptic camera, otherwise known as a "light-field camera," that uses a microlens array in the form of an adaptor rather than a fixed and integrated component as seen on what is arguably the most famous consumer light field camera, the Lytro. Founded by Stanford University computer science graduate student Ren Ng, Lytro Inc. has drawn funding from heavy-hitting investors despite its limited hardware lineup. Late Apple cofounder Steve Jobs famously showed interest in the tech and reportedly invited Ng to his Palo Alto, Calif., home for a live demonstration of the camera before it went public.

In fact, Apple's patent cites Ng's invention as prior art, but notes improvements can be made in regard to picture quality and resolution.

Lytro
Lytro light-field camera. | Source: Lytro


Light field technology uses so-called microlenses disposed in front of a CMOS or CCD sensor, which together form a "ray sensor" capable of capturing intensity of light as a function of position and angle. The microlens array directs light in a specific and predictable manner based on the lens aperture. In the case of Apple's patent, each microlens corresponds to a fixed set of imaging pixels. After the light information has been recorded, software can be used to focus and refocus the corresponding image.

Unlike Lytro, Apple's design employs a movable adaptor situated between the lens element and the imaging sensor. The adaptor holds the microlens array, meaning the camera can operate in two separate modes: high-resolution non-refocusable and low-resolution refocuasable. Lytro's microlens component can also be moved closer and farther from the sensor, but cannot be completely removed from the light path.

Apple's patent describes the adaptor as a switchable optical module mounted to a bracket, on which a glass plate is also positioned for normal high-resolution photos.

Plenoptic
Low-resolution refocusable mode with microlens array active.


When in low-resolution refocusing mode, as seen above, the optical module is positioned between the lens and the sensor, thus creating a ray sensor. The patent notes the sensor records light data from the microlens (positioning) and angle of incidence (direction). Spatial resolution is limited to that of the microlens array.

In high-resolution non-refocusing mode, seen below, the switchable optical module moves to a different position in which the glass plate is situated between the lens and imaging sensor. Light is redirected by the glass pane onto the sensor to create an image of the scene as would a normal camera. The glass has a thickness and refraction index that compensates for the focal plane adjustment made to accommodate the microlens array. This mode allows for the sensor's full resolution to be used.

Plenoptic
High-resolution non-refocusable mode with microlens array inactive.


In some embodiments, the hardware includes smartphone-like features, like the ability to place phone calls and connect to a wireless network. Other forms of the patent include a system that can be used as an add-on accessory for any existing camera, an interesting thought given the standalone Lytro starts at $299.

It is unknown if Apple will employ the technology in a later iPhone model, though the solution would be a welcome addition for many users who have trouble focusing on small screens. Perhaps most interesting is the possibility of a light-field video camera. Refocusable movies would be a great addition to the iPhone, which lacks tactile controls for focusing on the fly.

Apple's plenoptic adaptor patent was first filed for in 2011 and credits John Norvold Border and Richard D. Young as its inventors.
«134

Comments

  • Reply 1 of 79
    shogunshogun Posts: 362member
    I was just thinking about this yesterday in relation to Apple's acquisition of the 3D scanner company. Maybe they align with each other to create some cool new capacities?
  • Reply 2 of 79
    irelandireland Posts: 17,771member
    Did Apple acquire Lytro a while back?
  • Reply 3 of 79
    doxxicdoxxic Posts: 100member
    Kind of off topic, but it makes me wonder how many animals can see more depth than we can with single eyes, using tricks like this.
  • Reply 4 of 79
    hodarhodar Posts: 338member
    Just as Digital killed Film, a "light field" camera will kill the fixed focal length photography. There is nothing that the fixed focal length camera can do, that the "light field" does just as well; and many very neat effects that are available on the "light field" camera, that are impossible to do with the fixed focal length camera.

    Apple "gets it". The camera on the iPhone is disrupting the entire photography industry. This is the next revolutionary (rather than evolutionary) change.
  • Reply 5 of 79
    solipsismxsolipsismx Posts: 19,566member
    ireland wrote: »
    Did Apple acquire Lytro a while back?

    Apple doesn't appear to have either purchased or invested in Lytro. The only evidence of any interaction is Steve Jobs meeting with their CEO back in 2011.

  • Reply 6 of 79
    hodarhodar Posts: 338member
    Birds, especially non-preditory birds have various tricks for good depth perception. Pidgeons, for example move their heads back and forth as they walk, like they are peckin the air. Other birds, like ducks use a combination of moving thier head forward and back, as well as left/right. I'm unaware of any animal that has a removeable lense for this ability.
  • Reply 7 of 79
    Quote:

    Originally Posted by Hodar View Post



    There is nothing that the fixed focal length camera can do, that the "light field" does just as well

     

    Fixed focal length cameras have much higher resolution due to the fact that they do not need to split the sensor. This might somehow change in the future, but at the moment that's how it works.

  • Reply 8 of 79
    Quote:

    Originally Posted by Doxxic View Post



    Kind of off topic, but it makes me wonder how many animals can see more depth than we can with single eyes, using tricks like this.

    Some insects have multiple eyes, like a fly or a bee.

  • Reply 9 of 79
    No wonder there's a switch back to normal focusing because image resolution degrades so rapidly with light field on small mobile sensors that the result is almost unusable. Can't see 150 megapixel sensors in iPhones anytime soon that would allow for a decent light field photos. Still, cool tech and I wonder when we will see Blade Runner style tech that would allow to capture a true 3D picture.
  • Reply 10 of 79
    mknoppmknopp Posts: 257member
    How long until the crop of Android also rans announces that their phones will have light field cameras? It happens pretty much every time Apple does anything innovative, or possible innovation is even hinted at. Samsung and Google rush to release their product before Apple, and most of the time the rush job is apparent in the quality, or lack there of.
  • Reply 11 of 79
    MacProMacPro Posts: 19,426member
    mknopp wrote: »
    How long until the crop of Android also rans announces that their phones will have light field cameras? It happens pretty much every time Apple does anything innovative, or possible innovation is even hinted at. Samsung and Google rush to release their product before Apple, and most of the time the rush job is apparent in the quality, or lack there of.

    You got that right ...

    ... here we are all reading some of Samsung's and Google's R&D departments' research work ... err ... I mean Apple Insider ... ;)
  • Reply 12 of 79

    IT BEGINS. IT. BEGINS.

     

    Here goes Apple, revolutionizing photography. Again. This will be huge.

  • Reply 13 of 79
    MacProMacPro Posts: 19,426member
    I am certain we will see major advancements in this area in all high end digital photography in the years to come. Not only the ability to select the point of focus but to also select the view angle to some extent. It will no doubt come in several varieties, high end plug-ins for Aperture and Photoshop and automatic versions in iPhones etc. using AI. Much the same way HDR is today. I can't wait to dive into playing with this technology, I can imagine in macro photography it would be a blast. Heck imagine iMovie being able to make a movie traversing through the depth of focus and perhaps the angles viewed (see behind things) in a single image. I can imagine wee iOS apps allowing the same thing. Creativity will take a leap forward.

    Edit: Lytro already offer all of the above I now see ... OMG the future is now :)

    AI was lit up with some great comments about all this back in 2011 (I assume).
  • Reply 14 of 79
    muppetrymuppetry Posts: 3,331member
    Miniaturizing light-field imaging for use in a small device while maintaining acceptable resolution seems like a huge challenge to me - not just in terms of technology but because basic physics is working against you. In the microlens array solution, the image size (resolution) is equal to the dimensions of the microlens array. One can certainly envisage construction of a vey fine array of such lenses, but when one remembers that each microlens requires a separate 2-D array of sensors to capture the light field, both sensor light sensitivity and the diffraction limit become serious issues.

    Sensor sensitivity can conceivably be increased but, given the small sensor size in a cell phone, even the large-aperture diffraction limit for visible light of, at best, a couple of micron, limits raw sensor array size to around 10 MP. Reserving just a 10 x 10 array for each microlens to capture the directional light field data drops the spatial resolution by a factor of 100, leaving just a 0.1 MP image.

    Right now it seems to me that sensor sensitivity improvements may instead lead to something resembling high-speed burst mode focus stacking, which leaves spatial resolution unchanged, as a far more practical solution to the same problem.
  • Reply 15 of 79
    rogifanrogifan Posts: 10,669member
    IT BEGINS. IT. BEGINS.

    Here goes Apple, revolutionizing photography. Again. This will be huge.
    Because of a patent filing??
  • Reply 16 of 79
    MacProMacPro Posts: 19,426member
    rogifan wrote: »
    Because of a patent filing??

    Oh ye of little faith ;)

    Actually what Apple will do IMHO is make an esoteric feature like this, an everyday feature people, as they always do, take for granted. How many folks using an iPhone even know what HDR is for example, yet use it everyday? That said I want this on a high end Canon :)
  • Reply 17 of 79
    [IMG]http://i1.wp.com/allthingsd.com/files/2013/06/walt-mossberg-parallax-asiad-400.gif[/IMG]

    I think the parallax effect capable with a light field camera is a natural fit for the iPhone.
    Being able to control the 3D view using the gyroscopes in iPhones and iPads would be magical. Apple loves magical.
  • Reply 18 of 79
    Also being able to create your own wallpaper with parallax for your iPhone and iPad will be a novelty.
  • Reply 19 of 79
    MacProMacPro Posts: 19,426member
    walt-mossberg-parallax-asiad-400.gif

    I think the parallax effect capable with a light field camera is a natural fit for the iPhone.
    Being able to control the 3D view using the gyroscopes in iPhones and iPads would be magical. Apple loves magical.

    Great picture of Walt ... I hope anti sea sickness pills are included ... :D
  • Reply 20 of 79
    MacProMacPro Posts: 19,426member
    Also being able to create your own wallpaper with parallax for your iPhone and iPad will be a novelty.

    Ah you cracked the obviously hidden secret! That's how they did it! iOS7 obviously already has the play back ability ...
Sign In or Register to comment.