Apple invention uses spherically curved photosensor for smaller, better iPhone camera

Posted:
in General Discussion edited January 2016
In its quest for high-performing, flexible and -- most importantly -- small imaging systems, Apple on Tuesday was granted a patent for a spherical photosensor and lens array that provides high-resolution capture in an incredibly compact package.


Source: USPTO


As published by the U.S. Patent and Trademark Office, Apple's U.S. Patent No. 9,244,253 for a "Small form factor high-resolution camera" marries cutting edge sensor technology with similarly complex optics to create a compact, high-performance digital shooter suitable for deployment in iPhone and iPad.

In some embodiments Apple describes a photosensor that is spherically curved to receive incoming light, refracted through specially crafted lenses, onto a concave surface. An apt analogy would be a bowl filled with water; the bowl represents a spherically curved image sensor and the water a fitted lens system.

To correct for diffraction and visual aberrations that propagate within miniature cameras, the proposed lens system includes three lenses, two of which are convex or substantially convex. A third meniscus lens, or a lens with opposing convex and concave surfaces, is situated between the first two lens elements and the spherical photosensor. The meniscus lens' concave surface faces the first two lens elements, and thus incoming light rays, while the convex surface interfaces with the sensor, focusing light onto the sensing surface.




This unique arrangement provides a comparatively small ray fan spot size for all field heights at the image plane, allowing for sharp, low-distortion images. Apple says that employing a curved array limits diffraction across the image field (it scores high in point spread function and modulation transfer function metrics), thereby allowing for a smaller photosensor with equally tiny pixels. Further, with an axial length of two millimeters or less, the total camera package is incredibly compact.

The setup does come with a few drawbacks, however, the most prominent being native barrel distortion, or a bubble-like warping. Apple proposes a software solution for correcting such unwanted effects, which can appear in varying degrees of severity depending on focal length, aperture and other system settings.




It is unclear if Apple intends to apply its curved photosensor patent in future iPhones, as the company has long relied on Sony's stellar backside-illuminated modules for its imaging needs. The technology could, however, delay the inevitable as Apple races to cram more components into an ever-shrinking device lineup.

Apple's curved spherical photosensor patent was first filed for in 2013 and credits Xi Chen, David S. Gere and Matthew C. Waldon as its inventors.
«1

Comments

  • Reply 1 of 28
    calicali Posts: 3,494member
    nice! And people said smaller wasn't possible. smh.
    monstrosity
  • Reply 2 of 28
    cnocbuicnocbui Posts: 3,613member

    Ah - looks like Sony might be able to make them if they can put that much stress on the wafer to bend it that much, which I doubt.


    In 2013, Sony was awarded a patent for a number of lens designs, including the rather simple 35mm f/1.8 lens paired with a curved image sensor shown above.

    edited January 2016 Unco_Sam
  • Reply 3 of 28
    foggyhillfoggyhill Posts: 4,767member
    That's a bit what ultra-compact bright zoom lens like those on the Canon G7X/G5X do.

    To keep the light gathering capacity up, they accept a lot of of predictable distortion, which can then be fixed in software (because it is predictable).
     This introduces slight softness on the periphery in the widest end of the zoom, but it's a good
    tradeoff for being able to stick those large sensors and bright zooms in a small body.

    edited January 2016 netmage
  • Reply 4 of 28
    cnocbuicnocbui Posts: 3,613member
    foggyhill said:
    That's a bit what ultra-compact bright zoom lens like those on the Canon G7X/G5X do.

    To keep the light gathering capacity up, they accept a lot of of predictable distortion, which can then be fixed in software (because it is predictable).
     This introduces slight softness on the periphery in the widest end of the zoom, but it's a good
    tradeoff for being able to stick those large sensors and bright zooms in a small body.

    What?

    I don't see the relevance.

  • Reply 5 of 28
    bluefire1bluefire1 Posts: 1,302member
    Looks like they're already working on the iPhone 9s.
    cornchip
  • Reply 6 of 28
    irelandireland Posts: 17,798member
    When are iPhones going to use a camera tech that's stacked vertically inside the phone and then points out via a mirror. That would allow for a longer lens in a thinner form factor.
  • Reply 7 of 28
    uraharaurahara Posts: 733member
    bluefire1 said:
    Looks like they're already working on the iPhone 9s.
    Actually, it's for iPhone 11s Plus.
  • Reply 8 of 28
    radarthekatradarthekat Posts: 3,842moderator
    ireland said:
    When are iPhones going to use a camera tech that's stacked vertically inside the phone and then points out via a mirror. That would allow for a longer lens in a thinner form factor.
    I recall seeing Apple patents for such a system.  So, you never know.
    nolamacguy
  • Reply 9 of 28
    boredumbboredumb Posts: 1,418member
    cnocbui said:
    foggyhill said:
    That's a bit what ultra-compact bright zoom lens like those on the Canon G7X/G5X do.

    To keep the light gathering capacity up, they accept a lot of of predictable distortion, which can then be fixed in software (because it is predictable).
     This introduces slight softness on the periphery in the widest end of the zoom, but it's a good
    tradeoff for being able to stick those large sensors and bright zooms in a small body.
    What? I don't see the relevance.

    Perhaps the "relevance" will turn out to be a smaller protrusion on the back of the phone...?
  • Reply 10 of 28
    tmaytmay Posts: 6,329member
    boredumb said:
    cnocbui said:
    What? I don't see the relevance.

    Perhaps the "relevance" will turn out to be a smaller protrusion on the back of the phone...?
    Jony was never happy about that from his comments; a pragmatic compromise.

    Now the smartphone industry can get back on track to thinner and lighter, and I mean the in the best possible way; that's what the market wants, and its just waiting for battery tech to leap forward.
    edited January 2016
  • Reply 11 of 28
    cnocbuicnocbui Posts: 3,613member
    boredumb said:
    cnocbui said:
    What? I don't see the relevance.

    Perhaps the "relevance" will turn out to be a smaller protrusion on the back of the phone...?
    The patent is for a curved sensor + lenses to suit.  The whole point of doing that is you can get away with far simpler lens configurations and not have huge distortions, which you would get with a flat sensor without a complex multi element lens using a variety if different materials.  I don't see how the canon cameras mentioned relate to this patent as they don't have a curved sensor to reduce distortions.  Fixing distortions with software is not as good an approach as not having them in the first place - which is the point of the Sony and Apple patents.
    edited January 2016 cornchip
  • Reply 12 of 28
    tmaytmay Posts: 6,329member
    cnocbui said:
    boredumb said:
    Perhaps the "relevance" will turn out to be a smaller protrusion on the back of the phone...?
    The patent is for a curved sensor + lenses to suit.  The whole point of doing that is you can get away with far simpler lens configurations and not have huge distortions, which you would get with a flat sensor without a complex multi element lens using a variety if different materials.  I don't see how the canon cameras mentioned relate to this patent as they don't have a curved sensor to reduce distortions.  Fixing distortions with software is not as good an approach as not having them in the first place - which is the point of the Sony and Apple patents.
    You seem to be missing the point of the conversation; in both cases, the design is to reduce the lens stack depth, as well as bulk, weight and complexity, and as foggyhill notes, it is an easy correction in software if inaccuracies are predictable. It's common practice for camera manufacturer's to make such a compensation for barrel distortion for their own lenses in camera, but for third party lenses, Adobe and others provide the same corrections. Computation photography is something that is rapidly evolving now that smartphones are so powerful.
    edited January 2016 boredumb
  • Reply 13 of 28
    cnocbuicnocbui Posts: 3,613member
    tmay said:
    cnocbui said:
    The patent is for a curved sensor + lenses to suit.  The whole point of doing that is you can get away with far simpler lens configurations and not have huge distortions, which you would get with a flat sensor without a complex multi element lens using a variety if different materials.  I don't see how the canon cameras mentioned relate to this patent as they don't have a curved sensor to reduce distortions.  Fixing distortions with software is not as good an approach as not having them in the first place - which is the point of the Sony and Apple patents.
    You seem to be missing the point of the conversation; in both cases, the design is to reduce the lens stack depth, as well as bulk, weight and complexity, and as foggyhill notes, it is an easy correction in software if inaccuracies are predictable. It's common practice for camera manufacturer's to make such a compensation for barrel distortion for their own lenses in camera, but for third party lenses, Adobe and others provide the same corrections. Computation photography is something that is rapidly evolving now that smartphones are so powerful.
    I don't think I am missing the conversation, I do think some people are missing the 'point' behind the patent.  Software correction for distortion is not loss-less, for want of a better word, and isn't as good an approach as eliminating distortions before the image is recorded, in the same way as software image stabilisation isn't as good as mechanical image stabilisation.  This patent is about reducing distortions before image capture, though 3 elements is probably too good to be true and further software correction would probably be required also..
  • Reply 14 of 28
    I have prior art to invalidate this patent. My eye has a spherical retina with custom optics to reduce distortion as well. This idea has been around for millennia.
    cnocbui1983irnchriz
  • Reply 15 of 28
    19831983 Posts: 1,225member
    Sony was there first. And I think already have working prototypes in their labs.
    edited January 2016 Unco_Sam
  • Reply 16 of 28
    cnocbui said:

    Ah - looks like Sony might be able to make them if they can put that much stress on the wafer to bend it that much, which I doubt.


    In 2013, Sony was awarded a patent for a number of lens designs, including the rather simple 35mm f/1.8 lens paired with a curved image sensor shown above.


    I Imagine that they will be working with Sony on this one. 
  • Reply 17 of 28
    Mr_GreyMr_Grey Posts: 118member
    The setup does come with a few drawbacks, however, the most prominent being native barrel distortion, or a bubble-like warping. 
    The whole article is fairly poorly explained, but wouldn't the key innovation here (the curved sensor), be the very thing that corrects for barrel distortion?  Like in a human eye?  Yet the article implies that at the end of the day they would still have a distorted image that would have to be corrected in software. 
  • Reply 18 of 28
    steviestevie Posts: 956member
    There's no way Apple would produce any of these things - they are more expensive and produce significant barrel distortion.

    The good thing is that a patent prevents anybody else from using it, even if they could solve the engineering problems themselves.  And I bet that they are all working on such designs, and will now have to abandon them, no matter how close they were to perfection.

    Way to go Apple!
  • Reply 19 of 28
    foggyhillfoggyhill Posts: 4,767member
    cnocbui said:
    tmay said:
    You seem to be missing the point of the conversation; in both cases, the design is to reduce the lens stack depth, as well as bulk, weight and complexity, and as foggyhill notes, it is an easy correction in software if inaccuracies are predictable. It's common practice for camera manufacturer's to make such a compensation for barrel distortion for their own lenses in camera, but for third party lenses, Adobe and others provide the same corrections. Computation photography is something that is rapidly evolving now that smartphones are so powerful.
    I don't think I am missing the conversation, I do think some people are missing the 'point' behind the patent.  Software correction for distortion is not loss-less, for want of a better word, and isn't as good an approach as eliminating distortions before the image is recorded, in the same way as software image stabilisation isn't as good as mechanical image stabilisation.  This patent is about reducing distortions before image capture, though 3 elements is probably too good to be true and further software correction would probably be required also..
    They trade off, an advantage, a bright (larger sensors) compact optical zooms due to specialized optics, for a disadvantage, much distortion that can be corrected through software. The weird optics are there to fit a big ass sensor in a somewhat thin camera...

    They could have made it much thinner if they used a smaller sensor (like in a smartphone). But, past a certain point in thinness, focusing on a flat surface becomes nearly impossible without introducing distortion that can't be corrected; that is what limits the max size a flat sensor can be for a certain thinness.

    Here they get more light, with less depth (2mm) through optics and since they got so little space to work with, the distortion needed is so severe they need to curve the sensor too to partially correct for it. with the result barrel distortion that still needs to be corrected (The curved sensor wasn't able to fix everything).

    That's the parallel; that's the point,

    These compact bright zooms with weird optics of Canon is their main advantage over Sony, which is tops in sensors.

    That's what they say in the article:
    "The setup does come with a few drawbacks, however, the most prominent being native barrel distortion, or a bubble-like warping. Apple proposes a software solution for correcting such unwanted effects, which can appear in varying degrees of severity depending on focal length, aperture and other system settings"
    edited January 2016
  • Reply 20 of 28
    foggyhillfoggyhill Posts: 4,767member
    stevie said:
    There's no way Apple would produce any of these things - they are more expensive and produce significant barrel distortion.

    The good thing is that a patent prevents anybody else from using it, even if they could solve the engineering problems themselves.  And I bet that they are all working on such designs, and will now have to abandon them, no matter how close they were to perfection.

    Way to go Apple!
      The distortion is not a big problem, most high end compacts zooms have it by design (and it is then corrected for).
    They can use just part of the image (use a slightly bigger sensor) to compensate. That's what my Canon G7X does; the distortion which is corrected, creates a slight peripheral softness when in the 24-30mm range (but, unless doing architectural shots it's not noticeable).
Sign In or Register to comment.