Apple patents 'super resolution' multi-sensor cameras, display-integrated light sensor tech

Posted:
in General Discussion
Apple continues to win patents detailing advanced technology rumored to show up in next year's iPhone model, the most recent being a trio of inventions dealing with "super resolution" multi-sensor cameras and light sensors embedded into functional phone displays.









As awarded by the U.S. Patent and Trademark Office on Tuesday, Apple's U.S. Patent Nos. 9,467,666 and 9,466,653 detail a multi-sensor camera assembly capable of achieving maximum image quality in a minimal amount of space. The system splits incoming light into at least three (red, blue and green) wavelengths using a series of prisms, directs and captures the rays with independent light sensors and combines the resulting data into a "super resolution" image via specialized software.



In one embodiment described in the '666 patent covering "Miniature camera super resolution for plural image sensor arrangements," Apple notes the use of color splitters allows for enhanced image resolution compared to single-sensor solutions as a lower portion of light is absorbed by filters.



Conventional single-sensor cameras commonly apply a Bayer pattern filter to derive color (typically through a demosaicing or software interpolation process) from CCD or CMOS sensor data. By contrast, three-sensor cameras gather and apply nearly all incoming light -- in some cases three times the amount of a single-sensor array -- passing through the beam splitter to a final output image.



Apple's IP refers to the beam splitter as either a Philips prism or smaller stacked cube variation, the latter described in detail in the '653 patent for a "Digital camera with light splitter," itself an extension of a patent granted in 2015. In a cube configuration, sensors collecting red, green and blue light information are embedded into ceramic substrates and positioned on three sides of the prism. The light splitter is designed to direct specific wavelengths to each sensor with minimal waste.



In addition to enhanced light gathering capabilities, three-sensor arrays offer better performance in situations calling for polarization imaging. Typically, polarizing filters shed 50 percent of incident light to increase visibility of specific targets, but the same results can be achieved with the splitting cube as it enables polarization sum and difference imaging. Infrared imaging is another area that would benefit from multiple wavelength-dedicated sensors.



As for deployment in smartphones, Apple illustrates a "folded" camera design in both of today's patents. To minimize system depth, a mirror mechanism is used to bounce incoming light through lens elements positioned orthogonal to the phone's Z-axis. For example, the objective lens can be installed as it appears on current iPhones, while the imaging sensor is positioned at some other location in the chassis at a right angle to said lens.



This design holds advantages far beyond space savings. The mirror, for example, can be mounted on a miniature motor to enable optical image stabilization, while the long connecting tube offers additional room for zoom capabilities.









Whether Apple plans to implement the inventions in a shipping device remains unclear, though the company already owns a number of patents covering three-sensor, three-lens systems, and cube splitter designs and other imaging technology.



The '666 patent goes on to explain super resolution algorithms, which in part use natural misalignments in the relative positions of the arrays to sample a given scene at higher spatial resolution. Super samples are also gleaned from edges detected in sub-sampled sites, as well as color information and artifacts.



Apple's super resolution camera patent was first filed for in July 2015 and credits Richard J. Topliss and Richard H. Tsai as its inventors. The cube splitter patent update was filed for in March 2015 and credits Steven Webster and Ning Y. Chan as its inventors.









Apple's third patent deals with embedded sensors, specifically light-sensing components like ambient light and proximity sensors.



As noted in Apple's U.S. Patent No. 9,466,653 for "Electronic devices with display-integrated light sensors," light-sensing apparatuses are commonly displaced from the device display. While advantageous from a production standpoint, such implementations lead to wasted space, or in some cases force sleek designs to be modified. Indeed, the iPhone's proximity and ALS are positioned above the display near the handset speaker.



Apple proposes forming sensors on display layers that already support conductive traces. Some embodiments provide for sensor positioning at the periphery of a device display beyond the edge of touch sensitive traces.



For example, an ALS unit might be disposed at the extreme edge of an OLED display, then covered by a touch-sensitive layer and finally an encapsulation layer. Alternatively, a handset could include a dedicated TFT layer onto which a variety of sensors are embedded.



In any case, the sensor or sensors are disposed within the display itself, not above it as with current iPhone models. This design tweak alone would save precious millimeters off final design specifications and could pave the way to a true full-screen display.



Apple is said to be working on an advanced iPhone design with "full-screen face," meaning the rumored OLED display stretches across the device's entirety. The company last week patented technology detailing a fingerprint sensor that works through portable device displays, though the ear speaker remains a problem.



Apple's embedded light sensor patent was first filed for in June 2015 and credits Erik G. de Jong, Anna-Katrina Shedletsky and Prashanth S. Holenarsipur as its inventors.
«1

Comments

  • Reply 1 of 25
    badmonkbadmonk Posts: 1,285member
    Can't innovate anymore, my a$$-

    Phil Schiller


    watto_cobra
  • Reply 2 of 25
    SoliSoli Posts: 10,035member
    We've been talking about this concept in phones for years to allow for better optics in a thinner casing.
    SpamSandwich
  • Reply 3 of 25
    Soli said:
    We've been talking about this concept in phones for years to allow for better optics in a thinner casing.
    Except that concepts are easy to come up with based on needs. 
    HOW to make that concept to work is another matter.

    Everyone knows about teleportation from sci-fi. 
    Have you seen any actual device that can teleport more than just a single photon?
    edited October 2016 mike1nolamacguywatto_cobraloopychew
  • Reply 4 of 25
    SoliSoli Posts: 10,035member
    Soli said:
    We've been talking about this concept in phones for years to allow for better optics in a thinner casing.
    Except that concepts are easy to come up with based on needs. 
    HOW to make that concept to work is another matter.

    Everyone knows about teleportation from sci-fi. 
    Have you seen any actual devices that teleports more than just a single photon? 
    Is your post serious? You thought my comment about discussing how Apple may invent a viable process to further innovate their products was to be equated to inventing the technology? What part of the word "concept" is confusing?
    edited October 2016 gatorguyirelandSpamSandwichsuddenly newton
  • Reply 5 of 25
    irelandireland Posts: 17,798member
    The reflective periscope design is a no-brainier to get rid of the camera bump.
    berndogwatto_cobra
  • Reply 6 of 25
    Soli said:
    Soli said:
    We've been talking about this concept in phones for years to allow for better optics in a thinner casing.
    Except that concepts are easy to come up with based on needs. 
    HOW to make that concept to work is another matter.

    Everyone knows about teleportation from sci-fi. 
    Have you seen any actual devices that teleports more than just a single photon? 
    Is your post serious? You hought my comment about discussing how Apple may invent a viable process to further innovate their products was to be equated to inventing the technology? What part of the word "concept" is confusing?
    Agreed.
    It was a serious...brain fart.  :#

    SolifotoformatSpamSandwichloopychewcali
  • Reply 7 of 25
    chabigchabig Posts: 641member
    As usual, Appleinsider writes that Apple "wins" patents. As has been stated numerous times, patents are "granted" to inventors. They aren't won. It's not a competition.
    mike1SolinolamacguySpamSandwichjony0watto_cobracali
  • Reply 8 of 25
    SoliSoli Posts: 10,035member
    chabig said:
    It's not a competition.
    With the awful state of affairs the US patent system is in, that might be a good idea.
    cali
  • Reply 9 of 25
    fracfrac Posts: 480member
    The '666' patent? 
    If this transpires, it will surely be called the devil's work or some other dumb meme.
    watto_cobracali
  • Reply 10 of 25
    Can somebody explain me where the innovation is ? Three sensor (video) camera were commons in the end of the 80s beginning of the 90s. Probably the innovation is elsewhere, around resolution.
  • Reply 11 of 25
    am8449am8449 Posts: 392member
    This concept, although interesting, seems rather delicate. I can imagine the system not working properly due to any number of alignment issues with the mirror, lenses, or prism. Maybe that's why Apple decided on a two camera system for the iPhone 7 Plus.
  • Reply 12 of 25
    SoliSoli Posts: 10,035member
    am8449 said:
    This concept, although interesting, seems rather delicate. I can imagine the system not working properly due to any number of alignment issues with the mirror, lenses, or prism. Maybe that's why Apple decided on a two camera system for the iPhone 7 Plus.
    I see your point, but couldn't that also be said for the advanced camera modules they've been using for years?


    fastasleepwatto_cobracali
  • Reply 13 of 25
    jasenj1jasenj1 Posts: 923member
    Maurizio said:
    Can somebody explain me where the innovation is ? Three sensor (video) camera were commons in the end of the 80s beginning of the 90s. Probably the innovation is elsewhere, around resolution.
    The innovation could also be around the use of the prism, arrangement of the sensors, the periscope mirror. Three sensor video cameras do not have the space constraints of a cell phone so I expect the sensor arrangement is quite different. 

    See this earlier story on the patent.

    See also Wikipedia entry on 3 CCD cameras.
    edited October 2016
  • Reply 14 of 25
    ireland said:
    The reflective periscope design is a no-brainier to get rid of the camera bump.
    The real no brained is that the whole thing will fit into the hole left by removing the headphone jack!
    watto_cobra
  • Reply 15 of 25
    So tell me... why can't I use, say, a three-sided sensor with a red filter ed sensor and a blue filtered sensor and another sensor with no filter only, just b&w? The b&w sensor provides high low-light sensitivity due to the lack of any filter at all, and you should be able to extrapolate green by combining the outputs of the three sensors. (i.e. lot's of light here but little red and blue... hmmmm.... what color could it be???)
  • Reply 16 of 25
    SoliSoli Posts: 10,035member
    hmlongco said:
    So tell me... why can't I use, say, a three-sided sensor with a red filter ed sensor and a blue filtered sensor and another sensor with no filter only, just b&w? The b&w sensor provides high low-light sensitivity due to the lack of any filter at all, and you should be able to extrapolate green by combining the outputs of the three sensors. (i.e. lot's of light here but little red and blue... hmmmm.... what color could it be???)
    In terms of the color reproduction in photos, which seems to be important to many consumers as well as Apple, would this be good enough accuracy for photophiles?
  • Reply 17 of 25
    AppleBumAppleBum Posts: 37unconfirmed, member
    I believe three-sensor cameras still use filters. The innovation is in the prism, that refracts the light toward the sensors rather than just filtering out the unwanted bandwidths.
    nolamacguy
  • Reply 18 of 25
    chabig said:
    As usual, Appleinsider writes that Apple "wins" patents. As has been stated numerous times, patents are "granted" to inventors. They aren't won. It's not a competition.
    Agreed. That choice of words is annoying and unnecessary.
    edited October 2016 nolamacguy
  • Reply 19 of 25
    hmlongco said:
    So tell me... why can't I use, say, a three-sided sensor with a red filter ed sensor and a blue filtered sensor and another sensor with no filter only, just b&w? The b&w sensor provides high low-light sensitivity due to the lack of any filter at all, and you should be able to extrapolate green by combining the outputs of the three sensors. (i.e. lot's of light here but little red and blue... hmmmm.... what color could it be???)

    Maybe I'm not understanding your question, but cameras already use green for luminance. That's why the color space is still commonly 4:2:0. That's changing quickly of course and this new configuration would better support wider color gamuts such as 4:4:4. 
  • Reply 20 of 25
    jasenj1jasenj1 Posts: 923member
    AppleBum said:
    I believe three-sensor cameras still use filters. The innovation is in the prism, that refracts the light toward the sensors rather than just filtering out the unwanted bandwidths.
    No. Three sensor cameras already use prisms - see the Wikipedia entry I posted earlier.
Sign In or Register to comment.