Apple exploring multi-camera systems for 3D picture taking

Posted:
in Future Apple Hardware edited January 2014
Apple has toyed with the idea of adding two or more rear-facing cameras and sensors to a device like an iPhone, allowing pictures to be taken and displayed in 3D.



The U.S. Patent and Trademark Office this week revealed a patent application from Apple related to 3D picture taking, entitled "Systems and Methods for an Imaging System Using Multiple Image Sensors." First discovered by AppleInsider, the filing describes a system that would capture, process and render 3D images with the accompanying dual-camera hardware.



The application refers to Apple's approach as a "paradigm shift from the known software-based approaches." It notes that current software must "guess" at how to perform stereo disparity compensation for 3D images.



Those guesses, Apple said, can result in images with artifacts in a composed 3D image that would result in a poor quality photo. Apple's method would instead employ a hardware-based approach with a "deterministic calculation for stereo disparity compensation."



Apple would create hardware with multiple imaging systems, employing separate luma, chroma and depth/distance sensors to capture images that can be turned into a single, three-dimensional picture.



In addition to still images, Apple's system would also be able to record video in 3D. Setup and calibration of such a system would be a simple one-time event, the application claims.







3D picture taking capabilities for the average consumer just arrived in the U.S. this week, with the launch of the new Nintendo 3DS handheld gaming console. Nintendo's portable device features two rear cameras for capturing still pictures and playing augmented reality games in 3D.



On Nintendo's gaming system, the 3D content is shown on a glasses-free 3D display. Of course, if Apple were to offer 3D picture taking capabilities on a future device, it too would need to feature 3D display capabilities.







The patent application revealed this week is not the first time Apple has shown interest in 3D imaging. In 2009, the company explored 3D display capabilities with head tracking technology, while in 2008 it was revealed to be researching auto-stereoscopic 3D display hardware. The Cupertino, Calif., company has also shown interest in a 3D interface for Mac OS X.



The patent application revealed this week by the USPTO is credited to Brett Bilbrey and Guy Cote. It was first filed in September of 2009.
«1

Comments

  • Reply 1 of 25
    t0mat0t0mat0 Posts: 58member
    Would be interesting for a MacBook or iMac. I think it could also link to their previous patents on being able to move the desktop display through tracking potentially? Also depth perception, possible uses could link up to it. We haven't seen much from Apple in regard Kinect like body or hand gestures (or being able to identify hand position and gestures. Guess we'll see if they want to go in that direction.

    Sounds like they wanted to patent an idea to have a foot in the door, than push through an iPod Touch with a 3D camera?

    How would this fit today, or with Lion?
  • Reply 2 of 25
    ronboronbo Posts: 669member
    Quote:
    Originally Posted by AppleInsider View Post


    Of course, if Apple were to offer 3D picture taking capabilities on a future device, it too would need to feature 3D display capabilities.



    That's not true. Twin cameras make AR apps much more powerful, but you could happily interact with those apps using a regular 2D display. That's not to say people might not clamor for a 3D display, just because they wanted one. But twin cameras have utility without such a display. (Edit: I was a little too fast to reply, and I think it looks like I'm commenting about something just to the side of the point you're making. What I had in mind was a video I saw of a guy who used a 2-camera device to turn a flat real-life game board into an AR zombies game, where zombies are walking down the street, and the device screen acts as a camera on the scene. He moves around gets a different view of the streets below. It was very cool. The device was essentially taking a 3D picture over and over and used that to generate scene information for the AR)
  • Reply 3 of 25
    srathisrathi Posts: 29member
    Let's see if all the iFanboys will suddenly change their mind and start liking 3D on their phones. The same iFans who blasted this feature on Android phones.
  • Reply 4 of 25
    I'm interested, would be real cool if the Live Chat sites use it (until you click on a good-look'n thumbnail and the live 3D baaadddddd). Meaning maybe 3D isn't isn't the best for everything.
  • Reply 5 of 25
    stelligentstelligent Posts: 2,680member
    Photography in general has not been an area that Apple has shown consistent passion for.



    The only iDevice with a good camera is iP4. Then they followed up with disappointing sensors in the iPod Touch and iPad 2. Whether the rationale was costs or design (to keep the devices thin), it was still disappointing.



    Hopefully this signals a growing interest to conquer the hearts and imagination of more photo-buffs.
  • Reply 6 of 25
    Quote:
    Originally Posted by srathi View Post


    Let's see if all the iFanboys will suddenly change their mind and start liking 3D on their phones. The same iFans who blasted this feature on Android phones.



    Dude, I haven't read one article about a 3D video taking / stereoscopic display android device. I have read about android phones with silly interfaces wasting mass resources to present a OpenGL type rendered look. Notionally a 3G navigation system could be good, still waiting to see it done successfully yet.
  • Reply 7 of 25
    pmzpmz Posts: 3,433member
    Even if Apple could do it, and make it not suck, I still think it would remain the simple Fad that it is.



    From theaters, to TVs, to playstation, to Nintendo handhelds...I've yet to see a single implementation of 3D that does not absolutely suck.
  • Reply 8 of 25
    srathisrathi Posts: 29member
    Quote:
    Originally Posted by cy_starkman View Post


    Dude, I haven't read one article about a 3D video taking / stereoscopic display android device. I have read about android phones with silly interfaces wasting mass resources to present a OpenGL type rendered look. Notionally a 3G navigation system could be good, still waiting to see it done successfully yet.



    And why is OpenGL rendering bad?
  • Reply 9 of 25
    drdoppiodrdoppio Posts: 1,132member
    3D is the [very near] future. It would be nice to see Apple developing something in this area as well. As to the current patent application, it doesn't seem particularly original; it may not pass due to existing prior art.
  • Reply 10 of 25
    sciwizsciwiz Posts: 77member
    Quote:
    Originally Posted by cy_starkman View Post


    Dude, I haven't read one article about a 3D video taking / stereoscopic display android device. I have read about android phones with silly interfaces wasting mass resources to present a OpenGL type rendered look. Notionally a 3G navigation system could be good, still waiting to see it done successfully yet.



    HTC EVO 3D







    4.3" (960x540) glasses-free 3D display

    1.2GHz dual-core Qualcomm processor,

    Dual 5 megapixel cameras

    Front-facing 1.3 megapixel camera

    Support for 1080p video playback via the HDMI out (or 720p for 3D content)
  • Reply 11 of 25
    srathisrathi Posts: 29member
    Quote:
    Originally Posted by cy_starkman View Post


    Dude, I haven't read one article about a 3D video taking / stereoscopic display android device. I have read about android phones with silly interfaces wasting mass resources to present a OpenGL type rendered look. Notionally a 3G navigation system could be good, still waiting to see it done successfully yet.



    As usual, iFans are in denial. Unless Apple comes up with a feature, it is totally crap and worthless!
  • Reply 12 of 25
    SpamSandwichSpamSandwich Posts: 33,407member
    Current and near future 3D remains a novelty and it is also not recommended for kids to be exposed to for extended periods. Just like in the 50s, then again in the 70s, it'll fade away or remain a niche.
  • Reply 13 of 25
    Quote:
    Originally Posted by Ronbo View Post


    What I had in mind was a video I saw of a guy who used a 2-camera device to turn a flat real-life game board into an AR zombies game, where zombies are walking down the street, and the device screen acts as a camera on the scene. He moves around gets a different view of the streets below. It was very cool.



    I just picked up a Nintendo 3DS and it includes an AR game that works the same way. It uses a card to have a known position in the world and you walk around it to get different views. You can play the game with the 3D display on or off.

    http://www.youtube.com/watch?v=uOa3DHgoHs0



    I've also been having fun testing out the 3D camera

    http://www.atariage.com/forums/topic/179570-3ds-photos/
  • Reply 14 of 25
    flaneurflaneur Posts: 4,526member
    Quote:
    Originally Posted by DrDoppio View Post


    3D is the [very near] future. It would be nice to see Apple developing something in this area as well. As to the current patent application, it doesn't seem particularly original; it may not pass due to existing prior art.



    Maybe you know someone else who has taken this approach? If so, could you provide a link?



    It looks like this describes a method of using specific aspects of the sensor data (chroma, luminance) to calculate the disparity between left and right views with NEW levels of precision. Like using metadata at chip level to refine the parallax geometry.



    I think this is new, and if anybody has knowlege of 'prior art' can we hear about it?



    One imaginable application: calculate in real time and control by software the spacing between the two camera lenses (or sensor fields perhaps) according to whether foreground or background subjects are of interest, determined by autofocus routines in the viewfinder.



    I'm just guessing here. I'll be looking for this patent document, should have a link soon.
  • Reply 15 of 25
    quinneyquinney Posts: 2,528member
    Quote:
    Originally Posted by SpamSandwich View Post


    Current and near future 3D remains a novelty and it is also not recommended for kids to be exposed to for extended periods. Just like in the 50s, then again in the 70s, it'll fade away or remain a niche.



    Last time there was a rumor about 3D, someone posted a link to an article which explained that the way current electronic devices display 3D is different from the way the human eye is built to experience 3D. That basic incompatibility is responsible for the eye strain and vertigo many people get from viewing current 3D. Until there are some kind of holographic displays, I agree that 3D will remain a novelty. I do acknowledge that the presence of 3D cameras on an iPhone would be one way of differentiating between multiple models of iPhone, should Apple ever decide to do that.
  • Reply 16 of 25
    I’ve been taking digital 3D photos for years. I bolted identical digital cameras together at the tripod mount making the cameras look like a pair of strange binoculars. I try and snap the pictures at the same time, then combine the left and right images side by side in Photoshop making sure the horizontal details line up on the same line. I then print the combined image and look at it with a the aid of a stereoscope or a lorgnette. This form of 3D image making goes back to the 1860’s and is still exciting just for the sheer sense of seeing in 3D. It has to be seen to be really appreciated, even the most ordinary of shots has a fascination because of the perceived depth. Last summer a Yankee game was broadcast with a similar kind of twin lens visual experience. It would really be something to have an iPhone, iPod, or iPad that could shoot such images.
  • Reply 17 of 25
    flaneurflaneur Posts: 4,526member
    Quote:
    Originally Posted by quinney View Post


    Last time there was a rumor about 3D, someone posted a link to an article which explained that the way current electronic devices display 3D is different from the way the human eye is built to experience 3D. That basic incompatibility is responsible for the eye strain and vertigo many people get from viewing current 3D. Until there are some kind of holographic displays, I agree that 3D will remain a novelty. I do acknowledge that the presence of 3D cameras on an iPhone would be one way of differentiating between multiple models of iPhone, should Apple ever decide to do that.



    That would have been the Walter Murch letter to Roger Ebert. There's a rejoinder to Murch's 'misguided' conclusions here:



    http://www.crunchgear.com/2011/01/24...ignment-of-3d/



    Murch's point about the loss of brightness in current projection systems and TV viewing systems is valid, I think. The movie has to be worth it to transcend that loss. So far some are, many aren't.



    Edit: Headaches and eyestrain are caused by bad 3D, not 3D itself, OR, by one's own binocular misalignment, not detected in ordinary eye exams. (See Sue Barry's website on 'Fixing My Gaze' if you get headaches when others don't.) There's also a learning curve for those with marginal stereo vision, and a fatigue curve, depending on the viewing system. I felt tired around the eyes after Avatar, and I've been messing with stereo for many years. It was maybe too long, a dreadfully dark and noisome movie. Ambitious, though, and quite an accomplishment.
  • Reply 18 of 25
    Just give us ONE camera and Hipstamatic as standard camera, now THAT would make me really happy!
  • Reply 19 of 25
    Stereo cameras on Apple devices must be for something else besides making images. They must be for measuring distance for a game or some other motion intensive use.



    To create 3D photographs one needs special processing. Already the photographic film industry is almost dead. Three dimensional photography has been in existence with quality cameras for decades and it hasn't caught on.



    I don't like 3D photography. It is a novelty that is OK when looking at one image. It becomes annoying with many images. Movies in 3D just take away from the story. Good stories aren't made better with 3D. I watched Avatar on my computer without the 3D effects. I really liked the movie. Having 3D effects always takes attention off the story and puts it on the effects, thus lessening the impact of the story.



    The thing that makes 3D bad is that the parallax of a camera isn't the same as the one your eyes produce. All 3D therefore seems unnatural to each viewer.



    When holograms that resemble actual things can be produced, our own eyes will then observe the 3D without strain. Of course people on the left side of a theater will not see the movie the people on the right side of the screen see. It would be more like viewing a play than a movie.



    I just hope that 3D goes away and stays away.
  • Reply 20 of 25
    quinneyquinney Posts: 2,528member
    Quote:
    Originally Posted by Flaneur View Post


    That would have been the Walter Murch letter to Roger Ebert. There's a rejoinder to Murch's 'misguided' conclusions here:



    http://www.crunchgear.com/2011/01/24...ignment-of-3d/



    Murch's point about the loss of brightness in current projection systems and TV viewing systems is valid, I think. The movie has to be worth it to transcend that loss. So far some are, many aren't.



    Edit: Headaches and eyestrain are caused by bad 3D, not 3D itself, OR, by one's own binocular misalignment, not detected in ordinary eye exams. (See Sue Barry's website on 'Fixing My Gaze' if you get headaches when others don't.) There's also a learning curve for those with marginal stereo vision, and a fatigue curve, depending on the viewing system. I felt tired around the eyes after Avatar, and I've been messing with stereo for many years. It was maybe too long, a dreadfully dark and noisome movie. Ambitious, though, and quite an accomplishment.



    From the article (emphasis added):

    "Mr. Murch is perfectly right to point out the convergence/focus problem as a limitation of the medium, but for him and Ebert (whom I have rebutted before on the topic) to consider it fatal is, in my opinion, a lack of imagination and faith in the ingenuity of filmmakers."



    I don't interpret that as saying that Murch is misguided, rather, it seems to agree he has a point.

    No doubt filmmakers can film things differently to make the effect less painful on people, but the incompatibility with human physiology remains. If a person can view their real world surroundings without pain and only when viewing 3D electronically do the symptoms of binocular misalignment or marginal stereo vision occur, I place the blame on the medium, not the eyes.
Sign In or Register to comment.