Apple working on iPhone video feature that automatically switches cameras during a FaceTime calls
A newly published patent application from Apple outlines teleconferencing and video capture technology that would process video from an iPhone's front and rear cameras simultaneously, and automatically determine which feed to stream based on voice or visual cues.
Source: USPTO
Apple's "Automatic video stream selection" patent application, published by the U.S. Patent and Trademark Office on Thursday, describes a system that can intelligently determine whether to use the video feed from a smartphone's front or rear facing camera. The tech can be used in a video calls, such as those made via Apple's FaceTime, or for content stored locally on a device.
As noted in the document, many handhelds have two cameras for video and picture taking. The shooters are typically facing in opposite directions, with the rear-facing module used for higher-resolution shots, while the front-facing unit handles video call duties.
With advancements in mobile processing power, some smartphones are able to capture video streams from both cameras simultaneously, though the use cases for such a feature have been limited. The iPhone currently uses only one camera at a time, offering users the option to toggle between the two as desired.
Apple notes that current bandwidth restrictions would make it difficult to stream both video feeds at the same time, thus a method is needed to intuitively select one while keeping audio in sync.
What Apple proposes is a system that captures and monitors two video streams for voice or visual cues, then outputs one of the two depending on a set of predetermined criteria. One example would be a FaceTime call in which a rear-facing camera video stream is being used before the feed automatically switches to the front-facing camera when a person begins to talk.
In order to decide which camera to use, the technology detects speech by using data from microphones facing in the same direction as each camera, or facial expressions such as lip movements.
One embodiment describes an "Auto-Select" indicator that, when selected, will activate the system and begin outputting the multiplexed video.
Various implementations are described in the paper, including "report mode," which captures a a scene from the rear-facing camera and switches to the user when they start talking. When no speech or lip movement is detected, the video switches back to the scene. Both video streams can be interleaved throughout the process and output to local storage.
In "interview mode," the phone is located between a user, who is being captured by the front-facing camera, and another person facing the rear camera. The video switches between the two when one person is determined to be speaking.
Interview mode.
Finally, along with streaming the multiplexed video in a FaceTime call, the invention allows devices without cellular access to store the content, to be uploaded whenever a Wi-Fi connection is available.
While the invention relies heavily on software, there are hardware limitations, such as audio and video processors, that may restrict Apple from releasing such a feature in the near future. However, as the final specifications of the upcoming "iPhone 5S" have yet to be revealed, it is possible that the smartphone could have the necessary components baked in. As of now, iOS 7 beta does not carry such functionality.
Apple's automatic stream selection patent application was first filed for in May and credits Jae Han Lee and E-Cheng Chang as its inventors.
Source: USPTO
Apple's "Automatic video stream selection" patent application, published by the U.S. Patent and Trademark Office on Thursday, describes a system that can intelligently determine whether to use the video feed from a smartphone's front or rear facing camera. The tech can be used in a video calls, such as those made via Apple's FaceTime, or for content stored locally on a device.
As noted in the document, many handhelds have two cameras for video and picture taking. The shooters are typically facing in opposite directions, with the rear-facing module used for higher-resolution shots, while the front-facing unit handles video call duties.
With advancements in mobile processing power, some smartphones are able to capture video streams from both cameras simultaneously, though the use cases for such a feature have been limited. The iPhone currently uses only one camera at a time, offering users the option to toggle between the two as desired.
Apple notes that current bandwidth restrictions would make it difficult to stream both video feeds at the same time, thus a method is needed to intuitively select one while keeping audio in sync.
What Apple proposes is a system that captures and monitors two video streams for voice or visual cues, then outputs one of the two depending on a set of predetermined criteria. One example would be a FaceTime call in which a rear-facing camera video stream is being used before the feed automatically switches to the front-facing camera when a person begins to talk.
In order to decide which camera to use, the technology detects speech by using data from microphones facing in the same direction as each camera, or facial expressions such as lip movements.
One embodiment describes an "Auto-Select" indicator that, when selected, will activate the system and begin outputting the multiplexed video.
Various implementations are described in the paper, including "report mode," which captures a a scene from the rear-facing camera and switches to the user when they start talking. When no speech or lip movement is detected, the video switches back to the scene. Both video streams can be interleaved throughout the process and output to local storage.
In "interview mode," the phone is located between a user, who is being captured by the front-facing camera, and another person facing the rear camera. The video switches between the two when one person is determined to be speaking.
Interview mode.
Finally, along with streaming the multiplexed video in a FaceTime call, the invention allows devices without cellular access to store the content, to be uploaded whenever a Wi-Fi connection is available.
While the invention relies heavily on software, there are hardware limitations, such as audio and video processors, that may restrict Apple from releasing such a feature in the near future. However, as the final specifications of the upcoming "iPhone 5S" have yet to be revealed, it is possible that the smartphone could have the necessary components baked in. As of now, iOS 7 beta does not carry such functionality.
Apple's automatic stream selection patent application was first filed for in May and credits Jae Han Lee and E-Cheng Chang as its inventors.
Comments
I like the Second Person (310)
Interesting patent !
To summarize what this could do, iOS could listen to your audio stream and use voice or lip movements to figure out what the speaker is talking about and automatically switch between front and back cameras.
For example : Say the front camera's active at the moment. When the user says, "Hey Jim take a look at my new computer i got", iOS could auto switch to the rear camera without ever having to manually do it. Pretty Powerful. Can you guys think of other implementations ?
1. Listen to me, I'm only gonna say this once...
2. I once killed a man with thiiiiiis thumb
3. Read my lips, no more tax increase
I would prefer perfection or close to in their existing products that are used extensively.
Quote:
Originally Posted by PhilBoogie
1. Listen to me, I'm only gonna say this once...
2. I once killed a man with thiiiiiis thumb
3. Read my lips, no more tax increase
I hear you bro... I hear you
Quote:
Originally Posted by hfts
Shouldn't Apple be fixing things instead of mucking around with these fancy ideas that most likely won't see the light of day? The virtual keyboard is a joke. ICloud back ups that don't work. Home buttons that freeze up.
I would prefer perfection or close to in their existing products that are used extensively.
The virtual keyboard is deemed awesome by the rest of the world. Don't understand why you think its a joke (cause of ur if / of issues :P ).
iCloud backup works just fine if you haven't exceed ur backup quota on the cloud.
Home button that freezes up : Where did you get ur iPhone from, a gray market source ?
A tiny quibble. . .
I believe the actual quote should be "Read my lips. No new taxes".
The keyboard does not compare with android or the new Blackberry, this is fact.
Did not exceed limit, I think I know how to count, but it refused to back up on iCloud.
Updated to iOS 6 on a iPhone 4S, and it bricked the wi-fi, Apple admitted it was their fault, and a common one.
Returned two iPhone 5s where the home button would freeze up, again many complaints about this on the Internet.
No they are not purchased in the grey market, purchased through a Telco in Australia, I don't reside in China.
Happy now?
You are a funny guy, if of, the software is crap that auto corrects when it does NOT need to.
Plus starting to get lag in typing.
I only have Apple products at home, but I am getting annoyed that these common problems are not being fixed.
This seems overly complicated when all you have to do is hit a button >_>
Quote:
Originally Posted by hfts
You are so wrong on all counts.
The keyboard does not compare with android or the new Blackberry, this is fact.
Did not exceed limit, I think I know how to count, but it refused to back up on iCloud.
Updated to iOS 6 on a iPhone 4S, and it bricked the wi-fi, Apple admitted it was their fault, and a common one.
Returned two iPhone 5s where the home button would freeze up, again many complaints about this on the Internet.
No they are not purchased in the grey market, purchased through a Telco in Australia, I don't reside in China.
Happy now?
You are a funny guy, if of, the software is crap that auto corrects when it does NOT need to.
Plus starting to get lag in typing.
I only have Apple products at home, but I am getting annoyed that these common problems are not being fixed.
No, these are not problems.
Apple's virtual keyboard is the best you'll find anywhere, and hasn't changed much since day one, 6 years ago (because it doesn't need changing).
Auto correct is unbelievably useful, and gets even better with the smallest amount of user personalization (see: Keyboard settings).
iCloud works fine for millions of users.
iOS 6 didn't brick millions of devices.
iPhone 5 home buttons are fine on millions of devices.
Your limited unfortunate experiences are no indication of anything Apple needs to address.
"Many complaints on the internet" is always the first sign of someone who thinks 100 people with issues that establish a thread of 2000 comments = a problem. Its not even a rounding error compared to the actual number of devices in the wild. The sheer number of devices means all kinds of things are possible for a few units. A giraffe could have sneezed on both your defective iPhones. With these kinds of numbers, anything is possible.
If you only have Apple products how do you know the android keyboard is much better? The keyboard on my iPhone and iPad are far more accurate and easier to use than the keyboard on my Samsung Tab 2.
I also like the implied potential of recording video from both cameras and microphones simultaneously. For example, for a user-narrated segment, or an interview that you could later edit down, switching between the two camera POV… it would sure make it easy to sync things for continuity.
Bad bad bad girl/boy, that will teach you, never complain and you won't get banned.
Now run along like a good sheepie.