Apple exploring accelerometer, gyro stabilization for iPhone video recording

Posted:
in iPhone edited January 2014
Using the built-in accelerometer and gyroscope, future iPhones could compensate for shaky video recordings with new software-based stabilization.



Apple's interest in allowing users to record better video on their iPhone was revealed this week in a newly published patent application discovered by AppleInsider. Entitled "Accelerometer/Gyro-Facilitated Video Stabilization," it describes how an iPhone might use motion sensing data to compensate for any jittering in a recorded video.



The filing notes that software-based video stabilization already exists and can improve the perceptual quality of a video sequence, but not without consequences. For example, current stabilization techniques can use up "considerable resources," which can be particularly detrimental on a portable, battery-powered device like an iPhone.



In addition, while advanced algorithms can help offset any shakiness in a video, sometimes they can generate incorrect estimates that don't actually improve video quality at all.



But now, consumer devices like the iPhone include gyroscopes and accelerometers, providing motion data for software on the device. While this data can be helpful, even it isn't a perfect solution for video stabilization due to "noise" in the data, Apple said.



"Motion detection devices can provide metadata that indicates motion effects of a camera during video capture, however, even though the motion detectors provide data relating to global motion of the camera, the level of shakiness between frames is often comparable to the noise level of the motion detector data," the filing reads. "Such high level of the noise in data prohibits (direct use) of accelerometer data in video stabilization."







Apple's solution would be to selectively control the motion stabilization feature, only adjusting and improving the video captured when the system determines it is necessary. This determination would be made by comparing motion sensor data to a pre-set threshold.



"Based on the determination, motion stabilization may be suspended on select portions of a captured video sequence," the application states.







The described system could go frame by frame in a captured video, comparing the file with synced motion detection data. By doing this, an iPhone could determine if and when image normalization processes are required, only applying the effect when necessary.



The proposed invention, made public this week, was first filed in April of 2010. It is credited to Yuxin Liu, Xiaojin Shi, James Oliver Normile, and Hsi-jung Wu.
«1

Comments

  • Reply 1 of 27
    wardcwardc Posts: 150member
    Sounds good, this is needed.



    As of now, the recording quality is good but the lack of stabilization makes the videos very jittery and non-professional-looking, unless you use something to hold the phone stationary (like a tripod), so stabilization sounds like something that should be added, and could be done easily through software... (I am sure there are 3rd party apps for this already) -- but it should be built into the iOS by now.
  • Reply 2 of 27
    reganregan Posts: 474member
    Yup. Sounds good. Every little bit helps.



    I am surprised a third party hasnt come out wirh a Mini "steadi-cam" like device for the iphone and ipod touch yet. Call it an isteadi or a steadi-pod. I dunno.



    But that would be sweet, if it was small enough. :-)
  • Reply 3 of 27
    Quote:
    Originally Posted by regan View Post


    I am surprised a third party hasnt come out wirh a Mini "steadi-cam" like device for the iphone and ipod touch yet. Call it an isteadi or a steadi-pod. I dunno.



    The Internet has these things called "search engines."



    If you type in "iPhone steadicam" into one of these fancy search engines, you'll end up on a page showing a genuine Steadicam for the iPhone 4 (and priced at what one might expect from Steadicam).



    Same day, you should try these search engines. They can be quite helpful.



    A few suggestions are: Google, Bing, and Yahoo!
  • Reply 4 of 27
    Quote:
    Originally Posted by regan View Post


    Yup. Sounds good. Every little bit helps.



    I am surprised a third party hasnt come out wirh a Mini "steadi-cam" like device for the iphone and ipod touch yet. Call it an isteadi or a steadi-pod. I dunno.



    But that would be sweet, if it was small enough. :-)



    Somebody already did. Edit:Oops, someone beat me to it.

    http://www.steadicam.com/smoothee_home.html



    It would be seriously cool if Apple were able to put camera stabilization in iOS...
  • Reply 5 of 27
    What they need to patent and implement is the "Hey you! Your finger is covering the microphone and muffling all audio!" alert for the video you're recording.
  • Reply 6 of 27
    [insult removed]
  • Reply 7 of 27
    What are you talking about?



    I was responding to a person who clearly is not aware of a valuable Internet tool: the search engine.



    Not only did I provide a link to the device described, I also mentioned a couple of these search engine tools to get the person started, as well as the actual query string that I used to find the desired device. So in a way, I gave this person a cake, as well as the ingredients, recipe and oven for them to bake the next time they wanted a cake.



    You, on the other hand, provided zero help in your comment. I suggest you provide responses with more long term pedagogical value.
  • Reply 8 of 27
    Quote:
    Originally Posted by cvaldes1831 View Post


    What are you talking about?



    I was responding to a person who clearly is not aware of a valuable Internet tool: the search engine.



    Not only did I provide a link to the device described, I also mentioned a couple of these search engine tools to get the person started, as well as the actual query string that I used to find the desired device. So in a way, I gave this person a cake, as well as the ingredients, recipe and oven for them to bake the next time they wanted a cake.



    You, on the other hand, provided zero help in your comment. I suggest you provide responses with more long term pedagogical value.



    Look, if you are going to be condescending with your original response, stick with it. Nothing wrong with being arrogant and condescending unless you try to whitewash it after. It only makes you look stupid.
  • Reply 9 of 27
    sennensennen Posts: 1,465member
    Quote:
    Originally Posted by cvaldes1831 View Post


    The Internet has these things called "search engines."



    If you type in "iPhone steadicam" into one of these fancy search engines, you'll end up on a page showing a genuine Steadicam for the iPhone 4 (and priced at what one might expect from Steadicam).



    Same day, you should try these search engines. They can be quite helpful.



    A few suggestions are: Google, Bing, and Yahoo!



    Quote:
    Originally Posted by cvaldes1831 View Post


    What are you talking about?



    I was responding to a person who clearly is not aware of a valuable Internet tool: the search engine.



    Not only did I provide a link to the device described, I also mentioned a couple of these search engine tools to get the person started, as well as the actual query string that I used to find the desired device. So in a way, I gave this person a cake, as well as the ingredients, recipe and oven for them to bake the next time they wanted a cake.



    You, on the other hand, provided zero help in your comment. I suggest you provide responses with more long term pedagogical value.



    Ignore the haterz, good posts
  • Reply 10 of 27
    As for steady-cam software, I seem to recall a product demo of iMovie 8 (i think) that had built in image stabilization. So Apple could easily introduce this functionality.
  • Reply 11 of 27
    Skype it already on the ball with this
  • Reply 12 of 27
    Quote:
    Originally Posted by sennen View Post


    Ignore the haterz, good posts



    Yeah, as good a bludgeon as the steady cam he linked to on the amazon site.
  • Reply 13 of 27
    melgrossmelgross Posts: 31,491member
    I was wondering if this could be done, but I've always thought that the camera movement was too small to be picked up by the sensors. I guess it's just barely possible. Something is better than nothing.



    I'm just nit too sure that turning it off and on during a recording is a great idea. Just like audio noise reduction, when it's on and off, we get a spurt of noise which is more distracting than the noise itself too often.
  • Reply 14 of 27
    gatorguygatorguy Posts: 20,301member
    Quote:
    Originally Posted by DJinTX View Post


    As for steady-cam software, I seem to recall a product demo of iMovie 8 (i think) that had built in image stabilization. So Apple could easily introduce this functionality.



    I'm nearly certain that someone announced that they either do or soon will offer video stabilization with their chat application. Can't find the mention now and I can't remember if it was also for video recording. I can certainly see where it would benefit Facetime or other video chat apps though.



    EDIT: I found what I believe I may have read a few weeks back. SRI Tech does have image stabilization for live video, but it doesn't look like it's applicable to video recording, which is Apple's intended usage.



    http://www.sri.com/news/releases/07072011.html
  • Reply 15 of 27
    [insult removed]
  • Reply 16 of 27
    jd_in_sbjd_in_sb Posts: 1,483member
    Quote:
    Originally Posted by cvaldes1831 View Post


    The Internet has these things called "search engines."



    If you type in "iPhone steadicam" into one of these fancy search engines, you'll end up on a page showing a genuine Steadicam for the iPhone 4 (and priced at what one might expect from Steadicam).



    Same day, you should try these search engines. They can be quite helpful.



    A few suggestions are: Google, Bing, and Yahoo!



    Everybody knows about Google, including you. And everybody knows that everybody knows about Google, apparently except you.
  • Reply 17 of 27
    addaboxaddabox Posts: 12,660member
    Quote:
    Originally Posted by DJinTX View Post


    As for steady-cam software, I seem to recall a product demo of iMovie 8 (i think) that had built in image stabilization. So Apple could easily introduce this functionality.



    The problem with software image stabilization (and this is true of iMove as well) is that it degrades the image. In order to align frames you have to discard some number of pixels around the edges-- the more aggressive the stabilization, the more pixels. That's because if I shift a frame to the left to make it match the previous frame, there's nothing beyond the original right edge to show, and I just get a weirdly cropped frame. So I have to in effect zoom in on my image somewhat so I have a buffer area within which to shift things around. Electronically zoomed image = lower resolution.



    Better is optical image stabilization, which uses a software controlled moving mirror to align the image frame to frame. I doubt there's room in an iPhone for this approach, so I'm curious if Apple has some way to do this that doesn't negatively effect image quality.
  • Reply 18 of 27
    MacProMacPro Posts: 18,157member
    Quote:
    Originally Posted by jd_in_sb View Post


    Everybody knows about Google, including you. And everybody knows that everybody knows about Google, apparently except you.



    So are you saying, not everybody knows that not everybody know about Google?
  • Reply 19 of 27
    Quote:
    Originally Posted by addabox View Post


    The problem with software image stabilization (and this is true of iMove as well) is that it degrades the image. In order to align frames you have to discard some number of pixels around the edges-- the more aggressive the stabilization, the more pixels. That's because if I shift a frame to the left to make it match the previous frame, there's nothing beyond the original right edge to show, and I just get a weirdly cropped frame. So I have to in effect zoom in on my image somewhat so I have a buffer area within which to shift things around. Electronically zoomed image = lower resolution.



    Better is optical image stabilization, which uses a software controlled moving mirror to align the image frame to frame. I doubt there's room in an iPhone for this approach, so I'm curious if Apple has some way to do this that doesn't negatively effect image quality.



    But, isn't OIS doing essentially the same thing -- cropping the image to attain more stable results?



    I don't have enough enough experience to know whether OIS is better than after-the-fact "image analysis and stabilization".



    FWIW, FCPX (Final Cut Pro X) has a pretty good image stabilization capability. Video can be analyzed in the background as it is ingested (or later, after ingest). Then, you can turn stabilization on or off for individual clips. If you want, you can identify portions of a clip as sub-clips and turn stabilization on or off for the individual sub-clips. Then, If you really want to get granular, you can roll the edit point between stabilized and non-stabilized sub-clips -- a frame at a time.



    If you want to experiment, FCPX is available as a 30-day free trial.



    Try Final Cut Pro X free for 30 days.
  • Reply 20 of 27
    melgrossmelgross Posts: 31,491member
    Quote:
    Originally Posted by addabox View Post


    The problem with software image stabilization (and this is true of iMove as well) is that it degrades the image. In order to align frames you have to discard some number of pixels around the edges-- the more aggressive the stabilization, the more pixels. That's because if I shift a frame to the left to make it match the previous frame, there's nothing beyond the original right edge to show, and I just get a weirdly cropped frame. So I have to in effect zoom in on my image somewhat so I have a buffer area within which to shift things around. Electronically zoomed image = lower resolution.



    Better is optical image stabilization, which uses a software controlled moving mirror to align the image frame to frame. I doubt there's room in an iPhone for this approach, so I'm curious if Apple has some way to do this that doesn't negatively effect image quality.



    Sony, and others, use a method in their still camera's where the sensor itself moves around. The disadvantage to this is that they need a sensor with more sensing sites to cover the extra area needed at the edges. It wasn't thought they could do this for full frame, but they did.



    Canon invented optical stabilization which they and Sony, who licensed the technology, use in their video cameras. Canon and Nikon, who also licenses the tech from Canon, use it in their still lenses as well. It's a very expensive technology, but it's the best. It uses sensors and a module centered at the nodal point in the lens that contains lenses and a prism. This vibrates to counter the movement from handholding.



    I'm not certain as to how Apple intends to do this even though I read the patent application.



    It seems as though the software pulls the image around somewhat like vibrating the sensor in cameras. I would think it would require a bigger sensor as well. But I'm missing something.



    It would seem to me that this could be done easily if the sensor was larger, and the camera just detected where the image was, and used those pixels. But that's likely an oversimplification, as it would only work pixel to pixel and wouldn't be able to account for smaller shake.
Sign In or Register to comment.