Apple exploring accelerometer, gyro stabilization for iPhone video recording
Using the built-in accelerometer and gyroscope, future iPhones could compensate for shaky video recordings with new software-based stabilization.
Apple's interest in allowing users to record better video on their iPhone was revealed this week in a newly published patent application discovered by AppleInsider. Entitled "Accelerometer/Gyro-Facilitated Video Stabilization," it describes how an iPhone might use motion sensing data to compensate for any jittering in a recorded video.
The filing notes that software-based video stabilization already exists and can improve the perceptual quality of a video sequence, but not without consequences. For example, current stabilization techniques can use up "considerable resources," which can be particularly detrimental on a portable, battery-powered device like an iPhone.
In addition, while advanced algorithms can help offset any shakiness in a video, sometimes they can generate incorrect estimates that don't actually improve video quality at all.
But now, consumer devices like the iPhone include gyroscopes and accelerometers, providing motion data for software on the device. While this data can be helpful, even it isn't a perfect solution for video stabilization due to "noise" in the data, Apple said.
"Motion detection devices can provide metadata that indicates motion effects of a camera during video capture, however, even though the motion detectors provide data relating to global motion of the camera, the level of shakiness between frames is often comparable to the noise level of the motion detector data," the filing reads. "Such high level of the noise in data prohibits (direct use) of accelerometer data in video stabilization."
Apple's solution would be to selectively control the motion stabilization feature, only adjusting and improving the video captured when the system determines it is necessary. This determination would be made by comparing motion sensor data to a pre-set threshold.
"Based on the determination, motion stabilization may be suspended on select portions of a captured video sequence," the application states.
The described system could go frame by frame in a captured video, comparing the file with synced motion detection data. By doing this, an iPhone could determine if and when image normalization processes are required, only applying the effect when necessary.
The proposed invention, made public this week, was first filed in April of 2010. It is credited to Yuxin Liu, Xiaojin Shi, James Oliver Normile, and Hsi-jung Wu.
Apple's interest in allowing users to record better video on their iPhone was revealed this week in a newly published patent application discovered by AppleInsider. Entitled "Accelerometer/Gyro-Facilitated Video Stabilization," it describes how an iPhone might use motion sensing data to compensate for any jittering in a recorded video.
The filing notes that software-based video stabilization already exists and can improve the perceptual quality of a video sequence, but not without consequences. For example, current stabilization techniques can use up "considerable resources," which can be particularly detrimental on a portable, battery-powered device like an iPhone.
In addition, while advanced algorithms can help offset any shakiness in a video, sometimes they can generate incorrect estimates that don't actually improve video quality at all.
But now, consumer devices like the iPhone include gyroscopes and accelerometers, providing motion data for software on the device. While this data can be helpful, even it isn't a perfect solution for video stabilization due to "noise" in the data, Apple said.
"Motion detection devices can provide metadata that indicates motion effects of a camera during video capture, however, even though the motion detectors provide data relating to global motion of the camera, the level of shakiness between frames is often comparable to the noise level of the motion detector data," the filing reads. "Such high level of the noise in data prohibits (direct use) of accelerometer data in video stabilization."
Apple's solution would be to selectively control the motion stabilization feature, only adjusting and improving the video captured when the system determines it is necessary. This determination would be made by comparing motion sensor data to a pre-set threshold.
"Based on the determination, motion stabilization may be suspended on select portions of a captured video sequence," the application states.
The described system could go frame by frame in a captured video, comparing the file with synced motion detection data. By doing this, an iPhone could determine if and when image normalization processes are required, only applying the effect when necessary.
The proposed invention, made public this week, was first filed in April of 2010. It is credited to Yuxin Liu, Xiaojin Shi, James Oliver Normile, and Hsi-jung Wu.
Comments
As of now, the recording quality is good but the lack of stabilization makes the videos very jittery and non-professional-looking, unless you use something to hold the phone stationary (like a tripod), so stabilization sounds like something that should be added, and could be done easily through software... (I am sure there are 3rd party apps for this already) -- but it should be built into the iOS by now.
I am surprised a third party hasnt come out wirh a Mini "steadi-cam" like device for the iphone and ipod touch yet. Call it an isteadi or a steadi-pod. I dunno.
But that would be sweet, if it was small enough. :-)
I am surprised a third party hasnt come out wirh a Mini "steadi-cam" like device for the iphone and ipod touch yet. Call it an isteadi or a steadi-pod. I dunno.
The Internet has these things called "search engines."
If you type in "iPhone steadicam" into one of these fancy search engines, you'll end up on a page showing a genuine Steadicam for the iPhone 4 (and priced at what one might expect from Steadicam).
Same day, you should try these search engines. They can be quite helpful.
A few suggestions are: Google, Bing, and Yahoo!
Yup. Sounds good. Every little bit helps.
I am surprised a third party hasnt come out wirh a Mini "steadi-cam" like device for the iphone and ipod touch yet. Call it an isteadi or a steadi-pod. I dunno.
But that would be sweet, if it was small enough. :-)
Somebody already did. Edit:Oops, someone beat me to it.
http://www.steadicam.com/smoothee_home.html
It would be seriously cool if Apple were able to put camera stabilization in iOS...
I was responding to a person who clearly is not aware of a valuable Internet tool: the search engine.
Not only did I provide a link to the device described, I also mentioned a couple of these search engine tools to get the person started, as well as the actual query string that I used to find the desired device. So in a way, I gave this person a cake, as well as the ingredients, recipe and oven for them to bake the next time they wanted a cake.
You, on the other hand, provided zero help in your comment. I suggest you provide responses with more long term pedagogical value.
What are you talking about?
I was responding to a person who clearly is not aware of a valuable Internet tool: the search engine.
Not only did I provide a link to the device described, I also mentioned a couple of these search engine tools to get the person started, as well as the actual query string that I used to find the desired device. So in a way, I gave this person a cake, as well as the ingredients, recipe and oven for them to bake the next time they wanted a cake.
You, on the other hand, provided zero help in your comment. I suggest you provide responses with more long term pedagogical value.
Look, if you are going to be condescending with your original response, stick with it. Nothing wrong with being arrogant and condescending unless you try to whitewash it after. It only makes you look stupid.
The Internet has these things called "search engines."
If you type in "iPhone steadicam" into one of these fancy search engines, you'll end up on a page showing a genuine Steadicam for the iPhone 4 (and priced at what one might expect from Steadicam).
Same day, you should try these search engines. They can be quite helpful.
A few suggestions are: Google, Bing, and Yahoo!
What are you talking about?
I was responding to a person who clearly is not aware of a valuable Internet tool: the search engine.
Not only did I provide a link to the device described, I also mentioned a couple of these search engine tools to get the person started, as well as the actual query string that I used to find the desired device. So in a way, I gave this person a cake, as well as the ingredients, recipe and oven for them to bake the next time they wanted a cake.
You, on the other hand, provided zero help in your comment. I suggest you provide responses with more long term pedagogical value.
Ignore the haterz, good posts
Ignore the haterz, good posts
Yeah, as good a bludgeon as the steady cam he linked to on the amazon site.
I'm just nit too sure that turning it off and on during a recording is a great idea. Just like audio noise reduction, when it's on and off, we get a spurt of noise which is more distracting than the noise itself too often.
As for steady-cam software, I seem to recall a product demo of iMovie 8 (i think) that had built in image stabilization. So Apple could easily introduce this functionality.
I'm nearly certain that someone announced that they either do or soon will offer video stabilization with their chat application. Can't find the mention now and I can't remember if it was also for video recording. I can certainly see where it would benefit Facetime or other video chat apps though.
EDIT: I found what I believe I may have read a few weeks back. SRI Tech does have image stabilization for live video, but it doesn't look like it's applicable to video recording, which is Apple's intended usage.
http://www.sri.com/news/releases/07072011.html
The Internet has these things called "search engines."
If you type in "iPhone steadicam" into one of these fancy search engines, you'll end up on a page showing a genuine Steadicam for the iPhone 4 (and priced at what one might expect from Steadicam).
Same day, you should try these search engines. They can be quite helpful.
A few suggestions are: Google, Bing, and Yahoo!
Everybody knows about Google, including you. And everybody knows that everybody knows about Google, apparently except you.
As for steady-cam software, I seem to recall a product demo of iMovie 8 (i think) that had built in image stabilization. So Apple could easily introduce this functionality.
The problem with software image stabilization (and this is true of iMove as well) is that it degrades the image. In order to align frames you have to discard some number of pixels around the edges-- the more aggressive the stabilization, the more pixels. That's because if I shift a frame to the left to make it match the previous frame, there's nothing beyond the original right edge to show, and I just get a weirdly cropped frame. So I have to in effect zoom in on my image somewhat so I have a buffer area within which to shift things around. Electronically zoomed image = lower resolution.
Better is optical image stabilization, which uses a software controlled moving mirror to align the image frame to frame. I doubt there's room in an iPhone for this approach, so I'm curious if Apple has some way to do this that doesn't negatively effect image quality.
Everybody knows about Google, including you. And everybody knows that everybody knows about Google, apparently except you.
So are you saying, not everybody knows that not everybody know about Google?
The problem with software image stabilization (and this is true of iMove as well) is that it degrades the image. In order to align frames you have to discard some number of pixels around the edges-- the more aggressive the stabilization, the more pixels. That's because if I shift a frame to the left to make it match the previous frame, there's nothing beyond the original right edge to show, and I just get a weirdly cropped frame. So I have to in effect zoom in on my image somewhat so I have a buffer area within which to shift things around. Electronically zoomed image = lower resolution.
Better is optical image stabilization, which uses a software controlled moving mirror to align the image frame to frame. I doubt there's room in an iPhone for this approach, so I'm curious if Apple has some way to do this that doesn't negatively effect image quality.
But, isn't OIS doing essentially the same thing -- cropping the image to attain more stable results?
I don't have enough enough experience to know whether OIS is better than after-the-fact "image analysis and stabilization".
FWIW, FCPX (Final Cut Pro X) has a pretty good image stabilization capability. Video can be analyzed in the background as it is ingested (or later, after ingest). Then, you can turn stabilization on or off for individual clips. If you want, you can identify portions of a clip as sub-clips and turn stabilization on or off for the individual sub-clips. Then, If you really want to get granular, you can roll the edit point between stabilized and non-stabilized sub-clips -- a frame at a time.
If you want to experiment, FCPX is available as a 30-day free trial.
Try Final Cut Pro X free for 30 days.
The problem with software image stabilization (and this is true of iMove as well) is that it degrades the image. In order to align frames you have to discard some number of pixels around the edges-- the more aggressive the stabilization, the more pixels. That's because if I shift a frame to the left to make it match the previous frame, there's nothing beyond the original right edge to show, and I just get a weirdly cropped frame. So I have to in effect zoom in on my image somewhat so I have a buffer area within which to shift things around. Electronically zoomed image = lower resolution.
Better is optical image stabilization, which uses a software controlled moving mirror to align the image frame to frame. I doubt there's room in an iPhone for this approach, so I'm curious if Apple has some way to do this that doesn't negatively effect image quality.
Sony, and others, use a method in their still camera's where the sensor itself moves around. The disadvantage to this is that they need a sensor with more sensing sites to cover the extra area needed at the edges. It wasn't thought they could do this for full frame, but they did.
Canon invented optical stabilization which they and Sony, who licensed the technology, use in their video cameras. Canon and Nikon, who also licenses the tech from Canon, use it in their still lenses as well. It's a very expensive technology, but it's the best. It uses sensors and a module centered at the nodal point in the lens that contains lenses and a prism. This vibrates to counter the movement from handholding.
I'm not certain as to how Apple intends to do this even though I read the patent application.
It seems as though the software pulls the image around somewhat like vibrating the sensor in cameras. I would think it would require a bigger sensor as well. But I'm missing something.
It would seem to me that this could be done easily if the sensor was larger, and the camera just detected where the image was, and used those pixels. But that's likely an oversimplification, as it would only work pixel to pixel and wouldn't be able to account for smaller shake.