or Connect
AppleInsider › Forums › Mobile › iPhone › Apple exploring accelerometer, gyro stabilization for iPhone video recording
New Posts  All Forums:Forum Nav:

Apple exploring accelerometer, gyro stabilization for iPhone video recording

post #1 of 28
Thread Starter 
Using the built-in accelerometer and gyroscope, future iPhones could compensate for shaky video recordings with new software-based stabilization.

Apple's interest in allowing users to record better video on their iPhone was revealed this week in a newly published patent application discovered by AppleInsider. Entitled "Accelerometer/Gyro-Facilitated Video Stabilization," it describes how an iPhone might use motion sensing data to compensate for any jittering in a recorded video.

The filing notes that software-based video stabilization already exists and can improve the perceptual quality of a video sequence, but not without consequences. For example, current stabilization techniques can use up "considerable resources," which can be particularly detrimental on a portable, battery-powered device like an iPhone.

In addition, while advanced algorithms can help offset any shakiness in a video, sometimes they can generate incorrect estimates that don't actually improve video quality at all.

But now, consumer devices like the iPhone include gyroscopes and accelerometers, providing motion data for software on the device. While this data can be helpful, even it isn't a perfect solution for video stabilization due to "noise" in the data, Apple said.

"Motion detection devices can provide metadata that indicates motion effects of a camera during video capture, however, even though the motion detectors provide data relating to global motion of the camera, the level of shakiness between frames is often comparable to the noise level of the motion detector data," the filing reads. "Such high level of the noise in data prohibits (direct use) of accelerometer data in video stabilization."



Apple's solution would be to selectively control the motion stabilization feature, only adjusting and improving the video captured when the system determines it is necessary. This determination would be made by comparing motion sensor data to a pre-set threshold.

"Based on the determination, motion stabilization may be suspended on select portions of a captured video sequence," the application states.



The described system could go frame by frame in a captured video, comparing the file with synced motion detection data. By doing this, an iPhone could determine if and when image normalization processes are required, only applying the effect when necessary.

The proposed invention, made public this week, was first filed in April of 2010. It is credited to Yuxin Liu, Xiaojin Shi, James Oliver Normile, and Hsi-jung Wu.
post #2 of 28
Sounds good, this is needed.

As of now, the recording quality is good but the lack of stabilization makes the videos very jittery and non-professional-looking, unless you use something to hold the phone stationary (like a tripod), so stabilization sounds like something that should be added, and could be done easily through software... (I am sure there are 3rd party apps for this already) -- but it should be built into the iOS by now.
post #3 of 28
Yup. Sounds good. Every little bit helps.

I am surprised a third party hasnt come out wirh a Mini "steadi-cam" like device for the iphone and ipod touch yet. Call it an isteadi or a steadi-pod. I dunno.

But that would be sweet, if it was small enough. :-)
post #4 of 28
Quote:
Originally Posted by regan View Post

I am surprised a third party hasnt come out wirh a Mini "steadi-cam" like device for the iphone and ipod touch yet. Call it an isteadi or a steadi-pod. I dunno.

The Internet has these things called "search engines."

If you type in "iPhone steadicam" into one of these fancy search engines, you'll end up on a page showing a genuine Steadicam for the iPhone 4 (and priced at what one might expect from Steadicam).

Same day, you should try these search engines. They can be quite helpful.

A few suggestions are: Google, Bing, and Yahoo!
post #5 of 28
Quote:
Originally Posted by regan View Post

Yup. Sounds good. Every little bit helps.

I am surprised a third party hasnt come out wirh a Mini "steadi-cam" like device for the iphone and ipod touch yet. Call it an isteadi or a steadi-pod. I dunno.

But that would be sweet, if it was small enough. :-)

Somebody already did. Edit:Oops, someone beat me to it.
http://www.steadicam.com/smoothee_home.html

It would be seriously cool if Apple were able to put camera stabilization in iOS...
post #6 of 28
What they need to patent and implement is the "Hey you! Your finger is covering the microphone and muffling all audio!" alert for the video you're recording.
post #7 of 28
[insult removed]
post #8 of 28
What are you talking about?

I was responding to a person who clearly is not aware of a valuable Internet tool: the search engine.

Not only did I provide a link to the device described, I also mentioned a couple of these search engine tools to get the person started, as well as the actual query string that I used to find the desired device. So in a way, I gave this person a cake, as well as the ingredients, recipe and oven for them to bake the next time they wanted a cake.

You, on the other hand, provided zero help in your comment. I suggest you provide responses with more long term pedagogical value.
post #9 of 28
Quote:
Originally Posted by cvaldes1831 View Post

What are you talking about?

I was responding to a person who clearly is not aware of a valuable Internet tool: the search engine.

Not only did I provide a link to the device described, I also mentioned a couple of these search engine tools to get the person started, as well as the actual query string that I used to find the desired device. So in a way, I gave this person a cake, as well as the ingredients, recipe and oven for them to bake the next time they wanted a cake.

You, on the other hand, provided zero help in your comment. I suggest you provide responses with more long term pedagogical value.

Look, if you are going to be condescending with your original response, stick with it. Nothing wrong with being arrogant and condescending unless you try to whitewash it after. It only makes you look stupid.
post #10 of 28
Quote:
Originally Posted by cvaldes1831 View Post

The Internet has these things called "search engines."

If you type in "iPhone steadicam" into one of these fancy search engines, you'll end up on a page showing a genuine Steadicam for the iPhone 4 (and priced at what one might expect from Steadicam).

Same day, you should try these search engines. They can be quite helpful.

A few suggestions are: Google, Bing, and Yahoo!

Quote:
Originally Posted by cvaldes1831 View Post

What are you talking about?

I was responding to a person who clearly is not aware of a valuable Internet tool: the search engine.

Not only did I provide a link to the device described, I also mentioned a couple of these search engine tools to get the person started, as well as the actual query string that I used to find the desired device. So in a way, I gave this person a cake, as well as the ingredients, recipe and oven for them to bake the next time they wanted a cake.

You, on the other hand, provided zero help in your comment. I suggest you provide responses with more long term pedagogical value.

Ignore the haterz, good posts
"We're Apple. We don't wear suits. We don't even own suits."
Reply
"We're Apple. We don't wear suits. We don't even own suits."
Reply
post #11 of 28
As for steady-cam software, I seem to recall a product demo of iMovie 8 (i think) that had built in image stabilization. So Apple could easily introduce this functionality.
post #12 of 28
Skype it already on the ball with this
post #13 of 28
Quote:
Originally Posted by sennen View Post

Ignore the haterz, good posts

Yeah, as good a bludgeon as the steady cam he linked to on the amazon site.
post #14 of 28
I was wondering if this could be done, but I've always thought that the camera movement was too small to be picked up by the sensors. I guess it's just barely possible. Something is better than nothing.

I'm just nit too sure that turning it off and on during a recording is a great idea. Just like audio noise reduction, when it's on and off, we get a spurt of noise which is more distracting than the noise itself too often.
post #15 of 28
Quote:
Originally Posted by DJinTX View Post

As for steady-cam software, I seem to recall a product demo of iMovie 8 (i think) that had built in image stabilization. So Apple could easily introduce this functionality.

I'm nearly certain that someone announced that they either do or soon will offer video stabilization with their chat application. Can't find the mention now and I can't remember if it was also for video recording. I can certainly see where it would benefit Facetime or other video chat apps though.

EDIT: I found what I believe I may have read a few weeks back. SRI Tech does have image stabilization for live video, but it doesn't look like it's applicable to video recording, which is Apple's intended usage.

http://www.sri.com/news/releases/07072011.html
melior diabolus quem scies
Reply
melior diabolus quem scies
Reply
post #16 of 28
[insult removed]
post #17 of 28
Quote:
Originally Posted by cvaldes1831 View Post

The Internet has these things called "search engines."

If you type in "iPhone steadicam" into one of these fancy search engines, you'll end up on a page showing a genuine Steadicam for the iPhone 4 (and priced at what one might expect from Steadicam).

Same day, you should try these search engines. They can be quite helpful.

A few suggestions are: Google, Bing, and Yahoo!

Everybody knows about Google, including you. And everybody knows that everybody knows about Google, apparently except you.

Please update the AppleInsider app to function in landscape mode.

Reply

Please update the AppleInsider app to function in landscape mode.

Reply
post #18 of 28
Quote:
Originally Posted by DJinTX View Post

As for steady-cam software, I seem to recall a product demo of iMovie 8 (i think) that had built in image stabilization. So Apple could easily introduce this functionality.

The problem with software image stabilization (and this is true of iMove as well) is that it degrades the image. In order to align frames you have to discard some number of pixels around the edges-- the more aggressive the stabilization, the more pixels. That's because if I shift a frame to the left to make it match the previous frame, there's nothing beyond the original right edge to show, and I just get a weirdly cropped frame. So I have to in effect zoom in on my image somewhat so I have a buffer area within which to shift things around. Electronically zoomed image = lower resolution.

Better is optical image stabilization, which uses a software controlled moving mirror to align the image frame to frame. I doubt there's room in an iPhone for this approach, so I'm curious if Apple has some way to do this that doesn't negatively effect image quality.
They spoke of the sayings and doings of their commander, the grand duke, and told stories of his kindness and irascibility.
Reply
They spoke of the sayings and doings of their commander, the grand duke, and told stories of his kindness and irascibility.
Reply
post #19 of 28
Quote:
Originally Posted by jd_in_sb View Post

Everybody knows about Google, including you. And everybody knows that everybody knows about Google, apparently except you.

So are you saying, not everybody knows that not everybody know about Google?
Enjoying the new Mac Pro ... it's smokin'
Been using Apple since Apple ][ - Long on AAPL so biased
nMac Pro 6 Core, MacBookPro i7, MacBookPro i5, iPhones 5 and 5s, iPad Air, 2013 Mac mini.
Reply
Enjoying the new Mac Pro ... it's smokin'
Been using Apple since Apple ][ - Long on AAPL so biased
nMac Pro 6 Core, MacBookPro i7, MacBookPro i5, iPhones 5 and 5s, iPad Air, 2013 Mac mini.
Reply
post #20 of 28
Quote:
Originally Posted by addabox View Post

The problem with software image stabilization (and this is true of iMove as well) is that it degrades the image. In order to align frames you have to discard some number of pixels around the edges-- the more aggressive the stabilization, the more pixels. That's because if I shift a frame to the left to make it match the previous frame, there's nothing beyond the original right edge to show, and I just get a weirdly cropped frame. So I have to in effect zoom in on my image somewhat so I have a buffer area within which to shift things around. Electronically zoomed image = lower resolution.

Better is optical image stabilization, which uses a software controlled moving mirror to align the image frame to frame. I doubt there's room in an iPhone for this approach, so I'm curious if Apple has some way to do this that doesn't negatively effect image quality.

But, isn't OIS doing essentially the same thing -- cropping the image to attain more stable results?

I don't have enough enough experience to know whether OIS is better than after-the-fact "image analysis and stabilization".

FWIW, FCPX (Final Cut Pro X) has a pretty good image stabilization capability. Video can be analyzed in the background as it is ingested (or later, after ingest). Then, you can turn stabilization on or off for individual clips. If you want, you can identify portions of a clip as sub-clips and turn stabilization on or off for the individual sub-clips. Then, If you really want to get granular, you can roll the edit point between stabilized and non-stabilized sub-clips -- a frame at a time.

If you want to experiment, FCPX is available as a 30-day free trial.

Try Final Cut Pro X free for 30 days.
"...The calm is on the water and part of us would linger by the shore, For ships are safe in harbor, but that's not what ships are for."
- Michael Lille -
Reply
"...The calm is on the water and part of us would linger by the shore, For ships are safe in harbor, but that's not what ships are for."
- Michael Lille -
Reply
post #21 of 28
Quote:
Originally Posted by addabox View Post

The problem with software image stabilization (and this is true of iMove as well) is that it degrades the image. In order to align frames you have to discard some number of pixels around the edges-- the more aggressive the stabilization, the more pixels. That's because if I shift a frame to the left to make it match the previous frame, there's nothing beyond the original right edge to show, and I just get a weirdly cropped frame. So I have to in effect zoom in on my image somewhat so I have a buffer area within which to shift things around. Electronically zoomed image = lower resolution.

Better is optical image stabilization, which uses a software controlled moving mirror to align the image frame to frame. I doubt there's room in an iPhone for this approach, so I'm curious if Apple has some way to do this that doesn't negatively effect image quality.

Sony, and others, use a method in their still camera's where the sensor itself moves around. The disadvantage to this is that they need a sensor with more sensing sites to cover the extra area needed at the edges. It wasn't thought they could do this for full frame, but they did.

Canon invented optical stabilization which they and Sony, who licensed the technology, use in their video cameras. Canon and Nikon, who also licenses the tech from Canon, use it in their still lenses as well. It's a very expensive technology, but it's the best. It uses sensors and a module centered at the nodal point in the lens that contains lenses and a prism. This vibrates to counter the movement from handholding.

I'm not certain as to how Apple intends to do this even though I read the patent application.

It seems as though the software pulls the image around somewhat like vibrating the sensor in cameras. I would think it would require a bigger sensor as well. But I'm missing something.

It would seem to me that this could be done easily if the sensor was larger, and the camera just detected where the image was, and used those pixels. But that's likely an oversimplification, as it would only work pixel to pixel and wouldn't be able to account for smaller shake.
post #22 of 28
Quote:
Originally Posted by melgross View Post

Sony, and others, use a method in their still camera's where the sensor itself moves around. The disadvantage to this is that they need a sensor with more sensing sites to cover the extra area needed at the edges. It wasn't thought they could do this for full frame, but they did.

Canon invented optical stabilization which they and Sony, who licensed the technology, use in their video cameras. Canon and Nikon, who also licenses the tech from Canon, use it in their still lenses as well. It's a very expensive technology, but it's the best. It uses sensors and a module centered at the nodal point in the lens that contains lenses and a prism. This vibrates to counter the movement from handholding.

I'm not certain as to how Apple intends to do this even though I read the patent application.

It seems as though the software pulls the image around somewhat like vibrating the sensor in cameras. I would think it would require a bigger sensor as well. But I'm missing something.

It would seem to me that this could be done easily if the sensor was larger, and the camera just detected where the image was, and used those pixels. But that's likely an oversimplification, as it would only work pixel to pixel and wouldn't be able to account for smaller shake.

So, If you had a larger sensor -- wouldn't most people want the option of shooting a larger, unstabilized frame, in addition to shooting a smaller, stabilized frame.

I normally use a tripod when filming things like soccer games. That eliminates shake. But I frequently have problems when shooting midrange shots, where the center of action changes suddenly -- a pass or a long kick, for instance. Then, I have rapid side-to-side movement, followed by a a second (or so) to align the new center of action. The only way I have found to compensate for this is to add a dissolve.
"...The calm is on the water and part of us would linger by the shore, For ships are safe in harbor, but that's not what ships are for."
- Michael Lille -
Reply
"...The calm is on the water and part of us would linger by the shore, For ships are safe in harbor, but that's not what ships are for."
- Michael Lille -
Reply
post #23 of 28
Quote:
Originally Posted by Dick Applebaum View Post

So, If you had a larger sensor -- wouldn't most people want the option of shooting a larger, unstabilized frame, in addition to shooting a smaller, stabilized frame.

I normally use a tripod when filming things like soccer games. That eliminates shake. But I frequently have problems when shooting midrange shots, where the center of action changes suddenly -- a pass or a long kick, for instance. Then, I have rapid side-to-side movement, followed by a a second (or so) to align the new center of action. The only way I have found to compensate for this is to add a dissolve.

It's not a whole lot larger, perhaps 10%. just enough to cover the most extreme shake. With a fair number of cameras, you need to turn the anti shake off when on a tripod. There are some cameras that will compensate for horizontal shake when on a tripod, whether still or video. You would need to read the reviews to see which ones.
post #24 of 28
A larger sensor would work, but then you get into non-standard resolutions and what I assume would be higher prices. I don't think video functionality is central enough to the iPhone experience for Apple to invest in their own sensor tech, but who knows?

This is all more critical on tiny sensors with so so lenses where you're barely eking out an acceptable image to begin with. FCP can get good results because you're often working with material generated by a much larger sensor, or higher initial resolution, or both. One of the big advantages of the higher res pro motion cameras is that you can reframe a shot via cropping without any IQ loss, since your sensor is already overshooting the resolution of your deliverables. No pad like that on a phone camera, though.

I wonder if might be possible to sufficiently miniaturize the gimbal and actuators to move the sensor itself ala the scheme Mel mentioned. Again, it might not be important enough to the all over iPhone experience to merit the investment, but if anyone is going to improve on things like this it's going to be Apple.

Apple has a real motivation for this kind of thing because they're the only ones making enough money on their hardware to invest in basic improvements like these. It's where they can differentiate from Android handsets. A stabilized high quality video camera would be the sort of thing that Samsung, just to pick a name out of a hat, would be hard pressed to copy.
They spoke of the sayings and doings of their commander, the grand duke, and told stories of his kindness and irascibility.
Reply
They spoke of the sayings and doings of their commander, the grand duke, and told stories of his kindness and irascibility.
Reply
post #25 of 28
Thanks Guys...

I learned something(s).

That's why I visit and participate in these forums.
"...The calm is on the water and part of us would linger by the shore, For ships are safe in harbor, but that's not what ships are for."
- Michael Lille -
Reply
"...The calm is on the water and part of us would linger by the shore, For ships are safe in harbor, but that's not what ships are for."
- Michael Lille -
Reply
post #26 of 28
Quote:
Originally Posted by addabox View Post

A larger sensor would work, but then you get into non-standard resolutions and what I assume would be higher prices. I don't think video functionality is central enough to the iPhone experience for Apple to invest in their own sensor tech, but who knows?

This is all more critical on tiny sensors with so so lenses where you're barely eking out an acceptable image to begin with. FCP can get good results because you're often working with material generated by a much larger sensor, or higher initial resolution, or both. One of the big advantages of the higher res pro motion cameras is that you can reframe a shot via cropping without any IQ loss, since your sensor is already overshooting the resolution of your deliverables. No pad like that on a phone camera, though.

I wonder if might be possible to sufficiently miniaturize the gimbal and actuators to move the sensor itself ala the scheme Mel mentioned. Again, it might not be important enough to the all over iPhone experience to merit the investment, but if anyone is going to improve on things like this it's going to be Apple.

Apple has a real motivation for this kind of thing because they're the only ones making enough money on their hardware to invest in basic improvements like these. It's where they can differentiate from Android handsets. A stabilized high quality video camera would be the sort of thing that Samsung, just to pick a name out of a hat, would be hard pressed to copy.

All sensors have more pixels than are used. A few rows on the outside aren't. One reason is because of alignment. If the sensor is off by one row, then it will use one row on that edge. All sensors used for anti-shake have even more rows that normally wouldn't be in the image if stabilization was turned off. But this doesn't mean that non standard resolutions are used.

Two things I'd love to see. One is stabilization of some kind that is effective with such a small and lightweight device. And two is an optical zoom. Lack of optical zoom is the last thing preventing phone cameras from challenging compact cameras.

There have been, for years now, tiny lenses developed that are plastic in the sense that they change shape with the application of a small current. These lenses change focus that way. Phillips developed the first one a good ten years ago, and I have a prototype of that. But this concept could also be used for making a zoom. If Apple could just get a 2x zoom, it would change everything.
post #27 of 28
Quote:
Originally Posted by cvaldes1831 View Post

The Internet has these things called "search engines."

If you type in "iPhone steadicam" into one of these fancy search engines, you'll end up on a page showing a genuine Steadicam for the iPhone 4 (and priced at what one might expect from Steadicam).

Same day, you should try these search engines. They can be quite helpful.

A few suggestions are: Google, Bing, and Yahoo!

Cool (and expensive) as that is, there is a simple poor man's solution.
The very smallness that makes small cameras so handy is what also makes them shaky. The small distance from the lens to the points of support (hands) amplifies any small movement. The fact that the lens is not in the center of mass of the phone makes it worse (in fact it's really bad since it's out on the corner of the phone.) Finally, the small mass and therefor inertia of the camera also makes it very susceptible to shake and jitter.

If a handy person were to simply take an iPhone case and attach handles to it in such a way as to place the iPhone lens at the geometric center of the support points (handles) while keeping the shutter buttons convenient, one would have a very steady and superior point and shoot camera.
post #28 of 28
Quote:
Originally Posted by DESuserIGN View Post

Cool (and expensive) as that is, there is a simple poor man's solution.
The very smallness that makes small cameras so handy is what also makes them shaky. The small distance from the lens to the points of support (hands) amplifies any small movement. The fact that the lens is not in the center of mass of the phone makes it worse (in fact it's really bad since it's out on the corner of the phone.) Finally, the small mass and therefor inertia of the camera also makes it very susceptible to shake and jitter.

If a handy person were to simply take an iPhone case and attach handles to it in such a way as to place the iPhone lens at the geometric center of the support points (handles) while keeping the shutter buttons convenient, one would have a very steady and superior point and shoot camera.

There are some pretty bad ass iPhone mounting systems available, up to and including an actual Steadicam:

They spoke of the sayings and doings of their commander, the grand duke, and told stories of his kindness and irascibility.
Reply
They spoke of the sayings and doings of their commander, the grand duke, and told stories of his kindness and irascibility.
Reply
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: iPhone
  • Apple exploring accelerometer, gyro stabilization for iPhone video recording
AppleInsider › Forums › Mobile › iPhone › Apple exploring accelerometer, gyro stabilization for iPhone video recording