Apple looks to improve visual quality of streaming video

[AppleInsider]

Apple has investigated a method to make imperfections in compressed video files less visible to the human eye, a new patent application revealed this week shows.

The described technology would not do away with compression, which is necessary given bandwidth restrictions. Instead, it details a new technology that would mask artifacting that often appears when video files are compressed to reduce their file size.

"Disadvantageously, most video compression standards use lossy data compression techniques in which data determined by a particular compression algorithm to be of lesser importance to the overall content, but which is nonetheless discernible and objectionable to the user, is discarded," the application reads. As a result, certain video compression algorithms may introduce visual artifacts into the decoded video stream, which may be distracting to a user when viewing the decoded video data."

It continues: "Such visual artifacts are generally attributable to the latent error in lossy data compression and may appear more frequently as higher video compression rates are used. Moreover, such artifacts are exacerbated when the decoded video images are scaled to larger high definition displays. "

Earlier this year, Apple was rumored to be working on a new service called iTunes Replay. It was said to allow users to stream iTunes video purchases directly from the company's servers for playback anywhere. Such a system would prevent the need for local storage to save large video files.

Apple has also taken interest in streaming video on the iPhone, with the latest 3.0 software allowing the HTTP Live Streaming standard. Apple has provided the QuickTime Streaming Server for the last decade, but that format has run into issues as it is often blocked by firewalls. But the new HTTP streaming technology allows simple real-time streaming of content and can offer multiple versions of clips in different formats which could automatically switch based on bandwidth availability.



The patent application describes a system that would introduce "random noise" into a video stream to reduce or mask the visibility of the effects of compression on video. The technology would analyze the video and determine appropriate places based on the picture's characteristics to insert the noise and mask the artifacting, otherwise known as "fuzziness" in an overly compressed image. Such a system could be used on a computer, an Apple TV, or an iPhone.

"(A device), upon receiving streaming compressed video data from a network connection via network (device), may decode the compressed video data and temporarily buffer or cache the decoded video data in the (memory)," the patent application states. "In one embodiment, the noise generation (system) may sequentially process each frame of the video data buffered in the (memory) on a pixel-by-pixel basis. Each buffered pixel is read from the (memory) via an input data (bus) coupled to the noise generation."

[ View this article at AppleInsider.com ]

[huntercr]

So Apple wants to mask artifacts by introducing random noise that makes it less obvious that an artifact occurred? So I won't notice it because thoe whole screen will be snowy?
Uhhh.... ok.

[Zandros]

Or maybe they could start using higher levels of h.264 for more efficient compression and/or better encoders.

[teckstud]

Simple- make Macs handle FLASH properly. Streaming Flash looks great.

[Shookster]

The thread title is a bit misleading - this technology could be applied to all video, not just streaming video.

[Cubert]

I'm assuming this is somewhat analogous to one sound wave cancelling out another?

[Cubert]

Originally Posted by teckstud

Simple- make Macs handle FLASH properly. Streaming Flash looks great.

That should be "Make Adobe create a better Flash plug-in for Macs."

The problem is entirely on their end with shoddy coding and indifference to the platform.

[Marvin]

Originally Posted by huntercr

So Apple wants to mask artifacts by introducing random noise that makes it less obvious that an artifact occurred? So I won't notice it because thoe whole screen will be snowy?
Uhhh.... ok.

I'd expect it to be more like when you get blocky artifacts, it breaks those blocks down into noisier and less uniform shapes so they appear more natural and less digital. Like how an old VHS tape with image degradation still looks natural whereas a realplayer stream doesn't.

[oakie]

if you have the bandwidth to send randomized noise that attempts to "motion-blur the eff out of compression artifacts," why not just encode the original file with less compression, thus more effectively utilizing the bandwidth?

[kerryn]

Originally Posted by oakie

if you have the bandwidth to send randomized noise that attempts to "motion-blur the eff out of compression artifacts," why not just encode the original file with less compression, thus more effectively utilizing the bandwidth?

I think this is more a client technology in the player, rather than sending the noise in the stream.

[Zoolook]

Originally Posted by teckstud

Simple- make Macs handle FLASH properly. Streaming Flash looks great.

Flash is shit. I'd rather take Silverlight or HTML 5.

I don't know how Apple encodes its TV shows, but some look great and some look absolutely horrible. BSG Season 2 in SD is appalling (I managed to encode Season 1 from DVD at the same bit-rate with far better quality using free software). But Seasons 3 and 4 were better (although not perfect, even in HD there was a lot of pixelating and aliasing).

However Fringe looks great (even in SD), and Mad Men in HD varies between looking amazing and then looking like an AVI from the 1990s. I had Dexter Season 3, great until the penultimate episode which was encoded in the wrong aspect ratio and had lots of artifacts. To their credit, iTunes store has always given a credit for anything I have complained about.

iTunes videos definitely suffer from artifacts when there are lots of dark areas or when scenes are fairly still. The reason is the fixed bit-rate (1500kbps isn't a lot, even for SD) - a constant quality encoder (available on any x264 encoder) allows for bit-rates to rise when scenes require more quality, but keep the overall file sizes lower. These work fine on ATV and the iPhone, but not on older iPods that are limited to 1500kbps or 2500kbps decoding.

[slinberg]

I just hope that one day, in the far future, maybe there will be a 1,024-core mac that can play a youtube video over a broadband link without stuttering.

[LuisDias]

Originally Posted by Marvin

I'd expect it to be more like when you get blocky artifacts, it breaks those blocks down into noisier and less uniform shapes so they appear more natural and less digital. Like how an old VHS tape with image degradation still looks natural whereas a realplayer stream doesn't.

What the hell do you mean by a VHS tape image degradation looking "natural"?!? Whenever I look at reality, things do not look "degradated", mkay? Any kind of "degradation" due to it being "recorded" or "streamed" or whatever have you is not natural, it's by definition, artificial. mkay?

[samurai1999]

Sounds like a rubbish patent
- certainly from the information presented here, there's no actual algorithm, so it's impossible to say what it does, or how it does it
- saying 'add some noise to make the artefact go away' isn't enough for a patent...

[Booga]

Originally Posted by samurai1999

Sounds like a rubbish patent
- certainly from the information presented here, there's no actual algorithm, so it's impossible to say what it does, or how it does it
- saying 'add some noise to make the artefact go away' isn't enough for a patent...

AppleInsider, if they want to discuss patents responsibly, really needs to include the patent # or application # and preferrably a link.

The one in question appears to be patent application 20090257507. The patent application is actually fairly specific. As patents generally go, some of the later claims get more and more general to try to cover as broad an area as possible, but the major claims seem to be fairly interesting if the effect they're describing is real.

Would you rather watch a super-blocky degraded digital signal or a somewhat snowy degraded analog signal? This makes some attempt to find a happy medium.

[LuisDias]

Originally Posted by samurai1999

Sounds like a rubbish patent
- certainly from the information presented here, there's no actual algorithm, so it's impossible to say what it does, or how it does it
- saying 'add some noise to make the artefact go away' isn't enough for a patent...

Yeah, because hey, what's better than patenting an algorithm?

Don't know?

Why hell, to announce to the world the friggin' algorithm!

[sprockkets]

Originally Posted by oakie

if you have the bandwidth to send randomized noise that attempts to "motion-blur the eff out of compression artifacts," why not just encode the original file with less compression, thus more effectively utilizing the bandwidth?

Because some people have good bandwidth and some don't, and current systems will on the fly reduce the quality when the bandwidth gets constrained.

[newbee]

Originally Posted by huntercr

So Apple wants to mask artifacts by introducing random noise that makes it less obvious that an artifact occurred? So I won't notice it because thoe whole screen will be snowy?
Uhhh.... ok.

Introducing "white noise" into headphones for sound cancelling ability has been done for ages. This just sounds like a similar arrangement. Should be interesting to see the results.

[newbee]

Originally Posted by teckstud

Simple- make Macs handle FLASH properly. Streaming Flash looks great.

Don't you ever get tired of shoveling that same load of crap?

[newbee]

Originally Posted by LuisDias

What the hell do you mean by a VHS tape image degradation looking "natural"?!? Whenever I look at reality, things do not look "degradated", mkay? Any kind of "degradation" due to it being "recorded" or "streamed" or whatever have you is not natural, it's by definition, artificial. mkay?

It doesn't matter what it is ... only how it looks .... chill out.

[teckstud]

Originally Posted by newbee

Don't you ever get tired of shoveling that same load of crap?

Open your mind NEWBEE and embrace different forms of streaming- ones that already exists- ones that work excellently on 90% of computers out there and excellently on the other 10%, provided you don't use any other application simultaneously.

[Booga]

Originally Posted by teckstud

Open you mind NEWBEE and embrace different forms of streaming- ones that already exists- ones that work excellently on 90% of computers out there and excellently on the other 10%, provided you don't use any other application simultaneously.

Big difference between a ubiquity argument and a quality one. Yes, agreed, Flash video is everywhere. But no, the quality isn't very good in an absolute sense and it's absolutely awful if you factor in quality for consumed processing power.

[teckstud]

Originally Posted by Booga

Big difference between a ubiquity argument and a quality one. Yes, agreed, Flash video is everywhere. But no, the quality isn't very good in an absolute sense and it's absolutely awful if you factor in quality for consumed processing power.

The quality of FLash VOD is pretty decent- not HD- but decent. Much better than YouTube and HULU.

[Maestro64]

Originally Posted by huntercr

So Apple wants to mask artifacts by introducing random noise that makes it less obvious that an artifact occurred? So I won't notice it because thoe whole screen will be snowy?
Uhhh.... ok.

Okay spend all this time to clean up the video and get rid of noise only to re-introduce it back into the video stream. The basis principle is that human eye can easily filter out the noise verses missing blocks of information.

The problem is when the video blocks you can also get distorted audio and mixing noise into the audio stream would not be a good thing

[Zandros]

Originally Posted by Maestro64

Okay spend all this time to clean up the video and get rid of noise only to re-introduce it back into the video stream. The basis principle is that human eye can easily filter out the noise verses missing blocks of information.

Getting rid of noise like film grain isn't desirable, DNR destroys detail.

[faisalmali]

Compression artifacts and noise are not the same thing. Compression artifacts usually are present in the form of "blocks". This is a result of the compression routine "grouping" together like colors, and even going so far as to make like colors more similar than they originally are. I don't know if my explanation makes any sense to you, the reader, but it makes sense to me!

Noise on the other hand, is just that....it's noise. A bit like turning on the TV when the cable is out. Only in this case, the noise would be MUCH more subtle.

Applying noise to reduce compression artifacts or banding in still imagery or videos is a very old trick of digital artists. I do this in Photoshop and Final Cut Pro all the time. My guess is that Apple would like to apply a similar trick to heavily compressed streaming video. I doubt it will be part of the video file, but rather, a post effect handled by Quicktime or iTunes or whatever.

If this is the case, they're not really doing anything new. About the only thing that is new about this is that you won't need to manually apply the noise effect yourself. Apple's software will take care of it for you.

[Wil Maneker]

In the interim, Apple recommends users "stand further away from the screen than usual" to observe the desired effect.

[Marvin]

Originally Posted by LuisDias

What the hell do you mean by a VHS tape image degradation looking "natural"?!? Whenever I look at reality, things do not look "degradated", mkay? Any kind of "degradation" due to it being "recorded" or "streamed" or whatever have you is not natural, it's by definition, artificial. mkay?

It's a more acceptable form of noise and easier to ignore. Some film makers prefer to shoot on old-style film because it gives the footage character and a different mood. When the footage is shot on crisp HD, it often looks more fake - this is especially true for CG movies. I'm not suggesting that VHS quality is where we should aim for streaming video, just that the artifacts were more acceptable the way they appeared than how they appear in digital compression.

Originally Posted by teckstud

Simple- make Macs handle FLASH properly. Streaming Flash looks great.

Modern Flash streaming uses H264 and they can often use high bitrates for downstream-only. I reckon this patent is more for ichat real-time streaming where the bitrates are limited to your upstream connection (easily under 500k) and fast compression where artifacts can appear.

[ghostface147]

Just make sure no patent trolls have any possible things to work off of!

[TomHicks]

Long time lurker, first time poster.

In case anyone is interested in looking this up, it's essentially dithering, but performed on the client side, as well as being dynamic. I haven't read the entire patent, but I'd imagine that's what it is.

As for "Flash video is better", that's total nonsense. FLV compression simply isn't up there with the more advanced codecs.

[Zandros]

Originally Posted by Marvin

It's a more acceptable form of noise and easier to ignore.

As far as I understand the HVS model, artificial detail is preferable to softness. There's a reason why major studios use edge enhancement and sharpening on major blockbusters.

Some film makers prefer to shoot on old-style film because it gives the footage character and a different mood. When the footage is shot on crisp HD, it often looks more fake - this is especially true for CG movies.

This makes no sense unless you're talking about some film with very low resolution. Many movies in HD were shot on film, old ones at that.

I'm not suggesting that VHS quality is where we should aim for streaming video, just that the artifacts were more acceptable the way they appeared than how they appear in digital compression.

Well, I can apply blur filters on playback if I feel like it. Dot crawl and rainbowing might be harder.

[Stoobs]

Originally Posted by TomHicks

As for "Flash video is better", that's total nonsense. FLV compression simply isn't up there with the more advanced codecs.

FLV supports H.264

[ahmlco]

Originally Posted by Stoobs

FLV supports H.264

Does Flash support HARDWARE decoding of H.264?

[tawilson]

Originally Posted by teckstud

Simple- make Macs handle FLASH properly. Streaming Flash looks great.

Either you're trolling (unlike you :P), or you can't be bothered to do even the slightest bit of research,

1. Flash performance is Adobe's fault (as is evident from playing H.264 video from YouTube via ClickToFlash [10% usage per core, versus Flash embedded 65% usage per core).
2. Flash, at least on the desktop, has access to the higher quality "Main profiles" of H.264, mobile devices in particular use "baseline profile" which doesn't support some of the fancy quantizations and require less CPU cycles to decode (also making it cheaper to implement decoder in hardware), this baseline profile will also result in a lower quality image.

[teckstud]

Originally Posted by tawilson

Either you're trolling (unlike you :P), or you can't be bothered to do even the slightest bit of research,

1. Flash performance is Adobe's fault (as is evident from playing H.264 video from YouTube via ClickToFlash [10% usage per core, versus Flash embedded 65% usage per core).
2. Flash, at least on the desktop, has access to the higher quality "Main profiles" of H.264, mobile devices in particular use "baseline profile" which doesn't support some of the fancy quantizations and require less CPU cycles to decode (also making it cheaper to implement decoder in hardware), this baseline profile will also result in a lower quality image.

Are you talking about on a Mac, PC , OR both?

[quinney]

Originally Posted by Cubert

I'm assuming this is somewhat analogous to one sound wave cancelling out another?

I think it is more analogous to dithering in audio.

Originally Posted by Maestro64

... mixing noise into the audio stream would not be a good thing

It is routinely done in digital audio.

Originally Posted by TomHicks

Long time lurker, first time poster.

In case anyone is interested in looking this up, it's essentially dithering, but performed on the client side, as well as being dynamic. I haven't read the entire patent, but I'd imagine that's what it is.
.

You beat me to it.

[teckstud]

Originally Posted by Marvin

Modern Flash streaming uses H264 and they can often use high bitrates for downstream-only. I reckon this patent is more for ichat real-time streaming where the bitrates are limited to your upstream connection (easily under 500k) and fast compression where artifacts can appear.

OMG- THE post of the day with a real intelligent answer. Thank you , thank you. You've explained everything- really!

[GregAlexander]

Adding some blur is a normal way for graphic designers to clean up an image. It works great in some areas, and badly in others.

Likewise, if you make your TV less sharp (or even change the brightness and contrast on your TV), you can dramatically reduce the visibility of compression artefacts from digital cable TV.... and the picture looks clearer.

I would assume that the decoder has a great idea at where the edges of a compressed block of picture are, and can then look at the range of colours in a block and surrounding block and whether the edges predict a visible change in colouring - implying a high likelihood that it's a compression artefact rather than something from the movie itself. So applying a blur at those points would work rather well.

[AznZOFIA111]

Originally Posted by slinberg

I just hope that one day, in the far future, maybe there will be a 1,024-core mac that can play a youtube video over a broadband link without stuttering.

[success]

Originally Posted by Marvin

Modern Flash streaming uses H264 and they can often use high bitrates for downstream-only. I reckon this patent is more for ichat real-time streaming where the bitrates are limited to your upstream connection (easily under 500k) and fast compression where artifacts can appear.

Hmm iChat chatter is making the rounds again recently. I reckon we are going to see iChat on the iPhone sooner than later.