Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Increasing Video Detail Using Super-Resolution?

Cliff posted more than 11 years ago | from the sharper-image dept.

Graphics 41

Cecil Esquivel asks: "I'm looking for ways to increase the quality of video by using super-resolution algorithms which use the visual information across multiple frames of video to increase the resolution of individual frames. I have found very little on the web that can do this effectively for the entire length of video. There is commercial software, VideoFOCUS, which produces hi-res stills from video, but doesn't seem to have a product for producing hi-res video from video. There is a thesis from Duke U. which is 6 years old, monochrome only and is mostly proof of concept.) Anybody out there have more information or is willing to help me develop some software that can do this? Darwin/Mac OS X solution that can work with QuickTime DV, preferred." Typically, super-resolution uses image samples generated from low-resolution and high-resolution samples of the same source, which is then converted into source independent information that can be used to increase detail for other low resolution sources. Has anyone seen programs that use super-resolution techniques for increasing the resolution of your typical digital video clip?

Sorry! There are no comments related to the filter you selected.

Some deinterlacers do similar work already (4, Interesting)

mcgroarty (633843) | more than 11 years ago | (#6248766)

Some deinterlacers use the InterCine or Messe filters to approximate the data missing in alternate frames. There are other algorithms as well -- most interpolate the existing data to guess what would have been in the missing area, and some do look at previously known data for the missing area as well as analyzing moving ranges in previous/coming frames to guess where edges or patterns would continue.

Looking at the state of deinterlacing technology and some of the "detail enhancing" resizing filters would be a good area for study.

I'd *love* to see this used to help correct data errors in video streams as well. A DirecTV receiver with this built in would be cake++.

Re:Some deinterlacers do similar work already (1, Redundant)

Directrix1 (157787) | more than 11 years ago | (#6248900)

The duke U thesis has C source code to implement this. I don't understand what the problem is.

Re:Some deinterlacers do similar work already (1)

vadim_t (324782) | more than 11 years ago | (#6249279)

Just try to read the article. "...which is 6 years old, monochrome only and is mostly proof of concept". Turning this into something usable would take a good skill and quite a lot of time that he very possibly doesn't have.

Re:Some deinterlacers do similar work already (1)

Directrix1 (157787) | more than 11 years ago | (#6253095)

Bah, just process the RGB channels seperately and recombine. And also write image importers/exporters (or use appropriate codec) to get the images in there. Even a lowly person with a CS degree can do that ;-).

Re:Some deinterlacers do similar work already (1)

ces3001 (683719) | more than 11 years ago | (#6270412)

I emailed the author... he said that getting the settings and parameters right was an "art-form in of itself." There are also significant optimization issues with this algorithm. It really would need to be reworked from scratch... but serves, I suppose as a point of departure.

multiple sources? (1)

chocolatetrumpet (73058) | more than 11 years ago | (#6251552)

This might be a slightly related note -

I know many web sites have a video available in multiple formats; i.e. real, mp4, and sorenson, or whatever.

I'd like something that takes the different streams and assembles for me a high res final product, based on the idea that there are going to be different details available in each compression scheme... does this make sense? Does it sound like a reasonable idea?

Re:multiple sources? (1)

t (8386) | more than 11 years ago | (#6295355)

Maybe, maybe not. The problem is that most video compression schemes depend on the DCT. Thus the artifacts and such are similar, in other words those different formats are somewhat correlated. If the correlation between the artifacts is stronger than the signal then all you will be doing is amplifying the artifacts.

Think of a badly compressed video with lots of blocky shapes, combine several like that and you'll probably end up with something resembling a brick wall.

I've thought that a better method would be to figure out what the best undo-filtering process would be for the oh so common DCT blocking artifacts. It's like having an audio signal that is contaminated with a 60 Hz hum, you just filter it out.

Next generation imaging technology from X-10 (3, Funny)

VisorGuy (548245) | more than 11 years ago | (#6248806)

Just imagine, you too could have a Beowulf cluster of X-10 cameras...

Re:Next generation imaging technology from X-10 (0)

Anonymous Coward | more than 11 years ago | (#6249109)

Is there no mod for "BORING"?

Re:Next generation imaging technology from X-10 (0)

Anonymous Coward | more than 11 years ago | (#6249148)

What about a mod for "useless comment from an anonymous coward, that nobody gives a fuck about?" That's what we need for you.

Re:Next generation imaging technology from X-10 (0)

Anonymous Coward | more than 11 years ago | (#6249243)

speak for yourself.

Spatial vs. Temporal (3, Informative)

Wavy (11500) | more than 11 years ago | (#6248921)

I imagine increasing the spatial resolution for a frame of footage decreases the temporal resolution of the set of frames. My gut feeling is that you would see more after/pre-image than in the source.

Re:Spatial vs. Temporal (1)

ArmorFiend (151674) | more than 11 years ago | (#6249053)

I think there's a sort of heisenberg uncertainty principle for video. Of course I have no proof either, just, as you say, a "gut feeling".

Re:Spatial vs. Temporal (1)

Mr Z (6791) | more than 11 years ago | (#6251654)

Kinda, but not quite. You are bounded in your available information content. Total information is given by "spatial information * temporal information." If you increase one, you decrease the other, at least if you hold the bitrate constant.

You're right that it's like Heisenberg's Uncertainty Principle, in that its formulation is similar. (delta-X * delta-T == constant, so as you make delta-X smaller, delta-T gets bigger.) The difference is that Heisenberg's Uncertainty Principle is based on a single hard constant. For a video stream, the right-hand-side is basically dictated by "total bits" and varies by the stream.

--Joe

Re:Spatial vs. Temporal (3, Interesting)

MoOsEb0y (2177) | more than 11 years ago | (#6249225)

Yes, typically there would be. However, if you look at the Duke U thesis, it uses MPEG's own motion vectors to determine when there's motion video (vs. a still scene) and compensate for this. Thus, what it will accomplish, is making still shots look a LOT better (and more compressible!), and in the fast-moving scenes it will do very little (but you won't notice it that much, either).

If my programming skills were better, I've be VERY interested in seeing if this can be done with MPEG-2. Noise originating from the original video has always been my primary gripe with those kinds of sequences. if this could be compensated for and exported to MPEG-4.... MMM that would be tasty.

Re:Spatial vs. Temporal (3, Interesting)

NanoGator (522640) | more than 11 years ago | (#6249258)

"I imagine increasing the spatial resolution for a frame of footage decreases the temporal resolution of the set of frames."

Not necessarily, though it'd require two passes.

In the first pass, you generate a high res image by using motion tracking to figure out how far the camera turned. Then, using that motion, it can read the sub-pixel data. That's semi easy to do, it's been done before. Before long, you have a high-resolution image.

Then, you do the second pass where you take that high resolution image and paste it on top of the low resolution footage using the motion tracking data to move the new pixels around in the right position. As long as the motion tracking data is reasonably accurate, then you could theoretically create a higher resolution video without losing temporal resolution. It's not clear to me, though, that there wouldn't be situations where that would break down.

Man I hope I expressed that fairly clearly. I've got a little experience with digitally painting video to change the details of a scene.

Wow.. (1)

dr ttol (674155) | more than 11 years ago | (#6248943)

That sounds pretty interesting. Typically any software that deals with movies tend to reduce quality or at most keep it the same. Is this the first time software can actually be made to dramatically increase the quality of a movie, even if the source sucks?

The script... is $#!+. (1)

yerricde (125198) | more than 11 years ago | (#6254552)

Is this the first time software can actually be made to dramatically increase the quality of a movie, even if the source sucks?

Image processing systems have contained "noise reduction" processes for a long time. However, no software can save a bad script.

MOD PARENT UP!!!!! (0)

Anonymous Coward | more than 11 years ago | (#6254765)

what's the answer??

Surely this should be evident (1, Informative)

Anonymous Coward | more than 11 years ago | (#6248977)

There is commercial software, VideoFOCUS, which produces hi-res stills from video, but doesn't seem to have a product for producing hi-res video from video.

You can't figure out why that is?

Re:Surely this should be evident (1)

benjamindees (441808) | more than 11 years ago | (#6249236)

You can't figure out why that is?

No. Why don't you explain to us the magical quality of the universe that prevents us from doing that.

If you can enhance a single frame, you can enhance all the frames and re-encode an enhanced video.

algorithms which use the visual information across multiple frames of video to increase the resolution of individual frames.

Isn't that what the poster asks for?

Re:Surely this should be evident (0)

Anonymous Coward | more than 11 years ago | (#6252574)

Do you try to make gold from lead on the weekends? How's that going?

Re:Surely this should be evident (0)

Anonymous Coward | more than 11 years ago | (#6254335)

That's possible, too, dipshit.
Why don't you try opening a book once in a while, instead of trolling on Slashdot.

Why this is possible (1)

0x0d0a (568518) | more than 11 years ago | (#6251114)

It looks like you're claiming that this isn't possible. Technically, you're right --- you can't necessarily get bits from the air. However, you're ignoring the fact that video contains a good deal of redundant information. Video codecs try to avoid including redundant information this, but only do so in the simplest ways.

Suppose I show three consecutive frames of an eggbeater. Now, if you knew nothing about this video, if the frames had no correlation between each other, and were just each random static, then I couldn't improve the quality at all.

When I'm showing an object that I know is the same object from frame to frame, however, I can use detail from each of the frames and add it to the other frames.

Use VideoFocus (2, Insightful)

nomel (244635) | more than 11 years ago | (#6249003)

from the videofocus website:
http://www.salientstills.com/product/pro duct_intro .html

Increased editorial choice: VideoFOCUS digitizes every single frame. You choose exactly the right shot from a video sequence to capture the perfect moment.


So...just run it through video focus, make an avi with all the frames, add the original audio track, then compress them using some video compression...you have yourself a high res video! :)

I'm guessing something will look strange, like funny blurs, or background motion or something. Who knows. I don't think it would do good with fast moving objects in the video...probably blur it real bad.

It's worth a try though. :)

Re:Use VideoFocus (0)

Anonymous Coward | more than 11 years ago | (#6249344)

The idea is to automate the process. Thats what software is FOR.

Re:Use VideoFocus (0)

Anonymous Coward | more than 11 years ago | (#6270325)

VideoFOCUS requires a "significant" amount of manual intervention for each selected frame to enhance. At 30fps for Video, fuhgettaboutit. The process needs to be automated. The trick is to get software to recognize what is going on in the scene - background, foreground elements, movement of objects versus camera movement. And not just object movement, which is relatively simple, but changes in the shape of elastic objects, like people.

Oh yea. I've seen this. (-1, Troll)

Anonymous Coward | more than 11 years ago | (#6249277)

This is that cool shit that the NSA always uses in those thriller type movies and shit. You know the ones where the systems analyst in the back of the van types like a five letter command and the software knows which part of the picture he wants enhanced and zooms in and super resolution enhances that section of the picture and shit. They use is for stuff like zooming in on license plates from helicopters and shit. Also, they use it from satellites so that they can read the secret plan papers of the terrorists in the park and shit like that. It is really cool shit.

Re:Oh yea. I've seen this. (1)

MoOsEb0y (2177) | more than 11 years ago | (#6249304)

Okay. Let me debunk you. What you are describing is impossible. You can't get more "hidden" detail out of any image. What the poster of this article is describing, is using additional detail from nearby frames in a video sequence to gain additional pixels. In effect, a higher resolution image is created because the amount of data is there. It just needs to be juggled properly.

This principle applies to astronomy. If you want to find faint galaxies, you simply increase the exposure time for your video device (CCD or photo film) and make sure your tracking is right. Then you just let it go and in effect do the same thing as is done here digitally.

Debunk yo mama. (-1, Troll)

Anonymous Coward | more than 11 years ago | (#6249357)

You don't know what you are talking about. I seen this shit. It works real good. The software is like some advanced shit that knows what you want with just a few keystrokes and shit. The NSA boys just hit a few buttons and the picture turns all blocky and then like it zooms in and gets like real sharp and shit. I've seen it even if they say like they don't really have that and shit.

Re:Debunk yo mama. (0)

Anonymous Coward | more than 11 years ago | (#6249380)

It's got pretty lame security. Any hacker that can access the Internet can connect to the NSA server and download top secret documents without even having to know the PASSWORD.

Re:Debunk yo mama. (0)

Anonymous Coward | more than 11 years ago | (#6249506)

Dude! Can you like get me some pictures of my neighbor's yard. SHe is like so hot and shit. And she sunbathes in the nude all the time. If you can email them to me that would be like awesome.

Re:Oh yea. I've seen this. (2, Informative)

hawkstone (233083) | more than 11 years ago | (#6249491)

I realize the root was probably a troll (the AC, not you). Also, the submitter was technically asking about nearby frames. Realizing that, however, I'll respond anyway, because I feel the technique is at least worth a mention:

There is nevertheless a form of super-resolution which works on standalone single frames. It depends on what you mean by "additional detail". Normally when magnifying images you would use nearest-neighbor, or better yet, bilinear interpolation. But a magnification using these methods will still look blocky.

Now suppose you first detect the edges in your lo-res image. When you magnify, use that information to determine which pieces of the newly created pixels should be colored what. You can use a similar concept for textured areas without edges. The result? Through some simple assumptions, you get much better magnification than through other schemes.

If a star in your original image is a 5-pixel plus pattern, a super-resolution image might turn it into a circle. Though not strictly accurate, this is in fact not a bad result in many situations. However, you're right that there is really no more detail; if a star was not visible in the original image, it never will be.

Found some more information (3, Informative)

MoOsEb0y (2177) | more than 11 years ago | (#6249367)

Ok, after googling for a little bit (I will never MSN anything!), I found a site who has done just what you were describing with MPEG video.

http://www.ai.mit.edu/~brussell/research/sres/data / [mit.edu]

Anyways, it seems that without proper filtering, the output looks REALLY weird. (look at they guy walking in a circle in front of the quilt) It seems that the motion vectors from the MPEG get taken in as part of the detected edges somehow! Thus, this would be most useful for uncompressed analog video as an input.

VideoFOCUS may already do that (1)

Tyrdium (670229) | more than 11 years ago | (#6249612)

VideoFOCUSLite is a limited edition of Salient Stills VideoFOCUS software. It is an affordable way to enjoy the full capabilities of Salient Stills patented image enhancing features. It does NOT offer several features of the full version including : DV capture, multi-session video capability, various workflow enhancing floating menus, free-hand elliptical and inverting masking tools, advanced preview management, improved video deinterlacing, ability to create a new video movie from a selection of video frames, de-multiplexing of digitized video movie.

This seems to be implying that it can take a video, run it through the process, and spit out a higher quality video...

Interesting but here are the facts (-1, Offtopic)

Anonymous Coward | more than 11 years ago | (#6251303)

Conventional thinking makes us belive that the interwoven geometry of Visual simulation using semi-interlaced projection is the best solution.This mode of Visual projection leads to reduced Tube life and definitly not withstood by the Liquid crystal devices. LCDs require full interlaced video signals. My two cents.

This isn't much help, but.... (1, Informative)

nzyank (623627) | more than 11 years ago | (#6251732)

When I worked at HP (R&D) we had some pretty cool stuff for doing very similar things. Most of it came from Purdue or Palo Alto and then we made it actually work. All patented as fast as we got them working and sometimes sooner. Betcha you're going to run into that in real life. Even if you think of something 'novel' the patent's probably already filed or granted for what you're wanting to do.

Covered on slashdot last year (1)

grundy (151557) | more than 11 years ago | (#6252097)

Here [slashdot.org] . Granted this is about creating high resolution stills, but read the ideas in the /. comments.

Fractal compression can do similar tricks (0)

Anonymous Coward | more than 11 years ago | (#6253950)

I remember years back seeing a demo by Prof Barnsley where he demonstrated a low resolution image of a parrot that he had compressed using fractal compression. The resultant image compared with the original could be blown up in size to much higher resolution than you would expect and it seemed to add more detail than there was in the original. There was talk at the time of using this as a method for upconverting old video material to high resolution for projection screens. Not really a cure for noisy video of course.

What kind of video? (1)

poptones (653660) | more than 11 years ago | (#6255544)

To what are you applying this? If you are trying to make high rez from bad video caps or a DVD, that's a whole 'nother animal than astronomy.

I've been working on this for some time and have had a decent amount of success using multiple captures of digital broadcast (broadcasts are still generally encoded on the fly, so each stream will be different). This can also be applied (with great success) to analog captures of broadcast video. Just sum each capture with equal weighting and you'll be surprised how much detail you can gain. Of course, this won't work at all if you are trying to improve a DVD, since you have only the one stream.

If you want to combine frames temporally, a temporal filter will do this - that's what it's for. I tend to dislike temporal filters because they add all sorts of other artifacts I find exceedingly objectionable, but if you are working on relatively low motion video you can gain a bit of detail like this. I can't comment on mac software because I don't have one, but I've worked a good bit on and with AVISynth and I know a couple of people have been working on temporal filters that use motion vectors. I rolled my own, but it's a macro op specific to avisynth and I don't know if you could duplicate all the specific filter behaviors using mac software. Maybe if someone would port ASVISynth to mac or linux...

Visar (1)

Elm Tree (17570) | more than 11 years ago | (#6256475)

What you're looking for can be done by VISAR as displayed here [exn.ca]
Although, as it's apparently only available though Intergraph I'm sure it's rather costly.
Check for New Comments
Slashdot Login

Need an Account?

Forgot your password?