FPS Benchmarks No More? New Methods Reveal Deeper GPU Issues 125
crookedvulture writes "Graphics hardware reviews have long used frames per second to measure performance. The thing is, an awful lot of frames are generated in a single second. Calculating the FPS can mask brief moments of perceptible stuttering that only a closer inspection of individual frame times can quantify. This article explores the subject in much greater detail. Along the way, it also effectively illustrates the 'micro-stuttering' attributed to multi-GPU solutions like SLI and CrossFire. AMD and Nvidia both concede that stuttering is a real problem for modern graphics hardware, and benchmarking methods may need to change to properly take it into account."
Egad (Score:2)
Re: (Score:3)
Re: (Score:2)
Or perhaps the author has no background or experience with statistics, and wouldn't know what is meaningful about one or the other.
Re: (Score:1)
When I was an engineer, I presented everything to my management in text and numbers that "any idiot" could quickly understand. Just a couple of slides with the essential bullet points and some "standard deviations and percentiles". I never knew why my managers ended up getting frustrated, bored, or began working "on other projects" d
Re: (Score:2)
It's called "make it shiny." As an engineer/tech guy myself, watching PP presentations by manager types make me want to either barf or put my head through the table. They violate basically every single rule of graphical presentation. Yet "it's shiny," so everyone is happy.
Is a multi-GPU problem. (Score:3)
Our eyes detect 'deltas' better than 'speeds', so if the odd numbered frames have a delay shorter than others, our eye will detect it. But this only affect setups with multiple GPU's. And is easy to fix. Just calculate the delta of the latest frame, and force the same delta, maybe use a buffer. This is not a problem once has ben detected, It may need some minor changes on engines, but thats all. IMHO.
Re: (Score:2)
Right up until some disk IO causes your last frame to be 2 seconds long, now every frame in the future is forced to 2 seconds between updates! Awesome for the win.
Its only slightly (and by slightly, I mean a lot) more complicated than you think it is.
Re: (Score:2)
Outlier filtering is hardly complicated.
A smarter version of vsync (Score:2)
Re: (Score:2)
That's the weird thing... on Starcraft 2 (from the article), the jitter was more pronounced on the single chips than in the multi-GPU configurations. It's not that simple ;)
Re: (Score:2)
implying a single GPU is even a single processing core any longer
Re: (Score:2)
Actually, if you read TFA, it's not only multi-GPU setups doing it. Also, the 'solution' you describe has been used by Nvidia since the GeForce 8 era. They call it 'frame metering', and it's not a perfect solution either.
Consistency (Score:1)
tldr; benchmarks ignore consistency in their measurements and are therefore nonscientific marketing devices.
Feel of a given fps value (Score:5, Informative)
After all, your average geek tends to know that movies happen at 24 FPS
Movies happen at a motion-blurred 24 fps. Video games could use an accumulation buffer (or whatever they call it in newer versions of OpenGL and Direct3D) to simulate motion blur, but I don't know if any do.
and television at 30 FPS
Due to interlacing, TV is either 24 fps, when a show is filmed and telecined, or a hybrid between 30 and 60 fps, when a show is shot live or on video. Interlaced video can be thought of as having two frame rates in a single image: parts in motion run at 60 fps and half vertical resolution, while parts not in motion run at 30 fps and full resolution. It's up to the deinterlacer in the receiver's DSP to find which parts are which using various field-to-field correlation heuristics.
and any PC gamer who has done any tuning probably has a sense of how different frame rates "feel" in action.
Because of the lack of motion blur, 24 game fps doesn't feel like 24 movie fps. And because of interlacing, TV feels a lot more like 60 game fps than 30.
Re: (Score:2)
Movies happen at a motion-blurred 24 fps. Video games could use an accumulation buffer (or whatever they call it in newer versions of OpenGL and Direct3D) to simulate motion blur, but I don't know if any do.
Forced Unleashed 2 uses a technique similar to this, but there are probably others that do, too. It renders frames at about 30 frames a second, but updates the screen at a stable 60. It uses an interpolating motion blur to make the gameplay feel nice and smooth. This allows for more geometry to be drawn while still providing a "good" user experience. It is slightly bit different than a simple accumulation blur since they predict the motion when doing the blur instead of simply blurring previous frames.
Re: (Score:2)
The 24 fps rate is not because the human eye can't see faster than that (it can), it's a compromise due to technology limitations nearly 100 years ago.
Except for "special effects" I prefer that the stuff be updated much faster and let our eyes do the motion blurring. And I don't really like blurred special effects.
I dislike movie scenes where s
Re: (Score:2)
When you look at a fast moving object it's sharp (unless it's moving so fast that your eyes can't track it).
But the background will be blurred. Motion blurring usually has it's place, if done well you will not even notice it, it just seems natural. Of course that often involves no effects at all, the camera just uses the same limitations as your eyes. If you were to use a very high shutter speed in a scene with a fast moving object it will seem strange, your eyes won't do the motion blurring right because they're fed a completely sharp image to begin with, which is not what happens when your eyes pan in a real sc
Re: (Score:2)
The problem is that "film" based cameras can not film in infinite depth of field. This makes it rather impossible to do what you asking for in outdoor shots. for indoor or limited depth bright scenes it may be possible to use a higher Fstop and trade light amount for higher depth of field.
Anyways fully rendered stuff can be rendered with perfect camera separation, and infinite depth of field.
Re: (Score:2)
My complaint is about the situation where movie directors artificially and unnecessarily make bits of the picture out of focus. Sometimes there are technological limitations, but often nowadays most of the scene is rendered and then artificially blurred (e.g. Avatar). Sometim
Re: (Score:1)
1) You miss the point. The background might be blurred, but that's when I am not looking at it (e.g. not interested). BUT if I choose to look at it, it stops being blurred. At what is "background" depends on what/where I choose to look.
Maybe I misunderstood you. I agree that it shouldn't be overdone, but I think that if you were to remove motion blur (MB) altogether from such a scene it would look very strange. Try enabling/disabling MB and play a suitable scene in a modern video game. It *will* look strange without MB, also with high fps/resolution on a fast monitor. I don't think that having everything completely sharp will work (if that's what you mean).
My complaint is about the situation where movie directors artificially and unnecessarily make bits of the picture out of focus. Sometimes there are technological limitations, but often nowadays most of the scene is rendered and then artificially blurred (e.g. Avatar). Sometimes it's part of the story, or for good effect, in which case I don't mind.
It hurts my eyes when it's overdone.
Exactly.
2) I know they are not the same, I'm just complaining about them all. I wrote: "out of focus or motion blurred". I dislike both effects. And I also dislike lower frame rates (which is another separate thing - the low-frame-rate frames could be perfectly sharp and in focus).
I think we're agreeing in principle, but I took what you wrote to mean "do
Re: (Score:2)
to simulate motion blur, but I don't know if any do.
Pretty much all modern games use motion blur. Some of course use it better then others and it is often not quite as high quality as one might want it to be, but motion blur itself is almost everywhere these days (most noticeable when swinging the camera around in a third person shooter).
Re: (Score:1)
It's way more than motion blue that makes movies tolerable at 24 fps - motion blur is just one sign of the fundamental difference between recording and animation. If an animation runs at 60 fps, you are getting information about 60 frames in one second. This is not true for video - each frame of the 24 are actually an average over many million frames (look up the Planck time on Google - the fps is actually far more than a million). The universe runs at a tremendous amount of fps, and the 24 frames you see i
Re: (Score:2)
Did Voodoo cards apply any sort of motion blur? I've long noticed this stuttering, e.g. when circle strafing around a corner it's pretty easy to see that the corner doesn't move smoothly. It even happens when running very non-demanding games (prboom, openarena, UT, etc) at 75hz under vsync with modern hardware under any OS.
The only hardware I've ever used where I didn't see this effect is my Voodoo 2 setup. Those things are smooth as butter, as long as you don't throw too many polygons at them.
Re: (Score:2)
Again, an over simplification.
Movies are shot at 24 fps with a 1/180 second exposure time. When they make the exposure time longer, you notice and it gets hard to watch. Listen to the commentary by Joss Whedon on Serenity for more, but the scene on the reaver planet are shot at 1/120 and are a little unsettling. Things like panning include blur. the scene in Saving Private Ryan on the beach are shot at 1/60 and are noticeably difficult to watch--when the camera moves, everything blurs. When something flys b
Re: (Score:1)
Re: (Score:2)
Nope, unintentional.
Re: (Score:2)
Re: (Score:2)
I'm guessing you've not played any multitrheaded games then?
The controls are plenty responsive. The visual feedback is not.
Re: (Score:1)
Progessive is 25 or 30 fps (Score:1)
Note: Not all TV shows are at 50 or 60 Hz. Some TV shows use the progressive encoding scheme that halves framerate.
The two fields of interlace are hijacked to form a single still image. It's a single image built from two passes. Hence the name "progressive".
They sacrifice framerate for the sales pitch. I don't know why though. Possibly to make the freeze-frame look better, dunno.
The "film look" (Score:2)
They sacrifice framerate for the sales pitch. I don't know why though.
TV producers use film to make the show appear more cinematic. Audiences are used to seeing film-like frame rates and other artifacts of film for scripted programming as opposed to video for "reality" programming such as news and sports.
Possibly to make the freeze-frame look better
The freeze-frame doesn't look better when you hit it at the wrong field dominance.
What's more (Score:2)
You can sure as hell tell the difference between 24fps and 60fps video. It is real, REAL easy to see. I have an AVCHD cam that will shoot 24p, 30p, or 60p (HDC-TM900 if you are interested) and it is amazing how smooth the 60p video looks. You just don't get that kind of smoothness in movies.
Some people don't like it, they think it looks "too smooth" and "not cinematic". In other words, they've gotten used to the choppiness of 24fps and associate that with movies.
What's more, there's a market in TVs that try
Ahh, complexity... (Score:2)
You'll need the FPS value, as before, (ideally with a worst-case FPS reported); but you'll also want a measure of the deviation of every frame's draw time from the average draw time being reported. And likely a measure of how atypically bad frames are distributed(ie. 5 seconds of super-low framerate during some sort of loading is annoying. 20 25 millisecond frames scattered throught action-h
Re: (Score:2)
Re: (Score:2)
Even in the realm of game benchmarking, you can see some pretty dramatic differences, between engines, in how Nvidia's approach or ATI's ap
Re: (Score:2)
Ah, back to the good old days of 3D graphics, when output was measured in polygons per second, and all the jiggling about, until the big boys all standardized on 50 pixel polygons, gouraud-shaded, at a resolution of 1280x1024, with a colour depth of 24 bits... And then more stuff was tacked on...
Well, ok, almost all the big boys... Intergraph kept trying to pass 25 pixel polygon performance numbers as a valid comparison to 50 pixel polygons....
Re: (Score:2)
you'll also want a measure of the deviation of every frame's draw time from the average draw time being reported. And likely a measure of how atypically bad frames are distributed(ie. 5 seconds of super-low framerate during some sort of loading is annoying. 20 25 millisecond frames scattered throught action-heavy areas is really annoying...)
As a non-stats-guy I hereby invent the standard-deviation-for-slow-down-only-including-outliers metric. You're welcome :)
But seriously, couldn't they just include the whole distribution graph of time between frames, with a description of what it means? And maybe provide the underlying data as a spreadsheet so that you can run whatever analysis you're most comfortable with on it. Oh well, back to stats class.
This post is going nowhere, so I'm gonna save it for posterity.
The point of FPS-benchmarks (Score:1)
The point of doing a FPS-benchmark is to reveal how the graphic card performs in the games that most people play. People don't care about the theoretical performance. They just want to know if it can run the new cool game with the good graphics. Either a game renders fast enough, or it is so slow that you can't turn out the special effects that makes the game look really good. It's all about the game. The other stuff is not so important.
Re: (Score:2)
The point of doing a FPS-benchmark is to reveal how the graphic card performs in the games that most people play.
What about people who don't play a lot of first-person shooters? Some of these benchmarks measure only fps in FPS, not fps in other genres.
Either a game renders fast enough, or it is so slow that you can't turn out the special effects that makes the game look really good.
The point of the article is that an isolated frame that takes too long to render can jerk one out of being absorbed in the effects.
Re: (Score:2)
Wrong.
The point of an FPS benchmark is to get a higher number than some guy on a forum so you can brag about it.
FPS has never been about how well a system can render a scene because its a shitty measure of it. It ignores quality, complexity, accuracy, stuttering (why is this supposed to be new? GPU stuttering is less noticable than disk IO causing jitter, and NOW the framerate no longer matters? Thats just dumb.
There is nothing 'new' about the frame rate being a shitty method of rating performance, well,
What fraction of GPUs are used for graphics? (Score:2)
New benchmark: percentage of frames rendered at (Score:2)
less than 60 fps.
kthxbye.
Re: (Score:2)
Re: (Score:2)
Re: (Score:1)
I agree, I am not sure that 99 percentile is working or not now, but that still allows for a bad frame every other second.
More useful would be the average of that worse 1%, or to crank it up to 99.9% (bad frame every twenty seconds), or longer, to some acceptable rate.
Even if 99 percentile works now, it's ripe for abuse by the manufacturers if it becomes a common metric (I imagine anyway there could be trade-offs in a driver where 1 frame every 2 seconds is terrible, rather tan one frame a second being OK,
60 FPS (Score:1)
Don't you only need 60 FPS to have the illusion of animation anyway?
Re: (Score:2)
Just for the illusion of motion? That's around 12fps, which is what cartoons use. 60fps is the start of fairly smooth motion, but 120hz is in uncanny valley territory for me. Too smooth, but not smooth enough.
Re: (Score:3)
Are you sure about this, or are you basing it on experience with really poor TV frame interpolation? That's not 120Hz. That's more like 30Hz with lots of fake frames.
Re: (Score:2)
You have a point on the fake frames. I don't have anything capable of generating 120Hz as source material, or for displaying it. So in-store demo is really all I have. I suppose I'd have to watch the 120Hz interpolation in slow motion to see if the artifacts are the cause. Maybe it's because too much motion blur is missing after the interpolation (more than should be for that amount of interpolation).
Re: (Score:2)
We got an LG 3D TV at work which has LG's truemotion. At first glance, with slow FPS material this does 'smooth' the image a lot - the slow FPS stuff has noticeable flicker almost like a cinema screen - the truemotion looks more 'solid'. After a few seconds you see how they've done it - it's almost like MPEG motion detection - you will see certain blocks that move across the screen. Quite often this is effective, but some material confuses it - in one sequence following a bicycle down the street the grav
Re: (Score:2)
What I notice most on these things is how mangled mouths move during speech with the interpolation on. I have TVs that will do it, media players that will do it, etc. It all comes down to being creeped out by the fake mouth motions and I shut it off. *some* types of motion does look better, but the things that look wrong, look wrong enough that there's no way I'll use it. Give me a higher framerate source without trying to interpolate... Please.
Comment removed (Score:5, Informative)
Re: (Score:2)
Re: (Score:2)
Actually he measured the rendering just fine. But what nVidia told him is that they're doing some timing magic before they display it in SLI setups, which is currently not possible to measure with FRAPS or any of the other standard FPS tools. So right now you would need to get a high-speed camera to snap pictures of the screen to know what the user sees. But regardless of that they can't get rid of all the stuttering that easily, the slowest frame still takes much longer to render than the average. Also thi
Re:Obviously, no one read TFA (Score:5, Informative)
Re: (Score:2)
You should probably review videocards giving your insight, most tech sights (even anand) still just runs canned FPS games. Maybe you can even make a little $ from doing articles.
Re: (Score:1)
I'd like your five minutes and mine for taking the time to read your response too.
Re: (Score:1)
Disk IO (Score:1)
Re: (Score:1)
1) Pagefile/swapping. - the os thinks it is from the 90s and trying to do memory swapping (no matter how much ram you have).
2) disk controller drivers run in kernel mode. any rise in disk queue length (pending requests) hangs the system. arggh.
Re: (Score:2)
I'm sorry, what OS are you using to which these things don't apply to? OSX, Linux, Windows all are exactly the same here, they all do these things (well, except for hang the system, unless of course we're hanging because we're waiting on something to be paged so the app we're running appears frozen), and are expected too, for obvious reasons. Well, clearly not obvious to you.
So again, what OS are you using, that those things don't apply to? QNX or BeOS?
Re: (Score:1)
Most of the time when I notice any stuttering is also the same time my hard drive lights up
As the drive(s) get busy, pending operations get queued up (disk queue depth) and as such, the machine starts "hanging" waiting for the IOs to finish.
unless of course we're hanging because we're waiting on something to be paged so the app we're running appears frozen), and are expected too, for obvious reasons.
It is obvious. The point here is that the paging operation of the machine makes the problem worse. Especially if you have slow disk(s), or the pagefile/swap space dynamically changes.
primarily, linux does a much better job about not paging unless it starts running out of memory. whereas windows seems to start using it as soon as it boots.
Re: (Score:1)
Re: (Score:2)
Re: (Score:2)
Erm, isn't that what we already do, at least on Linux? I mean, the application simply uses read/write and possibly seek against an arbitrary filename - it's up to the OS and FS to handle actually getting or saving the data. Which means that the application doesn't know or care whether the file's on spinning media, a slow flash drive, SSD or cached in system ram!
The only thing that may not be optimal is that we have the sync() function, which usually won't return until the data has actually been saved to dis
Re: (Score:2)
Re: (Score:2)
Actually, isn't that what the whole debacle about ionice etc is for? Being able to give some tasks low priority, depending on command or w/e.
Admittedly, I was really disappointed with the stock behaivior on Debian - Copying over many GB of files with cp to a network share basically brought things to a standstill. Worst part was that it would attempt to aggressively cache these files, until it r out of spare ram and then swapping inactive applications. So, leaving the PC for a couple of hours... Yeah.
Fixed t
ugh (Score:2)
FWIW, FPS is still a fine benchmark. Like any benchmark, it only tells part of the story. That's why you use tools like 3dmark that run a battery of benchmarks to aggregate a rating, and then measure actual
Poor pressentation, but good content... (Score:2)
I find he content and discussion very interesting. For me, this was something obvious because of my line of work, but I can imagine that most people reading (and writing) GPU reviews had no clue what so ever about this.
As much as I find the content interesting, its presentation is awful. Although is is interesting the present some figures on a frame-count base, most of the overview figures should be on equivalent time base, allowing a proper comparison of the tests sequences. I'd have shown one frame based
Re: (Score:2)
Re: (Score:2)
i agree with you, much like the recent AMD bulldozer discussion where some anandtech benches were linked to. The slightly higehr clocked non-turbo quads beating the 6 ways in x264 encodes... seems to me they need to turn some more x264 options on.
No great loss. Benchmarks are BS. (Score:1)
The only way I have ever been able to test what real performance will be like in a given game or rendering in a given program is to play that game or render in that program. Even built-in benchmarks like in HL2 don't seem to take gameplay into account well enough. While (at best) benchmarks can be a help in deciding what to buy in a very general way, I have learned to be skeptical and trust my experience only. Even framerate monitors in games often don't reflect the smoothness of the experience of the game.
Inverse measure is what we want (Score:2)
I'm glad somebody started looking at ms per frame instead of frames per second. Since what we really care about for game performance is whether frames are rendered quickly enough to give satisfactory reaction times etc, using frames per second is misleading.
Another example where the same thing happens is fuel consumption: we keep talking about miles per gallon, but what we primarily care about is the fuel consumed in our driving, not the driving we can do on a given amount of fuel, so this is misleading. To
Re: (Score:2)
Changing from frames/sec to msec/frame doesn't fix the problem at all.
I was playing Minecraft the other day with a buggy mod installed, and was getting 240 fps, but choppy performance. Sometimes I'd get 1 second spikes, and the fps monitor would change to show something like 30fps before creeping back up to 240. If you convert that to msec/frame, those numbers still look really good.
Other games that run at 50fps looked much better than this buggy minecraft mod. Taking the inverse (as you suggest) doesn't
Re: (Score:2)
Of course just looking at an average of either quantity isn't going to be sufficient- you need to look at the distribution of values, not just the mean. (Looking at the 99th percentile, as they did in the article, is a start.) But at least this way we're looking at the distribution of the right numbers.
Pretty sure (Score:1)
Another source (Score:1)
FPS vs refresh rate (Score:2)
Why would anyone need a framerate faster then the refresh rate of the display refresh rate you're using?
I've never understood why anyone would push a graphics card faster then the refresh rate of the display you're using. Why not just cap it off at the max refresh rate, and let the card take more time in rendering each frame.....
It seems as though there should be some sort of "dynamic rendering" option. You want the framerate to match the refresh rate of the monitor, so why can't the rendering engine deci
COD Black Ops embodies this article (Score:1)
Re: (Score:1)
what about the standard deviation of the duration between frames?
Re: time/frame (Score:4, Informative)
The Crysis test loop measures the slowest frame, starting with the second loop (to avoid measuring disk performance). That "minimum FPS" number is what I personally use to benchmark graphics cards - it has always been the speed through the slow-to-render part of the map that matters.
Re: (Score:2)
This would work, though you should make sure to cut the first few seconds off. In just about every graphics benchmark I've ever seen, the initial few frames stutter madly as the process starts before the GPU has finished loading everything into VRAM.
Re: (Score:2)
Yes, the Crysis benchmark ignores the entire first test loop. (Didn't I just say that?)
Re: (Score:2)
You did... somehow my brain skipped right over that. Apologies.
Re: (Score:2)
I notice Crysis is a game that suffers strongly from the problem the article talks about. It runs smoothly most of the time and then occasionally, it'll bog down for a small fraction of a second, apparently skipping a few frames.
Also some games seem to slow down no matter what computer you run them on. GTA3, GTA:SA and NFS: Undercover all do this if there are too many cars nearby. You can fiddle with the graphics settings all you want, same behavior.
Re: (Score:2)
Also some games seem to slow down no matter what computer you run them on. GTA3, GTA:SA and NFS: Undercover all do this if there are too many cars nearby. You can fiddle with the graphics settings all you want, same behavior.
Sounds like it might be a CPU issue, not graphical. I would see the same thing running L4D on my 1201N [amazon.com], the ION was perfectly happy running at reasonable settings, but as soon as a horde came, the poor little Atom N330 choked and I'd be running at 10 FPS, no matter what the graphical settings were.
Re: (Score:3)
At least the GTA games exhibit the same behavior on computers that are ridiculously overpowered for those games (2x+ recommended processing power), and I'd bet the same will prove true of NFS:Uc - that slowdown happened on an i7 940. So it's not something you can take care of by throwing more processing power at the problem.
The L4D games all run silky smooth on my i7 940 system, so I don't think it's the same kind of problem.
Re: (Score:2)
This.
min and max ms between frames, mean, and standard deviation.
Re: (Score:2)
Yeah, having written multi-GPU drivers I'm amazed any of this stuff ever works, let alone actually improves performance. Back in our day there were a ton of things that game developers could do that would cut the performance to a tenth or less of what a single card would achieve due to triggering massive amounts of inter-card communication.
Re: (Score:2)
Voodoo 2 SLI is as good as advertised. Twice the FPS, higher resolution support, and it "just works". No stability problems, and the cards work fine with passive cooling.
I'd agree that what nVidia did with SLI has all been garbage though.
Re: (Score:2)
Voodoo 2 SLI is as good as advertised. Twice the FPS, higher resolution support, and it "just works".
That's because the Voodoo-2 was a cut-down piece of crap even in its heyday. You literally couldn't do anything with it that would make SLI difficult.
Most people I knew in graphics and gaming at the time hated 3dfx for crippling the industry with its refusal to add any features that weren't required to render Quake fast.
Re: (Score:2)
The Voodoo 2 still has the prettiest output of any graphics card I've ever seen. It may not have all the features you want, but what it has it does really well.
You have a point though. 3dfx didn't do much to hold onto its market leader position and paid the price.
Re: (Score:1)
Re: (Score:2)
No, because its consistent, every frame, at vsync and it doesn't hold back the rendering process.
Re: (Score:3)
Well played Sir. I used my overclocked Radeon HD 3650 to post this (almost as quick) reply...
by Anonymous Coward on Friday September 09, @10:10AM
by webmistressrachel (903577) Alter Relationship on Friday September 09, @10:39AM
19 minutes? Sounds about right.
Re: (Score:2)
Yea, or 29 minutes ... depending on if you want correct math or not. Hope thats not what you're using that GPU for :/
Re: (Score:2)
Re: (Score:1)
That was my point... whoosh...
Re: (Score:2)
Do you use a wireless mouse? Some of them update only at about 60Hz, so you'll not see the benefit in window dragging. And even with a wired mouse, the default 100Hz (Linux) or 125Hz (Windows) mouse update rate isn't fast enough to guarantee updating every frame on a 120Hz monitor, so you'll need to change mousepoll (Linux) or install a hacked hidusbf with test certificate (Windows). It's also possible your window compositing system is capped at 60fps. IIRC correctly Compiz does that unless you change some
Re: (Score:1)