bit-tech.net

An open letter to the graphics ecosystem

Comments 26 to 42 of 42

Reply
Meanmotion 21st October 2005, 21:14 Quote
Dude, i wasn't really suggesting you put 3dmark scores on there, it was demonstrative. I meant having a set of easy to run timedemos or single levels (so you're not there for hours) of the games of the moment - and a few older ones - so that you can quickly compare any number of cards. However, for the full review you can go more indepth. Something along those lines.
Tim S 21st October 2005, 21:16 Quote
Quote:
Originally Posted by Meanmotion
Dude, i wasn't really suggesting you put 3dmark scores on there, it was demonstrative. I meant having a set of easy to run timedemos or single levels (so you're not there for hours) of the games of the moment - and a few older ones - so that you can quickly compare any number of cards. However, for the full review you can go more indepth. Something along those lines.
We always attempt to include apples to apples, because not everyone likes the apples to oranges... I understand what you're saying now. I'll have a chew on it.

I'd need to think it through and work out how we can do it - there'd have to be a reference system set up that doesn't change.
WilHarris 21st October 2005, 21:59 Quote
On the subject of navel-gazing...

Yes, we've written about this subject a couple of times in the past couple of months. If you've read those articles, great stuff - glad you're with us. However, with the recent (high-profile) HardOCP editorial, we're at a point where if we just shout a bit louder, we might get lots more people involved.

As I write this, I'm reading a review on probably the biggest tech site in the world, where they attempt to draw authorative conclusions on the best cards to play a game with using a timedemo running with a no sound switch. How is that representative of what you guys will get when you go and buy a card?

So, apologies if you feel you've read this before. But we are trying to reach a wider audience with this.

Besides, there's plenty of other stuff on the site today, so don't feel like you've been deprived of your daily dose of bit-tech!
Da Dego 21st October 2005, 22:01 Quote
Quote:
Originally Posted by bigz
All that I asked in what you're referring to as a 'trumpet blowing session' is that motherboard manufacturers make the distinction between shipping products and pre-release hardware before it gets in to our hands. Pre release hardware shouldn't be shipped to a journalist in retail packaging, simple as.
Hehe...easy there, bigz. Either I'm misunderstood or you're blending my quoting him with my saying that. I said that I didn't know where it stood, it was just being talked about too much. :)

All I'm saying is that all of these articles have needed to turn into what BT is going to do about these industry problems. They have to, it's the inevidable conclusion to talking about review problems...how will our reviews correct this? I'm just saying that we're now up to three seperate and distinct articles, all of which focus on BT's new review methods when you get past the second or third paragraph (and that's being generous). I would like to see some different things covered, is all.
DivineSin 21st October 2005, 22:07 Quote
I think everyone stopped comparing 3Dmark scores when they found out that 3Dmark had a bias towards a certain company. It sure made me loose faith in any of there later releases, now i only look at 3Dmark as showing how beautifull a scene my card can render and for comparing me and my friends systems.
Sathy 22nd October 2005, 02:05 Quote
Was very nice to read some more detailed information about how you perform your tests, I've been wondering about it from time to time.

Also I would like to say that personally I like BT:s reviewing method better than any other sites, and I do read alot of different hardware related sites daily. Personally I'm pretty much had my share of those fps charts, especially after BT started using this now-already-familiar system. Luckily, for those who think that those fps charts are more important than what this real-world-testing-method tells us readers, you can find a dozen sites with that sort of information - be it that not all are too reliable.

About the mid-rage discussion:
It's true that it's hard nowdays to find reviews / tests that are run with average setups, and in that sense it would be a good idea to add that sort of testing to your repertoir. But as it has been mentioned here, it would take alot more time to test, and being on "the-bleeding-edge", well it would mix things up. There's no denying that alot of readers would like to see more affordable systems tested too, in the beginning of this year I too was searching for such reviews to get a clearer picture of what I should and what I shouldn't buy. Maybe I'm writing nonsence here - got a feeling that I might -, but eventhough I do agree that mid-range systems are too rarely used in reviews and that it would be a good way to reach more readers, I still think that the way BT reviews are done now - atleast for me - are more than enough to give enthusiast a more clearer view of is it good or not...but that's just my opinion.

About 'trumpet-blowing':
Since I value this 'innovative' method of reviewing graphics cards very highly, I really can't say I would've got an impression of BT's crew trying to 'blow-their-own-trumpets'.
I think the testing method is great, so why shouldn't BT crew think so too, because it's 'theirs'? It wouldn't be a good system if they themselves wouldn't have thought about it and fine tuned it to be as good as it is! So what's wrong with appreciating one's own 'innovations' so to speak?

But then again, that's just me.

I wish a good weekend for everyone!
Firehed 22nd October 2005, 02:31 Quote
I agree guys - a lot of review sites tend to try and maximize the card's potential rather than real-world use. Who the heck plays games with no sound? I mean, damn, I won't even play old DOS games without sound enabled (and yes, for the record, I'll break one out on occasion, a la dosbox). Many sites will cater to those who'll use LN2 just to break that record, alienating almost the entire gaming and enthusiast community.

Am I an enthusiast? Yes. Am I willing to spend a crapload of cash to have the best hardware? To a degree - if I can save cash by overclocking to the same point, I certainly will. Do I like squeezing every FPS out of my system? Sure, why not, I like to get my money's worth. Am I going to play games without sound and precalculated physics just to watch the number FRAPS gives me climb a tad higher? Umm... what do you think I am, retarded?

The only issue I see with the system as it is now is that it's quite strenuous to maintain, by the looks of it. Sure you could pubish reviews faster if you did timedemos, but I want to know how the hardware I'll be potentially buying performs in a realistic situtation, and I don't play timedemos. I play games, and that's where your numbers come from. I have different views on image quality - ie I'll take upping the resolution before upping the AA/AF, but I've played with enough settings to get the general idea whether I can knock off some filtering for a size bump.

Blow away, guys :P
careyd 22nd October 2005, 05:39 Quote
HEAR HEAR!!!

Wil Harris has just managed to collect and post all of my exact thoughts on the subject. While I haven't taken the time to post such, I have felt that Bit-Tech's methodologies of testing and reviewing GPUs of late is far and away the best methodology on the net.

I have, for years in fact!, stressed to other computing and gaming enthusiasts around me that I couldn't care less about hundreds of frames per second, but rather want image quality at any playable frame rate over 60 or 70.

Bottom line is, I think the best playable performance is the priority of the testing, but the inclusion of apples-to-apples comparisons in the review, while not the most important part of the review (as was stressed in the past), is still vital information for me, the consumer to factor in with all the rest of the information in the review to have a more complete understanding of the product.

Lazy consumers who just want to glance at charts need not reply...since the 'good stuff' from which you can develop a truly informed opinion is in the COPY, not the charts. Of course, the screenshot comparisons are vital and appreciated as well.

All this put together, and an open acknowledgement of the problems and trade-offs associated with your testing methodology is what makes bit-tech by far the most comprehensive, most trustworthy source for these reviews on the net. Kyle had it sorta right...but bit tech have got taken it way further and made it better.

Thanks to everyone at bit-tech for your excellent work!

Carey Dissmore
Dr. Strangelove 22nd October 2005, 16:13 Quote
I have to say I love the way you guys have started reviewing hardware! And now that you have given us a sneak peek into how much work is involved in your review process I'm just lost for words!

I have had enough of timedemo FPS hell, especially when authors go on about how one card is clearly better than another because it scores 3-5 fps more than the other, I would love to se how convinced they would be if they ran their timedemos a couple of times and put nice error bars on their graphs. This is especially bad when sites compare the same card from different manufactures.
K-man 22nd October 2005, 18:01 Quote
Great article. Finally someone realized what's important. I mean, who wants to play a game without sound? OK...you don't have a choice with Solitaire. :D

For the high-end system testing. I think it's absolutely ok to do so. It schows what the GPU is really capable of in "real world gaming". And if your own rig isn't that fast, then you must accept that you won't get the same results.

K
Tim S 22nd October 2005, 23:40 Quote
Quote:
Originally Posted by K-man
Great article. Finally someone realized what's important. I mean, who wants to play a game without sound? OK...you don't have a choice with Solitaire. :D

For the high-end system testing. I think it's absolutely ok to do so. It schows what the GPU is really capable of in "real world gaming". And if your own rig isn't that fast, then you must accept that you won't get the same results.

K
You probably will through 95% of the title - only the most intensive parts of the game (that we test) are where you're going to struggle a little. ;)
Ang3lFir3 23rd October 2005, 11:40 Quote
Well I don't post much on BT but I read a lot. And I must say as a mid-range afterwork and weekend gamer with a budget that I appreciate the new testing methods (they actually tell me something). I normally don't even look to upgrade until a title I REALLY want to play comes out that my card can't handle very well.......

However I would like to comment on a few things.... As bigz (i think it was) said AGP and Athlons are dead... but this isn't true.... I game on my 2500@2.4 w/512pc4000 just fine and often.....now since i have a now hard to find Epox 8rda6+pro (i kno you don't care but im telling you anyways) I get the pleasure of having AGP.... and you know what it still works..... and I can even buy some newer mid range cards in the AGP flavor.....and the real shocker...I'm not alone... I hear a lot of modest gamers who's real world concerns are "I gotta buy a new cpu and mobo just to go to PCI-E? screw that!"

So what does all that ranting have to do with midrange systems? Well I gotta say when I see reviews run on fx-57's I generally go look for something else...Its not at all representative of most people's "real world" (we are talking over $1,000 for a cpu..... I pay less than that a month in rent and car insurance combined). A decent mid range system say a 3000 at most would be an excellent comparision....nothing wrong with knowing once i OC i'll get better performance...i'd rather know that than know i'll get less performance and be less satisified....

(did i make any sense at all up there?)

and another thing i think would be of use to gamers that i think you can and may already infer from the data you are already collecting may be things like "average fps drop" (i know you don't want to count frames but it matters sometimes) ..... what i mean is that I as a mostly FPS (shooter) gamer sometimes suffer from FPS lagg (when the framerate drops drasticly and even a half sec of lagg can mean the difference between killing or being killed)... I would like to know how likely this is to occur since this definetly is part of my definition of "playable"....... also testing online games like BF2 and CSS etc online helps to determine the amount of playability i will really get when my machine is not just rendering the game but sending and recieving packets....

if i didn't make a damn bit of sense just ignore me.....

Lastly..... could someone possibly bring up the fact to game companies that those of us who can only honestly budget out a max $250(im having a hard time picking one cuz i still can't find a decent set of comprehensive reviews on all comparable card....ohh and it still needs to be AGP (go figure)) on a card upgrade every 9-12 mnths might actually like to get cards that play on today's titles..... (ie the min requirements for BF2 are outrageous imho)
Tim S 28th October 2005, 11:27 Quote
First off welcome to the forums Ang3lFir3. ;)
Quote:
Originally Posted by Ang3lFir3
Well I don't post much on BT but I read a lot. And I must say as a mid-range afterwork and weekend gamer with a budget that I appreciate the new testing methods (they actually tell me something). I normally don't even look to upgrade until a title I REALLY want to play comes out that my card can't handle very well.......

However I would like to comment on a few things.... As bigz (i think it was) said AGP and Athlons are dead... but this isn't true.... I game on my 2500@2.4 w/512pc4000 just fine and often.....now since i have a now hard to find Epox 8rda6+pro (i kno you don't care but im telling you anyways) I get the pleasure of having AGP.... and you know what it still works..... and I can even buy some newer mid range cards in the AGP flavor.....and the real shocker...I'm not alone... I hear a lot of modest gamers who's real world concerns are "I gotta buy a new cpu and mobo just to go to PCI-E? screw that!"

I understand your plight, but the problem is that manufacturers will be leaving you behind. I'm not sure about the mainstream and entry level cards, but there will be no more high end AGP parts. Socket A has been disbanded and nobody makes motherboards for that socket anymore.

Don't take this personally, but Athlon 64 has been around since September 2003 (over 2 years!), so the heart of your system is well over 2 years old now.
Quote:
So what does all that ranting have to do with midrange systems? Well I gotta say when I see reviews run on fx-57's I generally go look for something else...Its not at all representative of most people's "real world" (we are talking over $1,000 for a cpu..... I pay less than that a month in rent and car insurance combined). A decent mid range system say a 3000 at most would be an excellent comparision....nothing wrong with knowing once i OC i'll get better performance...i'd rather know that than know i'll get less performance and be less satisified....
We don't use FX-57 at its full speed - I clock it down to a "middle-of-the-range" Athlon 64 3700+ - 2.2GHz 1MB L2 Cache. Check on AMD's homepage and you'll see that the 3000+ doesn't even exist anymore. Sempron 2600+ is considered bottom-of-the-line entry level by AMD now, while 3200+ is the entry-level Athlon 64.

Athlon 64 3700+ is the middle of the Athlon 64 range, or the 3rd from slowest part in the entire Athlon 64 range that AMD are currently selling. I totally understand what you're saying, but the industry moves forwards at a frightening pace. It wasn't all that long ago when everyone was running 2500+'s because they were so cheap, but the 3200+ is pretty good value if you want a fast, overclockable gaming processor.
Quote:
Originally Posted by Ang3lFir3
and another thing i think would be of use to gamers that i think you can and may already infer from the data you are already collecting may be things like "average fps drop" (i know you don't want to count frames but it matters sometimes) ..... what i mean is that I as a mostly FPS (shooter) gamer sometimes suffer from FPS lagg (when the framerate drops drasticly and even a half sec of lagg can mean the difference between killing or being killed)... I would like to know how likely this is to occur since this definetly is part of my definition of "playable"....... also testing online games like BF2 and CSS etc online helps to determine the amount of playability i will really get when my machine is not just rendering the game but sending and recieving packets....
When we're doing our manual run throughs, that is something we take into account. I've been quite an avid first-person shooter fan since the mid-to-late 90's (ever since Doom 2, really speaking) and frame rate is certainly very important to me. For example, to this day, I still play Counter-Strike 1.6 at 640x480 -- I always have and always will, no matter what video card I'm playing the game on. Yes, before you ask, I've played at 640x480 on 7800 GTX SLI...

When I say something is 'smooth' it normally is unless I've highlighted a special case. I try not to do that, though, as I do spend quite a bit of time working out where the hardware shows its weak points in a particular title.
Meanmotion 28th October 2005, 19:39 Quote
Quote:
Originally Posted by bigz
For example, to this day, I still play Counter-Strike 1.6 at 640x480 -- I always have and always will, no matter what video card I'm playing the game on. Yes, before you ask, I've played at 640x480 on 7800 GTX SLI...

Lol, I'm an 800x600 man myself (might give 640x480 a try tonight). Can't beat never dropping below 100fps. ;)
Tim S 28th October 2005, 20:17 Quote
I've got another rig that I'm going to add in to the fray soon.... it's based on a Sempron 2600+. I'll be using that for entry-level reviews. ;)
[USRF]Obiwan 29th October 2005, 10:12 Quote
I just played the "Lost Coast" level of hl2. in a blazing 1600x1200 (my native lcd display res) with 16x anisotropic and 4x aa and all settings on "HIGH. There must be something wrong with my system because i only have a 3500 amd64 and a 6800GT running on a asus an8 with 1gb memory. It looks unbelievable slick, smooth and feels amazingly real. It almost wants to make you, to go to a church again to to compare the real deal with the virtual lost coast one :D
Michel Behna 31st October 2005, 20:17 Quote
A minimum frame rate of 24FPS should be mandatory for all cards. I am basing this on the fact that movies and film use that standard.

What I want in a graphics card review is the following:
1. what is the minimum fps at a certain resolution with all effects / no effects.
2. what is the maximum fps at a certain resolution with all effects / no effects.
3. how does turning on and off effects and features affect fps.
4. how does the image quality compare to another card.
5. Any issues with a particular card/vendor?

The fact that a card can achieve 120 fps or 176 fps is irrelevant to "real-world" performance. All that allows me to do is to "brag". Also make sure that when testing, to indicate whether the performance is bottlenecked by the CPU, GPU and/or the game/tool. Testing should be done using loops and gameplay.

I surmise that the reason most sites/reviews have focused on FPS alone is the ease/speed with which such reviews/evaluations can be produced.
Log in

You are not logged in, please login with your forum account below. If you don't already have an account please register to start contributing.



Discuss in the forums