An open letter to the graphics ecosystem

Good article I must say. Really puts it into perspective about how much work goes on behind the scenes of bit-tech! Can't say I have any answers, but it was a good read, thanks!

Well, I don't have the definitive answer to problems, but a good article, I do agree on the bit-tech method, and who knows, this article might lead to better games?

You mentioned about using mid-range systems or not, tbh I think a lot of people here have pretty much high end systems, I know you have limited time, so perhaps some of the forum members here with mid range systems could produce their own reviews using their system? Gives you more time to concentrate on creating reviews on high end systems?

Also, It would be interesting to know how a day goes in the life of a bit-tech staff, or the process you go through from hearing about a new graphics card or game, to the point you finish creating the article.

Nice article ;)

All makes sense, I really like the idea of real world testing. Its so annoying when you have to look at a bunch of specs and graphs and it doesnt actually answer the question, is this one actaully worth spending 25% on? is it really that much quicker?

Im not sure if its worth using mid range systems aswel, it would be a nice addition but I dunno if its worth it at your end, maybe if it was just a brief test just to find if there any potential bottlenecks. I suppose if your testing a 6600 GT its seems a bit stupid to stick it in a 2+ grand computer when your trying to do real world tests, it may work really well in it, but then when somones fits it in there fairly average pc runs nowhere near what they expected.

I've been trying to match CPUs up with graphics cards for a while now - it doesn't make sense to do 'real world' testing without using a CPU that is reasonably realistic.

I do not think that many reviews on hardware review sites, are testing realworld benchmarks. They say they are, thats true. But when i take a look at the spec list of the test system. I am thinking: woooott!? :(

Only 1 of the 1000 people who play games have a gamepc with an overclocked on crack: fx75, 2x 7800GTX EE in dual 16x SLI and 2 gigs of superiour timed ram. What are they thinking with realworld?

Well... Not my "real" world.

I tend to appreciate that mid-range systems are tested, and if they weren't I'd probably stop reading the reviews. Many people on these boards are enthusiasts that still have the 6800LE and 6600GT...and there are quite a few of us who are opposed to honestly going out and dropping $400 on a graphics card. So though nobody really needs to see FEAR running on a 6200, the mid-range setups still provide valuable information to many of us.

Originally Posted by [USRF]Obiwan
I do not think that many reviews on hardware review sites, are testing realworld benchmarks. They say they are, thats true. But when i take a look at the spec list of the test system. I am thinking: woooott!? :(

Only 1 of the 1000 people who play games have a gamepc with an overclocked on crack: fx75, 2x 7800GTX EE in dual 16x SLI and 2 gigs of superiour timed ram. What are they thinking with realworld?

Well... Not my "real" world.

I'm trying to grasp what you're saying here - whether this is a general comment about 'other' review sites, or about ourselves. Don't forget that our job is to 'show you' the newest products that are released - not just the fastest, the slower, more affordable ones too...

So, what would you like us to 'review' a GeForce 7800 GTX/Radeon X1800XT on? The slowest Sempron or Celeron that we can get our hands on with 256MB of your finest 'major' memory on the cheapest PCI-Express motherboard that we can find?

You may see the system specification for one of those reviews and think... "omfg, wtf" :|

That is because they are the fastest-available video cards around and we try and balance our system against what we're reviewing. When I look at something a bit more 'mid-range', like, say, the X800GT that was linked... We paired that with a '3700+'... ok, maybe a little bit on the high side from the PR ratings, but I'd hazard to guess that a lot of readers out there have Athlon 64 3000+'s overclocked higher than our 3700+, running at a steady 2.2GHz.

I'm just interested in yours and everybody elses thoughts - we stive to make things even better than they are now. If the information isn't useful and relevant, then we're barking up the wrong tree. :)

I'm not having a go, I'm merely trying to find more and more feedback to help make our systems a little bit more 'realistic' and balanced for what you guys would buy.

Good article. And I agree about more mid-end type review setups. Like me, I'm on a 2500+ with a 9800pro and a gig of Corsair value ram. Not exactly bleeding edge.

Yer id call mid-end an oc'd mobile barton/3000+ A64 (2ghz min), gig of branded ram.

The question is whether or not your going by price\components spec or actual performance, if you got a phased mobile barton at 3.0ghz with 2gig of ultra low latency ram i'd call that high end.

I suppose you could do what you done before and just use a lower clock and slacker timings.

nah, the barton core is history... just like AGP ;)

Yer true, but some of us are still stuck with that kit :( ... until the end of the month hopefully :D

yes, but with the way that the industry goes... anything that isn't Athlon 64 or Pentium 4 and PCI-Express based could well be considered legacy seeing as those two have been around for ~15-18 months now. ;)

I think the way forward for bit-tech would be to have a database of hardware that you can use to pull up a comparison with whatever cards you want. So if i wanna know if it's worth upgrading my old 8500, say, i can pull up a whole bunch of new cards (and some not so new) and compare them myself. If it's done well it'll be by far the best way of knowing what's the best course of action. That way you can make the initial review fairly basic - one reasonably high end rig tested in the way you do now - but then have a few raw numbers to throw in the db for stuff like timedemos, 3dmark, etc.

Hmmm, i was going to elaborate more and do a quick edit, but dinner's ready!

Originally Posted by bigz

So, what would you like us to 'review' a GeForce 7800 GTX/Radeon X1800XT on? The slowest Sempron or Celeron that we can get our hands on with 256MB of your finest 'major' memory on the cheapest PCI-Express motherboard that we can find?

Well... That would be something kewl to do!
Do not forget that users see/read a review about the fastest and laters videocard. And its comming out the benches like it is "the 10th wonder of the world". People buy this card and slap it into their system...

But wait! this system does not have the latest and greatest money can buy hardware inside to actualy make use of the 10th wonder...

See, It makes no sence that you need to buy an new proc, memory, mainboard and such, just the keep the also very expensive videocard running like the benches say it would. Maybe some people are diving in their money-pit and can do this, but most "normal" people do not. They just insert the card into the socket a amd athlon3000+ pc they bought last christmass.

I hope you understand what I am trying to point out here. (my english is not perfect but i am trying to)

I don't think we will be returning to the past in the way that we do our reviews, so there will not be any time demos, and no 3DMark results either. A database would be nice, but seeing as we focus on actual gameplay with systems that are reasonably representative of what we would say the 'fastest' system someone is likely to pair a particular video card with, it is increasingly difficult to actually put a huge comparison table down on paper.

I have some ideas as to how to cater for your needs, but in practice, I feel that there's at least another full time job required for both implementation and keeping it up to date...

My idea would revolve around the resolution you like to game at and then we could list a choice of video cards based on what we found at the time... again, the only problem is keeping it up to date when new games come out. I don't have the time to evaluate every video card we've ever reviewed (again) when new games come out that are more demanding (like FEAR, for example...)

Originally Posted by [USRF]Obiwan
Well... That would be something kewl to do!
Do not forget that users see/read a review about the fastest and laters videocard. And its comming out the benches like it is "the 10th wonder of the world". People buy this card and slap it into their system...

But wait! this system does not have the latest and greatest money can buy hardware inside to actualy make use of the 10th wonder...

See, It makes no sence that you need to buy an new proc, memory, mainboard and such, just the keep the also very expensive videocard running like the benches say it would. Maybe some people are diving in their money-pit and can do this, but most "normal" people do not. They just insert the card into the socket a amd athlon3000+ pc they bought last christmass.

I hope you understand what I am trying to point out here. (my english is not perfect but i am trying to)

If I remember back, people have found that our settings are actually below what they play the game at with slower hardware than what we were using in our testing system. People have always asked why our settings are so low, its because we're more concerned about worst-case than best-case, even if the system we're using to test the video cards is at the very high end. :)

Nice article, I really approve of the way gfx cards are reviewed at Bit and [H] now. I would *really* like to see more reviews of cards in what I would call the lower mid-range section - £40 to £80, as I think this is what a lot of people, especially those who like playing games now and again and like eyecandy but aren't prepared to pay £££'s for.

A big round-up of modern games on the likes of 9600 Pro\XT, X600, X700, GeForce 5700, 6200, 6600 etc, with lowest prices on these cards found from UK sites, would be brilliant. Setup on a more mid-range system - a 3000 Venice\2*512Mb would seem most appropriate to me. This information seems really hard to find in one place, yet I expect its the range of graphics card which sells in the highest quantity.

Congratulations on an excellent article! If I didnt know any better I'd swear you were putting the consumer's interest ahead of the manufacturer's. :D

Keep up the good work. ;)

Originally Posted by bigz
My idea would revolve around the resolution you like to game at and then we could list a choice of video cards based on what we found at the time... again, the only problem is keeping it up to date when new games come out. I don't have the time to evaluate every video card we've ever reviewed (again) when new games come out that are more demanding (like FEAR, for example...)

Well that would certainly be no easier!

My suggestion would actually be really easy to maintain and not too hard to setup - would need fine tuning for presentation etc. Thing is, if i want to upgrade it's no good just seeing a review that compares a 6600gt with a x800xl (or whatever) because I don't know what anything else is capable of (without looking at more reviews). What do i care about the minor difference between two cards when I could get a shitload better for another £50 or not much worse for £50 less. Everything's relative. By all means write a review the way you feel it's best but to 90% of people it's actually no good on it's own, so why not embrace that fact and create something of use.

tbh, I'm fed up of you guys blowing your own trumpets about how well you review stuff (i tink this is the third article in as many months covering this issue) when I'm sat here as a user/reader thinking how i don't like your reviews. It bugs me. It's your choice to make a rod for your own backs and play a game through completely, twice, but i can assure you i couldn't care less. Pick a section which is pretty intensive and play that instead - you'll still be 95% right. I'm one of the majority of people that doesn't actually upgrade all that much and seeing a review of the latest graphics card is of no real consequence to me until i'm in the market for one. So, at the minimum i just want to be able to scan a review, get a feel of what it's capable of and move on. Something i find hard to do with bit-tech's reviews.

I was going to actually delete most of that rant, but i feel like leaving most of it up just to see what reactions people have.

Originally Posted by Munterofamodder

A big round-up of modern games on the likes of 9600 Pro\XT, X600, X700, GeForce 5700, 6200, 6600 etc, with lowest prices on these cards found from UK sites, would be brilliant. Setup on a more mid-range system - a 3000 Venice\2*512Mb would seem most appropriate to me. This information seems really hard to find in one place, yet I expect its the range of graphics card which sells in the highest quantity.

Exactly!

The mid range - high end test subject is a hard one to really say which I'd rather see. It really depends where I am on my PURCHASE cycle. I buy a computer every 4-5 years. During that time, I may upgrade a few components.. or I may not.

The key is that when I'm looking to purchase a new computer, I want to see reviews on whats the FASTEST and BEEFIEST thing out there. When I'm upgrading I want to see what runs on par with what my computer currently is (usually mid range). In an ideal world, bit-tech would have the staff to be able to do a review on 3 different configurations. High end, medium, low end. But, this is not an ideal world, you guys do have lives and you do have limits.

So, essentially, both are very useful given the correct circumstance. Through my experience, bit-tech staff have proven to be among the most tech-savy I have come across. You guys are dedicated to your work and more than anything have a very level-headed and open minded approach to everything you look at. Its why I love your site and have been a faithful reader for almost 4 full years now (I think?? maybe a little longer.. time kinda blurs after a while).

So what am I trying to say? I'm saying that I trust your judgement and believe you will select a test system that is appropriate, and also that I really appreciate your interest in what I have to say about such things. Obviously it makes no sense to pair a flagship video card with a budget system. And respectively it makes no sense to pair a budget video card with a vastly superior computer. Try and pair, budget-budget, midrange-midrange, high end-high end. After all, no review or benchmark is the be-all end all in the world of technology. It comes down to the reader to interpret and apply (much easier to do that with your new method of going about such things) the data to their personal needs.

I think how you have been doing things so far is great. I now realize from this article just how much time you guys put into your reviews.. and WOW! I was very shocked and it makes me even more appreciative of what you do.

A small suggestion that might help give a wider range of readers the ability to relate to your hardware selections, and again this is only if you have time. At the end of a review you could do a quick blurb illustrating how the card did with a medium and budget system (assuming it was tested on a high end system). This would be more of a quick glance, and not half as in depth as your actual review.

Do a 15 minute test in the most stressful part of the game you discovered via the benchmark with the configuration used in the actual review. Then you could give a quick impression of how it faired just to give a point of reference for those that might apply to. You could even call it the "alternative configuration impressions" (i'm sure you can think up somethign way more nifty sounding than that) that will stress that its only a quick point of reference to how it will fair on other systems.

Regardless, what you guys are doing in my opinion is ABOVE AND BEYOND what the hardware/game review industry has been doing. You guys are, to me, setting a new standard and as is you are already doing a lot. Most people will be able to take what you present as is and gather more useful knowledge and advice from your reviews than ever before. But you asked what I thought, and thats what I think might help.

Great work bit-tech. Always appreciate your hard work and dedication. Its easy to just go with the flow.. its a real challange to stir things up and forcefully seek progress.

Originally Posted by Meanmotion
Well that would certainly be no easier!

If you want a 3DMark score, there are many sites out there that spend their time watching frames go by.

My suggestion would actually be really easy to maintain and not too hard to setup - would need fine tuning for presentation etc.

I don't disagree with that, but I don't believe that your suggestion is right for the direction that we are heading in.

tbh, I'm fed up of you guys blowing your own trumpets about how well you review stuff (i tink this is the third article in as many months covering this issue) when I'm sat here as a user/reader thinking how i don't like your reviews. It bugs me.

We don't force you to read anything, we're catering for a market of hardware enthusiasts that actually play games. We also cater for the modders, too. There are any number of sites out there that comment on playability without actually playing games - I think that's quite wrong.

A classic example of this was the recent FEAR coverage on Anandtech - some of the comments they'd made about playability are not true after me sitting down over the last 48 hours and playing several parts of the game with several different hardware configurations.

Speaking as someone who used to spend a lot of money on computer hardware, I'd never base a buying decision on the back of a single review. Why would you when there are hundreds of hardware sites out there? The more opinions you get the better, right? I certainly feel a lot more comfortable about spending £300 of my own money on a new piece of hardware for my computer if I know that two, three, four, five, six or however many reviews I read agree with my buying decision.

It's your choice to make a rod for your own backs and play a game through completely, twice, but i can assure you i couldn't care less. Pick a section which is pretty intensive and play that instead - you'll still be 95% right.

I don't understand the point you're making. Surely the whole idea of a 'playable setting' is just that... it is playable in a worst-case scenario in any particular game. As I've already said, most people will increase details on what we've suggested, because the gameplay sections we use are designed to stress the graphics cards.

I'm one of the majority of people that doesn't actually upgrade all that much and seeing a review of the latest graphics card is of no real consequence to me until i'm in the market for one. So, at the minimum i just want to be able to scan a review, get a feel of what it's capable of and move on. Something i find hard to do with bit-tech's reviews.

I understand, but with the time it takes to complete these things, we have to cut off somewhere. If I get the time, I do attempt to add in something that readers are a lot more familiar with.

I was going to actually delete most of that rant, but i feel like leaving most of it up just to see what reactions people have.

I'm glad you didn't, because I'd like to know if there are problems. I don't think 3DMark is the solution, though.

I've got an open question for you.

As an avid gamer looking to spend £300 on a new video card... would you base your buying decision on a 3DMark score? Secondly, being an avid gamer, would you rather see both ATI and NVIDIA's driver teams spending the time they spend optimising 3DMark, actually removing bugs and improving the performance in your favourite games?

Originally Posted by Meanmotion
tbh, I'm fed up of you guys blowing your own trumpets about how well you review stuff (i tink this is the third article in as many months covering this issue)

I disagree with a large portion of the rant except this. I would really like to see this be the last column/article space that is dedicated to the way reviews are done around here. This IS the third article in as many months by my recollection too... and even though I personally really like the way that the reviews have shifted to being done, I'd like to stop hearing about it and get on to other content. :p It's starting to feel like either self importance or sales pitch, and I haven't figured out which yet.

The end result is that you work hard to make the best review you feel you can write, and I don't think there are many people who don't feel comfortable making requests. If they need to, they have a feedback portion of the forum for it. But I see a whole lot of "Great job" and just a few complaints, so that tells me you're doing alright and we can move on. It would just be nice if the next column could instead go back to things going on in the industry instead of things going on in the site. I kinda miss that... :D

Originally Posted by bigz

As an avid gamer looking to spend £300 on a new video card... would you base your buying decision on a 3DMark score? Secondly, being an avid gamer, would you rather see both ATI and NVIDIA's driver teams spending the time they spend optimising 3DMark, actually removing bugs and improving the performance in your favourite games?

2-3 years ago. Probably.

3-5 years ago? Most definitely!

Now? Absolutely not!! I'm not sure if its because the industry and their way of doing business has caused this, or if I have simply grown up and become less impressionable. Either way, I much rather hear a trusted opinion style review of something than raw numbers.. because raw numbers dont always speak the truth, the whole truth and nothing but the truth.

For instance, a review of a home theater system. The reviewer can sit there and spew out measurements, frequency ranges, power handling limits, etc etc etc. But I get a lot more out of hearing that reviewers IMPRESSION and the overall experience they gained from it. I then cross reference that review with others to see how valid it is. After a time, I learn to trust the opinion.. but I will always cross reference because as you said.. when I spend money I like to make sure many people share the same perspective before I feel comfortable the information guiding my decision is valid.

Originally Posted by Da Dego
I disagree with a large portion of the rant except this. I would really like to see this be the last column/article space that is dedicated to the way reviews are done around here. This IS the third article in as many months by my recollection too... and even though I personally really like the way that the reviews have shifted to being done, I'd like to stop hearing about it and get on to other content. :p It's starting to feel like either self importance or sales pitch, and I haven't figured out which yet.

The end result is that you work hard to make the best review you feel you can write, and I don't think there are many people who don't feel comfortable making requests. If they need to, they have a feedback portion of the forum for it. But I see a whole lot of "Great job" and just a few complaints, so that tells me you're doing alright and we can move on. It would just be nice if the next column could instead go back to things going on in the industry instead of things going on in the site. I kinda miss that... :D

I think the last one you're referring to (i.e. last week) was actually a rant about things being sent to us that clearly aren't ready for our eyes, nevermind the consumer. We found out today that the motherboard that we spent a considerable amount of time debugging and attempting to review, isn't even going to use the same chipset as the board that will be mass produced and sold to the consumer.

If ATI, Intel, NVIDIA, VIA and ULi sent us motherboards that were never going to make it to retail we'd fully understand because, for the most part, they only send us motherboards when they release new chipsets in order to give an idea of how their chipset works/performs. However, in our opinion, a motherboard manufacturer shouldn't be sending us motherboards for review that are never ever going to make it to retail.

If they send us a motherboard for debugging/etc, that's a different kettle of fish. In that case, the motherboard shouldn't come in a retail packet - it should be a brown box job. The board that I was ranting about happened to be shipped to us in a retail packet - a retail packet should, by rights, mean that we're looking at a product. To say that I'm upset that both myself and Rich spent a large amount of time working on this 'review', only to find out that said motherboard manufacturer has decided to go so far as to change the version of the chipset that they're using. I don't have a problem with looking at pre-release products if we're sent them as-is in a brown box as a 'board-only' package. That's slightly different to being sent a 'pre-release' motherboard inside retail packaging.

All that I asked in what you're referring to as a 'trumpet blowing session' is that motherboard manufacturers make the distinction between shipping products and pre-release hardware before it gets in to our hands. Pre release hardware shouldn't be shipped to a journalist in retail packaging, simple as.