bit-tech.net

The Problem with Real World Benchmarking

Comments 26 to 50 of 50

Reply
Snips 5th July 2011, 15:50 Quote
Why would you ignore the biggest hardware player on the planet? Intel gets that respect for a reason.
SlowMotionSuicide 5th July 2011, 16:02 Quote
Quote:
Originally Posted by wuyanxu
call me cynical, but i thought the reason AMD walked out was due to their new processor was unable to compete.

My thoughts exactly. AMD can't compete on pure CPU performance with Intel so instead they go blowing out all the bells and whistles how GPGPU computing is everything.

Doesn't bode well for Bulldozer methinks :-(
Bleech 5th July 2011, 16:35 Quote
[QUOTE='[USRF]Obiwan;2735197']I agree with Pete, If you want basic stuff you could do with everything with a PC from 6 years ago. If you want to play the latest games just put in a middle segment recent video-cart and you can play it fine.

I on the other hand have just a amd quadcore with 8gb of ram and a GTX460, all more then capabple to furfill my needs. Like movie editing, fast unpacking of downloaded movies and series, playing games and other regular stuff you normally do. I call that a wise investment.

Why on earth do i want to have a 1200watt PSU feeding quad sli with a i7 990X processor on 32BG ram and a 400 dollar motherboard with lights and knobs on it which i will never use once it is inside the case and a 160GB SSD for which I can buy 6x 2TB drives.

So I can read my mail faster or start my PC faster? yeah right..

For the record, my PC is en sleep-mode 24/7 and starts in 3 seconds with the move of a mouse

When I was young and naive I always wanted the latest and the best. I even bought newer revision of a motherboard, Yeah crazy! I know better now...[/QUOT


Believe it or not. There IS a market for high end components, and folk will always buy them...because they can.
PingCrosby 5th July 2011, 16:49 Quote
Garcia Hotspur
KayinBlack 5th July 2011, 16:54 Quote
From what I've seen so far, AMD CAN compete. The issue is with rigged compilers, cheating benches and the fact that the head of BAPCO is also the head of performance benchmarking for Intel. NOW does it make sense why they walked out?
schmidtbag 5th July 2011, 17:01 Quote
Quote:
Originally Posted by SlowMotionSuicide
Quote:
Originally Posted by wuyanxu
call me cynical, but i thought the reason AMD walked out was due to their new processor was unable to compete.

My thoughts exactly. AMD can't compete on pure CPU performance with Intel so instead they go blowing out all the bells and whistles how GPGPU computing is everything.

Doesn't bode well for Bulldozer methinks :-(

I overall agree with this statement and everyone else who had one similar. HOWEVER, i don't think bulldozer has anything to do with it. i've seen linux benchmarks of bulldozer performance several months ago, and it is definitely good. bulldozer is meant to be cpu only, and if amd really thinks whining and complaining about how gpgpu tests aren't being done, those tests will not help bulldozer at all. llano's cpu portion is pretty bad. i am certain amd knew that llano was not going to defeat intel in cpu tests before they even benchmarked it themselves. but, i'm sure they definitely knew the video was going to win, and thats what they were really making a big deal out of. that being said, a benchmark suite that practically ignores gpu usage makes llano look terrible. when bulldozer is released i'm sure they won't be complaining so much.

i agree with some other posts saying you should pick the type of benchmark you actually perform. so if all i do is modern gaming and web browsing, why do i care about the other scores? if i only do ms office and virtualization, why do i care about the other scores? as bit-tech has mentioned, intel, amd, via, and even nvidia have all taken very different paths (mainly because intel is so anticompetitive so everyone was forced to). One of the greatest differences i've seen in cpu performance was intel's hex core xeon vs amd's 12 core opteron. both cpu models had drastically different strengths and weaknesses, to the point where they aren't really comparable, even though they are both server processors.
greypilgers 5th July 2011, 18:04 Quote
Meh... Owning a PC over the years is the same as the amount of time it takes to do chores around your house over the years. It takes no less time to clean your house now than it did 50 years ago, despite the vacuum cleaner, the 'special' dusters, techno-science polish sprays, non-smear window cleaners, etc, etc. By the same token, my pc NEVER loads quick enough, could always do with going just that little bit faster, and if I ever see that blue circley thing by my mouse cursor again...!!!

It is interesting that this benchmark is relied upon for mass Government and company purchases, because I thought those purchases were simply awarded to the company who offered the most units for the least money?
timevans999 5th July 2011, 18:44 Quote
What a load of guff I don't trust the reviews of this site and i don't think this site is impartial at all.
Plus benchmark talk from you lot is gay
Nexxo 5th July 2011, 18:52 Quote
People using the word 'gay' as a negative attribute are gay. :p
SexyHyde 5th July 2011, 19:41 Quote
Quote:
Originally Posted by Nexxo
People using the word 'gay' as a negative attribute are gay. :p

Oh darling. I am most certainly liking your work! +1
OCJunkie 5th July 2011, 20:55 Quote
I really think it's a moot point, seriously; any PC from the last few years will run youtube, web browsers and spreadsheets fine, and people don't die if it takes a second longer to load a video... face it, the user-perceivable difference between a slightly older machine and a raging multithreaded quadcore beast in average day-to-day apps is next to zero.

On the other hand...

If you are benchmarking it's precisely to see how it will run demanding applications that most users dont need or care about--otherwise you wouldn't bother because you already know it will run and it doesn't matter. So I agree that benchmarking common hardware with atypically demanding scenarios is unfair and completely misrepresentative of what your average user can expect, and I understand why companies are backing out...

And yes, though "can it run Crysis" is a completely abused joke that I wish would just go away already, it's still definitely relevant because even years after its release, alot of brand new latest-gen hardware still struggles with it.
cyrilthefish 5th July 2011, 23:53 Quote
Quote:
Originally Posted by Bauul
I personally think concentrating on CPU benchmarks is a mistake in itself. In my experience at work, a decent amount of fast RAM and a speedy harddrive is by far and away the most important part of getting a computer to run quickly.

We do quite typical office jobs (any high-end application is streamed via Citrix), and what I hear my staff complain about most are things like:

1) My computer takes too long to boot up
2) Opening *insert MS Office app* takes too long
3) My computer slows down when I have too much open

Above something quite simple, the CPU has pretty much no effect on any of the above.
Have to agree here.

After working as a IT helpdesk bod for most of a decade, a 'slow' pc is almost invariably caused by lack of ram / slow hdd, rather than cpu.

Only 6+yr old cheap celerons were ever a lack of cpu power problem that i came across, even then a ram/hdd upgrade would minimize the impact.

Out of the market now, but i could see cheap 32gb ssds keeping aging corporate pcs going for way past their usual working lifespan, especially if coupled with a ram upgrade later on. It only really needs to hold the OS and core apps locally, anything else can sit on a server.
law99 6th July 2011, 07:56 Quote
Quote:
Originally Posted by DbD
This is the same benchmark that Intel has had the same say in for many years. It's one that AMD championed for much of that time - making it the standard for government purchasing orders in America. Why did they do that? - because the A64 was a great cpu, it did well in BABCO and hence AMD could win some deals over Intel despite it being the smaller player.

So why are now are they really pulling out?

The strong suspicion is because bulldozer is weak, because if those same government dept's that keep using BABCO AMD will sell nothing. Hence it's not really to do with break down, or gpu compute or any of that it's to do with selling bulldozer. AMD need to minimise use of BABCO for competitive benchmarks of Xeon/SB vs bulldozer. It's all smoke, mirrors and marketing.

That's true of Llano, but total conjecture for bulldozer. And I don't think bulldozer is going to be all that great compared to Sandybridge and Ivybridge or even older Nehalem when considering how long it's been around.

The reality is, in the day and age of gpgpu, if office software is going to be the key selling point for cpu performance, which is what AMD wants to win some business, they are making total sense. Llano makes perfect sense for a PC in the office just like the lower end CoreIs, although AMD may be cheaper, finally have the power consumption to compete and should not be resting on their laurels when it comes to opencl or they might as well give up and let Intel steamroll them by not trying to prove their point.

Again, bulldozer has nothing to do with this...
Quote:
Originally Posted by cyrilthefish
Have to agree here.
Out of the market now, but i could see cheap 32gb ssds keeping aging corporate pcs going for way past their usual working lifespan, especially if coupled with a ram upgrade later on. It only really needs to hold the OS and core apps locally, anything else can sit on a server.

Exactly, the machines at my work would be so much faster with an ssd. All things over the network are load balanced and in raid configurations, so they all scream away... the problem is the 3.5" drives inside them. I just upgraded to an SSD and the difference is huge.
sandys 6th July 2011, 09:58 Quote
The last 3 companies I have worked for over the last 10yrs don't use computers like that anymore. So how useful is something like that to business I wonder.

We are all on virtual servers, I have a core i7 laptop with 8Gb of RAM, half the ram is useless as the machine is running Windows XP, the only process it runs is Citrix and the monitoring software, if there was a netbook with a 1920x1200 screen the company probably would have farmed those out and saved a lot of dough.
Xir 6th July 2011, 10:33 Quote
Quote:
Originally Posted by Nexxo
People using the word 'gay' as a negative attribute are gay. :p
It's used as a negative attribute in German but it doesn't have any meaning other than "negative attribute".....

The again, the german word for gay is also used as a "negative attribute" AND for describing people who are...you know...gay.

Nevermind
beckoner 6th July 2011, 10:58 Quote
I have a limited budget for P.C. at work : they must be able to access internet, run MS Office, decent security/protection and pretty much nothing else... and we use four or five excel sheets with 1.5Mcells (running the VBA update on one of them takes 20mins!). This is pretty much all a SBE/Government system ever needs - and thats where the manufacturers bread and butter is..
feathers 6th July 2011, 15:30 Quote
If a benchmark consisted of my own real world usage it would consist of the following tests:

1: Boot time into windows with at least 40 programs in startup.
2: Game performance with 109 background processes resident from various apps.
3: 1080HD Porn.
4: 1080HD Video editing.
5: Multitasking performance testing Firefox with 10 tabbed pages open, STEAM and MSN and of course Utorrent.
Adnoctum 6th July 2011, 17:33 Quote
Quote:
Originally Posted by DbD
This is the same benchmark that Intel has had the same say in for many years. It's one that AMD championed for much of that time - making it the standard for government purchasing orders in America. Why did they do that? - because the A64 was a great cpu, it did well in BABCO and hence AMD could win some deals over Intel despite it being the smaller player.

So why are now are they really pulling out?

The strong suspicion is because bulldozer is weak, because if those same government dept's that keep using BABCO AMD will sell nothing. Hence it's not really to do with break down, or gpu compute or any of that it's to do with selling bulldozer. AMD need to minimise use of BABCO for competitive benchmarks of Xeon/SB vs bulldozer. It's all smoke, mirrors and marketing.

AMD never championed SYSMARK. Anyone can join BAPCo. You just need to stump up the cash and you get a vote. AMD joined because at the time it was a well known Intel front organisation and they thought that by being a voting member they could influence development of a more representative benchmark.

Stupid AMD! Should have known that the reigning champion in influence peddling and its coterie of self-interested tools couldn't be swayed in making a fairer benchmark that would show the real value of 80% of their products.

The problem with SYSMARK is that it isn't being used to benchmark systems for office (or government) use, it is being used as a benchmark for marketing to average users, of which the "real world tasks" aren't as representative.

The issue is also not because the AMD CPUs are slower, and that the GPU isn't tested is only one of the problems. Another is the hidden weighting SYSMARK uses. The scores of some benchmarks are weighted higher than others.
For example, for each equally utilised (by the mythical "average" user) task:
* Neutral BM returns the score: Intel 2000fps and 5 weighting points with AMD 1800fps and 4.5 weighting points (or 0.5 points per 200fps),
* Intel-favoured BM returns the score: Intel 150sec and 10 weighting points with AMD 170sec and 6 weighting points (or 200 points - x seconds /5),
* GPU-favoured BM returns the score: Intel 150 units and 1 weighting point with AMD 300 units and 2 weighting points (or 0.1 points * x units/15),
Final score: (x weighting points * 10.5) Intel with 168 SYSMARKs and AMD with 131.25 SYSMARKs.

Does the AMD final score accurately represent the actual results of the benchmarks? In the neutral BM the performances are quite close. In the Intel-favoured BM, the results are close as well (20sec isn't much), but knowing Intel CPUs do well in this BM the weighting is inflated. In the GPU-favoured BM the AMD completed twice as many units, meaning twice the "performance", but the BM is weighted far lower. Remember that there is the assumption that these are three tasks of equal utility to the "average user". SYSMARK assigns greater importance to BMs that suit Intel features and diminishes AMD's. It is all in how the calculations are done.

We learnt about SYSMARK's weighting biases because the scores for each task used to be written to a file so a final score could be totalled. Someone found out that the file could be opened in Notepad and the weighting given to each task was there for all to see. BMs that favoured Intel CPUs were given higher importance in the final score, meaning that the SYSMARK score of a P4 was much closer to that of a A64 than the individual BM scores would indicate.

Once this dirty little BAPCo secret came out, the next version of SYSMARK released had started encrypting the storage file so the assigned weighting could no longer be scrutinised.

If this was just AMD I might agree with the suspicions about AMD's motives, but Nvidia and Via have left for similar reasons, and with the amount of GPGPU processing increasing this is going make this hardware capability more important. This now leaves Intel and Seagate as the ONLY hardware manufacturers left, all the others being system builders who have a vested interest in keeping the status quo.

Quite frankly SYSMARK is a bit pointless now, seeing as the system builders now have the choice of Intel Processor A or Intel Processor B to go with Intel Basic Chipset A (or Intel Mainstream Chipset B if they want to add $200 to system retail price) and Intel Wireless Controller C and Intel Peripheral Controller L. An Intel "Hardware Deccelerator" X that can do Flash and 1080p now comes for free, so bonus!
law99 6th July 2011, 19:38 Quote
Quote:
Originally Posted by Adnoctum
AMD never championed SYSMARK. Anyone can join BAPCo. You just need to stump up the cash and you get a vote. AMD joined because at the time it was a well known Intel front organisation and they thought that by being a voting member they could influence development of a more representative benchmark.

Stupid AMD! Should have known that the reigning champion in influence peddling and its coterie of self-interested tools couldn't be swayed in making a fairer benchmark that would show the real value of 80% of their products.

The problem with SYSMARK is that it isn't being used to benchmark systems for office (or government) use, it is being used as a benchmark for marketing to average users, of which the "real world tasks" aren't as representative.

The issue is also not because the AMD CPUs are slower, and that the GPU isn't tested is only one of the problems. Another is the hidden weighting SYSMARK uses. The scores of some benchmarks are weighted higher than others.
For example, for each equally utilised (by the mythical "average" user) task:
* Neutral BM returns the score: Intel 2000fps and 5 weighting points with AMD 1800fps and 4.5 weighting points (or 0.5 points per 200fps),
* Intel-favoured BM returns the score: Intel 150sec and 10 weighting points with AMD 170sec and 6 weighting points (or 200 points - x seconds /5),
* GPU-favoured BM returns the score: Intel 150 units and 1 weighting point with AMD 300 units and 2 weighting points (or 0.1 points * x units/15),
Final score: (x weighting points * 10.5) Intel with 168 SYSMARKs and AMD with 131.25 SYSMARKs.

Does the AMD final score accurately represent the actual results of the benchmarks? In the neutral BM the performances are quite close. In the Intel-favoured BM, the results are close as well (20sec isn't much), but knowing Intel CPUs do well in this BM the weighting is inflated. In the GPU-favoured BM the AMD completed twice as many units, meaning twice the "performance", but the BM is weighted far lower. Remember that there is the assumption that these are three tasks of equal utility to the "average user". SYSMARK assigns greater importance to BMs that suit Intel features and diminishes AMD's. It is all in how the calculations are done.

We learnt about SYSMARK's weighting biases because the scores for each task used to be written to a file so a final score could be totalled. Someone found out that the file could be opened in Notepad and the weighting given to each task was there for all to see. BMs that favoured Intel CPUs were given higher importance in the final score, meaning that the SYSMARK score of a P4 was much closer to that of a A64 than the individual BM scores would indicate.

Once this dirty little BAPCo secret came out, the next version of SYSMARK released had started encrypting the storage file so the assigned weighting could no longer be scrutinised.

If this was just AMD I might agree with the suspicions about AMD's motives, but Nvidia and Via have left for similar reasons, and with the amount of GPGPU processing increasing this is going make this hardware capability more important. This now leaves Intel and Seagate as the ONLY hardware manufacturers left, all the others being system builders who have a vested interest in keeping the status quo.

Quite frankly SYSMARK is a bit pointless now, seeing as the system builders now have the choice of Intel Processor A or Intel Processor B to go with Intel Basic Chipset A (or Intel Mainstream Chipset B if they want to add $200 to system retail price) and Intel Wireless Controller C and Intel Peripheral Controller L. An Intel "Hardware Deccelerator" X that can do Flash and 1080p now comes for free, so bonus!

Thanks for that info. Interesting.
Snips 6th July 2011, 23:37 Quote
I've found the answer, it was right in front of us all the time.

If you use SYSMARK and you're an AMD fanboi, just wear tinfoil on your head. (Don't tell anyone but it gives AMD better scores!)
Ipatinga 7th July 2011, 00:45 Quote
So... everybody is blablabla about a benchmark that:

1 - Launched in 2011;
2 - Has a name version of 2012;
3 - Can´t evaluate for real a component from 2010;
4 - And thinks that benchmarking a browser from 2008 and a cpu from 2006 is great?
5 - Jezzzzz... I miss the 90's...

Syswhat?
Meanmotion 7th July 2011, 09:32 Quote
I think several other people have touched upon it already but it really does strike me as the key overalll point here; if you're doing such basic workloads, you don't need the latest greatest kit! Any modern AMD APU or Intel nehelam chip will more than suffice. If you want gaming, well then the whole argument goes out the window as you'll need to invet in a separate gfx card anyway, no matter what AMD tries to convince us.
Adnoctum 7th July 2011, 12:06 Quote
Quote:
Originally Posted by Snips
I've found the answer, it was right in front of us all the time.

If you use SYSMARK and you're an AMD fanboi, just wear tinfoil on your head. (Don't tell anyone but it gives AMD better scores!)

It's not a UFO you twerp, if it's hovering over the front lawn of the White House and the driver is shaking hands with the President. If you still choose to think that it is just your own intestinal gas, then it reflects more about you than others.
You must be in a terrible amount of internal conflict. Your Intel-based "fanboi-ism" means you must rubbish everything AMD and defend SYSMARK as relevant, but your Nvidia-based "fanboi-ism" means that you must rubbish everything AMD and condemn SYSMARK as unrepresentative. I feel sympathy for your personal quandary.

Other benchmarks show that Intel CPUs are faster than AMD ones at the moment, but no one is claiming systemic bias and collusion about those.

Once again you accuse me of being an AMD "fanboi", and again I have to remind you that I am typing this on a C2D-based system. If I defend AMD it is because I defend a more reasonable and logical discourse that is all too lacking when discussing the attributes of computer components.
A few years ago AMD was on top and Intel could do nothing good, when the reality was that Intel was only slightly behind and most people wouldn't have known the difference when using the system. Today Intel is on top, and of course AMD can do nothing good. And people still wouldn't know the difference.

If I was building a basic PC for office or my parents, I would choose a lower end Llano-based dual core, an A4-3400 or so.
If I was building a HTPC, I would choose a mid-end Llano-based quad core, such as an A6-3800.
If I was building a gaming PC, I would use a SB-based Core i7, such as a 2600K.
This may change in six months. If you are building a system, choose what suits your needs, not what you're emotionally attached to.

@Meanmotion and @Ipatinga

Personally, SYSMARK is irrelevant to my decision making, as it can't be trusted. The individual scores for the applications used can be used as a guide, but it would be better to benchmark those applications without the dubious assistance of SYSMARK.

Your comments made me think about my own computer use, and in the last 18 months I haven't thought "I wish this went faster". All computer games have run at max settings at 1920x1200, and all programmes have run/encoded/compiled/compressed/ripped in a manner acceptable to someone as impatient as myself. Even encoding tens of GB of video files was painless because I set it going and came back 24hrs later to store the results.
XXAOSICXX 10th July 2011, 01:22 Quote
Quote:
Originally Posted by Tulatin
Unfortunately, it's now less funny than "lolcats".

Heyyyyy now. Go easy.

http://www.lolcats.com/view/113/

LOL :)

See!....still funny :p
xJohndoe001x 19th July 2011, 00:40 Quote
Quote:
Originally Posted by Tulatin
"But can it run Crysis?"
Congratulations, you're responsible for making the 4,877,921,204th post containing this joke. Unfortunately, it's now less funny than "lolcats". You could have at least wasted your comment by partaking in Internet tradition, spewing "First!" meaninglessly into the world.

If only we could have a time when people contribute meaningful discussion to articles, perhaps wondering if Intel influences Bapco's habit of only focusing on a tiny sliver of the real world?

Congradulation, you're the 4,877,921,204th Internet Troll to search out for someone to yap at like a rabit dog (poodle) talking nothing about the issue.

You could've easily found someone saying something you could relate to but I am sure that's no fun to you. You're so off subject. So, What do you think about the 2012 benchmarking?
Log in

You are not logged in, please login with your forum account below. If you don't already have an account please register to start contributing.



Discuss in the forums