The Problem with Real World Benchmarking

Written by Ben Hardwidge

July 5, 2011 // 7:53 a.m.

Tags: #2012 #amd #bapco #benchmark #benchmarks #leave #nvidia #real-world #resign #smark2012 #synthetic #sysmark #sysmark2012 #via

Ignoring GPGPU Computing

In fact, according to AMD's spokesperson a relatively large proportion of the SysMark 2012 score is based on system performance rated during optical character recognition (OCR) and file compression activities − things an average user will rarely if ever do.'

This isn't AMD's only complaint, though. The other is that the benchmark ' focuses only on the serial processing performance of the CPU, and virtually ignores the parallel processing performance of the GPU.' This is clear when you look at the browsers used in the benchmark – Internet Explorer 8 and Firefox 3.6.8. This is despite the fact that Internet Explorer 9 and Firefox 4 are both readily available, and both also support GPGPU acceleration in a number of areas.

The Problem with Real World Benchmarking Real World Benchmarks - Ignoring GPGPU ComputingNew GPGPU-accelerated browser features, such as the IMDB Video Panorama, shown here in Internet Explorer 9, won't be a part of SysMark 2012

On the one hand, you can see a possible justification for this. After all, many businesses are still using Internet Explorer 6, let alone version 8 or 9, and you could also argue that not many people are using GPGPU acceleration on a regular basis yet. However, you could also argue that new browsers and apps are now supporting GPGPU acceleration natively, and that home users will be using the latest versions of these browsers too.

Either way, this created a main point of conflict between several members of the BAPCo consortium. As AMD's spokesperson told us: 'Benchmarks will measure what they are designed to measure,' and in this case it ignores most of the benefits of GPGPU computing.

Of course, the idea behind the consortium is that individual members can vote on what's included in the benchmark, and BAPCo claims that AMD supported around 80 per cent of the benchmark's other development milestones. The problem is that AMD and Nvidia were clearly in the minority when it came to GPGPU computing. The consortium also includes many PC makers, including Samsung, Toshiba, Lenovo, Sony HP and Dell, and while these companies do use AMD products, most of their machines are based on Intel chips. What's more, with AMD's Fusion technology still very much in its infancy, you can see why these companies might not want to take the risk of their Intel-based machines looking bad in a benchmark.

The Problem with Real World Benchmarking Real World Benchmarks - Ignoring GPGPU ComputingAMD wants GPGPU computing to be a part of SysMark 2012 to show off the performance of its Llano chips

And herein lies the problem – it looks very much as though the consortium wasn't so much interested in measuring real world performance as it was interested in ensuring that the majority of its members' products came out of it looking good, and this includes AMD.

This attitude, including the whole CPU and GPU computing divide, also appears to be one of VIA's main complaints. 'I always think that it's better if the key players in the industry can work together to develop standardised benchmarks,' says VIA's Richard Brown. 'That would help reduce customer confusion and create a level playing field for everyone. However, that would require a highly transparent process in which companies are willing put aside pre-set agendas for the good of the industry and the user. Unfortunately, the reverse is currently happening with increasing polarisation between companies that have a CPU-centric view of the world and those with a GPU-centric view of the world. I don’t think that benefits anyone.'
Discuss this in the forums

QUICK COMMENT

SUBSCRIBE TO OUR NEWSLETTER

WEEK IN REVIEW

TOP STORIES

SUGGESTED FOR YOU