If you are the average person on the street that goes into a store, listens to the salesperson, and then makes their smartphone purchase, you probably do not care about the “benchmark” of your device – but then, you probably wouldn’t be reading this article, either. We’ve been saying for years that benchmarks do not mean much in relationship to the real-world use of your device, but there are some people that just have to compare those figures – they need to know that their device is at the top of the pecking order – bragging rights; I guess you could call it.
If you are one of those people, then you are aware of that recently Samsung and then HTC, were both accused of doctoring their test scores, in Samsung’s case, their Galaxy Note 10.1 and Galaxy Note 3 and in HTC’s case their HTC One and HTC One Mini. Because of that, Benchmark’s software operation, Futuremark, has “Delisted” those devices. Futuremark President Oliver Baltuch said:
“People rely on Futuremark benchmarks to produce accurate and unbiased results. That’s why we have clear rules for hardware manufacturers and software developers that specify how a platform can interact with our benchmark software. When a device is suspected of breaking our rules it is delisted.
The platform may not detect the launch of the benchmark executable. The platform must not alter, replace or override any parameters or parts of the test, nor modify the usual functioning of the platform based on the detection of the benchmark.”
Samsung denied the charges and said that, “The Galaxy Note 3 maximizes its CPU/GPU frequencies when running features that demand substantial performance. This was not an attempt to exaggerate particular benchmarking results. We remain committed to providing our customers with the best possible user experience.”
The problem is that our devices can sense when a benchmark test is being run, and can rev up the device’s processor to run “full-blast” causing a possible better reading than in everyday usage, and that is exactly what Samsung and HTC did. GameBench may have the solution to make everybody happy – they are coming out with an app that does not look like a benchmark test to the device – this application actually runs a game and takes test results during the use of the device when the processor is performing as it normally would while running an app. According to their test results of the Samsung Galaxy S4 and HTC One, there was no reason for Samsung to have “cheated,” as their scores were higher in the end.
So the moral of the story is “cheaters never win and winners never cheat” – Samsung lost because they “fudged” the results, when they could have won without all of the fudging. I like fudge as much as the next person, but enough already with the “fudging,” cheating, or whatever you call it – stand on your own merits from now on. Should Futuremark make Samsung and HTC “sit in the corner,” that is your call…personally, I do not believe that benchmarks are that important, but if Futuremark wants to publicly punish them, so be it – they did bend the rules.
Let us know in the comments or on Google+ what you think about benchmarks – do you put a lot of stock in them or as long as your device has the latest hardware and runs fast and smoothly, is that enough for you.