Samsung’s Benchmark Rigging Has Been Officially Removed in KitKat Updates

March 6, 2014 - Written By Nick Sutrich

As just about everyone knows Samsung is the biggest Android manufacturer by far.  They sell tens of millions more devices than any of their closest competitors, and this gives them an advantage when it comes to ordering components for their Next Big Thing.  As we’ve seen over the last couple of iterations of the Galaxy S and Galaxy Note line of phones, the processors Samsung is able to procure for these devices are usually a hair faster than the ones in competitor’s phones, and it allows Samsung to usually say they have the fastest device on the market.  So why then has Samsung been cheating at benchmarks for so long if its already got the hardware advantage?  There’s a number of reasons for this, and in actuality it’s not like Samsung is modifying benchmark software or scores artificially, rather they are using a trick with the device’s central processing unit to achieve the highest score possible.  To see exactly what I’m talking about it’s easiest to understand it in a visual manner:

If you’re not familiar with what a benchmark is or how it works, basically it’s a program that runs a set test in order to achieve a score for a device.  This score is used to compare to other devices, and it should be accurate since the tests are always the same no matter what device it’s run on.  Above is a screenshot of when you open Geekbench 3, one of the many popular benchmarking apps on the Google Play Store.  Anandtech ran an app in the bottom right hand corner of the screen showing exactly what the CPU is doing at any given time.  When the phone is sitting the home screen, or any other app where the CPU isn’t doing much, the processor is running at 300MHz, saving battery life as it doesn’t need to ramp up for any important tasks.  In Android 4.3 on the Galaxy Note 3 (pictured left) or the Galaxy S4 (pictured right) as soon as you open nearly any benchmarking app the CPU automatically turns on all available cores and ramps them up to their maximum speed.  This isn’t done in any other app, and it essentially gives Samsung an arbitrarily high score because the CPU isn’t behaving like normal and powering down while nothing is happening.

samsung android 4.4 benchmark-1

Above is the exact same app running under Android 4.4.2 on the Galaxy Note 3 (left) and the Galaxy S4 (right) again.  As you can see the CPU is behaving properly, and is even shutting off cores completely that aren’t needed.  This behavior is what the CPU is supposed to be doing, and the scores verify that the tricking of benchmarks is officially over as of the Android 4.4.2 KitKat update for Samsung phones.

samsung benchmark chart

Given that Samsung is updating 14 devices to Android 4.4.2 KitKat, it’ll be interesting to see which ones go up or down in standing on global benchmark rankings.  Samsung isn’t the only vendor that cheats like this though, HTC, ASUS and LG are all guilty of the exact same rigging.  While this move certainly helps give benchmarks a little more validity in the eyes of testers, you should always take benchmark results with a grain of salt and just use them as part of the overall picture of how a device performs.  When all is said and done though the device feels no different in everyday tasks because of this measure, it’s just simply to keep things fair in tests.  Thankfully Android 4.4 KitKat has a number of optimizations, including better memory usage, that will actually make your device feel faster and respond more quickly and efficiently, so this update definitely has more than meets the eye initially.