Benchamarks Can Be Manipulated and You Shouldn't Really Care About Them

Advertisement
Advertisement

Let's talk about benchmarks, specifically benchmark apps on your smartphone. I'll be up front here: benchmarking apps on your phone are easily manipulated and pretty worthless. We've known for at least 2 years that benchmarks can be easily manipulated. For some reason people still run them on their phones and then post screen shots to flex their internet muscles. I guess it feels good to compare proverbial penis-size with other anonymous commenters on a forum somewhere, but benchmark apps on your phone mean next to nothing.

If you are using benchmark app results to justify a phone purchase or flex your smartphone muscles, you're doing it wrong.

I'm not against benchmark tests. I mean real benchmark tests done with proper equipment that generate real results. It's fun and can be useful to test our devices and see how they perform. It's also helpful to verify the performance that manufacturers claim. What I'm not a fan of are the benchmark apps that don't actually test hardware performance. Pick your favorite benchmark app from the Google Play Store, I don't care which one it is. It's not going to give you results that matter for real world use. All that you're testing is how the software interacts with the hardware. I'm not against these apps or their developers. It just makes no sense to me that people put stock in the results.

Advertisement

Samsung is in some hot water after getting caught (again) doctoring the benchmark results of the new Galaxy Note 3. They did it with the Galaxy S4 as well. They've tweaked the device code to max out the clock speed when the phone detects that a benchmark app is running. According to Ron Amadeo at Ars Technica the phone only bumps up the CPU for benchmark apps. Pretty shady stuff. Other manufacturers do this as well, even if they have not been caught at it. If we can't trust benchmark app results, what's the point of running tests with them?

note3benchmarks

Android sites have long touted benchmark app results as proof that *insert new device* is the best thing since sliced bread. We've done it. I will admit to being a bit of a benchmark jock in my early Android days, taking screen shots and sharing them on Twitter so all 200 of my followers can see how big my smartphone muscles are. But what's the point in that? More importantly, OEMs are now attempting to manipulate consumers using these apps. It's time to stop the madness. Manufacturers need to be clear about how their devices interact with these apps. Benchmark app developers may want to implement some kind of notice that the phone has suddenly entered a "benchmark mode" and cranked the CPU to max. It would protect their integrity a little bit. I know one thing, though; I'm not wasting time running benchmark tests on my devices anymore.

Advertisement

Benchmark apps can be fun, sure. I'm even willing to admit that there are situations that I am overlooking where it could be helpful to run benchmark app tests. It's definitely time to re-think how we are using them, though. It's also well past the time to stop trusting results that you see from OEMs or anyone else. If you are using benchmark app results to justify a phone purchase or flex your smartphone muscles, you're doing it wrong.