Samsung Tweaked OS to Cheat Benchmarks on Galaxy S4

[IBNobody]

http://www.anandtech.com/show/7187/...ght.com&utm_medium=referral&utm_campaign=Feed

The folks at AnandTech caught Samsung red-handed in tweaking the Android OS to run certain benchmarks better. They found the behavior on both the Snapdragon and the Exynos S4 models, but the Exynos models cheated even worse.

Basically, Samsung added a "Benchmark Booster" feature to cheat benchmarks. (It's actually called that in the code!)

If a certain benchmark app was detected as being run, the S4 Exynos would overclock its GPU from 480MHz to 532MHz and both models would set the CPU governer to the maximum CPU clock speed. While the CPU clock speed was available to all apps, the overclocked GPU mode was only made available to these benchmarks.

Have a cool 3D high-end android game? The S4 Exynos's GPU will run at 480MHz.
Have a meaningless benchmark? The S4 Exynos's GPU will run at 532MHz.

UPDATE:

Samsung admits to tweaking the S4.
http://global.samsungtomorrow.com/?p=26314

Under ordinary conditions, the GALAXY S4 has been designed to allow a maximum GPU frequency of 533MHz. However, the maximum GPU frequency is lowered to 480MHz for certain gaming apps that may cause an overload, when they are used for a prolonged period of time in full-screen mode. Meanwhile, a maximum GPU frequency of 533MHz is applicable for running apps that are usually used in full-screen mode, such as the S Browser, Gallery, Camera, Video Player, and certain benchmarking apps, which also demand substantial performance.

The maximum GPU frequencies for the GALAXY S4 have been varied to provide optimal user experience for our customers, and were not intended to improve certain benchmark results.

Samsung Electronics remains committed to providing our customers with the best possible user experience.

Now, what they are saying is a bit duplicitous. The only apps that can use the 532/533 MHz mode are ones that Samsung approves and white-lists. Samsung's apps can use it. Benchmarks can use it. Everything else can't. What's the point of benching at 532MHz when your real world performance is artificially capped at 480MHz?


TL;DR:
Don't trust benchmarks!

 

[Joe88]

people actually care about cell phone benchmarks?

 

[IBNobody]

people actually care about cell phone benchmarks?

People care enough for Samsung to want to cheat.

 

[Rockhoundhigh]

Yeah, if there's anything I've learned from tinkering with my Android phone it's that benchmark scores mean nothing. Judge a phone only on its real world performance. In this case though Samsung was just being sleazy to get even more people to sing their praises.

 

[DinohScene]

Oh Nokia 3310.
Why are you so amazing!

 

[Joe88]

Yeah, if there's anything I've learned from tinkering with my Android phone it's that benchmark scores mean nothing. Judge a phone only on its real world performance. In this case though Samsung was just being sleazy to get even more people to sing their praises.

well its also the various review sites who mainly judge phones phones on benchmarks alone ignoring real world usage

 

[Lastly]

Samsung added a "Benchmark Booster" feature to cheat benchmarks. (It's actually called that in the code!)

Haha! I can actually imagine that:

// TOP SECRET - Do not let anyone see this code --- even your friend, wife, son, dog, cat, or your second part-time mean boss!
// All who fail to comply will be persecuted to the full extent of law!
// If this info is leaked, do not tell that the CEO, that's me, or head director explicitly instruct you to add this code. Blame it on the IDE you were using. Or that you accidentally multiplied it. Whatever it is, don't tattle on me.
 
realScore = getScore(); // The real score! Too bad no one will see it now!
fakeMultiplier = 2.5f; // This multiplier will increase the score by 2.5 times the real score.
benchmarkBooster = realScore * fakeMultiplier; // The final score our "benchmarkphiles" will see! This will convince them, which ultimately increase our sales by 5,000 units!
printf(@"benchmarkBooster %i", benchmarkBooster);

 

[jonthedit]

Hah. I can't imagina tat:

// If this info isn't leafed, do not tell that the CEO, that's me, or head director told you to do so. Blame it on the willy you were shaking. Or that you accidentally entered it insideout.
 
realScore = getShake(); // The real score! Too bad no one will see it now!
fakeMultiplier = 2.5f; // This multiplier will increase the score by 2.5 times the real score.
benchmarkBooster = realScore * jerkit; // The final score our "benchmarkphiles" will see! This will convince them, which ultimately increase our sales by 5,000 units!
printf(@"benchmarkBooster %i", benchthatweight);

Even without the overclocking the phone is still great for N64 emulation

 

[IBNobody]

Haha! I can actually imagine that:

// TOP SECRET - Do not let anyone see this code --- even your friend, wife, son, dog, cat, or your second part-time mean boss!
// All who fail to comply will be persecuted to the full extent of law!
// If this info is leaked, do not tell that the CEO, that's me, or head director explicitly instruct you to add this code. Blame it on the IDE you were using. Or that you accidentally multiplied it. Whatever it is, don't tattle on me.
 
realScore = getScore(); // The real score! Too bad no one will see it now!
fakeMultiplier = 2.5f; // This multiplier will increase the score by 2.5 times the real score.
benchmarkBooster = realScore * fakeMultiplier; // The final score our "benchmarkphiles" will see! This will convince them, which ultimately increase our sales by 5,000 units!
printf(@"benchmarkBooster %i", benchmarkBooster);

Their formula is wrong. It didn't take into account the competitor's benchmark score.

benchmarkBooster = realScore * fakeMultiplier + 0.1f * competitorBenchmark;

 

[Lastly]

Even without the overclocking the phone is still great for N64 emulation

A Galaxy S2 have no problem running most games, too. Don't get the wrong idea. I think the Galaxy S4 is great.

 

[Thanatos Telos]

Still more honest than apple.

 

[Lastly]

Still more honest than apple.

As expected from Thanatos.

I remember reading that faking the benchmark, actually slow down the device. At least from what I remember reading a year ago.

 

[Thanatos Telos]

As expected from Thanatos.

I remember reading that faking the benchmark, actually slow down the device. At least from what I remember reading a year ago.

Feel the cold treachery!

 

[shakirmoledina]

Welcome to the world of marketing.

Samsung will just release a new iteration with no 'tampering'.

 

[DarkSpace]

Remember it's not unethical until someone catches you doing cheating the system.

 

[FAST6191]

Remember it's not unethical until someone catches you doing cheating the system.

I am going instead with if the system is meaningless then cheating the meaningless system is itself meaningless and thus outside ethics.

 

[Ethevion]

People care enough for Samsung to want to cheat.

Considering they go head to head with Apple in the smart phone segment, I'd say yes.

 

[Foxi4]

I don't think it's cheating per-se and here's my reasoning:

• The benchmarking process is supposed to measure the maximum possible performance of hardware
• Normally portable handsets run on underclocked values to preserve battery life and prevent overheating which may damage your handset in the long run
• While running a benchmark, the hardware goes full-blast. Those values are not overclocked - they're the maximum possible values for the chips
• If you're skilled enough, you can manually select the higher stepping, however it's locked out for normal operation to prevent the forementioned overheating and draining the battery

Now, what Samsung should've done to be 100% "clean" was to embed a so-called "High Performance" mode which the user could switch to after agreeing (by checking a checkbox) that doing so may cause damage to the handset in high temperature/low air circulation conditions as well as severely lower the battery life of the overall setup. That way it'd be up to to the user to use those values and risk damaging the handset if it does go into total meltdown. The way it's been handled makes it seem fishy but I can see the logic behind it.

 

[IBNobody]

I've updated the OP with Samsung's response.

Still more honest than apple.

To be fair, everybody would be jumping on Apple if this were them.

 

[Gahars]

I always thought Samsung was a bench warmer, but this is ridiculous.