4 Comments

Summary:

It looks like Samsung has written code into the Galaxy Note 3 that causes it to artificially inflate benchmark scores. Could artificial benchmark boosting become as common among mobile phone makers as it was among PC vendors?

Galaxy Note3 FlipCover_004_Open Pen_Wild Orange

When Ars Technica tested the Samsung Galaxy Note 3 earlier this week, it noted that performance was fast. Maybe too fast. The phone showed a significant gain in speeds over the LG G2, which uses the same processor. And after some poking around the site discovered that Samsung appears to be artificially inflating benchmark scores through a code that makes the CPU act differently when running certain benchmarking programs.

This isn’t the first time that this has happened. Samsung was caught doing something very similar a year ago, with the GPU in the international version of the Galaxy S 4.

That’s strike two for Samsung, and I wonder if it’s the start of something greater. Back in the height of the PC era, news like this was a lot more common. Manufacturers would “optimize” their products to perform better on certain benchmarks. If mobile phones follow the same path and start competing on performance, will these benchmarking shenanigans start to happen all over again?

Samsung’s benchmarking “adjustments”

When stacked up next to the LG G2, the Samsung Galaxy Note 3 should show similar, if not equal, performance. Instead it was benchmarking anywhere from 20 to 50 percent faster than the LG G2, which uses the same quad-core Qualcomm Snapdragon 800 chip.

To account for the difference, Ars Technica decided to look into CPU idle speeds. By using a system monitor app it discovered that, when idle, the Galaxy Note 3 only uses one of its four cores in order to conserve power, and that core drops down to 300MHz. When running a benchmark, on the other hand, all four cores engage at the maximum 2.3GHz at all times, even when the phone would otherwise be idling.

Galaxy Note 3 Geekbench

Ars Technica was able to locate the file that triggers the phone to boost its CPU performance, and discovered a number of popular mobile benchmarks written into it, including the ones it tested. The language used in the coding refers to a “boost all adjustment.”

The important thing to understand is that the benchmarks scores achieved by the Galaxy Note 3 aren’t fake. Instead, the hardware has been “adjusted” to run at its peak capacity to achieve the highest score possible. But these scores aren’t accurate because the Galaxy Note 3 almost never runs like this otherwise. It’s still a very fast phone — its Snapdragon 800 chip makes it one of the fastest phones available — but artificially boosting its benchmark scores gives the wrong impression for what users can expect from real-world performance.

A quick look back

Back in the PC’s heyday, stories like this were a lot more common. Here’s one from just over 10 years ago that sounds nearly identical to what we’re hearing now.

3DMark developer Futuremark accused Nvidia of modifying the software drivers for its products to make them perform differently when running the 3DMark03 benchmark. Nvidia’s drivers would replace certain shaders with ones that its graphics cards could render more easily, thus boosting its 3DMark score.

Nvidia didn’t budge and refused to change its drivers. Eventually Futuremark and Nvidia basically agreed to disagree. “There are application-specific optimizations in every CPU and every GPU,” an Nvidia spokesman said at the time.

Why now?

Now that mobile processors are faster than ever, why would Samsung want to artificially inflate its benchmark scores? Well, with smartphone manufacturers touting PC-like performance, perhaps it’s because this is the first time that smartphones are truly being sold based on performance. Apple, for instance, recently made a big push for its new A7 chip in the iPhone 5s, which is the first smartphone processor with 64-bit architecture. Naturally, people are going to want to make performance comparisons, and mobile chip vendors are talking about developing formal benchmark standards.

For many users, specs and performance matter just as much as physical design and app availability. With new processors coming out on such a regular basis, it makes sense to buy a phone that will remain a strong performer through the length of a two-year contract. Think about it – two years ago one of the top Android phones was the Galaxy S II. It should still run fine now if you have one, but I wouldn’t expect top performance out of a new game like Need for Speed Most Wanted.

But adjusting phones to inflate benchmark scores isn’t going to help anyone. Instead, what manufacturers like Samsung should do is focus more on user experience. I’ve seen plenty of phones with powerful processors bogged down by too much bloatware and a bad software overlay. And I’d personally rather use a phone with a great overall experience than a phone with the absolute highest AnTuTu score.

  1. Well, we all can fully trust Samsung…… cause their convicted CEO says we can….. And that should be good enough for us…….. right?

    Samsung,,, more than happy to s@rew its customers in the pursuit of profits!!! Just saying!!

    Share
  2. This is shocking. I just feel that the only company I can trust these days is Apple. Thanks to Phil Shiller for dusting off “shenanigans” too. Keep it real.

    Share
  3. This is much ado about nothing. Asus, HTC and LG do it as well; http://www.anandtech.com/show/7376/samsung-galaxy-note-3-review/4

    Share
  4. The amazing thing is how Samsung gets a pass on this. If Apple did this, the Senate would actually pass the budget, get back to work so they could convene a hearing on how Apple is ruining America’s schoolchildren … but Samsung, no big deal. EVERYONE does this, right? right? Android, Oracle’s borrowed OS, lowering ethical standards at twice the usual speed!

    Share

Comments have been disabled for this post