Stay on Top of Enterprise Technology Trends
Get updates impacting your industry from our GigaOm Research Community
When Ars Technica tested the Samsung Galaxy Note 3 earlier this week, it noted that performance was fast. Maybe too fast. The phone showed a significant gain in speeds over the LG G2, which uses the same processor. And after some poking around the site discovered that Samsung appears to be artificially inflating benchmark scores through a code that makes the CPU act differently when running certain benchmarking programs.
This isn’t the first time that this has happened. Samsung was caught doing something very similar a year ago, with the GPU in the international version of the Galaxy S 4.
That’s strike two for Samsung, and I wonder if it’s the start of something greater. Back in the height of the PC era, news like this was a lot more common. Manufacturers would “optimize” their products to perform better on certain benchmarks. If mobile phones follow the same path and start competing on performance, will these benchmarking shenanigans start to happen all over again?
Samsung’s benchmarking “adjustments”
When stacked up next to the LG G2, the Samsung Galaxy Note 3 should show similar, if not equal, performance. Instead it was benchmarking anywhere from 20 to 50 percent faster than the LG G2, which uses the same quad-core Qualcomm(s qcom) Snapdragon 800 chip.
To account for the difference, Ars Technica decided to look into CPU idle speeds. By using a system monitor app it discovered that, when idle, the Galaxy Note 3 only uses one of its four cores in order to conserve power, and that core drops down to 300MHz. When running a benchmark, on the other hand, all four cores engage at the maximum 2.3GHz at all times, even when the phone would otherwise be idling.
Ars Technica was able to locate the file that triggers the phone to boost its CPU performance, and discovered a number of popular mobile benchmarks written into it, including the ones it tested. The language used in the coding refers to a “boost all adjustment.”
The important thing to understand is that the benchmarks scores achieved by the Galaxy Note 3 aren’t fake. Instead, the hardware has been “adjusted” to run at its peak capacity to achieve the highest score possible. But these scores aren’t accurate because the Galaxy Note 3 almost never runs like this otherwise. It’s still a very fast phone — its Snapdragon 800 chip makes it one of the fastest phones available — but artificially boosting its benchmark scores gives the wrong impression for what users can expect from real-world performance.
A quick look back
Back in the PC’s heyday, stories like this were a lot more common. Here’s one from just over 10 years ago that sounds nearly identical to what we’re hearing now.
3DMark developer Futuremark accused Nvidia of modifying the software drivers for its products to make them perform differently when running the 3DMark03 benchmark. Nvidia’s drivers would replace certain shaders with ones that its graphics cards could render more easily, thus boosting its 3DMark score.
Nvidia didn’t budge and refused to change its drivers. Eventually Futuremark and Nvidia basically agreed to disagree. “There are application-specific optimizations in every CPU and every GPU,” an Nvidia spokesman said at the time.
Now that mobile processors are faster than ever, why would Samsung want to artificially inflate its benchmark scores? Well, with smartphone manufacturers touting PC-like performance, perhaps it’s because this is the first time that smartphones are truly being sold based on performance. Apple(s aapl), for instance, recently made a big push for its new A7 chip in the iPhone 5s, which is the first smartphone processor with 64-bit architecture. Naturally, people are going to want to make performance comparisons, and mobile chip vendors are talking about developing formal benchmark standards.
For many users, specs and performance matter just as much as physical design and app availability. With new processors coming out on such a regular basis, it makes sense to buy a phone that will remain a strong performer through the length of a two-year contract. Think about it – two years ago one of the top Android phones was the Galaxy S II. It should still run fine now if you have one, but I wouldn’t expect top performance out of a new game like Need for Speed Most Wanted.
But adjusting phones to inflate benchmark scores isn’t going to help anyone. Instead, what manufacturers like Samsung should do is focus more on user experience. I’ve seen plenty of phones with powerful processors bogged down by too much bloatware and a bad software overlay. And I’d personally rather use a phone with a great overall experience than a phone with the absolute highest AnTuTu score.