Samsung issues second, unsubstantiated denial of Galaxy S4, Note 3 benchmark cheating

Posted:
in iPhone edited January 2014
In the face of a second set of investigative reports detailing exactly how Samsung leads other Android makers in exaggerating its performance on specific benchmarks, the company has issued its second denial this year, and which still fails to address any specifics.

Samsung cheat.jar


In a statement issued to CNET UK today, Samsung stated, "The Galaxy Note 3 maximises its CPU/GPU frequencies when running features that demand substantial performance. This was not an attempt to exaggerate particular benchmarking results. We remain committed to providing our customers with the best possible user experience."

Samsung's benchmarks for the Galaxy Note 3 do indeed indicate that the device "maximises its CPU/GPU frequencies when running" benchmarks, which certainly are "features that demand substantial performance."

However, Samsung's claim that this "was not an attempt to exaggerate particular benchmarking results" is belied by the fact that the maximization only occurs when running specific benchmarks and stops happening when the name of benchmark app is changed.

Or when the benchmark maker specifically changes their benchmark to stop such cheating, as was the case this week with AnTuTu, one of the benchmark apps Samsung's flagship devices identify by name when running, expressly in order to exaggerate their results.

This all happened before

Back in July, Samsung offered the same explanation to John Paczkowski of the Wall Street Journal "All Things Digital" blog, which the site characterized as "Samsung?s Bizarre Benchmark-Boosting Explanation."

Samsung's statement then was nearly identical: "the maximum GPU [graphical processing unit] frequencies for the GALAXY S4 have been varied to provide optimal user experience for our customers, and were not intended to improve certain benchmark results."

However, the company also contradicted itself and confirmed what AnandTech had reported about specifically juicing certain benchmarks within the same statement, adding:

"A maximum GPU frequency of 533MHz is applicable for running apps that are usually used in full-screen mode," Samsung said, "such as the S Browser, Gallery, Camera, Video Player, and certain benchmarking apps, which also demand substantial performance."

Additionally, the facts showed back then that tests involving Samsung's Browser, Gallery and Video Player didn't results in a GPU boost, instead leaving the processor clocked at 266MHz. And of course, the name of the software controlling the change was "BenchmarkBooster."

So Samsung admitted boosting benchmarks back in July while also saying it didn't (in the same statement!), and said it was also boosting other apps that it actually wasn't. This time around, Samsung has only removed the explicit admission that it considers benchmarks among the apps that "demand substantial performance" and require special handling from its BenchmarkBooster code.

Why Samsung's cheating doesn't win


Computing benchmarks are similar to a road test designed to compare the speed and agility of different cars driving on a specific route. However, mobile benchmarks also aim to test not just the brute speed, but rather the overall performance typical of running within real-world efficiency parameters related to battery consumption and heat dissipation.

Mobile benchmarks are therefore like a test track that involves not just a speed element, but also additional constraints related to mileage, emissions and overheating, to show not just what a souped up modified vehicle could do, but instead how well a customer's car will actually perform in real world use. Samsung does not activate "maximized frequencies" when running real apps, because if it did, its other benchmarks would suffer, particularly battery life and product reliability.

Samsung activates various performance modes when running specific benchmarks, including activating all idle cores, increasing the clock rate and making other, unknown changes that boost its graphics performance over identical hardware.

While these changes boost scores by 20 to up to 50 percent (in AnandTech testing), they do so at the cost of overheating components and running down the battery. These impacts are so costly to the overall experience that Samsung does not activate "maximized frequencies" when running real apps, because if it did, its other benchmarks would suffer, particularly battery life and product reliability.

Samsung's efforts to shoo attention away from investigations by AnandTech and ArsTechnica weren't enough to convince Wall Street Journal or CNET, but did seem to pacify the concerns of some Android fans.

"Which is true? Will we ever know? Does it even matter?" asked Zach Epstein of BGR after reciting Samsung's statement.

"Bam! Straight from the horses mouth, they don't cheat!" one reader commented. Another, represented by an Android avatar with the caption "hope," wrote, "I agree. I will believe anything from Google/DROID/Samsung. They are honest. I will never listen to or believe anything from or about Apple."

A higher rated comment observed, "Cheating or not on benchmarks, their Exynos and Qualcomm chips they use in their Galaxy line up are getting beat on every benchmark by Apple's A7. So Samsung is playing catch up."
«134

Comments

  • Reply 1 of 62

    Exactly how does a user get better "user experience" by only getting the performance boost when a benchmark runs? Their statement might make sense if the devices actually did this more than just for a whitelist of benchmark apps. I'm sure we'll see this press release make the rounds on all the Fandroid sites as they try to suppress this as some sort of "non-story" despite the fact that they'll crow every time some random Android device beats the iPhone in any random benchmark.

  • Reply 2 of 62
    slurpyslurpy Posts: 5,382member
    Samsung lies through their teeth, yet again. What a shocking development. It's scary how little integrity this company has.
  • Reply 3 of 62
    sockrolidsockrolid Posts: 2,789member

    Originally Posted by AppleInsider View Post



    This was not an attempt to exaggerate particular benchmarking results. We remain committed to providing our customers with the best possible user experience.

     

    That's like saying "I did not rob Bank of America.  I was simply very committed to providing myself with a large amount of cash."

  • Reply 4 of 62
    Hey Scumsung,

    Add my apps to the boost list please, they don't run as fast as they could and I'm feeling discriminated. It's not fair !
  • Reply 5 of 62

    "I didn't cheat," said the kid caught cheating.

  • Reply 6 of 62
    maestro64maestro64 Posts: 5,043member
    Here is how you fix this problem, as this article points out the benchmark number if just one of the things that tell you how good the product is. So if Sammy want to play this game let them, run the benchmark test until the battery die and record the time so report the performance as a function of usable life and see what happens.

    In a prior career I did performance testing on Laptops which we would measure battery performance as a function of doing everyday things like word processing and such. But our tool was more of a simulation of these activities not actually having someone sit their and type and such. As part of this we did real world testing and make sure the tool closely match the real world applications.

    We would test hundred of systems over time so we had good statistical data. The tool data was very close to real world except with on groupd of applications. It was MS office, we did not understand when running the office programs on a Powerbook the battery life was 1/2 as expected. After some research we found that MS was making direct calls to the processor to do a display update ever 10 to 20 secs. When MS was asked why they were talking directly to the processor and not operating through the tool box they said they did not want their customer seeing a performance hit because apple implemented power saving modes. By doing this they did not allow the process to idle or sleep so it was on all the time using up battery power. In the end the PowerBooks battery life was not better then a PC if you were using Office.
  • Reply 7 of 62

    Looks like Samsung is having a well-deserved bad week!

     

    About time that all the shady stuff finally caught up with them.....

  • Reply 8 of 62
    I hope Samsung's ethics don't represent that part of the world in general. I am sure they don't though, corporate entities seem to consistently set new lows for morals, ethics and social well-being when it comes to the effects of their activities.
  • Reply 9 of 62
    Quote:

    Originally Posted by AppleInsider View Post



    Computing benchmarks are similar to a road test designed to compare the speed and agility of different cars driving on a specific route.

    Apropos of that.... here's an interesting news item on Korean car manufacturers: http://www.forbes.com/sites/jimgorzelany/2012/11/02/epa-slams-hyundai-and-kia-for-overestimating-mpg/

  • Reply 10 of 62
    yojimbo007yojimbo007 Posts: 1,165member
    alienzed wrote: »
    I hope Samsung's ethics don't represent that part of the world in general. I am sure they don't though, corporate entities seem to consistently set new lows for morals, ethics and social well-being when it comes to the effects of their activities.
    It is a sad fact..... And last thing i want to do is discriminate and stereotype!
    But having seen Samsungs lying, cheating, bribing , copying , misleading behavior over and over has made me not only not want to touch a samsung product... But i am even turned off by the Kia sportage i have. ....... And often feel /question... Is this how koreans are?
  • Reply 11 of 62
    xflarexflare Posts: 199member

    What a shady company.

  • Reply 12 of 62
    nagrommenagromme Posts: 2,834member
    Who cares, if you're Samsung? 90% of the public will never remember the cheating story. Of those that to, 90% will remember the story that "others do it too" and assume that includes even Apple. Of those that really WERE worried for moment, 90% will now believe Samsung's false denials, no doubt bolstered by classic astroturfers.

    That adds up to GOOD marketing!

    It's not the first time in public discourse that facts have proven unimportant in driving opinions...
  • Reply 13 of 62
    poochpooch Posts: 768member
    maestro64 wrote: »
    Here is how you fix this problem, as this article points out the benchmark number if just one of the things that tell you how good the product is. So if Sammy want to play this game let them, runn the benchmark test until the battery die and record the time so report the performance as a function of useable life and see what happens.

    they'd just leave it plugged in, and claim battery-life improvements.
  • Reply 14 of 62
    It seems that according to Samsung, vociferous and repeated denials of copying and blatant cheating will somehow invoke an Obi-Wan Jedi mind trick over the tech press and anyone dumb enough to swallow their lies.
  • Reply 15 of 62
    Quote:

    "Bam! Straight from the horses mouth, they don't cheat!" one reader commented. Another, represented by an Android avatar with the caption "hope," wrote, "I agree. I will believe anything from Google/DROID/Samsung. They are honest. I will never listen to or believe anything from or about Apple."

     

    And they say Apple fans live in a reality distortion field and worship Steve Jobs like an idol. These Fandroids are insane!!

  • Reply 16 of 62
    Gnumas: "I am not a crook".

    Yeah right, Mr. Nixon.
  • Reply 17 of 62
    These things add up: the cheating, poor marks on innovation for the S4, failed smart watch, painting their plastic gold, all scream 2nd class products. They are running scared and are trying to figure out what to do.
  • Reply 18 of 62
    rissriss Posts: 47member
    I wouldn't be surprised if those 'positive' comments Daniel is mentioning were from paid Samsungtrolls.
  • Reply 19 of 62
    jpmiajpmia Posts: 63member
    I dont see cnbc taking about this...
  • Reply 20 of 62
    yojimbo007yojimbo007 Posts: 1,165member
    sog35 wrote: »
    Did Enron stop you from believing in American products?
    Or Bear Sterns?
    Or Goldman Sacs?

    Yes.. Definitely shook my confidence... ... In the system.

    Also another thing to note here is i live here in usa..i am more familiar with the US mentality than i am of some country i have never lived in or been too.. So naturally , samsung like behavior effects my feeling more for cultures i am not familiar with and Raise the question of " is this how they are " more easily!
Sign In or Register to comment.