or Connect
AppleInsider › Forums › Mobile › iPhone › Samsung issues second, unsubstantiated denial of Galaxy S4, Note 3 benchmark cheating
New Posts  All Forums:Forum Nav:

Samsung issues second, unsubstantiated denial of Galaxy S4, Note 3 benchmark cheating

post #1 of 59
Thread Starter 
In the face of a second set of investigative reports detailing exactly how Samsung leads other Android makers in exaggerating its performance on specific benchmarks, the company has issued its second denial this year, and which still fails to address any specifics.

Samsung cheat.jar


In a statement issued to CNET UK today, Samsung stated, "The Galaxy Note 3 maximises its CPU/GPU frequencies when running features that demand substantial performance. This was not an attempt to exaggerate particular benchmarking results. We remain committed to providing our customers with the best possible user experience."

Samsung's benchmarks for the Galaxy Note 3 do indeed indicate that the device "maximises its CPU/GPU frequencies when running" benchmarks, which certainly are "features that demand substantial performance."

However, Samsung's claim that this "was not an attempt to exaggerate particular benchmarking results" is belied by the fact that the maximization only occurs when running specific benchmarks and stops happening when the name of benchmark app is changed.

Or when the benchmark maker specifically changes their benchmark to stop such cheating, as was the case this week with AnTuTu, one of the benchmark apps Samsung's flagship devices identify by name when running, expressly in order to exaggerate their results.

This all happened before



Back in July, Samsung offered the same explanation to John Paczkowski of the Wall Street Journal "All Things Digital" blog, which the site characterized as "Samsung?s Bizarre Benchmark-Boosting Explanation."

Samsung's statement then was nearly identical: "the maximum GPU [graphical processing unit] frequencies for the GALAXY S4 have been varied to provide optimal user experience for our customers, and were not intended to improve certain benchmark results."

However, the company also contradicted itself and confirmed what AnandTech had reported about specifically juicing certain benchmarks within the same statement, adding:

"A maximum GPU frequency of 533MHz is applicable for running apps that are usually used in full-screen mode," Samsung said, "such as the S Browser, Gallery, Camera, Video Player, and certain benchmarking apps, which also demand substantial performance."

Additionally, the facts showed back then that tests involving Samsung's Browser, Gallery and Video Player didn't results in a GPU boost, instead leaving the processor clocked at 266MHz. And of course, the name of the software controlling the change was "BenchmarkBooster."

So Samsung admitted boosting benchmarks back in July while also saying it didn't (in the same statement!), and said it was also boosting other apps that it actually wasn't. This time around, Samsung has only removed the explicit admission that it considers benchmarks among the apps that "demand substantial performance" and require special handling from its BenchmarkBooster code.

Why Samsung's cheating doesn't win



Computing benchmarks are similar to a road test designed to compare the speed and agility of different cars driving on a specific route. However, mobile benchmarks also aim to test not just the brute speed, but rather the overall performance typical of running within real-world efficiency parameters related to battery consumption and heat dissipation.

Mobile benchmarks are therefore like a test track that involves not just a speed element, but also additional constraints related to mileage, emissions and overheating, to show not just what a souped up modified vehicle could do, but instead how well a customer's car will actually perform in real world use. Samsung does not activate "maximized frequencies" when running real apps, because if it did, its other benchmarks would suffer, particularly battery life and product reliability.

Samsung activates various performance modes when running specific benchmarks, including activating all idle cores, increasing the clock rate and making other, unknown changes that boost its graphics performance over identical hardware.

While these changes boost scores by 20 to up to 50 percent (in AnandTech testing), they do so at the cost of overheating components and running down the battery. These impacts are so costly to the overall experience that Samsung does not activate "maximized frequencies" when running real apps, because if it did, its other benchmarks would suffer, particularly battery life and product reliability.

Samsung's efforts to shoo attention away from investigations by AnandTech and ArsTechnica weren't enough to convince Wall Street Journal or CNET, but did seem to pacify the concerns of some Android fans.

"Which is true? Will we ever know? Does it even matter?" asked Zach Epstein of BGR after reciting Samsung's statement.

"Bam! Straight from the horses mouth, they don't cheat!" one reader commented. Another, represented by an Android avatar with the caption "hope," wrote, "I agree. I will believe anything from Google/DROID/Samsung. They are honest. I will never listen to or believe anything from or about Apple."

A higher rated comment observed, "Cheating or not on benchmarks, their Exynos and Qualcomm chips they use in their Galaxy line up are getting beat on every benchmark by Apple's A7. So Samsung is playing catch up."
post #2 of 59

Exactly how does a user get better "user experience" by only getting the performance boost when a benchmark runs? Their statement might make sense if the devices actually did this more than just for a whitelist of benchmark apps. I'm sure we'll see this press release make the rounds on all the Fandroid sites as they try to suppress this as some sort of "non-story" despite the fact that they'll crow every time some random Android device beats the iPhone in any random benchmark.

post #3 of 59
Samsung lies through their teeth, yet again. What a shocking development. It's scary how little integrity this company has.
post #4 of 59
Originally Posted by AppleInsider View Post

This was not an attempt to exaggerate particular benchmarking results. We remain committed to providing our customers with the best possible user experience.

 

That's like saying "I did not rob Bank of America.  I was simply very committed to providing myself with a large amount of cash."

Sent from my iPhone Simulator

Reply

Sent from my iPhone Simulator

Reply
post #5 of 59
Hey Scumsung,

Add my apps to the boost list please, they don't run as fast as they could and I'm feeling discriminated. It's not fair !
post #6 of 59

"I didn't cheat," said the kid caught cheating.

post #7 of 59
Here is how you fix this problem, as this article points out the benchmark number if just one of the things that tell you how good the product is. So if Sammy want to play this game let them, run the benchmark test until the battery die and record the time so report the performance as a function of usable life and see what happens.

In a prior career I did performance testing on Laptops which we would measure battery performance as a function of doing everyday things like word processing and such. But our tool was more of a simulation of these activities not actually having someone sit their and type and such. As part of this we did real world testing and make sure the tool closely match the real world applications.

We would test hundred of systems over time so we had good statistical data. The tool data was very close to real world except with on groupd of applications. It was MS office, we did not understand when running the office programs on a Powerbook the battery life was 1/2 as expected. After some research we found that MS was making direct calls to the processor to do a display update ever 10 to 20 secs. When MS was asked why they were talking directly to the processor and not operating through the tool box they said they did not want their customer seeing a performance hit because apple implemented power saving modes. By doing this they did not allow the process to idle or sleep so it was on all the time using up battery power. In the end the PowerBooks battery life was not better then a PC if you were using Office.
Edited by Maestro64 - 10/3/13 at 11:00am
post #8 of 59

Looks like Samsung is having a well-deserved bad week!

 

About time that all the shady stuff finally caught up with them.....

post #9 of 59
I hope Samsung's ethics don't represent that part of the world in general. I am sure they don't though, corporate entities seem to consistently set new lows for morals, ethics and social well-being when it comes to the effects of their activities.
post #10 of 59
Quote:
Originally Posted by AppleInsider View Post

Computing benchmarks are similar to a road test designed to compare the speed and agility of different cars driving on a specific route.

Apropos of that.... here's an interesting news item on Korean car manufacturers: http://www.forbes.com/sites/jimgorzelany/2012/11/02/epa-slams-hyundai-and-kia-for-overestimating-mpg/

post #11 of 59
Quote:
Originally Posted by alienzed View Post

I hope Samsung's ethics don't represent that part of the world in general. I am sure they don't though, corporate entities seem to consistently set new lows for morals, ethics and social well-being when it comes to the effects of their activities.
It is a sad fact..... And last thing i want to do is discriminate and stereotype!
But having seen Samsungs lying, cheating, bribing , copying , misleading behavior over and over has made me not only not want to touch a samsung product... But i am even turned off by the Kia sportage i have. ....... And often feel /question... Is this how koreans are?
post #12 of 59

What a shady company.

post #13 of 59
Who cares, if you're Samsung? 90% of the public will never remember the cheating story. Of those that to, 90% will remember the story that "others do it too" and assume that includes even Apple. Of those that really WERE worried for moment, 90% will now believe Samsung's false denials, no doubt bolstered by classic astroturfers.

That adds up to GOOD marketing!

It's not the first time in public discourse that facts have proven unimportant in driving opinions...
post #14 of 59
Quote:
Originally Posted by Yojimbo007 View Post


It is a sad fact..... And last thing i want to do is discriminate and stereotype!
But having seen Samsungs lying, cheating, bribing , copying , misleading behavior over and over has made me not only not want to touch a samsung product... But i am even turned off by the Kia sportage i have. ....... And often feel /question... Is this how koreans are?

 

Did Enron stop you from believing in American products?

Or Bear Sterns?

Or Goldman Sacs?

post #15 of 59
Quote:
Originally Posted by Maestro64 View Post

Here is how you fix this problem, as this article points out the benchmark number if just one of the things that tell you how good the product is. So if Sammy want to play this game let them, runn the benchmark test until the battery die and record the time so report the performance as a function of useable life and see what happens.

they'd just leave it plugged in, and claim battery-life improvements.
"Personally, I would like nothing more than to thoroughly proof each and every word of my articles before posting. But I can't."

appleinsider's mike campbell, august 15, 2013
Reply
"Personally, I would like nothing more than to thoroughly proof each and every word of my articles before posting. But I can't."

appleinsider's mike campbell, august 15, 2013
Reply
post #16 of 59
It seems that according to Samsung, vociferous and repeated denials of copying and blatant cheating will somehow invoke an Obi-Wan Jedi mind trick over the tech press and anyone dumb enough to swallow their lies.
post #17 of 59
Quote:
"Bam! Straight from the horses mouth, they don't cheat!" one reader commented. Another, represented by an Android avatar with the caption "hope," wrote, "I agree. I will believe anything from Google/DROID/Samsung. They are honest. I will never listen to or believe anything from or about Apple."

 

And they say Apple fans live in a reality distortion field and worship Steve Jobs like an idol. These Fandroids are insane!!

Help! I'm trapped in a white dungeon of amazing precision and impeccable tolerances!

Reply

Help! I'm trapped in a white dungeon of amazing precision and impeccable tolerances!

Reply
post #18 of 59
Gnumas: "I am not a crook".

Yeah right, Mr. Nixon.
post #19 of 59
These things add up: the cheating, poor marks on innovation for the S4, failed smart watch, painting their plastic gold, all scream 2nd class products. They are running scared and are trying to figure out what to do.
post #20 of 59
I wouldn't be surprised if those 'positive' comments Daniel is mentioning were from paid Samsungtrolls.
post #21 of 59
I dont see cnbc taking about this...
post #22 of 59
Quote:
Originally Posted by sog35 View Post

Did Enron stop you from believing in American products?
Or Bear Sterns?
Or Goldman Sacs?

Yes.. Definitely shook my confidence... ... In the system.

Also another thing to note here is i live here in usa..i am more familiar with the US mentality than i am of some country i have never lived in or been too.. So naturally , samsung like behavior effects my feeling more for cultures i am not familiar with and Raise the question of " is this how they are " more easily!
post #23 of 59
Quote:
Originally Posted by Pooch View Post


they'd just leave it plugged in, and claim battery-life improvements.

 

In the new smart phones you can actually disable charging in software so plug it in the tool can disable the charging. But if independent are running the test they are not going to things to help anyone supplier look better.

post #24 of 59
So if benchmarks don't matter, why does Samsung (and others) boost them?

"Apple should pull the plug on the iPhone."

John C. Dvorak, 2007
Reply

"Apple should pull the plug on the iPhone."

John C. Dvorak, 2007
Reply
post #25 of 59
Quote:
Originally Posted by Suddenly Newton View Post

So if benchmarks don't matter, why does Samsung (and others) boost them?

 

Because Samsung and others are focused on the wrong things, which is why their marketing sucks and their product designs suck (unless they are copying Apple). 

 

Even if Apple were totally amoral, I doubt Apple would bother cheating on these benchmarks because from a marketing standpoint it's a stupid thing to do and from an engineering standpoint there are better things to do with their time. 

post #26 of 59

All I'm interested in is a reassurance that when I add fabric softener to the special slot that it really does get added at the right time and not just "cheated" in at the start. Come on Sammie; I'm counting on your washer/dryer honesty here!

post #27 of 59
Pathetic. Simply pathetic.
post #28 of 59

of course this once again proves the blatent double standard in the tech media. if Apple did this you know it would be an instant "BenchGate" sensation, top headline news at, say, the Verge - which so far has not posted any report about this at all. and these guys claim to be "journalists." Consumer Reports would issue a press release, like they did for so-called "AntennaGate."

 

but instead the reactions are mostly So What? and related lame excuses. i guess in one sense they are right - Samsung has been cheating for years in every imaginiable way (including dominating Korea's government), and everyone should know that already. so this isn't "news."

 

let's see if they all go back and "update" the reviews of the identified cheater smartphones still on the market today that include benchmark results and comparisons with the compeition - especially iPhone. wanna bet?

post #29 of 59
Quote:
Originally Posted by Blastdoor View Post

Even if Apple were totally amoral, I doubt Apple would bother cheating on these benchmarks because from a marketing standpoint it's a stupid thing to do and from an engineering standpoint there are better things to do with their time. 

Apple's customers don't tend to look for benchmarks that ignore other usability features like an overheating device or reduced battery life. That's simply not in Apple's DNA.
post #30 of 59
Quote:
Originally Posted by sog35 View Post
 

 

Did Enron stop you from believing in American products?

Or Bear Sterns?

Or Goldman Sacs?

 

 

Quote:
Originally Posted by Yojimbo007 View Post


Yes.. Definitely shook my confidence... ... In the system.

Also another thing to note here is i live here in usa..i am more familiar with the US mentality than i am of some country i have never lived in or been too.. So naturally , samsung like behavior effects my feeling more for cultures i am not familiar with and Raise the question of " is this how they are " more easily!

Spoken like a true xenophobe. 

 

So not disclosing that their phones run a benchmark at full capacity is worse than bankrupting families and businesses, causing millions of people to lose their jobs, and cheating investors of hundreds of billions of their life savings?  Yeah, I can see how a benchmark number would be much worse than those things.  Let's not forget that BP destroyed the Gulf with their oil spill while lying to the US public about the amount of the oil being release while simultaneously dumping large quantities of dispersants to cover up their lies.  But I bet you are fine with British companies and products because they speak English with a great accent!

post #31 of 59
Quote:
Originally Posted by akqies View Post


Apple's customers don't tend to look for benchmarks that ignore other usability features like an overheating device or reduced battery life. That's simply not in Apple's DNA.

 

I suspect that's true of most of Samsung's customers, too. Samsung, like so many other tech companies, does not really understand its customers. I suspect that's one reason Samsung (and Dell et al) throw so many products up on the wall to see what sticks -- they just have no idea what real people want or need. They design for and market to the tech nerd press. 

post #32 of 59

Seems the popular press doesn't want to pick up on this story.  

 

CNN Tech's two main tech. headlines right now are:

 

"Atari Founder: Tim Cook isn't the next Steve Jobs" and

"5 problems with iOS 7, and how to fix them".

 

Nice huh?

 

 I wonder if Google will reprimand them for this in any way?   

post #33 of 59
Quote:
Originally Posted by Maestro64 View Post

Here is how you fix this problem, as this article points out the benchmark number if just one of the things that tell you how good the product is. So if Sammy want to play this game let them, run the benchmark test until the battery die and record the time so report the performance as a function of usable life and see what happens.

In a prior career I did performance testing on Laptops which we would measure battery performance as a function of doing everyday things like word processing and such. But our tool was more of a simulation of these activities not actually having someone sit their and type and such. As part of this we did real world testing and make sure the tool closely match the real world applications.

We would test hundred of systems over time so we had good statistical data. The tool data was very close to real world except with on groupd of applications. It was MS office, we did not understand when running the office programs on a Powerbook the battery life was 1/2 as expected. After some research we found that MS was making direct calls to the processor to do a display update ever 10 to 20 secs. When MS was asked why they were talking directly to the processor and not operating through the tool box they said they did not want their customer seeing a performance hit because apple implemented power saving modes. By doing this they did not allow the process to idle or sleep so it was on all the time using up battery power. In the end the PowerBooks battery life was not better then a PC if you were using Office.

Great historical story, thanks! So, MS really doesn't know how to code well. Such a pity that they never became this big because they excel at what they do. They merely became big because of other ways of doing business. Bit like Google, and quite sad.

Oh well, true colors will shine through at some point, and people will vote with their wallet.
"Fibonacci: As easy as 1, 1, 2, 3..."
Reply
"Fibonacci: As easy as 1, 1, 2, 3..."
Reply
post #34 of 59
Quote:
Originally Posted by Maestro64 View Post

Here is how you fix this problem, as this article points out the benchmark number if just one of the things that tell you how good the product is. So if Sammy want to play this game let them, run the benchmark test until the battery die and record the time so report the performance as a function of usable life and see what happens.

That would work - if you could get any of the reviewers to do it. Unfortunately, most of the reviewers are so deeply in Samsung's camp that they'd never go along.
"I'm way over my head when it comes to technical issues like this"
Gatorguy 5/31/13
Reply
"I'm way over my head when it comes to technical issues like this"
Gatorguy 5/31/13
Reply
post #35 of 59

The popular press had best be careful in reporting on this subject or they risk losing hundreds of millions in $camscum advertising dollors.

Who will be first to lambaste them, hmmm? (The will be the first to receive my subscription money.)

post #36 of 59
Quote:
Originally Posted by Blastdoor View Post

I suspect that's true of most of Samsung's customers, too. Samsung, like so many other tech companies, does not really understand its customers. I suspect that's one reason Samsung (and Dell et al) throw so many products up on the wall to see what sticks -- they just have no idea what real people want or need. They design for and market to the tech nerd press. 

I concede to your point and agree that very few customers look at raw benchmarks.

From my anecdotal experience (at least on tech forums) Android users that have a more limited comprehension of the technology which allows them to make superficial judgements about comparative capabilities. One needs only to look at AI's recent article about Qualcomm saying the A7 is only for marketing to get plenty of choice examples.
post #37 of 59
Quote:
Originally Posted by Jafo View Post


Spoken like a true xenophobe. 

So not disclosing that their phones run a benchmark at full capacity is worse than bankrupting families and businesses, causing millions of people to lose their jobs, and cheating investors of hundreds of billions of their life savings?  Yeah, I can see how a benchmark number would be much worse than those things.  Let's not forget that BP destroyed the Gulf with their oil spill while lying to the US public about the amount of the oil being release while simultaneously dumping large quantities of dispersants to cover up their lies.  But I bet you are fine with British companies and products because they speak English with a great accent!

Lol.. Did u even read what my response was? Lol... U seem to want to argue with yourself !

As for my xenophobia.. Lol..lets see.. I am . Part British , part Armenian, born in Iran. .. Lived in France and Usa. ..........
post #38 of 59
Quote:
"A maximum GPU frequency of 533MHz is applicable for running apps that are usually used in full-screen mode," Samsung said, "such as the S Browser, Gallery, Camera, Video Player, and certain benchmarking apps, which also demand substantial performance."

I've got some bad news for you sunshine...
post #39 of 59
Quote:
Originally Posted by AppleInsider View Post

"Which is true? Will we ever know? Does it even matter?" asked Zach Epstein of BGR after reciting Samsung's statement.

"Bam! Straight from the horses mouth, they don't cheat!" one reader commented. Another, represented by an Android avatar with the caption "hope," wrote, "I agree. I will believe anything from Google/DROID/Samsung. They are honest. I will never listen to or believe anything from or about Apple."

 

 

I find it amazingly obnoxious when NBC Nightly News or MSM newspapers quote a forum or Twitter comment in the midst of what should be real journalism.  Not that AI projects itself as real journalism, but anyone who has spent more than 5 minutes scanning comments on BGR on any given day knows that Norm or any of his clones "represented by an Android avatar with the caption 'hope'" are all known trolls who haven't written a serious comment ever.  I guess I should be embarrassed for knowing that because the BGR comments section is so flawed and filled with refuse that your brain cells die just from looking at it.  It really makes me laugh that Norm is getting quoted on AI as source material.

post #40 of 59
Quote:
Originally Posted by AppleInsider View Post

Another, represented by an Android avatar with the caption "hope," wrote, "I agree. I will believe anything from Google/DROID/Samsung. They are honest. I will never listen to or believe anything from or about Apple."

How could the wiring of anyone's brain be so completely haywire as to make them capable of writing something like this?
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: iPhone
  • Samsung issues second, unsubstantiated denial of Galaxy S4, Note 3 benchmark cheating
AppleInsider › Forums › Mobile › iPhone › Samsung issues second, unsubstantiated denial of Galaxy S4, Note 3 benchmark cheating