or Connect
AppleInsider › Forums › Mobile › iPhone › Samsung again caught doping benchmarks for Galaxy Note 3
New Posts  All Forums:Forum Nav:

Samsung again caught doping benchmarks for Galaxy Note 3

post #1 of 103
Thread Starter 
An early review of Samsung's Galaxy Note 3 noted that it "scores really, really well in benchmark tests," but also explained that edge is achieved through some phony doping that kicks in a steroid boost only when running synthetic benchmark apps.

Samsung cheat.jar


The review, by Ron Amadeo of ArsTechnica, details that Samsung is up to the same shenanigans it was called out on in an earlier report by AnandTech.

Snapdragon 800 vs Apple A7



Without any cheating, Geekbench 3 portrays the Samsung SM-N900P Note 3 as being essentially tied with LG's Optimus G2, with the same 2.3 GHz quad core chip and slightly more RAM (2.38 GB vs LG's 1.38 GB).

The AppleInsider review of Apple's iPhone 5s, running a dual core A7 at half the clock speed (1.3 GHz) and using less than half the RAM (1 GB), reported 64-bit scores that were essentially tied between the two others, and slightly higher in memory performance.

The two Android phones from Samsung and LG can't run 64-bit versions of the GeekBench 3 app because they are based on older technology. Instead, they are designed to scream through 32-bit benchmarks with more cores clocked twice as fast. That generates heat and wastes battery life, necessitating a larger device with a bigger battery.



Samsung cheating to look better than it does



Back in July, Anand Lal Shimpi & Brian Klug noted that versions of Samsung's Galaxy S 4 equipped with the company's Exynos 5 Octa were designed explicitly to report better that real-world scores that were not actually attainable.

Amadeo now points out that Samsung's repeat performance with the Galaxy Note 3 does the same for the company's latest flagship phablet even on the U.S. version powered by Qualcomm's Snapdragon 800 rather than Samsung's own chip.Samsung is now covering its tracks, making it more difficult for reviewers to access the true CPU clock speed and discern that the company is faking performance.

Simply changing the name of the benchmarking app turned in scores that were between 20 and 50 percent lower, a phenomenon explained by extracting code on the device that revealed Samsung expressly designed the phone to turn in faster that real-world results when running known benchmark tests.

The report noted that Samsung earlier issued an official statement that claimed its "benchmark boosting" on the Galaxy S 4 was actually used in other processor intensive apps such as the web browser, camera and in video playback, not just in benchmarking apps. But that isn't true, ArsTechnica noted.

Instead, according to Samsung's configuration files, "the function is used exclusively for benchmarks, and it seems to cover all the popular ones. There's Geekbench, Quadrant, Antutu, Linpack, GFXBench, and even some of Samsung's own benchmarks," Amadeo wrote.

Samsung also appears to be faking its graphics performance, tipped off by a "suspicious 'frame rate adjustment' string" and the inclusion of GFXBench, an OpenGL ES benchmark targeting GPU performance.

Samsung doubles down on doping



Amadeo also reports that Samsung is now covering its tracks, making it more difficult for reviewers to access the true CPU clock speed and discern that the company is faking performance.

Doped benchmarks don't just show up in reviews; they also appear on benchmark sites as user submitted rankings, making Samsung's falsifying of its performance a case of false advertising. But Samsung has also launched its own benchmark app, branded "MobileBench."

Samsung cheat.jar


"To measure and monitor system performance, you must install many applications for many different purposes; however, MobileBench? provides an all-in-one solution," the company explains.

"MobileBench? measures H/W performance, scores user experience, and compares results with other devices," the company says. "Take advantage of your results to promote your products and optimize your devices."

Now that Samsung is heavily invested in the design of its Exynos chip it can only sell on a fraction of its high end handsets, and faces a race to catch up with Apple's entirely different approach to performance (using a more modern, custom designed A7 engine) the leading Android phone maker is now focusing on fooling its customers with phony performance numbers.
post #2 of 103
Finally - something that Sammy can definitively show they did NOT copy from Apple.
post #3 of 103
This will become like baseball records. Will geekbench purge the submitted benchmarks from its database? Put an asterisk by them? So absolutely blatant. I mean, looking up a text string for god's sake. Its not even trying to algorithmically balance... or even HIDE THE CHEAT all that well.

Also looking at that chart makes me drool wondering what the A7X is gonna do.
Doodle Dice iPhone puzzle game: A fun, free physics-laden collection of dice games.  Greatest app made yet?  Perhaps young man... Perhaps.
Reply
Doodle Dice iPhone puzzle game: A fun, free physics-laden collection of dice games.  Greatest app made yet?  Perhaps young man... Perhaps.
Reply
post #4 of 103

Schiller can say  "shenanigans" on Twitter...Samsung will reply back with "20+ millions sold"

post #5 of 103
Well, technically, they aren't cheating. They are handicapping all non benchmark testing apps. The hardware is, obviously, capable of these speeds in some manner.

But to make the user experience better, they put on these governors. Without them, we get the "cheat" scores...

...ok, I have to go shower now after writing all that...
post #6 of 103
Is this another Samsung "feature?"

Also, I like this "new" Apple. Cagey. I dig it.
post #7 of 103
Quote:
Originally Posted by Ricardo Dawkins View Post
 

Schiller can say  "shenanigans" on Twitter...Samsung will reply back with "20+ millions sold"

 

I know. It's terrible when they just can't stop lying ;)

post #8 of 103
Shameless
post #9 of 103

I don't think Samsung was surprised by Apple's 64 bit architecture (lol, they built it).  What caught them off guard was the media hype Apple was able to get out of it.  Many tech oriented people questioned why you'd even need it with the state of current phones' hardware (there will be a need in the future).

 

The 32 bit note is faster than the 64 bit flagship Apple 5s.

 

Except for when Samsung cheats- then it is a lot faster.

post #10 of 103
Quote:
Originally Posted by Ricardo Dawkins View Post
 

Schiller can say  "shenanigans" on Twitter...Samsung will reply back with "20+ millions sold"

 

'20+ millions sold accomplished by lying to our customers.'

post #11 of 103

These devices that are locked down and have proprietary code that can be used to hide the actual GPU/CPU performance characteristics are the opposite of freedom.  If only those sheeple would use Android phones which are completely open source and nothing can be hidden from the user.   

 

Oh Wait...

post #12 of 103
First Lance Armstrong and now Samsung. When will this end?
post #13 of 103
What i would like to see is a giant, well publisized class action law suit against samsung for misleading consumers and fraudulant behavior .
Something that will expose this low life company wide open to the whole world !
post #14 of 103
Quote: Frood

The 32 bit note is faster than the 64 bit flagship Apple 5s.

 

Except for when Samsung cheats- then it is a lot faster.

 

Did you even read the benchmarking article?  The 5S is much faster than the Note.   The Note is sweating to beat the LG.  Reading is fundamental.

post #15 of 103
Samsung is digging its own grave.. Through fraud!
post #16 of 103
The next thing we will hear is that they sold 18 million phones over a weekend.
post #17 of 103
Yup. U get the speed and 1 hour battery life!
post #18 of 103
Quote:
Originally Posted by Frood View Post
 

I don't think Samsung was surprised by Apple's 64 bit architecture (lol, they built it).  What caught them off guard was the media hype Apple was able to get out of it.  Many tech oriented people questioned why you'd even need it with the state of current phones' hardware (there will be a need in the future).

 

The 32 bit note is faster than the 64 bit flagship Apple 5s.

 

Except for when Samsung cheats- then it is a lot faster.

 

Interesting point on the apple surprise.  That may have been the impetus.

 

However note that both samsung and the other high end phones are absolutely pounding clock-rate to stay near Apple and its quiet and cool A7.

Doodle Dice iPhone puzzle game: A fun, free physics-laden collection of dice games.  Greatest app made yet?  Perhaps young man... Perhaps.
Reply
Doodle Dice iPhone puzzle game: A fun, free physics-laden collection of dice games.  Greatest app made yet?  Perhaps young man... Perhaps.
Reply
post #19 of 103
Quote:
Originally Posted by starbird73 View Post

Well, technically, they aren't cheating. They are handicapping all non benchmark testing apps. The hardware is, obviously, capable of these speeds in some manner.

But to make the user experience better, they put on these governors. Without them, we get the "cheat" scores...

...ok, I have to go shower now after writing all that...
It is cheating when other companies aren't doing the same. It's deliberately trying to mislead so when tech sites do reviews and post benchmark scores Samsung comes out on top even though consumers will never see that performance in real world use.
post #20 of 103

Scamsung once again shows its' true colors.:lol:

post #21 of 103

Fandroids and Android got nothing on Apple.

 

They even need to lie about their benchmarks now.:lol:

 

What a bunch of lowlife losers!:D 

post #22 of 103
Quote:
Originally Posted by Frood View Post

I don't think Samsung was surprised by Apple's 64 bit architecture (lol, they built it).  What caught them off guard was the media hype Apple was able to get out of it.  Many tech oriented people questioned why you'd even need it with the state of current phones' hardware (there will be a need in the future).

The 32 bit note is faster than the 64 bit flagship Apple 5s.

Except for when Samsung cheats- then it is a lot faster.
Take a hike bribed troll!

Samsung is digging its own grave.. Through fraud and trolls like you!
post #23 of 103
Quote:
Originally Posted by Frood View Post
 

The 32 bit note is faster than the 64 bit flagship Apple 5s.

 

Base on which metric?  The dual core A7 @ 1.3Ghz beats the Samsung Exynos 8-cores @ 1.8Ghz on both efficiency and performance hands down. 

post #24 of 103
Quote:
Originally Posted by starbird73 View Post

Well, technically, they aren't cheating. They are handicapping all non benchmark testing apps. The hardware is, obviously, capable of these speeds in some manner.

But to make the user experience better, they put on these governors. Without them, we get the "cheat" scores...

...ok, I have to go shower now after writing all that...

Sure it is cheating. At best those speeds in continued processing would result in abhorrent battery life, and at worst—and not terribly unlikely over some short span of time—it could result in hardware failure. Benchmarks are only meaningful if they work under the same conditions that any other app would.
The true measure of a man is how he treats someone that can do him absolutely no good.
  Samuel Johnson
Reply
The true measure of a man is how he treats someone that can do him absolutely no good.
  Samuel Johnson
Reply
post #25 of 103
Quote:
Originally Posted by starbird73 View Post

Well, technically, they aren't cheating. They are handicapping all non benchmark testing apps. The hardware is, obviously, capable of these speeds in some manner.

But to make the user experience better, they put on these governors. Without them, we get the "cheat" scores...

...ok, I have to go shower now after writing all that...

 

Actually it is cheating. Benchmarks are meant to test real world performance... that is performance that all other apps are capable of achieving. Clearly that's not the case when Samsung singles out specific apps and sets the CPU into boost mode just for those apps, especially when those apps are mostly benchmarking apps. They can claim that a few other apps make use of this, but those are specialized cases and not indicative of how all other apps will perform.

Disclaimer: The things I say are merely my own personal opinion and may or may not be based on facts. At certain points in any discussion, sarcasm may ensue.
Reply
Disclaimer: The things I say are merely my own personal opinion and may or may not be based on facts. At certain points in any discussion, sarcasm may ensue.
Reply
post #26 of 103
Quote:
Originally Posted by Frood View Post

I don't think Samsung was surprised by Apple's 64 bit architecture (lol, they built it).  What caught them off guard was the media hype Apple was able to get out of it.  Many tech oriented people questioned why you'd even need it with the state of current phones' hardware (there will be a need in the future).

That's some mighty fine apologetics. See how long you can keep reality at bay by telling yourself that 64-bit doesn't matter. You only need to do that until Samsung/Google can come around with their own 64-bit phone, then you can flip-flop and extol the virtues of 64-bit phones. Just remember who got their first.

"Apple should pull the plug on the iPhone."

John C. Dvorak, 2007
Reply

"Apple should pull the plug on the iPhone."

John C. Dvorak, 2007
Reply
post #27 of 103
Quote:
Originally Posted by BigMac2 View Post

Base on which metric?  The dual core A7 @ 1.3Ghz beats the Samsung Exynos 8-cores @ 1.8Ghz on both efficiency and performance hands down. 

According to the chart in the article, you're wrong. The non-cheating Note scores higher than the 5S in all but one metric. If you look at performance per core or per clock, then the 5S pulls ahead.
post #28 of 103
Quote:
Originally Posted by Frood View Post
 

The 32 bit note is faster than the 64 bit flagship Apple 5s.

 

Except for when Samsung cheats- then it is a lot faster.

 

It clearly scores higher, but that's not the point here.

 

The A7 is a dual core SoC running at 1.3GHz and it can definitely hold its own against the "real" Note 3 performance, which is a 4 core SoC running at 2.3GHz.

 

Throw a 4 core A7 running at 2.3GHz up there and see just how "fast" the Note 3 is. An A7 beefed up that much would approach the power of the CPU in the MacBook Air.

 

In mobile devices, efficiency takes precedence over performance and it is clear that the A7 is a much, much more efficient chip than the Snapdragon 800.

Disclaimer: The things I say are merely my own personal opinion and may or may not be based on facts. At certain points in any discussion, sarcasm may ensue.
Reply
Disclaimer: The things I say are merely my own personal opinion and may or may not be based on facts. At certain points in any discussion, sarcasm may ensue.
Reply
post #29 of 103
Quote:
Originally Posted by starbird73 View Post

Well, technically, they aren't cheating. They are handicapping all non benchmark testing apps. The hardware is, obviously, capable of these speeds in some manner.

But to make the user experience better, they put on these governors. Without them, we get the "cheat" scores...

...ok, I have to go shower now after writing all that...

 

Problem is with raising the frequency rapidly decrease the efficiency as it's greatly increase the power consumption and heat.  So every mobile SoC has do be throttle to maximize efficiency and current sweat spot is under 1.5 Ghz.  With the A7, Apple has made clear choices with a lower frequency and lower cores counts but with greater performance and efficiency then its competitor. 

post #30 of 103

Shameful behavior.  Samsung I can believe as that sham of a company decides every time that no level is too low to stoop too.

Fandroids and iHaters show exceptional hypocrisy and shameful denial.  Had Apple did this, they would be screaming at the top of their lungs and demanding lawsuits.  These jackasses are actually defending Samsung's reasonings.  Unfrickenbelievable.  


Edited by sflocal - 10/1/13 at 12:59pm
post #31 of 103
Quote:
Originally Posted by Frood View Post
 

I don't think Samsung was surprised by Apple's 64 bit architecture (lol, they built it).  What caught them off guard was the media hype Apple was able to get out of it.  Many tech oriented people questioned why you'd even need it with the state of current phones' hardware (there will be a need in the future).

 

The 32 bit note is faster than the 64 bit flagship Apple 5s.

 

Except for when Samsung cheats- then it is a lot faster.

 

I hope it's faster. It's twice as large, the battery is twice as large, it has twice the cores, at twice the clock speed, three times the memory, a fraction of the battery life and it still BARELY beats out the iPhone 5s. All in a package the size of a book.

Truly, a marvel.

post #32 of 103

If the point of the test is to measure *normal* processing power, then any conspicuous throttle-up by one of entrants above and beyond the norm, when it shouldn't be happening under normal operation, *is* artificially inflating the power of that processor. 

 

If it's not cheating, it's using a feature of your processor that will operate in a certain manner under conditions that don't warrant it, for the purpose of getting a higher benchmark. 

 

It's as synonymous with cheating as you can possibly get. 

 

And oh how very Samsung of Samsung. 

post #33 of 103

You expect doping scandals in sports but who ever thought they would also show up in smart phone benchmarks.

 

I hope this story gains a lot of traction beyond just a few Apple sites like this one and makes it into mainstream news outlets as well. Samsung can not get away with lying about how their phone will perform with these amazing benchmark scores and then when people buy and actually use them they fall far short of that. 

 

Talk about a great ad campaign for HTC, LG, or even Microsoft attacking this fraud by Samsung. Apple is above that but those other companies should run with it. 

 

Never argue with an idiot. They will only bring you down to their level and beat you with experience. 

Reply

 

Never argue with an idiot. They will only bring you down to their level and beat you with experience. 

Reply
post #34 of 103
Quote:
Originally Posted by Suddenly Newton View Post


That's some mighty fine apologetics. See how long you can keep reality at bay by telling yourself that 64-bit doesn't matter. You only need to do that until Samsung/Google can come around with their own 64-bit phone, then you can flip-flop and extol the virtues of 64-bit phones. Just remember who got their first.

 

More precisely, ARM64 programs derive much of their performance boost from the new instructions available only to A64 builds. If ARM had decided instead to take the Intel route of making the new instructions available to 32-bit programs as well, the phrase "64-bit" wouldn't be getting this much attention. That's why the transition to 64-bit computing on the desktop happened without fanfare.

post #35 of 103
Quote:
Originally Posted by d4NjvRzf View Post
 

 

More precisely, ARM64 programs derive much of their performance boost from the new instructions available only to A64 builds. That's why the improved performance is associated with the phrase "64-bit", which technically describes merely the register width. If ARM had decided instead to take the Intel route of making the new instructions available to 32-bit programs as well, the phrase "64-bit" wouldn't be getting this much attention.

 

This is a non-issue, 

 

There is a lot goings on with the new AArch64 and AArch32 ISA for ARM SoC. here is a good in-depth Q&A

http://www.mikeash.com/pyblog/friday-qa-2013-09-27-arm64-and-you.html

post #36 of 103
Quote:
Originally Posted by BigMac2 View Post
 

 

There is a lot goings on with the new AArch64 and AArch32 ISA for ARM SoC. here is a good in-depth Q&A

http://www.mikeash.com/pyblog/friday-qa-2013-09-27-arm64-and-you.html

 

Very informative article, thanks. This summarizes things nicely.

Quote:

The "64-bit" A7 is not just a marketing gimmick, but neither is it an amazing breakthrough that enables a new class of applications. The truth, as happens often, lies in between.

 

The simple fact of moving to 64-bit does little. It makes for slightly faster computations in some cases, somewhat higher memory usage for most programs, and makes certain programming techniques more viable. Overall, it's not hugely significant.

 

The ARM architecture changed a bunch of other things in its transition to 64-bit. An increased number of registers and a revised, streamlined instruction set make for a nice performance gain over 32-bit ARM.

 
post #37 of 103

If I were developing apps for the Galaxy devices, I would want to enable that speed boost myself. I wonder what happens if it is enabled for all apps all the time? Obviously it saps your battery life but not when the app is idle right? What is the down side to having it on all the time? Does it melt the CPU or something?

post #38 of 103
I really hate the fact that this conniving, snake in the grass company is the only Android vendor making money.
"I got the answer by talking in my brain and I agreed of the answer my brain got" a 7 yr old explaining his math HW
"Just because something is deemed the law doesn't make it just" - SolipsismX
Reply
"I got the answer by talking in my brain and I agreed of the answer my brain got" a 7 yr old explaining his math HW
"Just because something is deemed the law doesn't make it just" - SolipsismX
Reply
post #39 of 103
Quote:
Originally Posted by Srice View Post
 

 

Did you even read the benchmarking article?  The 5S is much faster than the Note.   The Note is sweating to beat the LG.  Reading is fundamental.

 

What is more fundamental than reading is comprehension, or even being able to draw your own conclusions based on data.  This is AI and usually is pretty good at spinning the facts to support their team.  Yes, if you 'read' the article it makes it sound like the 5s would really like to be faster- so I'm assuming that's what you're making your 'The 5S is much faster than the Note' claim despite the article showing the data that proves otherwise?

 

If you look at the actual data provided.....  The Note 3 was faster in 3 out of the 4 tests, without cheating (the cheating tests are no contest, and without question dubious on Samsung's part).  The article references this "essentially tied' but is quick to point out that 'Apple won' in the one metric that it actually did slightly beat the others.  Its pretty good kindergarden spin when 'winning is winning, but losing is a tie'

 

The author is quick to make the tests seem skewed or almost unfair because Apple ran them with 'half the clock speed and half the RAM' and then nicely schmoozes the fact that the test was run on a native 64 bit benchmark instead of the running the same test as the other phones.  In my book both tests are perfectly valid- the Apple 5s ships with half the clock speed and half the RAM... and the other phones don't support 64 bit so there you go.  But even running the native 64 bit benchmark it was still slower in 3 of the 4 tests as unpopular a result as that might be on this site.

post #40 of 103
Quote:
Originally Posted by wakefinance View Post


According to the chart in the article, you're wrong. The non-cheating Note scores higher than the 5S in all but one metric. If you look at performance per core or per clock, then the 5S pulls ahead.

 

 

lolwut? per core or per clock????  That would be taking spin to new levels.  That would be like saying Android devices are twice as fast 'per bus bit.'  Both would be true statements... but pointless.

New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: iPhone
AppleInsider › Forums › Mobile › iPhone › Samsung again caught doping benchmarks for Galaxy Note 3