But calling 'benchmark booster', 'faking performance', 'benchmark shenanigans' to one of Android manuracturers and accusing them as 'cheater' because they tested benchmarks at the CPU speed that was advertised for is pure evil journalism to me.
Just so you are clear, I will remind you that the hardware, you so oh hail as "advertised" was advertised to deliver yay amount of performance, while delivering yay amount of battery backup. In Note 3's case, they promise certain amount of battery backup under normal usage conditions. These usage conditions employ the CPU to run at a power efficient state (which is normally what you use). It is just to call it "faking" because by putting the CPU to a performance state, it is delivering its full performance, yes, but at the cost of reduced battery life.
?
For the result not be a fake, either:
?1. Samsung use the normal CPU state policy across all apps
?2. Or, Samsung use the power policy and openly state a much reduced battery life.
?
?The fact that Samsung uses power policy just for benchmarks (to improve results), and still claim that the CPU runs at normal power (which is given by way of official battery backup specifications), is pure unadulterated fraud.
Just so you are clear, I will remind you that the hardware, you so oh hail as "advertised" was advertised to deliver yay amount of performance, while delivering yay amount of battery backup. In Note 3's case, they promise certain amount of battery backup under normal usage conditions. These usage conditions employ the CPU to run at a power efficient state (which is normally what you use). It is just to call it "faking" because by putting the CPU to a performance state, it is delivering its full performance, yes, but at the cost of reduced battery life.
?
For the result not be a fake, either:
?1. Samsung use the normal CPU state policy across all apps
?2. Or, Samsung use the power policy and openly state a much reduced battery life.
?
?The fact that Samsung uses power policy just for benchmarks (to improve results), and still claim that the CPU runs at normal power (which is given by way of official battery backup specifications), is pure unadulterated fraud.
I will be calling them 'cheater' only if they have overclocked its CPU when performing the benchmark tests. But they did not as long as I know.
And any manufacturer can limit its CPU or any of its components to optimize their user experience for normal users and of course all have been doing it ages. What is wrong with it?
Car manufactures advertise how fast they could go like 150miles per hour as a benchmark, but would anyone be complaining why they do not set benchmark 60miles an hour?
First of all, I am not a Samsung apologist. And that's fine that you blocked me for whatever the reason.
This limiting CPU speed for some of its applications in Android are not new as Ars notified. So this is a hardly news.
But calling 'benchmark booster', 'faking performance', 'benchmark shenanigans' to one of Android manuracturers and accusing them as 'cheater' because they tested benchmarks at the CPU speed that was advertised for is pure evil journalism to me.
The LG G2 having absolutely the same CPU seems to lag behind in those performance tests. So, it seems that only Samsung is capable of "limiting CPU speeds"?
While scaling down the CPU is obviously an industry practice, boosting it is not. Intel does that, calling it turbo boost, but it is available for each and every application running on your computer, no matter if it is a Windows, Linux or Mac OS X box.
Samsung does that specifically and only for performance tests. Those applications don't do any real work, but measure CPU capabilities. So, yes, that is the exact definitionof cheating. It is a fake performance, because it is not achievable in actual applications. And it is "benchmark shenanigans", since Samsung did not explicitly state they have special configuration for benchmark apps.
1. 64-bit CPUs do not have advantage over 32-bit CPUs when running 32-bit tests/benchmarks as the software does not take advantage of the additional registers and their greater size. ARM Cortex A15 already has a 64-bit data path between RAM and CPU cache.
2. Snapdragon is running at a 77% higher frequency, meaning more than twice the power needed (non-linear increase).
3. Snapdragon has twice the cores, meaning twice the consumption when all cores work (as is the case with benchmarks). That's why ARM created the big.LITTLE architecture. Samsung's implementation, however, is awful.
So, for close to 5 times the power consumption you get mind-blowing 3% better performance scores.
4. Benchmarks usually scale extremely well (close to perfect) on multi-core platforms. That is never the case with real-world applications, except for some cases that are usually run on supercomputers, or on GPUs.
So, it seems to me that the iPhone 5s has the most powerful mobile CPU. Apple chose not to over-do it with 4 cores, nor boost the frequency. Because A7's purpose is to power aphone, not a data center.
Well...the above post ended all the argument. Good job!
I will be calling them 'cheater' only if they have overclocked its CPU when performing the benchmark tests. But they did not as long as I know.
And any manufacturer can limit its CPU or any of its components to optimize their user experience for normal users and of course all have been doing it ages. What is wrong with it?
Car manufactures advertise how fast they could go like 150miles per hour as a benchmark, but would anyone be complaining why they do not set benchmark 60miles an hour?
Benchmark is just a benchmark.
I still don't think that you get what I am saying. Either Samsung should use the same state policy across all apps (to deliver the advertised performance with the advertised battery life), or use the power policy that they use in benchmarks, in all apps and openly admit that the battery life, CPU life have significantly decreased and there is a lot of heat generation. The fact that Samsung uses the power policy only for benchmarks is the act of window-dressing and that alone is misrepresentation.
Didn't you see the post? When they changed the name of the app, the performance decrease was 35% at an average, or put simply, the real performance of Note 3 is actually 35% less than what it has you believe in benchmarks. Isn't that misrepresentation?
Benchmark boosting is widespread among Android OEMs, cut throat market I suppose, reminds me a little of the dynamic contrast ratio ratings of monitors, we end up seeing ridiculous numbers and it becomes meaningless. It already is kind of meaning less between OS platforms, if not just for the reason that if say you are loyal to Android, why does it matter to you what a Windows phone or iPhone scores?
Benchmark boosting is widespread among Android OEMs, cut throat market I suppose, reminds me a little of the dynamic contrast ratio ratings of monitors, we end up seeing ridiculous numbers and it becomes meaningless. It already is kind of meaning less between OS platforms, if not just for the reason that if say you are loyal to Android, why does it matter to you what a Windows phone or iPhone scores?
Did you note this additional comment he made?
"Google Experience (Nexus) devices aren't affected" which shows it's being done intentionally by some of the others. Google plays it straight with Nexus models, as they should anyway especially since they're used to premier new OS versions.
Why manufacturer should use the same power policy across the board? It is up to them for optimisation.
What Sammy has done wrong again?
Can any app opt into the aggressive power policy that Samsung uses for those benchmarks? If so, why does Samsung hardcode those benchmarks in a special whitelist? If not, wouldn't the benchmark scores lead consumers to wrongly conclude that the Note 3 can run real-world apps significantly faster than other similarly equipped devices?
First, Note 3 runs benchmarks at 2.3Ghz that SD800 is advertised for. So, there is no such thing like "benchmark boosting". ie. Note 3 does not overclock when performing benchmark tests.
I am not royal to Android. My family still enjoy more than 2 years old iPad2 and recommends iPad to anyone, although I prefer Android for smartphones over iOS.
As you might see from above, I do not have any interest what a Windows phone or iPhone scores, for now of course.
First, Note 3 runs benchmarks at 2.3Ghz that SD800 is advertised for. So, there is no such thing like "benchmark boosting". ie. Note 3 does not overclock when performing benchmark tests. I am not royal to Android. My family still enjoy more than 2 years old iPad2 and recommends iPad to anyone, although I prefer Android for smartphones over iOS. As you might see from above, I do not have any interest what a Windows phone or iPhone scores, for now of course.
[SIZE=12px]First of all, I am not a Samsung apologist. And that's fine that you blocked me for whatever the reason. [/SIZE]
[SIZE=12px]This limiting CPU speed for some of its applications in Android are not new as Ars notified. So this is a hardly news. [/SIZE]
[SIZE=12px]But calling <span style="line-height:22.386363983154297px;"> </span> 'benchmark booster', 'faking performance', 'benchmark shenanigans' to one of Android manuracturers and accusing them as 'cheater' because they tested benchmarks at the CPU speed that was advertised for is pure evil journalism to me.[/SIZE]
Why manufacturer should use the same power policy across the board? It is up to them for optimisation.
What Sammy has done wrong again?
Optimization is one thing. 'Optimization' would involve figuring out what the app does and then assigning the appropriate conditions for that app. But when the same app is 'optimized' differently by merely changing it's name, that's not optimization. It's artificially giving known benchmark apps an advantage that other apps don't have.
The purpose of benchmarks is to give you an easily measured performance metric that allows you to compare one device to another. Ideally, if device A is 30% faster in real life, then it should be 30% faster on the benchmark. With this type of gamesmanship, the benchmark loses all value. It is 'optimized' differently than the apps that people are using and is therefore useless as an indicator of which device is faster.
"Google Experience (Nexus) devices aren't affected" which shows it's being done intentionally by some of the others. Google plays it straight with Nexus models, as they should anyway especially since they're used to premier new OS versions.
I didn't read every comment, but hey he wrote them, not me. And gooooood for you if you own a Nexus, have a cookie.
The purpose of benchmarks is to give you an easily measured performance metric that allows you to compare one device to another. Ideally, if device A is 30% faster in real life, then it should be 30% faster on the benchmark. With this type of gamesmanship, the benchmark loses all value. It is 'optimized' differently than the apps that people are using and is therefore useless as an indicator of which device is faster.
I didn't read every comment, but hey he wrote them, not me. And gooooood for you if you own a Nexus, have a cookie.
Has nothing to do with which device you use. It speaks to how reliable the benchmarking results are. If it's a Nexus device they're not gaming the tests. If it's Samsung it's likely they are.
Comments
But calling 'benchmark booster', 'faking performance', 'benchmark shenanigans' to one of Android manuracturers and accusing them as 'cheater' because they tested benchmarks at the CPU speed that was advertised for is pure evil journalism to me.
Just so you are clear, I will remind you that the hardware, you so oh hail as "advertised" was advertised to deliver yay amount of performance, while delivering yay amount of battery backup. In Note 3's case, they promise certain amount of battery backup under normal usage conditions. These usage conditions employ the CPU to run at a power efficient state (which is normally what you use). It is just to call it "faking" because by putting the CPU to a performance state, it is delivering its full performance, yes, but at the cost of reduced battery life.
?
For the result not be a fake, either:
?1. Samsung use the normal CPU state policy across all apps
?2. Or, Samsung use the power policy and openly state a much reduced battery life.
?
?The fact that Samsung uses power policy just for benchmarks (to improve results), and still claim that the CPU runs at normal power (which is given by way of official battery backup specifications), is pure unadulterated fraud.
Just so you are clear, I will remind you that the hardware, you so oh hail as "advertised" was advertised to deliver yay amount of performance, while delivering yay amount of battery backup. In Note 3's case, they promise certain amount of battery backup under normal usage conditions. These usage conditions employ the CPU to run at a power efficient state (which is normally what you use). It is just to call it "faking" because by putting the CPU to a performance state, it is delivering its full performance, yes, but at the cost of reduced battery life.
?
For the result not be a fake, either:
?1. Samsung use the normal CPU state policy across all apps
?2. Or, Samsung use the power policy and openly state a much reduced battery life.
?
?The fact that Samsung uses power policy just for benchmarks (to improve results), and still claim that the CPU runs at normal power (which is given by way of official battery backup specifications), is pure unadulterated fraud.
I will be calling them 'cheater' only if they have overclocked its CPU when performing the benchmark tests. But they did not as long as I know.
And any manufacturer can limit its CPU or any of its components to optimize their user experience for normal users and of course all have been doing it ages. What is wrong with it?
Car manufactures advertise how fast they could go like 150miles per hour as a benchmark, but would anyone be complaining why they do not set benchmark 60miles an hour?
Benchmark is just a benchmark.
First of all, I am not a Samsung apologist. And that's fine that you blocked me for whatever the reason.
This limiting CPU speed for some of its applications in Android are not new as Ars notified. So this is a hardly news.
But calling 'benchmark booster', 'faking performance', 'benchmark shenanigans' to one of Android manuracturers and accusing them as 'cheater' because they tested benchmarks at the CPU speed that was advertised for is pure evil journalism to me.
The LG G2 having absolutely the same CPU seems to lag behind in those performance tests. So, it seems that only Samsung is capable of "limiting CPU speeds"?
While scaling down the CPU is obviously an industry practice, boosting it is not. Intel does that, calling it turbo boost, but it is available for each and every application running on your computer, no matter if it is a Windows, Linux or Mac OS X box.
Samsung does that specifically and only for performance tests. Those applications don't do any real work, but measure CPU capabilities. So, yes, that is the exact definition of cheating. It is a fake performance, because it is not achievable in actual applications. And it is "benchmark shenanigans", since Samsung did not explicitly state they have special configuration for benchmark apps.
1. 64-bit CPUs do not have advantage over 32-bit CPUs when running 32-bit tests/benchmarks as the software does not take advantage of the additional registers and their greater size. ARM Cortex A15 already has a 64-bit data path between RAM and CPU cache.
2. Snapdragon is running at a 77% higher frequency, meaning more than twice the power needed (non-linear increase).
3. Snapdragon has twice the cores, meaning twice the consumption when all cores work (as is the case with benchmarks). That's why ARM created the big.LITTLE architecture. Samsung's implementation, however, is awful.
So, for close to 5 times the power consumption you get mind-blowing 3% better performance scores.
4. Benchmarks usually scale extremely well (close to perfect) on multi-core platforms. That is never the case with real-world applications, except for some cases that are usually run on supercomputers, or on GPUs.
So, it seems to me that the iPhone 5s has the most powerful mobile CPU. Apple chose not to over-do it with 4 cores, nor boost the frequency. Because A7's purpose is to power a phone, not a data center.
Well...the above post ended all the argument. Good job!
I will be calling them 'cheater' only if they have overclocked its CPU when performing the benchmark tests. But they did not as long as I know.
And any manufacturer can limit its CPU or any of its components to optimize their user experience for normal users and of course all have been doing it ages. What is wrong with it?
Car manufactures advertise how fast they could go like 150miles per hour as a benchmark, but would anyone be complaining why they do not set benchmark 60miles an hour?
Benchmark is just a benchmark.
I still don't think that you get what I am saying. Either Samsung should use the same state policy across all apps (to deliver the advertised performance with the advertised battery life), or use the power policy that they use in benchmarks, in all apps and openly admit that the battery life, CPU life have significantly decreased and there is a lot of heat generation. The fact that Samsung uses the power policy only for benchmarks is the act of window-dressing and that alone is misrepresentation.
Didn't you see the post? When they changed the name of the app, the performance decrease was 35% at an average, or put simply, the real performance of Note 3 is actually 35% less than what it has you believe in benchmarks. Isn't that misrepresentation?
Why manufacturer should use the same power policy across the board? It is up to them for optimisation.
What Sammy has done wrong again?
Crysisftw.
Why manufacturer should use the same power policy across the board? It is up to them for optimisation.
What Sammy has done wrong again?
See this comment by Anand of Anandtech:
http://anandtech.com/comments/7376/samsung-galaxy-note-3-review/331108
Benchmark boosting is widespread among Android OEMs, cut throat market I suppose, reminds me a little of the dynamic contrast ratio ratings of monitors, we end up seeing ridiculous numbers and it becomes meaningless. It already is kind of meaning less between OS platforms, if not just for the reason that if say you are loyal to Android, why does it matter to you what a Windows phone or iPhone scores?
Did you note this additional comment he made?
"Google Experience (Nexus) devices aren't affected" which shows it's being done intentionally by some of the others. Google plays it straight with Nexus models, as they should anyway especially since they're used to premier new OS versions.
Why manufacturer should use the same power policy across the board? It is up to them for optimisation.
What Sammy has done wrong again?
Can any app opt into the aggressive power policy that Samsung uses for those benchmarks? If so, why does Samsung hardcode those benchmarks in a special whitelist? If not, wouldn't the benchmark scores lead consumers to wrongly conclude that the Note 3 can run real-world apps significantly faster than other similarly equipped devices?
I am not royal to Android. My family still enjoy more than 2 years old iPad2 and recommends iPad to anyone, although I prefer Android for smartphones over iOS.
As you might see from above, I do not have any interest what a Windows phone or iPhone scores, for now of course.
That was to murman.
Cheating in more ways than one
Yeah, nothing whatsoever to see here.
Samsung: "This is how we are wired…" ;-)
I beg to differ. You are a Samsung apologist.
Optimization is one thing. 'Optimization' would involve figuring out what the app does and then assigning the appropriate conditions for that app. But when the same app is 'optimized' differently by merely changing it's name, that's not optimization. It's artificially giving known benchmark apps an advantage that other apps don't have.
The purpose of benchmarks is to give you an easily measured performance metric that allows you to compare one device to another. Ideally, if device A is 30% faster in real life, then it should be 30% faster on the benchmark. With this type of gamesmanship, the benchmark loses all value. It is 'optimized' differently than the apps that people are using and is therefore useless as an indicator of which device is faster.
Did you note this additional comment he made?
"Google Experience (Nexus) devices aren't affected" which shows it's being done intentionally by some of the others. Google plays it straight with Nexus models, as they should anyway especially since they're used to premier new OS versions.
I didn't read every comment, but hey he wrote them, not me. And gooooood for you if you own a Nexus, have a cookie.
The purpose of benchmarks is to give you an easily measured performance metric that allows you to compare one device to another. Ideally, if device A is 30% faster in real life, then it should be 30% faster on the benchmark. With this type of gamesmanship, the benchmark loses all value. It is 'optimized' differently than the apps that people are using and is therefore useless as an indicator of which device is faster.
^This.
Has nothing to do with which device you use. It speaks to how reliable the benchmarking results are. If it's a Nexus device they're not gaming the tests. If it's Samsung it's likely they are.
EDIT: I see DED has posted an article here at AI pointing out the exact same thing. Daniel can haz cookies too?
http://appleinsider.com/articles/13/10/02/asus-htc-lg-and-other-android-licensees-join-samsung-in-faking-test-results