Apple's Benchmarks misleading?

1246789

Comments

  • Reply 61 of 178
    powerdocpowerdoc Posts: 8,123member
    Quote:

    Originally posted by Eugene

    The G5 mobo DOES use dual-channel DDR memory, and that's a good thing.



    You are right, my mistake, i have corrected it in an another thread but miss that one.
  • Reply 62 of 178
    sc_marktsc_markt Posts: 1,402member
    As far as I'm concerned, I'm glad as heck that we are in the performance ballpark with Intel based computers. AND, while running a whole 1GHz slower! To me, this says a lot in regards to the design of the 970 and the new G5 powermacs.
  • Reply 63 of 178
    albertwualbertwu Posts: 10member
    remember back when apple was the first to break the 300 mhz barrier?



    apple had the fastest computer in the world
  • Reply 64 of 178
    blestblest Posts: 24member
    wow alot of people get really heated in here. as for the post about it being taken into comparison to itanium and opteron machines. those are server and workstation chips. They said desktop. I know many avid amd fans and they don't own an opteron rig they wait for the athlon 64.. amd is the king of paper releases and delays so really as for now i think what apple is doing is fine. everyone lies, skews the truth sh ows the benchmarks it wins at. big deal. If you want an apple the g5 is faster, if y ou don't want an apple or want a pc tahts fine too. go buy one and live with windows and it's integrated webbrowser which is no longer a standalone client. i really don't care. I;ve been using pc's for a few years now and really i dont give a $hit if the g5 isnt the fastest when i get it. it will be enough for me.



    it's your money, anyway you spend it on computers is a waste.
  • Reply 65 of 178
    Quote:

    Originally posted by BLeST





    it's your money, anyway you spend it on computers is a waste.




    OK!!



    I think we can all agree on that.



    Good post to put things a little more into perspective.



    "it's your money, anyway you spend it on computers is a waste"



    8)

    that could be a good signature line

    but I am too lazy to paste it into my profile.



    -tom w
  • Reply 66 of 178
    Just in case this thread is not contentious enough there is MORE here:



    http://www.overclockers.com/tips00408/





    "Documenting the Dung"

    Ed Stroligo - 6/24/03

    ------------------------------------------------------------------------





    "Even in today's world, there's some things you can always count on. The sun shows up every day. The earth revolves around the sun.



    And Apple comes up with some nonsense that whatever it just came out with is faster than x86 processors.



    Apple just announced its new systems. These are what I call the MacHammer systems, simply because they are at least superficially rather alike. They both use SOI, both have a 32/64-bit architecture, and both run at around the same speed.



    Anyway, Apple presented some numbers to indicate that the G5 is faster than the PIV or Xeon in single and dual-CPU configurations. These measurement included Spec2000 measurements.



    Spec2000 is a platform-independent benchmark often used to compare the CPU performance of different platforms. Manufacturers submit their scores to Spec for posting on their website. You can see the lastest posting of results here.



    Anyone familiar with recent x86 spec scores would have smelled something rotten; the x86 scores presented by Apple were just too low. Chris Tom from AMDZone has already taken the official spec scores and written an article about it "





    snip.....

    (see charts and graphs on web page from link posted above )



    "No Credibility



    If Apple were part of the x86 market, they'd be ripped to shreds in a second presenting this kind of data. It would be a huge scandal. PC companies get torn a new one for attempting even a small fraction of what we have here.



    But it's a different world in Macdom. The average Macster is woefully ignorant of hardware, and tends to believe anything and everything Apple tells him. For many, it is guru and flock.



    This is not an honest business, but nobody in the PC world has ever approached Apple in the level of sheer and consistent technically accurate but totally misleading information when it comes to performance.



    It's really a shame. I thought the G5 would be pretty good. From what I can figure out, it isn't, and I don't trust the company to tell the honest truth for the rest of it.



    P.S. I've seen some pretty bizarre defenses of Apple so far. One said that the scores from Dell et. al. weren't proper because those machines were tweaked. Uhhh, just what did Apple did, they even said so deep in the fine print. What's fairer, measuring tweaked machine to tweaked machine, or tweaked machine to non-tweaked machine.



    Another said that some version of Linux had to be used to compare apples to apples. Well, MacOS X isn't Linux, and the desktop standard for x86 machines is Windows (not that using a properly optimized Linux bothered the Opterons very much). You want to know what machine is fastest, you test in their native environment.



    "
  • Reply 68 of 178
    alcimedesalcimedes Posts: 5,486member
    sounds like apple's reply is really taking the wind out of some sails in regards to their "cheating"
  • Reply 69 of 178
    jccbinjccbin Posts: 476member
    Klinux,



    The writer was being a detractor.



    What part of that is hard to understand? He even says as much.



    Rub some orajel on that sensitive spot.
  • Reply 70 of 178
    klinuxklinux Posts: 453member
    jccbin: Sure, I will rub some Orajel on it which still doesn't change the fact you prefer to remain blissfully ignorant and will take anything Apple says at fair value.
  • Reply 71 of 178
    nebcon65nebcon65 Posts: 47member
    First off Klinux. I would restrain from using the word "ignorant" unless you know the people you are talking about on an individual basis. I take offense to being called ignorant just because I disagree with all that you're saying. You know nothing of my educational background or anything else about me. I am a Mac enthusiast but I'm not an apologist for Apple.



    Secondly. I don't care much about benchmarks any more than the gamers that I have talked to care about 3DMark tests. However, having read the detailed article on Ars Technica about the 970, I am not surprised by the Apple benchmarks. The article pretty much predicts the results. In particular the integer results in comparison to the P4 ISA. Logical reasons are given as to how they might perform in general compared to one another. The key is bandwidth and I think that the difference there is born out in the SMP tests.



    Having said that. Being a video professional I find real world application comparisons more useful. I would like to know exactly how there was a cheat in the Photoshop tests as this is the first I have heard of this charge. An identical script of actions played on both machines is a perfectly good way to do a comparison. Yes, it's true that compiler differences can pop up just as easily in real apps but that isn't the point. Adobe doesn't allow you to recompile Photoshop or After Effects with the compiler of your choice. You get what you get. Professionals or semi Pro/hobbyists and gamers are the only ones that REALLY need to be this concerned about speed. I saw with my own eyes what was done with Photoshop in those demos and I'm convinced.



    I don't care how a Wintelhead or anyone else twists or optimizes. What the big venders supply is what people use.



    A word about GCC. I don't know a lot about it but it seems that it surely has benefited just as other open source ventures have, from Apple input. So how optimized GCC is for PPC at this point I think is not as well known as people may think. Ask someone that works on the project if you care, would be my suggestion.
  • Reply 72 of 178
    jjhlkjjhlk Posts: 3member
    Want to know what a "PC"* user thinks? If not I'll bugger off.



    * I consider the Mac a PC too.. so it's hard to come up with a name other than "non-mac". Oh well.
  • Reply 73 of 178
    kecksykecksy Posts: 1,002member
    Quote:

    Originally posted by alcimedes

    sounds like apple's reply is really taking the wind out of some sails in regards to their "cheating"



    Most of the PC sites are dismissing it and calling Joswiak an idiot. I think what Apple did was fair and they aren't trying to deceive anyone (they provided full disclosure"). I guess the guys at AMDZone and Overclockers feel threatened now that Apple has competitive hardware and see this as an opportunity to raise hell. I hope Apple continues to push the issue and publishes some more benchmarks. It might shut them up.
  • Reply 74 of 178
    bartobarto Posts: 2,246member
    Quote:

    Originally posted by jccbin

    I don't think Apple cheated.



    I've changed my mind. I don't think Apple have cheated either. The SSE2 issue was ignorance on the part of the accuser, the optimisation switches will be turned on by default in the final G5 and both computers are using the GCC complier.



    Apple finally builds a machine that thrashes PC, and what do insecure PC users do? "Apple sucks, it must be a hoax!"



    The benchmarks are misleading in that they do not include important information (ie the dell is running linux) next to the benchmarks, and Apple will of course only ever show positive benchmarks.



    However, what's the big deal? Did any of you honestly believe a DP 500 is two times faster most of the time than a Pentium 4 based on a benchmark testing a few photoshop actions?



    Barto
  • Reply 75 of 178
    herbivoreherbivore Posts: 132member
    The AMD and Intel detractors can have their day for the present time. I for one look forward to the delivery of the G5.



    One thing is for sure. IBM is back and ready to extract vengeance. The Apple tweaks aside, in another year, the PowerPC will simply dominate. No amount of Intel or AMD tweaks will be able to overcome the sheer performance built into the PowerPC processors.



    Intel was there before with the delivery of the G3. Unfortunately, this time they have OS X to also contend with.



    It makes one wonder where Apple would be if they had worked more seriously with IBM in the first place instead of going with Motorola and Altivec.



    From what I understand, the bus speed of the G5 runs at 1/2 processor speed. So a 3 GHz 970 would have a bus of 1.5 GHz. I am not at all certain that AMD and Intel will be able to duplicate this.



    Real world performance is the key in all things. We will have to wait for the machines to ship to identify the true performance. However, I don't see fantastic improvement in the Intel/AMD side of processor market by the time these machines ship. I am not even sure that we'll see large quantities of available Athlon 64 processors either.



    I am certain that IBM is moving to produce faster processors as quickly as they can also. In essence, by the time these machines ship, Apple might have plans for shipping 2.5 GHz machines a few months down the road again.



    The 980 intrigues me. IBM seems to have a fairly aggressive roadmap for this processor also. I am wondering if there are plans afoot to migrate the professional machines over to this processor as the consumer migration over to the 970 is completed.



    This could be a very good year for Apple.
  • Reply 76 of 178
    klinuxklinux Posts: 453member
    Quote:

    Originally posted by nebcon65

    First off Klinux. I would restrain from using the word "ignorant" unless you know the people you are talking about on an individual basis. I take offense to being called ignorant just because I disagree with all that you're saying. You know nothing of my educational background or anything else about me. I am a Mac enthusiast but I'm not an apologist for Apple.



    nebcon65: I am not calling you personally ignorant. Despite not having touched a Mac since Apple II, I am glad that I got the iBook 12 months ago. I too consider myself a Mac enthusisat and I am constantly defending Apple/Mac at work and at PC forums. With that said, like you, I am not an apologist for Apple either. My problem is with people who take what Apple (or any other company for that matter) says at face value without questions.



    In the PC world, people constantly question the #s that manufacturer put out, ATI vs NVIDIA, Intel vs AMD, etc. In this case, neither Intel nor AMD has publicly benchmarked itself against G5 yet. If it did, trust me, people all over the net, on Slashdot, Ars, etc would all dig into the Intel or AMD's result just like they did with Apple's too. This is nothing personal; I just want empirical evidence beside what Apple has shown.



    Unfortunately, none of us can really know until the G5 ships.
  • Reply 77 of 178
    ensoniqensoniq Posts: 131member
    In this whole benchmark issue, I'm not terribly bothered that some people call Apple's benchmarks into question...that is reasonable. Apple gave full disclosure, and thusly opened themselves up to a debate on certain settings they chose.



    What bothers me is:



    1 - People flat out accuse Apple of LYING about the results...how can you LIE about the results when you publish specific instructions on how anyone can duplicate the tests once they have the machines in their hands? Do we believe Apple actually thought that people wouldn't READ the test design? They wouldn't have allowed something which they felt "proved they cheated" to get published. To believe so is moronic.



    2 - People accuse Apple of manipulating the PC settings in order to hurt the P4 and Xeon tests, when Joswiak's explanation makes it clear that Apple tested multiple PC setting configurations, and published the settings that gave the PC the BEST results! Again...without any evidence at all, the PCers claimed the results were rigged against the PC when in fact the PC was shown in the best light possible in this GCC scenario.



    3 - People say Apple's entire benchmark suite is suspect because their scores don't match other published SPEC scores...and yet Apple again clearly points out how they did their tests, why they did their tests that way, and admits that SPEC tests done with Intel optimized compilers might be better. (I'm not going to debate the GCC issue here...those who still don't understand/agree with why Apple used GCC will not be convinced by anything I can offer.) Like what he said or not, Joswiak was honest..now he's being called a liar too.



    4 - If Apple had run the tests using the Intel compiler and optimizations the PC users are frothing at the mouth over, but had also used a PPC 970 optimized compiler on the Mac side, the PCers would have sworn up and down that Apple somehow cooked the books by using 970 optimization that would be "unfair"...yet these same people claim that the way Apple tried to run the tests in a FAIR and UNOPTIMIZED method across the board is ALSO unfair!



    5 - In one of the biggest conspiracy theories ever, even if you leave out the SPEC scores entirely, the PCers accuse all of the other application specific tests (Photoshop, Logic vs. Cubase, Mathematic, etc.) of being faked. The PCers believe that the presidents of multiple technology companies that are happy to sell their software to Mac AND PC users would stand up on the stage with Steve and LIE about the G5 being the fastest platform for their software. They are all in collusion with Steve, and are purposely trying to lie to the world because...uh...well just because!



    Even when/if Slashdot, ArsTechnica, Overclockers, and other "respected" PC-oriented sites get their hands on G5 machines and by some miracle manage to do their own testing which proves the G5 scores Apple produced are legitimate, there will be a flood of people who would begin saying somehow those PC-oriented sites made their own testing mistakes. Because to those people, anything which claims the Mac outperforms the PC will ALWAYS be a lie...even if it's not.



    -- Ensoniq
  • Reply 78 of 178
    zazzaz Posts: 177member
    Quote:

    Originally posted by Kecksy

    Most of the PC sites are dismissing it and calling Joswiak an idiot. I think what Apple did was fair and they aren't trying to deceive anyone (they provided full disclosure"). I guess the guys at AMDZone and Overclockers feel threatened now that Apple has competitive hardware and see this as an opportunity to raise hell. I hope Apple continues to push the issue and publishes some more benchmarks. It might shut them up.



    Is this really a surprise?



    The PC hardware model of assumptions was just seriously called into question...if not broken



    It is a spiraling case of dismay.



    In one day literally 1000s of armchair posts at the myriad of PC hardware sites that have been about compilers, SPEC and so on.



    Not one of these 'experts' has any experience with the machines, etc. They criticize blindly accepting Apples word ( and a plethora of other companies as well) yet insist that somehow, with no primary exposure of their own and one whole day to gather knowledge on the subject, that they know better and you are an idiot if you contradict them.



    Nameless, faceless citizens of geekdom chanting against visible, accountable individuals who have first hand knowledge.



    Is the G5 really faster? I have no idea.



    But it is obvious that there are a lot of those out there that find this prospect a little more than just trivial.



    Z
  • Reply 79 of 178
    jjhlkjjhlk Posts: 3member
    OK I cannot hold it in any longer.



    The reason that the G5 beat the P4 in the SPEC benchmark was because of the compiler. It's as simple as that. Ensoniq, perhaps you can tell me why that isn't a valid point?



    Just be thankful that there is an Apple produced computer that is no longer so clearly inferior to an even-priced Intel or AMD system.



    But don't think that Intel or AMD will stop producing CPUs. They have roadmaps too, you know.
  • Reply 80 of 178
    amorphamorph Posts: 7,112member
    Quote:

    Originally posted by jjhlk

    OK I cannot hold it in any longer.



    The reason that the G5 beat the P4 in the SPEC benchmark was because of the compiler. It's as simple as that. Ensoniq, perhaps you can tell me why that isn't a valid point?




    Of course. In fact, it's Apple's point.



    It's true that Intel uses a compiler specifically tweaked to score well on SPEC, and it's true that Apple didn't use that compiler.



    It's also true that the number of real-world x86 applications that use Intel's compiler is a lot smaller than the number that use GCC (not to mention MS), and those that do might just use different settings, so Apple's change of compiler is not irrelevant. Besides, it's supposed to be more "honest" to use the same application across platforms, so you end up with people using Premiere to establish a Dell's superiority in video editing, and with Apple using GCC to establish a Mac's superiority in SPEC.



    The main complaint, in other words, is that Apple isn't playing by Intel's rules. I see no reason why they should have to. Their rules make as much sense as cross-platform benchmarking ever does, and they spelled out their methodology quite specifically for anyone curious. Now, I'm not going to say that Apple should be taken at their word; not after the last couple of years of "Pentium crushing G4s." But nothing I can see reveals anything dishonest about what Apple did this time around. If anything, they seem to be refreshingly open and honest now that they have a CPU they don't have to make any apologies for.



    This is all academic anyway, since nobody can buy a G5 and see for themselves, and as the x86 fanatics are all too happy to point out, the x86 landscape will have changed slightly by the time the PowerMac G5 actually ships. Furthermore, no-one has convinced me that SPEC scores are of any particular interest to people interested in measuring real-world machine performance, especially across platforms, and especially when one platform relies fairly heavily on SIMD for its performance.
Sign In or Register to comment.