First Apple Silicon Developer Transition Kit benchmarks show Rosetta performance impact

Posted:
in Future Apple Hardware edited August 2020
The initial benchmarks for the Apple Silicon Developer Transition Kit reveal Apple is very slightly under-clocking the A12Z Bionic chip inside the developmental hardware, with Rosetta 2 also impacting performance.

Tim Cook introducing the Developer Transition Kit at WWDC 2020
Tim Cook introducing the Developer Transition Kit at WWDC 2020


One week after Apple announced its move from Intel chips to Apple Silicon and the creation of the Developer Transition Kit at WWDC 2020, the first benchmarks of the DTK have surfaced. Results found in Geekbench 5's online browser for "VirtualApple" reveals systems bearing the name "eperm-d995af6e2ef02771," which is thought to be the identifier for the DTK.

The eight results found and uploaded on June 20 are all listed as having "VirtualApple 2,400MHz (4 cores)" as the processor, referencing the A12Z being used inside the DTK. A full breakdown of the results indicates the device uses "macOS 11.0" as its operating system, and has 16 gigabytes of memory.

In terms of performance, the single-core tests resulted in a range of between 736 and 844 points and an average of 811. For multi-core benchmarks, the range is from 2,582 to 2,962, with an average score of 2,781.

These results are considerably lower than the 1,118 single-core and 4,625 multi-core benchmarks of the 12.9-inch iPad Pro which also uses the A12Z Bionic, but there are a few reasons that could be the case. For a start, Apple is running the A12Z at a slower 2.4GHz in the DTK, instead of the 2.5GHz clock in the iPad Pro line, though at this time it is unclear why this is the case.

Another interfering factor is the benchmark itself. Rather than using a native ARM-based app that runs on Apple Silicon, it is likely that benchmarks are being carried out using Geekbench 5's macOS client. As Apple introduced Rosetta 2 as a a translator to convert apps meant for Intel-based Macs to function on Apple Silicon, it is highly likely that this introduces some overhead that affects the benchmark's score.

Though the benchmarks may be disappointing to some, as they are below the results of the current range of Mac mini models, there are still a few more points to bear in mind. For a start, the DTK is intended as a system for developers to test out software in development for the new Apple Silicon-based Macs, and are not meant to be used as a demonstration of Apple Silicon's Performance.

It is also highly likely that Apple will make changes to the hardware before it ships its first consumer-based Apple Silicon model, including altering the chip itself as well as refining other components. Furthermore, performance improvements are almost certainly going to be made to speed up macOS Big Sur itself and Rosetta 2.
«13

Comments

  • Reply 1 of 48
    tommy65tommy65 Posts: 56member
     Being a developer ARM speed is not important for now. It’s all about testing your builds & that’s more than sufficient for now. This is not a consumer ARM to compare with the current lineup.
    aderutterMacProforegoneconclusionchiarcfaMisterKitgodofbiscuitsmwhitejony0doozydozen
  • Reply 2 of 48
    polymniapolymnia Posts: 1,077member
    I wonder: is it possible that the upcoming Mac-specific Apple Silicon chips might include features specifically designed to accelerate Rosetta 2?

    They can design anything they like now. Maybe smoothing the transition by accelerating Rosetta 2 would be a good move. Perhaps just for the first couple generations. They could drop it once MacOS drops support for Rosetta 2. 
    watto_cobra
  • Reply 3 of 48
    hmlongcohmlongco Posts: 423member

    I know people are waiting to see what kind of numbers this machine puts out, but in a recent interview with John Gruber Apple’s Craig Federighi wanted to stress that the Developer Transition Kit that was released this week should not be compared to any future consumer product.

    “Even that DTK hardware, which is running on an existing iPad chip that we don’t intend to put in a Mac in the future, it’s just there for the transition, the Mac runs awfully nice on that system.”

    “It’s not a basis on which to judge future Macs, of course, but it gives you a sense of what our silicon team can do when they’re not even trying,” Federighi continued. “And they’re going to be trying.”

    aderuttermuthuk_vanalingamnetroxforegoneconclusionRayz2016MplsPlkruppchiaMisterKitFidonet127
  • Reply 4 of 48
    chaickachaicka Posts: 256member
    Wondering what if the production A-based Macs coming end of the year actually ship with an A14Z chip instead (since A14 is going to be in this year's iPhone 12s), and that chip includes plentiful of specific-advancements meant for Rosetta 2 and Big Sur as compared to the A12Z in DTK?

     :D 
  • Reply 5 of 48
    jdb8167jdb8167 Posts: 626member
    I’m pretty sure Rosetta only uses the “big” cores which is why these results show up as a 4 core CPU and not 8. You also can’t rely on the reported clock speed. These results are from a kit that isn’t supposed to be benchmarked running an x86-64 translation to Aarch64. They shouldn’t be used to infer anything about native ARM performance. 

    It’s also interesting that the date of the tests were from before WWDC which likely means they came from a developer with very early access. 
    rundhvid
  • Reply 6 of 48
    hattighattig Posts: 860member
    Read the MacRumors thread on this, it's comprehensive.

    TL;DR: MacMini A12Z at iPad TDP and clock speeds matches 2016 MBP15 *in Rosetta*. Achieves about 75% native performance within Rosetta. Final hardware will be two generations newer (A14Z), have more cores (8+4), run faster (maybe not in MBA).

    TL;RDR: All good, keep calm.

    Aside: The bottom-of-article comment form is so horribly broken I'd rather that there was just a link to the forum only.
    jdb8167headfull0wineMplsPdoozydozenrundhviddelorean
  • Reply 7 of 48
    jdb8167jdb8167 Posts: 626member
    polymnia said:
    I wonder: is it possible that the upcoming Mac-specific Apple Silicon chips might include features specifically designed to accelerate Rosetta 2?

    They can design anything they like now. Maybe smoothing the transition by accelerating Rosetta 2 would be a good move. Perhaps just for the first couple generations. They could drop it once MacOS drops support for Rosetta 2. 
    I think it is a possibility but because of machine learning used by the transpiler it may not have been necessary. Another possibility is that they will add future hardware that will support features to improve Rosetta but are not available on the DTK. 
  • Reply 8 of 48
    hattighattig Posts: 860member
    jdb8167 said:
    I’m pretty sure Rosetta only uses the “big” cores which is why these results show up as a 4 core CPU and not 8. You also can’t rely on the reported clock speed. These results are from a kit that isn’t supposed to be benchmarked running an x86-64 translation to Aarch64. They shouldn’t be used to infer anything about native ARM performance. 

    It’s also interesting that the date of the tests were from before WWDC which likely means they came from a developer with very early access. 
    Rosetta is a translator, rather than a runtime emulation. Even the JIT feature will be implemented by translating finalised Intel Code into ARM (when you mark a memory page as executable post-JIT, Rosetta kicks in and does its work - I guess). **HOWEVER** there will be some things that need to be trapped as they are not emulateable, and therefore I suspect Rosetta provides a very thin VM/system layer around the executable. Also binary translators really cannot infer intent, so there is only so much they can do - they will generate code that is less optimal, especially when coming from x86 with all its quirks.

    Yes, these are likely in-Apple or in-close-third-party leaks.
    rundhvid
  • Reply 9 of 48
    MacProMacPro Posts: 19,523member
    tommy65 said:
     Being a developer ARM speed is not important for now. It’s all about testing your builds & that’s more than sufficient for now. This is not a consumer ARM to compare with the current lineup.
    Exactly.  I seriously suspect the actual released Apple Silicon hardware will have new totally new versions.  I wonder given how inexpensive they will be for Apple compared to Intel CPUs if we will see Macs with multiple copies of what ever they will be called.
    edited June 2020
  • Reply 10 of 48
    DAalsethDAalseth Posts: 2,564member
    hattig said:
    Aside: The bottom-of-article comment form is so horribly broken I'd rather that there was just a link to the forum only.
    that is so true. I don't even bother trying to post there. I just go the the forum and go from there. Like the update to the AI site. Hate how kludged the comments are now.
    mwhitejony0SpamSandwich
  • Reply 11 of 48
    zimmiezimmie Posts: 628member
    VirtualApple is 100% a Rosetta thing. The reported processor speed is probably just a lie to software to give it an expectation of performance level.
  • Reply 12 of 48
    tundraboytundraboy Posts: 1,845member
    We should take these benchmark results seriously because Apple is stupid enough to release a new product that performs worse than the product it replaces.

    /s
    jony0
  • Reply 13 of 48
    netroxnetrox Posts: 1,284member
    A12Z is literally two generations old already, it's just a binned chip. I imagine the performance will be amazing with A14Z. 


    jony0
  • Reply 14 of 48
    macbootxmacbootx Posts: 60member
    Developer: “Hey! I gots a good idea. Let’s benchmark this new Apple Silicon before we recompile a single line of code!” 

    ....SMH 

    jony0
  • Reply 15 of 48
    macbootxmacbootx Posts: 60member
    tundraboy said:
    We should take these benchmark results seriously because Apple is stupid enough to release a new product that performs worse than the product it replaces.

    /s
    But this isn’t a new product. It’s a testing and development platform. 
    docbburk
  • Reply 16 of 48
    lkrupplkrupp Posts: 10,323member
    Okay, here we go. This is precisely why Apple didn't want benchmarks published on a test machine. Of course benchmarks were inevitable and the reactions to them were predictable by a two year old child, a contstant flow of uninformed bullshit
    rcfamacbootxmwhitewelshdogjony0matrix077Beats
  • Reply 17 of 48
    lkrupplkrupp Posts: 10,323member
    macbootx said:
    Developer: “Hey! I gots a good idea. Let’s benchmark this new Apple Silicon before we recompile a single line of code!” 

    ....SMH 

    Websites are probably paying developers to ignore their NDAs. I hope Apple really cracks down on the offenders but unlikely. 
    macbootxjony0Beats
  • Reply 18 of 48
    cpsrocpsro Posts: 3,061member
    The DTK is a good use for rejected iPad Pro chips. They may be slower in ways other than clock speed would indicate, too.
    Rosetta 2 speed seems pretty good compared to Rosetta 1. And in a very few years (probably more like several months), that won't matter at all.
    jony0
  • Reply 19 of 48
    rcfarcfa Posts: 1,124member
    polymnia said:
    I wonder: is it possible that the upcoming Mac-specific Apple Silicon chips might include features specifically designed to accelerate Rosetta 2?

    They can design anything they like now. Maybe smoothing the transition by accelerating Rosetta 2 would be a good move. Perhaps just for the first couple generations. They could drop it once MacOS drops support for Rosetta 2. 
    Whatever would help Rosetta 2 would very likely also help a variety of virtualization solutions. As such, if they were to make such additions to the ARM instruction set, there would be very little incentive to ever remove them again.
    i would however expect Apple to tread very carefully with extending the ARM programming model, particularly in an unilateral move.
  • Reply 20 of 48
    hattighattig Posts: 860member
    zimmie said:
    VirtualApple is 100% a Rosetta thing. The reported processor speed is probably just a lie to software to give it an expectation of performance level.
    It's a thin VM/system layer around the translated application, because not everything can be translated. It will trap the issues to do them properly.
    I suspect the clock speed is gained from a syscontrol call so is valid.
Sign In or Register to comment.