Apple plans self-sufficient cloud infrastructure with 'Project McQueen'

AppleInsiderAppleInsider
Posted:
in iCloud edited March 2016
A day after reports linked Apple to a massive cloud services deal with Google Cloud Platform, sources have come forward with information of a comprehensive in-house data center initiative dubbed "Project McQueen."


Apple's data center in Maiden, NC.


With Project McQueen, Apple is looking to wean off third-party cloud providers like Amazon Web Services and Microsoft in favor of its own infrastructure, an investment expected to pay for itself within three years of going live, a person familiar with the company's plans told VentureBeat.

Apple currently relies on AWS and Microsoft's Azure for its content serving needs, including data-intensive products like iTunes and iCloud. The massive user base of iTunes and its various music, video and app storefronts is well known, and in February Apple revealed it services more than 782 million active iCloud accounts worldwide.

Sources said, however, that the iPhone maker is not pleased with Amazon's performance, specifically the inability to quickly expedite photo and video requests from iOS devices. In addition, Apple wants to bring iTunes under one roof. Currently, third-party services handle a majority of iTunes' data needs, with Azure being the primary provider, sources said.

Interestingly, the idea behind Project McQueen was purportedly formulated after a conversation with a Microsoft employee, who told an Apple employee that Azure would be unable to keep up with Apple's continued growth. To cope with the overflow, Apple would likely have to invest in an expansion to Azure's data center network, the Microsoft employee reportedly said at the time.

Instead of sinking capital into infrastructure owned and operated by outside firms, Apple is now mulling a buildout of its own. To that end, the company has purchased plots of land in China and Hong Kong for future data centers, the report said. Aside from the financial benefits, Apple would be able to optimize its data centers to suit customer needs, resulting in an improved user experience.

It is unclear when Apple plans to announce or break ground on the supposed data centers. Reports yesterday claimed the company struck a multi-year deal with Google Cloud Platform worth between $400 and $600 million, though the partnership could prove a stopgap until Project McQueen goes online.
«1

Comments

  • Reply 1 of 38
    melgrossmelgross Posts: 33,510member
    One reason why Apple purchased land within China was because of the demand from the Chinese government that all data from Chinese citizens remain within China.
    gatorguycalisrice
  • Reply 2 of 38
    why-why- Posts: 305member
    Lightning or Steve?
    gtr
  • Reply 3 of 38
    rob53rob53 Posts: 3,248member
    Fine with me. Now Apple can encrypt everything under their control. 
    cornchip
  • Reply 4 of 38
    quinneyquinney Posts: 2,528member
    The great escape (from Amazon and Google) is underway.
    tmaypscooter63cornchip
  • Reply 5 of 38
    canukstormcanukstorm Posts: 2,699member
    About time.
  • Reply 6 of 38
    sockrolidsockrolid Posts: 2,789member
    AppleInsider said:
    ... Interestingly, the idea behind Project McQueen was purportedly formulated after a conversation with a Microsoft employee, who told an Apple employee that Azure would be unable to keep up with Apple's continued growth. ...
    And I imagine that the Microsoft employee continued with: "Besides, this whole Internet thing is just a fad anyway."
    MacProradarthekatcoolfactorpscooter63calicornchip
  • Reply 7 of 38
    sockrolidsockrolid Posts: 2,789member
    "We believe that we need to own and control the primary technologies behind the products we make, and participate only in markets where we can make a significant contribution."
    - Tim Cook,  January 2009 FQ1 2009 Earnings Call
    Source: http://http//www.asymco.com/2011/01/17/the-cook-doctrine/

    Maybe Apple can re-animate the Xserve project too. 
    But this time with A9 SoCs instead of G4 chipsets, running RISC-enabled OS X.
    cornchip
  • Reply 8 of 38
    rob53rob53 Posts: 3,248member
    sockrolid said:
    "We believe that we need to own and control the primary technologies behind the products we make, and participate only in markets where we can make a significant contribution."
    - Tim Cook,  January 2009 FQ1 2009 Earnings Call
    Source: http://http//www.asymco.com/2011/01/17/the-cook-doctrine/

    Maybe Apple can re-animate the Xserve project too. 
    But this time with A9 SoCs instead of G4 chipsets, running RISC-enabled OS X.


    I appreciate your quote.

    As for Xserves, I bought several for work and they worked fine for years, until the new IT manager got rid of them and replaced them with used Windows servers. So much for supporting a large group of Mac users with proper equipment.

    I would like to see IBM servers being sold to SMB installations running a collection of Apple and IBM software. I'd also like to see these being sold to individuals. Regular customers should be able to reap the benefits of the IBM and Apple partnership. 
  • Reply 9 of 38
    SpamSandwichSpamSandwich Posts: 33,407member
    Incidentally...

    Wean

    1. to accustom (as a young child or animal) to take food otherwise than by nursing

    2. to detach from a source of dependence

  • Reply 10 of 38
    jameskatt2jameskatt2 Posts: 720member
    melgross said:
    One reason why Apple purchased land within China was because of the demand from the Chinese government that all data from Chinese citizens remain within China.
    But eventually, there will be more iPhone users in China than all of the United States. For example, there are 400 MILLION affluent Chinese. This compares to the 150 MILLION Apple customers in the United States. These affluent Chinese buy 1/3 of all the luxury goods in the world. They are ripe to become all Apple customers. They LOVE the gold iPhones. And cheap Android phones make them look bad. It makes sense to serve them locally so that Apple's services can be customized for their needs and a better user experience may be provided.
  • Reply 11 of 38
    jameskatt2jameskatt2 Posts: 720member
    sockrolid said:
    "We believe that we need to own and control the primary technologies behind the products we make, and participate only in markets where we can make a significant contribution."
    - Tim Cook,  January 2009 FQ1 2009 Earnings Call
    Source: http://http//www.asymco.com/2011/01/17/the-cook-doctrine/

    Maybe Apple can re-animate the Xserve project too. 
    But this time with A9 SoCs instead of G4 chipsets, running RISC-enabled OS X.


    That is wishful thinking. A9 SoCs are not built for the heavy duty data crunching and multitasking that Intel Processors are designed for. They won't work for XServes. Apple limits the A9 to 2 CPU cores since this is optimal for handset use to balance speed versus energy, and focus on single thread tasks. Intel Processors are built for brute force. The Xeon E7 for example can have up to 18 cores and 36 threads. In a server, a single Xeon kills the A9. The i7 can have up to 8 cores and 16 threads and runs up to 4 GHZ. Again, a single Xeon kills the A9. Horses for courses. As Steve Jobs said, some people need trucks (desktops and servers), and some people need cars (iPhones and iPads).
  • Reply 12 of 38
    foggyhillfoggyhill Posts: 4,767member
    If they're ready to build out, they likely already have the tech solution to do so.
    Now, it's more a matter of scaling properly.
    edited March 2016
  • Reply 13 of 38
    brakkenbrakken Posts: 687member
    Leave em behind, Apple!
    cali
  • Reply 14 of 38
    staticx57staticx57 Posts: 405member
    sockrolid said:
    "We believe that we need to own and control the primary technologies behind the products we make, and participate only in markets where we can make a significant contribution."
    - Tim Cook,  January 2009 FQ1 2009 Earnings Call
    Source: http://http//www.asymco.com/2011/01/17/the-cook-doctrine/

    Maybe Apple can re-animate the Xserve project too. 
    But this time with A9 SoCs instead of G4 chipsets, running RISC-enabled OS X.


    You do realize that the G4 CPUs were RISC right? So you want OSX to be OSX from 2007?
  • Reply 15 of 38
    dick applebaumdick applebaum Posts: 12,527member

    sockrolid said:
    "We believe that we need to own and control the primary technologies behind the products we make, and participate only in markets where we can make a significant contribution."
    - Tim Cook,  January 2009 FQ1 2009 Earnings Call
    Source: http://http//www.asymco.com/2011/01/17/the-cook-doctrine/

    Maybe Apple can re-animate the Xserve project too. 
    But this time with A9 SoCs instead of G4 chipsets, running RISC-enabled OS X.


    That is wishful thinking. A9 SoCs are not built for the heavy duty data crunching and multitasking that Intel Processors are designed for. They won't work for XServes. Apple limits the A9 to 2 CPU cores since this is optimal for handset use to balance speed versus energy, and focus on single thread tasks. Intel Processors are built for brute force. The Xeon E7 for example can have up to 18 cores and 36 threads. In a server, a single Xeon kills the A9. The i7 can have up to 8 cores and 16 threads and runs up to 4 GHZ. Again, a single Xeon kills the A9. Horses for courses. As Steve Jobs said, some people need trucks (desktops and servers), and some people need cars (iPhones and iPads).
    Maybe not the A9 -- but possibly a custom variant of the A10.  Say the A10C for Cloud, A10D for Distributed or A10S for Server.

    Multiple of these SoCs (1-n) could easily be combined in a server and outperform Intel's best.

    Then the "data crunching" could be distributed across multiple servers/geographic locations.


    I suspect that Apple's acquisition of FoundationDB is integral to this plan:

    FoundationDB: 14.4 Million Write Transactions per Second

    FoundationDB has released a new version of their database product, aimed at enabling a new generation of Internet of Things and device-driven interactive applications to be built that keeps a single view of a massive distributed database while allowing a constant stream of read and writes to the data.

    “One of the hardest things to scale is write transactions,” says Dave Rosenthal, CEO and Founder of FoundationDB, who has been working on version three of the database from for the past year. “Scaling transactions with lots of writes happening all at the same time is difficult: in the past, we have been able to manage 300-400,000 random writes to the database every second. That’s a pretty good number, but there have been some businesses pushing bigger numbers than that.”

    Rosenthal cites a recent Netflix post that last year stood out as the industry’s best practice. In the documented test, Netflix were able to run Cassandra at scale on a thousand core cluster that maintained 1.1 million writes per second.

    “That was one of the really cool benchmarks that caused a lot of people to stand up,” says Rosenthal. “It was about three times faster than our 2.0 product.”

    According to Rosenthal, many doubted FoundationDB’s ability to take on that level of transactional capacity, especially given FoundationDB’s architecture which is built on a single node.



    http://thenewstack.io/databases-high-volume-transactions-scale-part-two/


    Finally, Apple's development and release of Swift is a key component to their cloud offerings.

    patchythepiratecornchip
  • Reply 16 of 38
    dick applebaumdick applebaum Posts: 12,527member

    HP Extends Benefits of ARM Architecture into the Datacenter with New Servers

    Introduces first enterprise-class 64-bit ARM-based servers; while expanding the ARM developer ecosystem

    PALO ALTO, Calif., Sept. 29, 2014 — HP today announced two ARM®-based servers, including the first enterprise-class 64-bit ARM-based server, for customers that value choice in their compute strategy. Additionally, HP is making available a production-ready platform to enable software developers to develop, test and port applications to the 64-bit ARM-based server.

    Part of the HP ProLiant Moonshot portfolio, the new servers underpin a major milestone in designing the next-generation infrastructure platform, which addresses IT demands with a vast pool of processing resources that can be located anywhere, scaled to any workload and available at any time. The HP ProLiant Moonshot servers deliver high-density, ARM-based systems for hyperscale, datacenter environments to help customers improve application performance, drive business innovation and deliver breakthrough datacenter economics.

    The new HP ProLiant Moonshot 64-bit server, the HP ProLiant m400, is the result of a multi-year effort involving customer input and ARM ecosystem expansion to deliver solutions that meet HP’s rigid engineering standards and testing for exceptional performance and quality to ensure a product is production ready for deployment in enterprise datacenters.  For specific workloads, the ARM architecture will optimize a customer’s most critical compute requirement – balanced memory at a lower total cost of ownership.

    “ARM technology will change the dynamics of how enterprises build IT solutions to quickly address customer challenges,” said Antonio Neri, senior vice president and general manager, Servers and Networking, HP. “HP’s history, culture of innovation and proven leadership in server technology position us as the most qualified player to empower customers with greater choice in the server marketplace.”

    ARM technology delivers unprecedented levels of energy-efficient processing and a diverse range of solutions needed for mobile devices, and brings compelling economies of scale when combined with HP ProLiant Moonshot server technologies. HP ProLiant Moonshot is a precision-engineered system that optimizes application performance by ensuring server hardware is tuned to the task, providing the perfect compute balance for specific workloads. By leveraging energy-efficient system-on-a-chip (SOC) technology, a dense, converged form factor and highly flexible fabrics, HP ProLiant Moonshot provides high-speed connectivity between cartridges, as well as balanced computing with maximum throughput and memory to improve performance and compute efficiency.




    http://www8.hp.com/us/en/hp-news/press-release.html?id=1800094#.Vuu9XGN5gUE
  • Reply 17 of 38
    techprod1gytechprod1gy Posts: 838member
    This should be good for all of us from a performance standpoint. I have 50,000 photos and who knows how many videos. I am not happy right now with the time it takes to sync these across devices. Sometimes speed is ok. Sometimes it takes a day. A whole day!
  • Reply 18 of 38
    tmaytmay Posts: 6,328member

    That is wishful thinking. A9 SoCs are not built for the heavy duty data crunching and multitasking that Intel Processors are designed for. They won't work for XServes. Apple limits the A9 to 2 CPU cores since this is optimal for handset use to balance speed versus energy, and focus on single thread tasks. Intel Processors are built for brute force. The Xeon E7 for example can have up to 18 cores and 36 threads. In a server, a single Xeon kills the A9. The i7 can have up to 8 cores and 16 threads and runs up to 4 GHZ. Again, a single Xeon kills the A9. Horses for courses. As Steve Jobs said, some people need trucks (desktops and servers), and some people need cars (iPhones and iPads).
    Maybe not the A9 -- but possibly a custom variant of the A10.  Say the A10C for Cloud, A10D for Distributed or A10S for Server.

    Multiple of these SoCs (1-n) could easily be combined in a server and outperform Intel's best.

    Then the "data crunching" could be distributed across multiple servers/geographic locations.


    I suspect that Apple's acquisition of FoundationDB is integral to this plan:



    http://thenewstack.io/databases-high-volume-transactions-scale-part-two/


    Finally, Apple's development and release of Swift is a key component to their cloud offerings.

    I"m thinking that personal iCloud devices with ad hoc networking could be the next big thing, and I even imagine an AppleTV growing into a personal iCloud media server.

    Let's see what Apple could do with these;

    http://www.theregister.co.uk/2016/03/09/qualcomm_red_hat_enterprise_linux/
    http://www.theregister.co.uk/2016/01/14/amd_arm_seattle_launch/
    http://www.theregister.co.uk/2015/11/16/appliedmicro_x_gene_3/

    I think that HP is using an earlier version of the x gene in the Moonshot servers.

    The tools are certainly in place to make this happen.

    http://www.mactrast.com/2016/03/survey-developers-now-use-os-x-linux-swift-second-loved-language/
    edited March 2016 rcfa
  • Reply 19 of 38
    gatorguygatorguy Posts: 24,211member

    sockrolid said:
    "We believe that we need to own and control the primary technologies behind the products we make, and participate only in markets where we can make a significant contribution."
    - Tim Cook,  January 2009 FQ1 2009 Earnings Call
    Source: http://http//www.asymco.com/2011/01/17/the-cook-doctrine/

    Maybe Apple can re-animate the Xserve project too. 
    But this time with A9 SoCs instead of G4 chipsets, running RISC-enabled OS X.


    That is wishful thinking. A9 SoCs are not built for the heavy duty data crunching and multitasking that Intel Processors are designed for. They won't work for XServes. Apple limits the A9 to 2 CPU cores since this is optimal for handset use to balance speed versus energy, and focus on single thread tasks. Intel Processors are built for brute force. The Xeon E7 for example can have up to 18 cores and 36 threads. In a server, a single Xeon kills the A9. The i7 can have up to 8 cores and 16 threads and runs up to 4 GHZ. Again, a single Xeon kills the A9. Horses for courses. As Steve Jobs said, some people need trucks (desktops and servers), and some people need cars (iPhones and iPads).
    Maybe not the A9 -- but possibly a custom variant of the A10.  Say the A10C for Cloud, A10D for Distributed or A10S for Server.

    Multiple of these SoCs (1-n) could easily be combined in a server and outperform Intel's best.
    "Google is said to be working with Qualcomm to design servers based on ARM processors, which would be a significant endorsement for ARM as it tries to challenge Intel's dominance in data centers."
  • Reply 20 of 38
    rcfarcfa Posts: 1,124member
    sockrolid said:
    "We believe that we need to own and control the primary technologies behind the products we make, and participate only in markets where we can make a significant contribution."
    - Tim Cook,  January 2009 FQ1 2009 Earnings Call
    Source: http://http//www.asymco.com/2011/01/17/the-cook-doctrine/

    Maybe Apple can re-animate the Xserve project too. 
    But this time with A9 SoCs instead of G4 chipsets, running RISC-enabled OS X.


    Totally! ARM64 is more than adequate.
    Actually, what they should really do for power users of whom there will be more and more: morph the AppleTV and MacMini into an ARM64 home server, where all HomeKit and user data live, i.e. a self-administered iCloud server at home, with Apple's iCloud acting as rendezvous server (akin to back to my Mac), encrypted internet cache, and encrypted backup.
    Once this tech exists, OS X server can also host the full iCloud functionality for organizations who have sensitive data they don't want to entrust any third parties.

    Apple would solve the scalability issue, privacy issue in one go while giving AppleTV and HomeKit a boost at the same time...
    propodcornchip
Sign In or Register to comment.