Apple-owned FoundationDB open sources the core technology at the heart of iCloud

Posted:
in General Discussion
Apple's FoundationDB on Thursday announced that its namesake distributed datastore tech for organizations has been turned into an open-source project, allowing public-driven customization and expansion.

Apple's FoundationDB


Source code is now available on GitHub, and developers can communicate with each other on official forums, the company said. FoundationDB is built to be deployed on hardware clusters, for instance making it easier to add machines or bounce back from device failures.

The open-source material represents the key-value store, on which "layers" can be added to scale functionality. The Github listing for the project calls it a "distributed database designed to handle large volumes of structured data across clusters of commodity servers."

"By open sourcing the FoundationDB core, we expect the quantity and variety of layers to develop rapidly," the company said. "All major development will be done in the open," it added, noting that it has outlined a design document process and project governance it promises will offer input and transparency. The community is also expected to follow a code of conduct.

Apple now has a variety of open-source initiatives under its wing. Perhaps the best known is Swift, its programming language. While not strictly limited, the language is almost exclusively used for iOS, macOS, watchOS, and tvOS development.

FoundationDB was originally a multi-model NoSQL database, designed around a "core" database. The core database allowed for read or write transactions with multiple keys stored on any machine in the cluster. Apple purchased the company and all of its intellectual property in 2015 to expand on the iCloud infrastructure.

In the past year Apple has made a strong push to put Swift in colleges and high schools -- mostly in the U.S., but internationally as well. The company has positioned the program as a way of boosting coding skills, even as it benefits from encouraging development for its products.
«1

Comments

  • Reply 1 of 22
    rob53rob53 Posts: 3,241member
    So, does this mean those companies who wish to have local iCloud instances will be able to build and use their own locally-controlled iCloud? Would these include encryption at all levels so schools, businesses and government agencies could leverage the complete Apple environment totally ended their control?

    What other parts of iCloud will be open-sourced so users can build their own iCloud?

    Sounds encouraging to me. Might look into this for fun.
    cornchip
  • Reply 2 of 22
    No announced Swift support... Yet!

    I posted a question to the FoundationDB forum asking if Swift support is planned:

    https://forums.foundationdb.org/c/development/fdb-core
  • Reply 3 of 22

    rob53 said:
    So, does this mean those companies who wish to have local iCloud instances will be able to build and use their own locally-controlled iCloud? Would these include encryption at all levels so schools, businesses and government agencies could leverage the complete Apple environment totally ended their control?

    What other parts of iCloud will be open-sourced so users can build their own iCloud?

    Sounds encouraging to me. Might look into this for fun.
    Interesting Questions?

    I suspect, that if the API is rich enough, independent developers could build the equivalent of local iCloud implementations.  I'm unsure whether  Apple could open up iCloud to 3rd-parties without compromising Apple's iCloud services...

    But, how about IBM's cloud offerings... Or Google's... Or Microsofts...
  • Reply 4 of 22
    I just posted another question to the  FDB forums, Asking whether ARM support is planned.

    https://forums.foundationdb.org/c/development/fdb-core

    FDB's core implementation is based on an Ordered Key-Value Pair -- basically an Ordered Dictionary.  macOS, iOS and most OSes do not support an Ordered Dictionary.

    FWIW, A while back, I was experimenting with iOS Swift Playgrounds and was able to approximate what FDB did with their Ordered Key-Value Pair.


    If FDB/Swift/ARM is viable, it opens up many opportunities, e.g. small business Point-Of-Sale-Terminals (iPads) WiFi connected to a FDB transaction server (cluster) running in the back room on a Mac Mini or an iPad while simultaneously running other clusters in the cloud.

    This could really be a game changer.
    doozydozencornchipjony0
  • Reply 5 of 22
    asciiascii Posts: 5,936member
    Microsoft beat Apple once, not by making a better computer, but by standardising everyone else on one OS and therefore making Apple non-standard. There is potential for them to pull the same trick again by standardising everyone on the same cloud. So anything Apple can do to encourage some kind of open cloud is good from a strategic point of view.
    doozydozen
  • Reply 6 of 22

    rob53 said:
    So, does this mean those companies who wish to have local iCloud instances will be able to build and use their own locally-controlled iCloud? Would these include encryption at all levels so schools, businesses and government agencies could leverage the complete Apple environment totally ended their control?

    What other parts of iCloud will be open-sourced so users can build their own iCloud?

    Sounds encouraging to me. Might look into this for fun.
    But, how about IBM's cloud offerings... Or Google's... Or Microsofts...
    Or, you know, that other one ... AWS ...
    randominternetperson
  • Reply 7 of 22
    I just posted another question to the  FDB forums, Asking whether ARM support is planned.

    https://forums.foundationdb.org/c/development/fdb-core

    FDB's core implementation is based on an Ordered Key-Value Pair -- basically an Ordered Dictionary.  macOS, iOS and most OSes do not support an Ordered Dictionary.

    FWIW, A while back, I was experimenting with iOS Swift Playgrounds and was able to approximate what FDB did with their Ordered Key-Value Pair.


    If FDB/Swift/ARM is viable, it opens up many opportunities, e.g. small business Point-Of-Sale-Terminals (iPads) WiFi connected to a FDB transaction server (cluster) running in the back room on a Mac Mini or an iPad while simultaneously running other clusters in the cloud.

    This could really be a game changer.
    There are a couple things wrong with this. First off, an ordered dictionary/key-value pair is a data structure, not something that needs to be supported on an OS level. You can write one on whatever OS or in whatever language you like. 

    Second, this doesn’t really enable any new applications. Businesses could already use iPads connected to something like a MySQL (a database that’s been around for a long time and has APIs in nearly every language under the sun) server. Or, more likely, a cloud service that integrates payment processing and inventory management. 

    Third, you could probably run this on an ARM server if you wanted to, as long as the server is running Linux. A quick look at the git repo shows that they’ve got it all set up with Docker containers. You don’t need “ARM support” to be able to interact with it from an ARM based device like an iPad - how do you think iCloud works now? All you need are the language bindings for the language that you’re using to program that ARM device. 

    This is just another database that happens to be good for massively distributed applications (like iCloud). It’s really not applicable to the small business like you suggested. 
    edited April 2018 randominternetpersonboxcatcherdoozydozen
  • Reply 8 of 22

    rob53 said:
    So, does this mean those companies who wish to have local iCloud instances will be able to build and use their own locally-controlled iCloud? Would these include encryption at all levels so schools, businesses and government agencies could leverage the complete Apple environment totally ended their control?

    What other parts of iCloud will be open-sourced so users can build their own iCloud?

    Sounds encouraging to me. Might look into this for fun.
    But, how about IBM's cloud offerings... Or Google's... Or Microsofts...
    Or, you know, that other one ... AWS ...
    Yeah... AFAICT,  Apple’s iCloud runs on AWS, Google and Apple servers.
  • Reply 9 of 22
    I just posted another question to the  FDB forums, Asking whether ARM support is planned.

    https://forums.foundationdb.org/c/development/fdb-core

    FDB's core implementation is based on an Ordered Key-Value Pair -- basically an Ordered Dictionary.  macOS, iOS and most OSes do not support an Ordered Dictionary.

    FWIW, A while back, I was experimenting with iOS Swift Playgrounds and was able to approximate what FDB did with their Ordered Key-Value Pair.


    If FDB/Swift/ARM is viable, it opens up many opportunities, e.g. small business Point-Of-Sale-Terminals (iPads) WiFi connected to a FDB transaction server (cluster) running in the back room on a Mac Mini or an iPad while simultaneously running other clusters in the cloud.

    This could really be a game changer.
    I've never needed or used an ordered key-value pair data structure.  Under what circumstances is this better than a simple dictionary with a sortable key?
    boxcatcher
  • Reply 10 of 22
    nathreed said:
    I just posted another question to the  FDB forums, Asking whether ARM support is planned.

    https://forums.foundationdb.org/c/development/fdb-core

    FDB's core implementation is based on an Ordered Key-Value Pair -- basically an Ordered Dictionary.  macOS, iOS and most OSes do not support an Ordered Dictionary.

    FWIW, A while back, I was experimenting with iOS Swift Playgrounds and was able to approximate what FDB did with their Ordered Key-Value Pair.


    If FDB/Swift/ARM is viable, it opens up many opportunities, e.g. small business Point-Of-Sale-Terminals (iPads) WiFi connected to a FDB transaction server (cluster) running in the back room on a Mac Mini or an iPad while simultaneously running other clusters in the cloud.

    This could really be a game changer.
    There are a couple things wrong with this. First off, an ordered dictionary/key-value pair is a data structure, not something that needs to be supported on an OS level. You can write one on whatever OS or in whatever language you like. 

    Second, this doesn’t really enable any new applications. Businesses could already use iPads connected to something like a MySQL (a database that’s been around for a long time and has APIs in nearly every language under the sun) server. Or, more likely, a cloud service that integrates payment processing and inventory management. 

    Third, you could probably run this on an ARM server if you wanted to, as long as the server is running Linux. A quick look at the git repo shows that they’ve got it all set up with Docker containers. You don’t need “ARM support” to be able to interact with it from an ARM based device like an iPad - how do you think iCloud works now? All you need are the language bindings for the language that you’re using to program that ARM device. 

    This is just another database that happens to be good for massively distributed applications (like iCloud). It’s really not applicable to the small business like you suggested. 
    Ehhh... mas o menos... difference of opinion here.   

    Sure, the ordered dictionary is a data structure not an OS-level construct... but one has to wonder why it isn't implemented at the OS level... You can accomplish the implementation with a Kludgey combination of arrays and dictionaries -- a more standard implementation (0S level or not) could improve performance and reliability.  FWIW SQLite4 reportedly uses an ordered dictionary.

    Agreed, this doesn't enable new applications -- but it does offer the potential of bringing down the cost and complexity so that almost anyone could run a reliable processor for transactions, payments, inventory, etc.  I suspect that 3rd-parties will implement these solutions.

    Yeah, you can run on an ARM Server... Maybe even a Raspberry PI.  AFAIK, Some parts of FDB are written pretty close to the metal © -- optimizing for ARM could maintain FDB advantages while bringing down the cost of implementation.

    Yes, FDB is a db that can scale to massively distributed applications -- but that does not mean that it only applies to large scale implementations.

    Here's an interesting FDB video that FDB released before they were acquired by Apple:



    Those are pretty small/inexpensive hardware boxes (could be headless Raspberry PIs, Mac minis, AppleTVs, etc.)...  You could put the whole thing in a shoebox that would work at a street fair or swap meet whether the Internet was available or not.
  • Reply 11 of 22
    I just posted another question to the  FDB forums, Asking whether ARM support is planned.

    https://forums.foundationdb.org/c/development/fdb-core

    FDB's core implementation is based on an Ordered Key-Value Pair -- basically an Ordered Dictionary.  macOS, iOS and most OSes do not support an Ordered Dictionary.

    FWIW, A while back, I was experimenting with iOS Swift Playgrounds and was able to approximate what FDB did with their Ordered Key-Value Pair.


    If FDB/Swift/ARM is viable, it opens up many opportunities, e.g. small business Point-Of-Sale-Terminals (iPads) WiFi connected to a FDB transaction server (cluster) running in the back room on a Mac Mini or an iPad while simultaneously running other clusters in the cloud.

    This could really be a game changer.
    I've never needed or used an ordered key-value pair data structure.  Under what circumstances is this better than a simple dictionary with a sortable key?
    The main advantage is range search performance.  With an ordered structure the search stops at => -- with a standard dictionary you search the entire dictionary to detect a not-found condition -- or some undefined number of records (randomly ordered) until you find a hit (or you have to access and sort the keys before each search).

    This is critical in a database that supports a large number of concurrent transactions with out locking the DB to other transactions.  When Apple acquired FDB, their latest release supported 14 million random writes per second.
    edited April 2018 monstrosity
  • Reply 12 of 22
    jidojido Posts: 125member
    1. Why does it need all of C++ Boost, Mono and Java 8 to compile?
    2. What's the point of putting private keys in public source code?
  • Reply 13 of 22
    Replicated, Distributed (, Partitioned) database. I like the dynamic part of it, which look interesting. But having spend the last 20 years dealing with such beasts, I am convinced that it's not that simple as they try to make it.
  • Reply 14 of 22
    That is great news!
    I am looking forward to trying it out.  It has the potential to take the OSS DB World by storm.

    It is interesting how Apple is moving all their server modules into open source.
    MacOS can be turned in a full fledged server by downloading the server modules. (Web, DB, Directory Services, Wiki, File/Print etc...)
    This could turn out to be very interesting for Mac Hardware.

    The available API are listed below.
    It must integrate seamlessly with CoreData hence no Swift nor Objective C APIs.
    edited April 2018
  • Reply 15 of 22
    tallest skiltallest skil Posts: 43,388member
    It’d be great if iCloud Drive indexed the contents of your files so you could, you know, actually search through them while on the go, instead of having to go back to your Mac to use Spotlight on local machine’s copy of them…
  • Reply 16 of 22
    It’d be great if iCloud Drive indexed the contents of your files so you could, you know, actually search through them while on the go, instead of having to go back to your Mac to use Spotlight on local machine’s copy of them…
    Jeeze TS... what you describe is FDBs forte — you can have dbs with an unlimited number of indexes tailored to fast, ad hoc search needs... in fact, a FDB db can be indexes-only — with no underlying data dB.
    tallest skil
  • Reply 17 of 22
    mattinozmattinoz Posts: 2,299member
    It’d be great if iCloud Drive indexed the contents of your files so you could, you know, actually search through them while on the go, instead of having to go back to your Mac to use Spotlight on local machine’s copy of them…
    Even better if we had our own private siri instance that could then use this index as starting point then go out to other sources as needed.
    tmay
  • Reply 18 of 22
    mattinoz said:
    It’d be great if iCloud Drive indexed the contents of your files so you could, you know, actually search through them while on the go, instead of having to go back to your Mac to use Spotlight on local machine’s copy of them…
    Even better if we had our own private siri instance that could then use this index as starting point then go out to other sources as needed.
    What do you need a private Siri instance for — actually aren’t all Siri instances private?

    No matter, FDBs indexing would be excellent for analyzing/searching Siri queries.
    edited April 2018
  • Reply 19 of 22
    mattinozmattinoz Posts: 2,299member
    mattinoz said:
    It’d be great if iCloud Drive indexed the contents of your files so you could, you know, actually search through them while on the go, instead of having to go back to your Mac to use Spotlight on local machine’s copy of them…
    Even better if we had our own private siri instance that could then use this index as starting point then go out to other sources as needed.
    What do you need a private Siri instance for — actually aren’t all Siri instances private?

    No matter, FDBs indexing would be excellent for analyzing/searching Siri queries.
    So Siri and I could have a conversation and it be private between us. Yet I could delete with confidence down the track and know Siri can't use that information outside. Building confidence about what information I can share. So more like the relationship you have with an assistant.  I mean If I'm cooking steak I want siri to understand and set up timers for as I go then remember if for next time, including maybe tweaking the timing for next time by telling siri 1/2 hour later while washing up that steak was slightly over/under. Over time build up lots of shorthand between us, even surface information for me without being asked.

    The privacy of current Siri the learning and features Siri is criticized for not having.
    If FoundationDB could run on device then that store could sync to my devices and never need to be on Apple's servers or could just be there encrypted.
  • Reply 20 of 22
    mattinoz said:
    mattinoz said:
    It’d be great if iCloud Drive indexed the contents of your files so you could, you know, actually search through them while on the go, instead of having to go back to your Mac to use Spotlight on local machine’s copy of them…
    Even better if we had our own private siri instance that could then use this index as starting point then go out to other sources as needed.
    What do you need a private Siri instance for — actually aren’t all Siri instances private?

    No matter, FDBs indexing would be excellent for analyzing/searching Siri queries.
    So Siri and I could have a conversation and it be private between us. Yet I could delete with confidence down the track and know Siri can't use that information outside. Building confidence about what information I can share. So more like the relationship you have with an assistant.  I mean If I'm cooking steak I want siri to understand and set up timers for as I go then remember if for next time, including maybe tweaking the timing for next time by telling siri 1/2 hour later while washing up that steak was slightly over/under. Over time build up lots of shorthand between us, even surface information for me without being asked.

    The privacy of current Siri the learning and features Siri is criticized for not having.
    If FoundationDB could run on device then that store could sync to my devices and never need to be on Apple's servers or could just be there encrypted.
    It is likely that Siri uses a FDB database on Apple servers to search for binary sound matches to determine what you are asking Siri to do.  

    I think that Siri/FDB could be used in a similar way on your iPhone using a database local to the iPhone — and satisfy privacy requirements... if Apple chooses to implement it that way.  

    AFAICT, the A11 chip with 6 cores and 256GB has enough power to do the job.


    https://apple.github.io/foundationdb/benchmarking.html

    Without getting too far into the weeds, the above benchmark was done on a single Intel core.  They ran 100 parallel tasks, each doing 1,000-key range-reads (searches) resolving 3.6 million keys per second.  *

    A similar process could be used for music matches (Shazam?), face matchings (Photos) or other AI and search optimization.

    It appears likely that an A11 chip could resolve thousands of keys/second. 


    * It should be noted that, for performance reasons, FDB has a default 100KB maximum for values in its ordered key/value core db.  100KB, likely isn't large enough to hold large sound or image values which may be in the range of 5MB. The developer can mitigate this by splitting the large value over multiple keys -- where each successive key stores the next block of the value.
     
    mattinoz
Sign In or Register to comment.