Apple-owned FoundationDB open sources the core technology at the heart of iCloud
Apple's FoundationDB on Thursday announced that its namesake distributed datastore tech for organizations has been turned into an open-source project, allowing public-driven customization and expansion.
Source code is now available on GitHub, and developers can communicate with each other on official forums, the company said. FoundationDB is built to be deployed on hardware clusters, for instance making it easier to add machines or bounce back from device failures.
The open-source material represents the key-value store, on which "layers" can be added to scale functionality. The Github listing for the project calls it a "distributed database designed to handle large volumes of structured data across clusters of commodity servers."
"By open sourcing the FoundationDB core, we expect the quantity and variety of layers to develop rapidly," the company said. "All major development will be done in the open," it added, noting that it has outlined a design document process and project governance it promises will offer input and transparency. The community is also expected to follow a code of conduct.
Apple now has a variety of open-source initiatives under its wing. Perhaps the best known is Swift, its programming language. While not strictly limited, the language is almost exclusively used for iOS, macOS, watchOS, and tvOS development.
FoundationDB was originally a multi-model NoSQL database, designed around a "core" database. The core database allowed for read or write transactions with multiple keys stored on any machine in the cluster. Apple purchased the company and all of its intellectual property in 2015 to expand on the iCloud infrastructure.
In the past year Apple has made a strong push to put Swift in colleges and high schools -- mostly in the U.S., but internationally as well. The company has positioned the program as a way of boosting coding skills, even as it benefits from encouraging development for its products.
Source code is now available on GitHub, and developers can communicate with each other on official forums, the company said. FoundationDB is built to be deployed on hardware clusters, for instance making it easier to add machines or bounce back from device failures.
The open-source material represents the key-value store, on which "layers" can be added to scale functionality. The Github listing for the project calls it a "distributed database designed to handle large volumes of structured data across clusters of commodity servers."
"By open sourcing the FoundationDB core, we expect the quantity and variety of layers to develop rapidly," the company said. "All major development will be done in the open," it added, noting that it has outlined a design document process and project governance it promises will offer input and transparency. The community is also expected to follow a code of conduct.
Apple now has a variety of open-source initiatives under its wing. Perhaps the best known is Swift, its programming language. While not strictly limited, the language is almost exclusively used for iOS, macOS, watchOS, and tvOS development.
FoundationDB was originally a multi-model NoSQL database, designed around a "core" database. The core database allowed for read or write transactions with multiple keys stored on any machine in the cluster. Apple purchased the company and all of its intellectual property in 2015 to expand on the iCloud infrastructure.
In the past year Apple has made a strong push to put Swift in colleges and high schools -- mostly in the U.S., but internationally as well. The company has positioned the program as a way of boosting coding skills, even as it benefits from encouraging development for its products.
Comments
What other parts of iCloud will be open-sourced so users can build their own iCloud?
Sounds encouraging to me. Might look into this for fun.
I posted a question to the FoundationDB forum asking if Swift support is planned:
https://forums.foundationdb.org/c/development/fdb-core
Interesting Questions?
I suspect, that if the API is rich enough, independent developers could build the equivalent of local iCloud implementations. I'm unsure whether Apple could open up iCloud to 3rd-parties without compromising Apple's iCloud services...
But, how about IBM's cloud offerings... Or Google's... Or Microsofts...
https://forums.foundationdb.org/c/development/fdb-core
FDB's core implementation is based on an Ordered Key-Value Pair -- basically an Ordered Dictionary. macOS, iOS and most OSes do not support an Ordered Dictionary.
FWIW, A while back, I was experimenting with iOS Swift Playgrounds and was able to approximate what FDB did with their Ordered Key-Value Pair.
If FDB/Swift/ARM is viable, it opens up many opportunities, e.g. small business Point-Of-Sale-Terminals (iPads) WiFi connected to a FDB transaction server (cluster) running in the back room on a Mac Mini or an iPad while simultaneously running other clusters in the cloud.
This could really be a game changer.
Second, this doesn’t really enable any new applications. Businesses could already use iPads connected to something like a MySQL (a database that’s been around for a long time and has APIs in nearly every language under the sun) server. Or, more likely, a cloud service that integrates payment processing and inventory management.
Third, you could probably run this on an ARM server if you wanted to, as long as the server is running Linux. A quick look at the git repo shows that they’ve got it all set up with Docker containers. You don’t need “ARM support” to be able to interact with it from an ARM based device like an iPad - how do you think iCloud works now? All you need are the language bindings for the language that you’re using to program that ARM device.
This is just another database that happens to be good for massively distributed applications (like iCloud). It’s really not applicable to the small business like you suggested.
Sure, the ordered dictionary is a data structure not an OS-level construct... but one has to wonder why it isn't implemented at the OS level... You can accomplish the implementation with a Kludgey combination of arrays and dictionaries -- a more standard implementation (0S level or not) could improve performance and reliability. FWIW SQLite4 reportedly uses an ordered dictionary.
Agreed, this doesn't enable new applications -- but it does offer the potential of bringing down the cost and complexity so that almost anyone could run a reliable processor for transactions, payments, inventory, etc. I suspect that 3rd-parties will implement these solutions.
Yeah, you can run on an ARM Server... Maybe even a Raspberry PI. AFAIK, Some parts of FDB are written pretty close to the metal © -- optimizing for ARM could maintain FDB advantages while bringing down the cost of implementation.
Yes, FDB is a db that can scale to massively distributed applications -- but that does not mean that it only applies to large scale implementations.
Here's an interesting FDB video that FDB released before they were acquired by Apple:
Those are pretty small/inexpensive hardware boxes (could be headless Raspberry PIs, Mac minis, AppleTVs, etc.)... You could put the whole thing in a shoebox that would work at a street fair or swap meet whether the Internet was available or not.
This is critical in a database that supports a large number of concurrent transactions with out locking the DB to other transactions. When Apple acquired FDB, their latest release supported 14 million random writes per second.
2. What's the point of putting private keys in public source code?
I am looking forward to trying it out. It has the potential to take the OSS DB World by storm.
It is interesting how Apple is moving all their server modules into open source.
MacOS can be turned in a full fledged server by downloading the server modules. (Web, DB, Directory Services, Wiki, File/Print etc...)
This could turn out to be very interesting for Mac Hardware.
The available API are listed below.
It must integrate seamlessly with CoreData hence no Swift nor Objective C APIs.
No matter, FDBs indexing would be excellent for analyzing/searching Siri queries.
The privacy of current Siri the learning and features Siri is criticized for not having.
If FoundationDB could run on device then that store could sync to my devices and never need to be on Apple's servers or could just be there encrypted.
I think that Siri/FDB could be used in a similar way on your iPhone using a database local to the iPhone — and satisfy privacy requirements... if Apple chooses to implement it that way.
AFAICT, the A11 chip with 6 cores and 256GB has enough power to do the job.
https://apple.github.io/foundationdb/benchmarking.html
Without getting too far into the weeds, the above benchmark was done on a single Intel core. They ran 100 parallel tasks, each doing 1,000-key range-reads (searches) resolving 3.6 million keys per second. *
A similar process could be used for music matches (Shazam?), face matchings (Photos) or other AI and search optimization.
It appears likely that an A11 chip could resolve thousands of keys/second.
* It should be noted that, for performance reasons, FDB has a default 100KB maximum for values in its ordered key/value core db. 100KB, likely isn't large enough to hold large sound or image values which may be in the range of 5MB. The developer can mitigate this by splitting the large value over multiple keys -- where each successive key stores the next block of the value.