Apple's FoundationDB takes new Record Layer open source, confirms tech underpins CloudKit

Posted:
in General Discussion edited January 2019
Apple database arm FoundationDB on Monday announced the open source release of FoundationDB Record Layer, a relational database management technology that powers CloudKit.

FoundationDB


Explained in a post to the official FoundationDB blog, FoundationDB Record Layer provides schema management, indexing facilities and query capabilities to the eponymous distributed datastore.

Beyond features commonly found in a relational database, Record Layer offers support for complex nested data types, indexes on the commit-time of records and indexes and queries that span different types of records, FoudationDB says. As a layer built on FoundationDB, the open source release incorporates ACID semantics and transactional semantics, the latter of which allows for a massive distributed datastore.

Record Layers is stateless, which allows for highly scalable implementations and speedy functionality. According to FoundationDB, database instantiation and subsequent operation can be performed in "milliseconds."

Apple acquired FoundationDB in 2015, taking the scalable NoSQL, ACID-compliant architecture in-house for previously undisclosed purposes. Reports at the time suggested the database software would be used as a basis for Apple's various cloud services, including iCloud and iTunes content management.

Today's announcement confirms FoundationDB and Record Layer act as a foundation for CloudKit. An accompanying white paper details Apple's integration, as well as additional nuances of Record Layer's underlying technology.

The open sourcing of Record Layer represents the second such release from Apple's distributed database subsidiary. In 2018, Apple released FoundationDB as an open source project, hoping to build a community around the scalable system.

Comments

  • Reply 1 of 8
    Wow...  This is awesome.  Gotta check it out.
    The C APIs for FDB were very low level, this layer should provide higher level APIs and hopefully seamless CoreData integration.
    Keep the layers coming.  SQL?
    This DB has the potential to put some established DBs to shame in terms of scalability, performance, redundancy and ease of use.

    Go Apple!
    cgWerks
  • Reply 2 of 8
    By Codd, this is MegaBig...

    From the Wait Paper:

    The Record Layer is used by multiple systems at Apple. We demonstrate the power of the Record Layer at scale by de- scribing how CloudKit, Apple’s cloud backend service, uses it to provide strongly-consistent data storage for a large and diverse set of applications [43]. Using the Record Layer’s abstractions, CloudKit offers multi-tenancy at the extreme by maintaining independent record stores for each user of each application. As a result, we use the Record Layer on FoundationDB to host billions of independent databases shar- ing thousands of schemas. In the future, we envision that the Record Layer will be combined with other storage mod- els, such as queues and graphs, leveraging FoundationDB as a general purpose storage engine and remaining transac- tionally consistent across all these models. In summary, this work makes the following contributions:

    • An open source layer on top of FoundationDB with se- mantics akin to those of a relational database.

    • The record store abstraction and a suite of techniques to manipulate it, enabling billions of logical tenants to operate independent databases in a FoundationDB cluster.

    • A highly extensible architecture, clients can customize core features including schema management and indexing.

    • A lightweight design that provides rich features on top of

      the underlying key-value store.

    netmageRayz2016patsu
  • Reply 3 of 8

    "Represents records as Protocol Buffer messages"

    This should go very nicely with CoreData and may actually enhance CoreData.
    Google's Proto compilers will generate getter and setters for various languages to facilitate CRUD.
    This is a game changer.  Googlers will be all over this DB.
  • Reply 4 of 8
    netmagenetmage Posts: 314member
    Instead of SQL how about a LINQ driver?
  • Reply 5 of 8
    Tantalizing read...

    Interesting to note that Apple is building rich layers for abstraction.  Got me thinking about a relational system for AI, AR and Siri.
    Never mind, I know what they’re doing with it now.
  • Reply 6 of 8
    netmage said:
    Instead of SQL how about a LINQ driver?
    How about not. Stick to SQL.
  • Reply 7 of 8
    Using protobuf is great. 
    Looking forward to a Swift protobuf compiler for Ubuntu Linux. 
    XCode already has a super elegant GUI for this in Mac OS X
    Swift also offers a significant advantage over C++ by using Extensions to add intelligence to protobuf classes instead of wrappers. 

    Swift + Foundation DB  could become  Linux favorites for DB development. 
  • Reply 8 of 8
    Using protobuf is great. 
    Looking forward to a Swift protobuf compiler for Ubuntu Linux. 
    XCode already has a super elegant GUI for this in Mac OS X
    Swift also offers a significant advantage over C++ by using Extensions to add intelligence to protobuf classes instead of wrappers. 

    Swift + Foundation DB  could become  Linux favorites for DB development. 

    Oh, Yeah...

    I've been fiddle-farting around trying to install the FoundationDB example with the CLI -- I am not very good at the CLI, but every 2-3 years I need to do something which results in a few days or weeks of intense CLI activity... then my mind numbs 'til the next fit attacks :#

    I would really like to play with FDB in Swift Playgrounds on the Mac or iPad.

    mdriftmeyer said: netmage said: Instead of SQL how about a LINQ driver? How about not. Stick to SQL.
    Yeah, it appears that the FDB Record Layer can evolve to be an efficient superset of an SQL relational DB.  It appears that FDB is much more granular (can be very useful) but that granualarity can be hidden with higher-level functions (as in Swift).

    From what I'm reading, the Record Layer syntax can closely parallel SQL and then some -- and has the  advantage of being designed for today's OSes, Hardware and programming languages.
Sign In or Register to comment.