Inside APFS: new Apple File System detailed at WWDC to replace HFS+ in 2017

Posted:
in macOS
The new Apple File System announced at WWDC highlights how the company is planning for the future. Here's a look under the hood at what the company will use to replace HFS+ on Macs, iOS, Apple Watch and Apple TV.




Apple's existing HFS+



Over the last three decades of Macs, Apple's file system has spanned a history where data was first stored on 800K floppies, then on magnetic hard drives where 40MB of storage was considered a lot. Today, hard drives are measured in terabytes, and most of the devices Apple now sells use Flash or SSD storage.

HFS+ has been modified several times since it was "plus" refreshed in the late 1990s (largely to accommodate bigger drives). Apple has bolted on file system Journaling, which records a continuous record of changes to files so that if something bad happens--such as a loss of power--the file system can restore a disk to a known good condition.

Apple also created a version of HFS+ called HFSX that is case-sensitive (like most Unix systems) for use with server software that expects to be able to create multiple files in the same directory that differ in only in case (file1h and file1H, for example).

And when Apple introduced iOS, it created a specialized file system based on the Mac's HFSX with support for individual file encryption used for data protection on mobile devices.

Apple has also developed storage technologies that exist below HFS+, ranging from EFI storage partitions (Intel's equivalent to OpenFirmware, which Apple pioneered adoption of when it moved to Intel chips a decade ago) to CoreStorage, the logical volume format Apple developed to support features such as Full Disk Encryption and Fusion Drive. Logical volumes are created on disk partitions and then initialized with a file system.

However, HFS+ itself still has some significant shortcomings, and continuing to bolt on new features would eventually break compatibility with the past because the file system uses some rigid data structures that can't effectively be stretched much further without requiring a non-backward compatible volume format change.

HFS+ also uses largely single-threaded data structures which require catalog file locks that that make it hard for multiple processes to access the same data at once.

Apple File System to become default in 2017



Apple currently calls APFS, its replacement to HFS+, a developer preview in macOS Sierra, but it expects to complete support across its platforms and make it the default file system for all Apple products next year.

At WWDC, the company emphasized that APFS was designed with all of Apple's platforms in mind, scaling from watchOS to iOS and tvOS to macOS, and taking advantage of the fact that most of the company's products are now using Flash or SSD storage. It also was designed with encryption in mind as a primary feature.When complete, users will be able to upgrade from HFS+ to APFS in place

When complete, users will be able to upgrade from HFS+ to APFS in place, similar to how Microsoft migrated PC users from FAT to NTFS. User data will remain in place, and APFS will create new metadata within free space on the HFS+ volume before converting it into a APFS volume.

Developing a new Apple File System for the future



The new APFS opens up a variety of new features to support technologies on the horizon over the next decades. Apple developed its own new APFS rather than using an existing file system because most existing file systems are tuned for use cases that differ from Apple's.

For example, many high performance modern file systems are tuned to the needs of servers with multiple disks, and often optimize for throughput rather than latency. On mobile devices, latency is more critical--users want to see immediate results when they tap, more so than they need to move huge files quickly. Most Apple devices also only have one disk.


Xsan


Apple already supports a high performance, shared file system for Storage Area Networks on macOS: Xsan, which supports multiple platform users accessing the same volume via FibreChannel, using a dedicated metadata controller.

To address the more common needs of its existing HFS+ users, Apple developed an advanced replacement file system tuned specifically for the typical usage scenarios of its install base of consumer devices--now exceeding 1 billion worldwide.

Advanced new structures in APFS



APFS not only introduces new features, but is designed for future expansion in mind. This includes a jump to a 64-bit inode number (used to track each file; compare to HFS+ and its 32-bit catalog record). On APFS, this enables more than 9 quintillion (a million trillion) files on a disk.

The expanded new inode structure of APFS also appears to address a core problem in HFS+ related to file and directory information being kept in a single catalog file, which serializes access and creates a logjam when multiple threads are all trying to access the file system.

APFS also improves upon the one second time stamps of HFS+ with new nanosecond timestamps, making it possible to distinguish the order of files that may be written in parallel. APFS also has an expansive block allocator that allows users to set up a new very large disk or RAID array and begin using it before the system finishes slowly initializing the entire disk (as HFS+ requires).

APFS also adds support for Sparse files (HFS+ only includes virtual support) and implements a new copy-on-write metadata scheme that's used to maximize performance over the overhead required by HFS+ journaling. It also builds in support for extended file attributes, which on HSF+ were tacked on as a B-Tree Attributes file (which HFS+ uses to store Access Control List permission attributes, for example).

APFS will also harmonize support for full disk and individual file encryption across all of Apple's platforms, using multiple keys for user data and metadata designed to protect data access even if a device is physically obtained by another party.

APFS can be used with existing Apple RAID volumes to provide software-based striping (for performance) and mirroring (data protection) or simple disk spanning (for cost savings), and can also be used on hardware RAID devices.

While the current developer preview does not yet support Apple's Fusion drives (which combine an SSD and a conventional spinning hard drive in a RAID-like hybrid device), that's clearly in the cards. Apple also stressed that its new file system is designed around the modern reality of Flash and SDD storage, which presents different characteristics than spinning magnetic disk storage.

New features designed into Apple File System



In addition to greatly advancing and refining the basic design of HFS+, APFS also introduces a variety of completely new features.

This includes Space Sharing, which enables multiple volumes to claim the free space on a disk. That means a 4TB disk could have 4, 4TB volumes created on it, allowing each volume to expand as needed without creating a problem of having to repartition the volumes to assign more space to one of them. This could particularly benefit users of virtual machines (or Boot Camp Windows users) where a given disk volume on disk needs to create a very large file.

Another new feature, Clones, allows files or even entire directories to be instantly "copied" without using up storage space. This supports the idea of "duplicating" a document or an entire workflow, and then making multiple sets of independent changes to each copy. Because the changes are saved separately, the bulk of the parallel projects can be stored as one pristine copy, saving lots of otherwise wasted disk space.

APFS also supports Snapshots, which work something like Clones on an entire file system level; they preserve the state of an entire file system at a given point in time, recording new changes separately so that the system can efficiently backup and roll everything back to a clean slate Snapshot. This has applications in education and other shared systems, as well as being a more efficient and less convoluted way to backup devices compared to Apple's existing Time Machine, which saves backup points in time using on hard links.

Another new feature is Fast Directory Sizing, which enables the file system to calculate the space used by a folder and all of its contents and then keep this figure up to date. HFS+ has to regularly recalculate this information any time you Get Info on a folder because it can't simply store this data in a folder directory itself or percolate this information up the file system hierarchy (because child items can't lock their parent).

Because many files or apps on Apple's platforms are actually directories of contents ("Bundles") that merely pose as a singular item for simplicity, APFS introduces a feature called Atomic Safe-Save that ensures that when updates are being saved, the process can't be interrupted, resulting in a partially-saved (and possibly corrupt) item. The process is "Atomic" because it either successfully completes or doesn't, but won't end up in a split condition of uncertain status.

Origins and evolution of Apple File System



It's notable that Apple has been expanding upon HFS+ for years. Apple hired file system guru Dominic Giampaolo back in 2002, before the company even had a significant mobile device business and at at time when Macs were just transitioning from the classic Mac OS 9 to the new OS X, and would remain PowerPC based for another four years.


BeOS File Types let users set MIME types, icons, application binding, filename extensions and attributes


Giampaolo was perhaps best known for developing BeFS in 1996, an advanced file system of its time designed for BeOS (above). It relied heavily on extended file attribute metadata that the system could index to enable users to quickly search for data like a relational database. Today, iTunes and Photos similarly present collections of items in an easy to sort and search fashion.

Giampaolo literally wrote the book on file system design. After joining Apple, he worked on adding file system journaling to HFS+ (which was turned on by default in Mac OS X 10.3), and then initiated the Spotlight project to bring BeFS-style, indexed metadata search to the Mac (a feature which debuted in 2005).

Giampaolo also worked on Time Machine and the wireless Time Capsule product for backups, and helped develop the initial Documents and Data implementation for iCloud in 2011.

He also cited work on a "file system that never shipped," and Apple separately worked on adding support for ZFS on Macs, before abandoning the effort in 2009 (apparently related to licensing issues, although the sudden rise of iPhone likely also shifted Apple's strategy).

Giampaolo is now working on APFS as a senior software engineer in Storage and File Systems, applying the experience of a couple decades of developing file system-related features.

Once deployed next year, APFS will not only enable new new features and optimize performance and the utilization of modern storage systems, but will also serve as an extendable platform for introducing new file system advances in the future.
propod
«1

Comments

  • Reply 1 of 25
    sirdirsirdir Posts: 114member
    Huh I don't think the 'flexible partitioning' will help BootCamp users, as I doubt Bootcamp will be APFS formatted...

  • Reply 2 of 25
    polymniapolymnia Posts: 900member
    sirdir said:
    Huh I don't think the 'flexible partitioning' will help BootCamp users, as I doubt Bootcamp will be APFS formatted...

    Not that I'm any expert on such things, but can't you imagine a file system that is being just invented could be designed in such a way as to completely contain another file system and create virtual/logical Partitions, Volumes, Drive Letters or whatever silliness Windows requires be present.

    Virtualization is a thing that I don't understand at all, but I know its a thing, and its kind of a big deal.
    latifbp
  • Reply 3 of 25
    correctionscorrections Posts: 1,246member
    sirdir said:
    Huh I don't think the 'flexible partitioning' will help BootCamp users, as I doubt Bootcamp will be APFS formatted...

    Even if you have a Windows on an NTFS volume in your partition map, the flexible partitioning of APFS should allow you to share the remaining free space, removing barriers.
    moreck
  • Reply 4 of 25
    SoliSoli Posts: 8,550member
    sirdir said:
    Huh I don't think the 'flexible partitioning' will help BootCamp users, as I doubt Bootcamp will be APFS formatted...

    Containers are interesting because of the obvious potential to put different things within them. I don't see anything outright that would make a NTFS partition impossible within the APFS container. Think about how it's possible to create a Windows NTFS or FAT* partition in a VM as a separate file on a disparate OS where the partition can be resized as needed. That said, I doubt Apple will work so hard to make this new and still unfinished file system work that well with a Windows dual-boot. I'd bet the cost far exceeds the benefit for Apple.

    edited June 2016
  • Reply 5 of 25
    kevin keekevin kee Posts: 965member
    I wonder if APFS will improve indexing and increase searching efficiency and speed across all devices/networks attached, including wireless devices.
    edited June 2016 williamlondon
  • Reply 6 of 25
    lmasantilmasanti Posts: 58member
    Just curious… APFS…

    APple File System…

    Apple Private File System…

    Apple Protected File System…

    My best guest: Apple Platforms File System…

    (Although they said they will —probably— open sourced it to be used in servers —Linux, Windows—).
  • Reply 7 of 25
    ksecksec Posts: 1,549member
    Since they are going to use this across product range, i am thinking there will be an Time Capsule design with APFS snapshot in mind
  • Reply 8 of 25
    SoliSoli Posts: 8,550member
    ksec said:
    Since they are going to use this across product range, i am thinking there will be an Time Capsule design with APFS snapshot in mind
    Do you also predict an Time Capsule, or AirPort products using iOS? I hope that happens (as well as an Apple Home Server), but I am very doubtful for either iOS or APFS coming to those products. The only saving grace are the rumours that Apple may finally be trying to catch up with the Amazon's Echo, which I can't imagine would run anything other than iOS.
  • Reply 9 of 25
    mattinozmattinoz Posts: 1,003member
    ksec said:
    Since they are going to use this across product range, i am thinking there will be an Time Capsule design with APFS snapshot in mind
    Seeing they are going deploy across platforms, including the AppleTV, why not just make an AppleTVHub with a raid of 2.5" inch drives instead of the single drive.
  • Reply 10 of 25
    mattinozmattinoz Posts: 1,003member
    ksec said:
    Since they are going to use this across product range, i am thinking there will be an Time Capsule design with APFS snapshot in mind
    Seeing they are going deploy across platforms, including the AppleTV, why not just make an AppleTVHub with a raid of 2.5" inch drives instead of the single drive.
  • Reply 11 of 25
    lmasantilmasanti Posts: 58member
    Soli said:
    ksec said:
    Since they are going to use this across product range, i am thinking there will be an Time Capsule design with APFS snapshot in mind
    Do you also predict an Time Capsule, or AirPort products using iOS? I hope that happens (as well as an Apple Home Server), but I am very doubtful for either iOS or APFS coming to those products. The only saving grace are the rumours that Apple may finally be trying to catch up with the Amazon's Echo, which I can't imagine would run anything other than iOS.
    APFS will be in macOS X… that is in Time Capsule.
  • Reply 12 of 25
    SoliSoli Posts: 8,550member
    lmasanti said:
    Soli said:
    Do you also predict an Time Capsule, or AirPort products using iOS? I hope that happens (as well as an Apple Home Server), but I am very doubtful for either iOS or APFS coming to those products. The only saving grace are the rumours that Apple may finally be trying to catch up with the Amazon's Echo, which I can't imagine would run anything other than iOS.
    APFS will be in macOS X… that is in Time Capsule.
    Time Capsule does not run any version of macOS and that APFS will span macOS, iOS, tvOS, and watchOS. Note that ksec mentioned Time Capsule in his post, not a Mac.

    It appears all the current AirPort products run a version of NetBSD.

    edited June 2016
  • Reply 13 of 25
    Will this file system address bit rot?
    Habi_tweet
  • Reply 14 of 25
    appexappex Posts: 687member
    ZFS?
  • Reply 15 of 25
    mubailimubaili Posts: 385member
    Soli said:
    ksec said:
    Since they are going to use this across product range, i am thinking there will be an Time Capsule design with APFS snapshot in mind
    Do you also predict an Time Capsule, or AirPort products using iOS? I hope that happens (as well as an Apple Home Server), but I am very doubtful for either iOS or APFS coming to those products. The only saving grace are the rumours that Apple may finally be trying to catch up with the Amazon's Echo, which I can't imagine would run anything other than iOS.
    The "only saving grace"? Please, give me a break. As with Swift, APFS, WebKit, Darwin, LLVM, iOS etc have demonstrated plenty, Apple is very patient and dance to its own rhythm. 
    mattinozcoolfactordocno42ai46
  • Reply 16 of 25
    polymnia said:
    sirdir said:
    Huh I don't think the 'flexible partitioning' will help BootCamp users, as I doubt Bootcamp will be APFS formatted...

    Not that I'm any expert on such things, but can't you imagine a file system that is being just invented could be designed in such a way as to completely contain another file system and create virtual/logical Partitions, Volumes, Drive Letters or whatever silliness Windows requires be present.

    Virtualization is a thing that I don't understand at all, but I know its a thing, and its kind of a big deal.
    This is the very definition of virtual file systems. Virtual file systems rarely ever sit on the raw device partition, but instead are layered on top of another file system. Lets take a rudimentary example of having VMWare fusion on your mac. You run HFS+ as the primary file system. when you go to create a new VM, you don't resize the HFS+ filesystem and format a small block for NTFS. You simply create the VMDK that sits on top of the HFS+ system and windows formats the volume inside the installer. the VMDK isn't formatted in NTFS, but the volume INSIDE of it is. This would be the same principle for Apple's APFS. APFS is simply acting like a logical volume manager and its creating "fences" for you to drop data in. those "fences" are dynamic in size, or "thinly provisioned." The same concept applies to ZFS volumes or LVMs in Linux. Nothing new in terms of concept. As for bootcamp, you literally resize the partition and leave a blank partition to be filled and written to by windows. So the raw partition in this case has the actual NTFS file system on it. There would be no way to format the partition as APFS and expect windows boot loader to read the partition properly unless there was some foo that made it possible for the EFI firmware to present the NTFS volume without making windows read the APFS volume first.
  • Reply 17 of 25
    coolfactorcoolfactor Posts: 1,400member
    APFS is probably the most exciting announcement that's come out this year. The future is very bright for the iOS and Mac platforms... if Apple can deliver on their promises.
    docno42steveh
  • Reply 18 of 25
    MarvinMarvin Posts: 14,202moderator
    lmasanti said:
    Just curious… APFS…

    APple File System…

    Apple Private File System…

    Apple Protected File System…

    My best guest: Apple Platforms File System…

    (Although they said they will —probably— open sourced it to be used in servers —Linux, Windows—).
    Apple's just referring to it as Apple File System. They couldn't use AFS as that's already another filesystem:

    https://en.wikipedia.org/wiki/Andrew_File_System

    Soli said:
    sirdir said:
    Huh I don't think the 'flexible partitioning' will help BootCamp users, as I doubt Bootcamp will be APFS formatted...

    Containers are interesting because of the obvious potential to put different things within them. I don't see anything outright that would make a NTFS partition impossible within the APFS container. Think about how it's possible to create a Windows NTFS or FAT* partition in a VM as a separate file on a disparate OS where the partition can be resized as needed. That said, I doubt Apple will work so hard to make this new and still unfinished file system work that well with a Windows dual-boot. I'd bet the cost far exceeds the benefit for Apple.

    It would be very useful if they supported other formats or just get Windows to be able to support it natively using a driver. It would be good for VM software as they wouldn't need to make fixed disk images any more. The Mac system will benefit from this as people can install multiple systems easily to test them out. They can make an entire bootable partition with a clone of the existing system in seconds and upgrade the clone partition to a new OS, even a beta OS and test it out. It will only write the changes to the new partition. If it's not working right, people can just wipe the partition. It has a lot of benefits for autosave, snapshots, version control.

    I'd like to see them implement filesystem compression too. That's something NTFS has and it saves a lot of space. It can just be like their RAM compression. The performance hit is minimal and could give people large amounts of extra storage space, especially if they store lots of uncompressed file formats.
  • Reply 19 of 25
    SoliSoli Posts: 8,550member
    mubaili said:
    Soli said:
    Do you also predict an Time Capsule, or AirPort products using iOS? I hope that happens (as well as an Apple Home Server), but I am very doubtful for either iOS or APFS coming to those products. The only saving grace are the rumours that Apple may finally be trying to catch up with the Amazon's Echo, which I can't imagine would run anything other than iOS.
    The "only saving grace"? Please, give me a break. As with Swift, APFS, WebKit, Darwin, LLVM, iOS etc have demonstrated plenty, Apple is very patient and dance to its own rhythm. 
    None of those things speak directly to Apple bringing out an Apple Home Server running iOS, which means the only saving grace for this decades long desire is the rumour of a Siri-based device for the home. 
  • Reply 20 of 25
    SoliSoli Posts: 8,550member
    @appex ;;;said:
    ZFS?
    What about it? APFX is similar to ZFS in many ways, but what Apple is doing with APFS is undeniably better suited for their operating systems, devices, and customers.
    edited June 2016
Sign In or Register to comment.