Mac OS X Snow Leopard Server to pioneer ZFS ahead of desktop
Apple's expanded support for ZFS will premiere in Snow Leopard Server before trickling down to the desktop version, according to sources familiar with the company's plans.
Read-only support for the sophisticated new 128-bit file system, created by Sun and shared under its open source license, originally appeared in Mac OS X Leopard. It lacked the ability to create new ZFS pools or write data to them however, severely limiting it usefulness.
Because the software is open source, users can manually install the latest version of the software from Apple's MacOS Forge site and use ZFS from the command line to create pools and write data, although that lacks the interface polish Mac users expect to see.
For example, users can't currently empty the trash graphically in the Finder when working with a ZFS drive. Unplugging a device connected to a ZFS pool will also cause a panic unless the disk is properly closed out from the command line first.
In Snow Leopard Server, sources familiar with the new software say Apple will not only include the latest build of ZFS for Mac OS X but will also expose ZFS features within the graphical Disk Utility, making it easier to set up ZFS storage pools and file systems without resorting to command line utilities. The Finder also needs to be made fully ZFS savvy, as does any other software that makes assumptions about the underlying file system.
The Finder and Disk Utility app used by the desktop and server versions of Mac OS X have always been identical. Even so, Apple may initially keep full ZFS support associated with its server product, because server users have a greater practical need for the features related to ZFS, and also have the support resources to handle working with the new system. A similar thing happened when Apple released its IP over Firewire implementation, which was first incorporated into Mac OS X Server before being added to the desktop version in a later free update.
Despite the giddy buzzword interest in ZFS that was fanned into an active swarm by pundits around the release of Leopard (and touched off rumors that ZFS would become the default file system in Leopard), many applications of the new file system will require sophisticated reworking of lots of associated software. As a consumer-centric company, it would be expensive in terms of customer support for Apple to dump its desktop users on top of an alpha quality, brand new file system just to check off a feature box in its marketing. Most consumers, and in particular the notebook users who make up most of Apple's sales, have little or no real need for ZFS, but would be burned by its complexity and additional demands.
While ZFS support has made major headway on the Mac since the initial release of Leopard, there's still a long way to go before non-technical users can make real use of its new features. That will likely result in Apple focusing its development efforts to make ZFS practical initially to its server audience. That is reflected in the company's Snow Leopard marketing, which only mentions ZFS features in relation to Snow Leopard Server in "business critical server deployments." More technically-savvy desktop users will likely be able to begin experimenting with the new file system on the Snow Leopard desktop however.
ZFS' server-oriented features
As an enterprise vendor with no real consumer-facing business, Sun developed ZFS to solve the needs of server users. Implementing its features on the desktop to benefit less technical consumers will require significant work on Apple's part. Once the company completes a solid foundation on Mac OS X Server, making the technology accessible to end users will be easier. Among the potential benefits will be more flexible use of multiple drives, data redundancy, error correction, and snapshots. Among the challenges to overcome are greater disk waste (due to redundancy and snapshots) and greater complexity in managing storage pools.
ZFS uses a storage pool system that allows it to use a combination of block devices (abstracted as virtual devices) to build a logical drive that can contain a file system. This results in a RAID-like system that can span different drives and provide a level of redundancy to survive drive failure. Unlike a typical RAID, ZFS allows different disks to be tied together in a pool, and new drives to be added to existing pools without reformatting the drive.
The file system also provides continuous integrity checking and automatic repair to aid in file corruption, and supports massively large volumes. Its support for snapshots, which capture old data and retain it as a past "version" for later use, and clones, which enable two separate file systems to share overlapping blocks of identical data, has led some to speculate that Apple would use ZFS in conjunction with its Time Machine backup system.
However, Apple implemented features in the Mac's native HFS+ to accommodate parallel instances of backups for Time Machine. Further, the system is based on copying newly changed files (as reported by a system auditing process) to a new disk, not retaining old data on the same disk as it is updated.
For users with a single drive in their system (as is the case with most desktop Mac users), shadow copy snapshots would solve the wrong problem. Users want to be protected from the crisis that might befall their hard drive, not a system that would only eat up their hard drive faster and then go down with the ship in the case of drive failure.
For now, that mostly leaves ZFS as a technical curiosity for most Mac users outside of those who manage very large disk arrays in a server environment. Even so, progress on the server side will eventually result in trickle down engineering for Apple's consumer users, too, just as the company's work in Directory Services has resulted in Parental Controls for desktop users. A variety of sharing services that started out as server products have also made their way to the Mac OS X desktop, including Apache web services and remote management screen sharing.
Read-only support for the sophisticated new 128-bit file system, created by Sun and shared under its open source license, originally appeared in Mac OS X Leopard. It lacked the ability to create new ZFS pools or write data to them however, severely limiting it usefulness.
Because the software is open source, users can manually install the latest version of the software from Apple's MacOS Forge site and use ZFS from the command line to create pools and write data, although that lacks the interface polish Mac users expect to see.
For example, users can't currently empty the trash graphically in the Finder when working with a ZFS drive. Unplugging a device connected to a ZFS pool will also cause a panic unless the disk is properly closed out from the command line first.
In Snow Leopard Server, sources familiar with the new software say Apple will not only include the latest build of ZFS for Mac OS X but will also expose ZFS features within the graphical Disk Utility, making it easier to set up ZFS storage pools and file systems without resorting to command line utilities. The Finder also needs to be made fully ZFS savvy, as does any other software that makes assumptions about the underlying file system.
The Finder and Disk Utility app used by the desktop and server versions of Mac OS X have always been identical. Even so, Apple may initially keep full ZFS support associated with its server product, because server users have a greater practical need for the features related to ZFS, and also have the support resources to handle working with the new system. A similar thing happened when Apple released its IP over Firewire implementation, which was first incorporated into Mac OS X Server before being added to the desktop version in a later free update.
Despite the giddy buzzword interest in ZFS that was fanned into an active swarm by pundits around the release of Leopard (and touched off rumors that ZFS would become the default file system in Leopard), many applications of the new file system will require sophisticated reworking of lots of associated software. As a consumer-centric company, it would be expensive in terms of customer support for Apple to dump its desktop users on top of an alpha quality, brand new file system just to check off a feature box in its marketing. Most consumers, and in particular the notebook users who make up most of Apple's sales, have little or no real need for ZFS, but would be burned by its complexity and additional demands.
While ZFS support has made major headway on the Mac since the initial release of Leopard, there's still a long way to go before non-technical users can make real use of its new features. That will likely result in Apple focusing its development efforts to make ZFS practical initially to its server audience. That is reflected in the company's Snow Leopard marketing, which only mentions ZFS features in relation to Snow Leopard Server in "business critical server deployments." More technically-savvy desktop users will likely be able to begin experimenting with the new file system on the Snow Leopard desktop however.
ZFS' server-oriented features
As an enterprise vendor with no real consumer-facing business, Sun developed ZFS to solve the needs of server users. Implementing its features on the desktop to benefit less technical consumers will require significant work on Apple's part. Once the company completes a solid foundation on Mac OS X Server, making the technology accessible to end users will be easier. Among the potential benefits will be more flexible use of multiple drives, data redundancy, error correction, and snapshots. Among the challenges to overcome are greater disk waste (due to redundancy and snapshots) and greater complexity in managing storage pools.
ZFS uses a storage pool system that allows it to use a combination of block devices (abstracted as virtual devices) to build a logical drive that can contain a file system. This results in a RAID-like system that can span different drives and provide a level of redundancy to survive drive failure. Unlike a typical RAID, ZFS allows different disks to be tied together in a pool, and new drives to be added to existing pools without reformatting the drive.
The file system also provides continuous integrity checking and automatic repair to aid in file corruption, and supports massively large volumes. Its support for snapshots, which capture old data and retain it as a past "version" for later use, and clones, which enable two separate file systems to share overlapping blocks of identical data, has led some to speculate that Apple would use ZFS in conjunction with its Time Machine backup system.
However, Apple implemented features in the Mac's native HFS+ to accommodate parallel instances of backups for Time Machine. Further, the system is based on copying newly changed files (as reported by a system auditing process) to a new disk, not retaining old data on the same disk as it is updated.
For users with a single drive in their system (as is the case with most desktop Mac users), shadow copy snapshots would solve the wrong problem. Users want to be protected from the crisis that might befall their hard drive, not a system that would only eat up their hard drive faster and then go down with the ship in the case of drive failure.
For now, that mostly leaves ZFS as a technical curiosity for most Mac users outside of those who manage very large disk arrays in a server environment. Even so, progress on the server side will eventually result in trickle down engineering for Apple's consumer users, too, just as the company's work in Directory Services has resulted in Parental Controls for desktop users. A variety of sharing services that started out as server products have also made their way to the Mac OS X desktop, including Apache web services and remote management screen sharing.
Comments
As a consumer-centric company, it would be expensive in terms of customer support for Apple to dump its desktop users on top of an alpha quality, brand new file system just to check off a feature box in its marketing.
Are you inferring that this "consumer-centric" company would give an alpha-quality file system to business users?
I think Apple would lose a lot more money in lawsuits vs enterprises that lost data.
I don't think Apple could ever release an "alpha-quality" file system. If it's in Server, then it's definitely not alpha and ready for the consumer OS X version...but it's just a matter of how Apple wants to differentiate OS X and OS X Server.
I personally think it's a mistake that it's not offered to consumers. Sure, most are using MacBooks or iMacs...however, there's quite a few people that have Mac Pros with all bays filled.
I wouldn't mind having my 4 drives act as a single volume or two.
I think you greatly underestimate the usefulness of snapshots. Snapshots only use disk space for the block-level differences between snapshots and the current state so they take relatively little space. OpenSolaris has already implemented a Time Machine type interface that lets users go back in time in 15 minute increments and find files that they have deleted or are in a previous state. More importantly, snapshots make Time Machine backups simpler, allowing point-in-time backups instead of the problematic file-at-a-time method it now uses.
+1
I thought the commentary on snapshots was the weakest of the post. Snapshots, or more appropriately efficient snapshots, are very important. In fact as much as I like Time Machine it is rather pedestrian in how it manages delta changes.
I'm going to guess that we're going to see the ZFS distinction for Snow Leopard remain at Server=yes Desktop=no but that by the time 10.7 hits ZFS will indeed be much more important to the future of OS X than what we realize today.
Another thing that is important is Copy on Write. What this does in a nutshell is allow read access to a file from multiple sources without having to copy the file. The file is only copied when it has to be modified. Sounds rather plain but I imagine that there are plenty of resources in OS X that could be made more efficient by a Copy on Write model.
ZFS isn't just about multiple drives. The checksum features, snapshots and COW will benefit even the most basic user even if they may be unaware of the bestowing features.
If Apple is serious about ZFS they will most likely integrate ZFS into OS X and perhaps we will see more fine grain control in Time Machine, better snapshot capabilities for MobileMe and a more responsive OS in general.
Are you inferring that this "consumer-centric" company would give an alpha-quality file system to business users?
I think Apple would lose a lot more money in lawsuits vs enterprises that lost data.
I don't think Apple could ever release an "alpha-quality" file system. If it's in Server, then it's definitely not alpha and ready for the consumer OS X version...but it's just a matter of how Apple wants to differentiate OS X and OS X Server.
I personally think it's a mistake that it's not offered to consumers. Sure, most are using MacBooks or iMacs...however, there's quite a few people that have Mac Pros with all bays filled.
I wouldn't mind having my 4 drives act as a single volume or two.
Alpha quality? Really? It's standard in OpenSolaris 2008.11.
Do you honestly think Apple's state of ZFS is alpha? Try again.
why doesn't Apple just license NTFS? Wonder if MS would even? Seems like NTFS has all this and more
ZFS obliterates anything from Microsoft or Apple concerning filesystems.
http://www.sun.com/software/solaris/zfs.jsp
Being POSIX compliant makes it even simpler for Apple to incorporate it into it's fully POSIX compliant operating sytem.
How XSan evolves with regards to ZFS will be interesting.
why doesn't Apple just license NTFS? Wonder if MS would even? Seems like NTFS has all this and more
Simple.
ZFS is bigger better faster meaner.
In Solaris here's how you create storage across 32 disks
# zpool create -f tank (32 disks)
# zfs create tank/fs
Here's how you do it with NTFS
diskpart> create volume stripe disk= (32 disks)
DiskPart successfully created the volume
diskpart> assign letter=s
DiskPart successfully assigned the driver letter or mount point.
diskpart> exit
c:/> format s: /fs:ntfs
The type of the file system is RAW
The new file system is NTFS
WARNING, ALL DATA ON NON-REMOVABLE DISK DRIVES S:
WILL BE LOST!
Proceed with Format (Y/N)? Y
Verifying 840012M
Volume label (32 characters, ENTER for none)? <rtn>
Creating file system structures
Format complete
860172284 KB total disk space
860080288 KB are available
ZFS took 17.5 seconds
Windows Server 2003 and NTFS took 4.5 hours.
ZFS is really the ultimate file system. 128 bit native. unlimited file size. copy on write with full checksumming. self-healing abilities, snapshots and best of all, datapools that create safe, fast dynamic storage with virtually any combination of drives.
can you imagine what that kind of data integrity would mean for artists, designers, video editors, photographers, musicians and the like? it would be invaluable.
HFS+ is a good file system, but ZFS is far better.
the average consumer with a macbook may not know ZFS from PDQ, but when data corruption is elimiinated and the OS is fast, safe and stable...that's something any user will notice.
Simple.
ZFS is bigger better faster meaner.
In Solaris here's how you create storage across 32 disks
# zpool create -f tank (32 disks)
# zfs create tank/fs
Here's how you do it with NTFS
diskpart> create volume stripe disk= (32 disks)
DiskPart successfully created the volume
diskpart> assign letter=s
DiskPart successfully assigned the driver letter or mount point.
diskpart> exit
c:/> format s: /fs:ntfs
The type of the file system is RAW
The new file system is NTFS
WARNING, ALL DATA ON NON-REMOVABLE DISK DRIVES S:
WILL BE LOST!
Proceed with Format (Y/N)? Y
Verifying 840012M
Volume label (32 characters, ENTER for none)? <rtn>
Creating file system structures
Format complete
860172284 KB total disk space
860080288 KB are available
ZFS took 17.5 seconds
Windows Server 2003 and NTFS took 4.5 hours.
Nice post.
Shoehorning the semantics of an advanced file system like ZFS into existing APIs may introduce some changes in assumptions that are easy to work around with a limited vendor selection and a lot of open source. Doing the same thing for the thousands of small developers for the consumer version is a bigger task.
Other than that, there's really no reason why consumers wouldn't want it. Snapshots for Time Machine, better performance, data integrity, etc., are all things consumers like too.
Nice post.
Let me give credit where credit is due
http://blogs.sun.com/Peerapong/entry...fs_performance
Yes he may be a tad biased due ot his employer but his results mirror
what I've read elsewhere about the ease of setting up pools.
ZFS uses a storage pool system that allows it to use a combination of block devices (abstracted as virtual devices) to build a logical drive that can contain a file system. This results in a RAID-like system that can span different drives and provide a level of redundancy to survive drive failure.
Not entirely true. Pooling drives alone is equivalent to concatonating drives with HFS. They all appear as a single file system, but there is no redundancy. For that, you need to turn on RAID-Z, which is functionally similar to RAID-5.
I can't imagine anyone with multiple drives not using RAID-Z to achieve redundancy, but it can be turned off if desired.
The file system also provides continuous integrity checking and automatic repair to aid in file corruption
This feature alone is why I want ZFS now! As has been pointed out, artists and others with lots of important content (actually anyone with valuable content) can benefit from this immediately, if not sooner. The larger drives become, the greater the statistical chance for data corruption (often 'silent', which means you'll never know it even happened.) I have 2TB of music and movies in storage, and every time I have to migrate that to larger drives I run the risk of corrupting one or more of the files, just through the simple and seemingly-innocent act of copying from one disk to another.
The problem with discussions on ZFS is there are so many new features in this file system that people tend to focus on one part and use that as a blanket justification for supporting it or not. And while it makes sense for Apple to introduce it on machines that are run by more technical-savvy people (namely servers), there are still components of ZFS that will benefit the common user even if they don't understand the technology. Unfortunately, file systems are so foundational to a computer platform that we can't casually implement bits and pieces without considering the whole package. Unless Apple wants to continue to develop HFS to be more ZFS-like (which is ultimately a lost effort), they have to make the leap carefully and with consideration to how it effects every corner of usability and software compatibility.
While ZFS support has made major headway on the Mac since the initial release of Leopard, there's still a long way to go before non-technical users can make real use of its new features. That will likely result in Apple focusing its development efforts to make ZFS practical initially to its server audience. That is reflected in the company's Snow Leopard marketing, which only mentions ZFS features in relation to Snow Leopard Server in "business critical server deployments." More technically-savvy desktop users will likely be able to begin experimenting with the new file system on the Snow Leopard desktop however.
It would be nice for Apple to include ZFS as an option for consumer desktops too. I like to play with a file system or beta software *at my own risk* and I'm sure that I'm not alone. By adding ZFS as an option to format a Mac drive and install Mac OS X, Apple would create an incentive for developpers to update the code of their applications to take advantage of the new features made possible by ZFS.
Assuming that Apple wants ZFS to be the default file system for Mac OS X at some point in the future, it would make sense for Apple to allow customers and developpers to discover the possibilities of ZFS before it is implemented as the default file system for all Macs.
Simple.
Here's how you do it with NTFS
diskpart> create volume stripe disk= (32 disks)
DiskPart successfully created the volume
diskpart> assign letter=s
DiskPart successfully assigned the driver letter or mount point.
diskpart> exit
c:/> format s: /fs:ntfs
The type of the file system is RAW
The new file system is NTFS
WARNING, ALL DATA ON NON-REMOVABLE DISK DRIVES S:
WILL BE LOST!
Proceed with Format (Y/N)? Y
Verifying 840012M
Volume label (32 characters, ENTER for none)? <rtn>
Creating file system structures
Format complete
860172284 KB total disk space
860080288 KB are available
ZFS took 17.5 seconds
Windows Server 2003 and NTFS took 4.5 hours.
It seems the formatting type used is "full" rather than the "quick." Full formatting will always take longer than quick format with NTFS. Quick format of a single hard drive takes roughly 5-10 seconds. If I remember correctly, full format checks each sector for read/write errors. This is why the operation take so long. Quick format assumes that everything is "okay" and just mark the disk as NTFS.
Would the formatting work with the quick parameter with the stripe disk? If so, how long does that take?
Full format: format s: /fs:ntfs
Quick format: format s: /fs:ntfs /q
See the AMAZING VIDEO!!!
http://www.opensolaris.org/os/commun...demos/selfheal
This demo shows how ZFS's self healing features can repair silent data corruption which occurs in mirrored volumes. This is in contrast to traditional volume managers which won't notice that the corruption ever occurred. In this demo, we splat random data over one side of a mirror.
It seems the formatting type used is "full" rather than the "quick." Full formatting will always take longer than quick format with NTFS. Quick format of a single hard drive takes roughly 5-10 seconds. If I remember correctly, full format checks each sector for read/write errors. This is why the operation take so long. Quick format assumes that everything is "okay" and just mark the disk as NTFS.
Would the formatting work with the quick parameter with the stripe disk? If so, how long does that take?
Full format: format s: /fs:ntfs
Quick format: format s: /fs:ntfs /q
http://opensolaris.org/os/community/...s/zfsadmin.pdf
The administration guide for ZFS should answer your questions.
When I read this article I was thinking "AppleTV, Apple Home Server, Apple iTunes Server". ZFS seems to fit in with a home server product and the AppleTV like potatoes and gravy. The question is when will it happen.
I'm waiting for this also. I think it may come sooner than we are expecting. This just makes so much sense to do for a product like a media server.
A question for the more technical folk out there, how does ZFS differ from what the people at Drobo are doing? Is it the same technology?