ZFS Shows Up in New Leopard Build

Udo Schmitz writes "As a follow-up to rumours from May this year, World of Apple has a screenshot showing Sun's Zettabyte File System in "the most recent Build of Mac OS X 10.5 Leopard". Though I still wonder: If it is not meant to replace HFS+, could there be any other reasons to support ZFS?"

Exciting!

Now that Vista is finalized, expect Apple to show more and more of the 'secret' features of leopard!

--jeffk++

Otherwise...

Now that Vista is finalized,

Because if Apple showed them before, there was a risk that Microsoft tried to announce them as future features in their soon-to-be-released perfect Windows Vista ?

Re:Otherwise...

Nah, Microsoft would never do anything like that... They are a respectable company that does not misrepresent their products.

--jeffk++

copy-on-write

FTA:

Makes use of copy-on-write; rather than overwriting old data with new data, it writes new data to a new location and then overwrites the pointer to the old data

Wouldn't that pose a problem for mmap?

Re:copy-on-write

No. Mmap lives above the filesystem layer. Unless you are doing mmap on the block device, in which case you should realise that not everyone works for oracle...

Mmap simple maps pages of a disk file into memory. If the disk file changes its physical location then the mapping is updated. When you call mmap, you give it a disk file, an offset, and an extent. It is up to the VFS layer to translate this into physical mappings. LFS has the same issues, and these were solved well over a decade ago.

If you invoke mmap with MAP_PRIVATE, this actually makes it easier; if someone else updates the file then you just keep the existing mapping.

Re:copy-on-write

FTA:

Makes use of copy-on-write; rather than overwriting old data with new data, it writes new data to a new location and then overwrites the pointer to the old data

Wouldn't that pose a problem for mmap?

It may do, but like many things there are alternative approaches.

From working on embedded hardware with flash memory, this makes me wonder whether possible addition of ZFS is meant to be for flash storage? Let me explain: flash memory has a fairly limited write-count, relative to hard disks, so to compensate for this memory is written in a circular fashion, to ensure that a given sector is written the least often possible. In addition to this, from what I can tell, Apple's main sales point are low profile computers and portables. The latter would benefit from flash storage as means of extending battery life, even if it is for a certain elements, such as for the OS which is accesed far more frequently than anything else on disk. Given this I wouldn't be surprised to see flash memory in future models of Apple portables, using ZFS, while HFS+ is still used for the hard disks.

This is pure speculation, but I feel that it has a high probabilty of being near the mark.

Reasons to support? Servers

I will be soon converting my Linux server to Solaris just for ZFS. Although ZFS may not terribly useful on a normal desktop, on a server, it's very powerful.... The idea of parity data actually being used actively to ensure data isn't corrupted is brilliant imho. So is the idea of on-the-fly recovery (I remember a video of some guy writing 30 megs of junk to a partition using dd, ZFS detecting it, and repairing it). *ends rant since all this can be read up about online*

Not a likely replacement...

If it is not meant to replace HFS+, could there be any other reasons to support ZFS?"

Well, OSX 10.4 already supports FAT16, FAT32, and HFS, HFS+ (case sensitive and case-insensitive) and UFS. I don't see any obvuious conclusion that HFS+ is on the way out. Now if the OSX kernel AND os both support a ZFS-formatted partition as a boot partition, we might find it as an accceptable replacement, but otherwise I would think ZFS will be added for large enterprise SAN RAIDs and such.

ZFS is overkill for a laptop - for now

A few years ago, I sat down and worked out exactly what I thought a filesystem should do, and how I would implement it. At the time no filesystem came close. Then Sun released ZFS. Real documentation on it is hard to find (behind the marketing hype), but when I did track it down I discovered two things:

1. They had implemented everything I thought they should, and
2. That only accounted for about 40% of the features of ZFS.

Calling it the last word in filesystems might be hyperbole, but I expect ZFS to last a good 10-20 years, which is quite respectable for a filesystem, and I wouldn't be surprised if it lasted longer. Is it a replacement for HFS+? Not yet.

HFS+ is a very nice filesystem for single user systems with a single disk. It implements journalling, has reasonable performance, and has good metadata support. For the average users at the moment, the only real advantage of ZFS would be snapshots, and these are not too difficult to implement for other filesystems.

ZFS, however, is much better when you have multiple physical disks. At the moment, only the top-end Macs have more than one disk. This is likely to change in two ways:

1. Cheap flash,
2. Network storage

For a home user, ZFS could handle backups trivially by plugging in a large flash drive and adding it to the pool. I suspect this will be one mechanism Time Machine will use. Due to the way ZFS works, you can just mirror a part of the directory tree (e.g. /Users/aUser) onto the external disk. With a big external drive, you could mirror the entire disk onto it and also save snapshots (another Time Machine feature...). The same could be done with network storage. With the current price of hard drives, I wouldn't be surprised if .Mac started offering 10-20GB of storage space for remote backups using this mechanism (take a look at the NFS4 integration in ZFS to see how this could be done).

ZFS is not needed as a replacement for HFS+ in 2007, but it probably will be in 2008-9. ZFS is a 128-bit filesystem, which means it is designed to last for a long time. We will probably never need a 128-bit filesystem (unless we actually want to build hard drives the size of planets with single-atom sectors), but we will need a 65-bit filesystem once we get to around 10 Exabytes. This won't happen with single drives for a while, but it will with RAID arrays.

Re:Storeage size - speak for yourself

"That's no moon -- that's a file server!"

ZFS vs HFS vs NTFS?

The tech behind ZFS [sun.com] at least sounds very impressive, but I have to wonder how useful it is for workstation drives.

I've never found plain-Jane posix permissions to be all that useful on anything other than the most basic of server environments.

HFS has going for it all the fun stuff we've come to love apple for, such as transparent file customization like icons, labels, meta data, and whatnot through resource forks. I assume that these can be made to work with ZFS by making hidden files.

What I'd really like to see is both that kind of functionality along with NTFS's really excellent ACL permission system implemented. ACL permissions are a godsend for people responsible for running a file store that's used by humans as opposed to automated processes. NTFS also has a great deal of capacity for meta-data, although not to the same level as HFS.

NTFS is one of the few worthwhile things that's ever come out of Redmond. I wish more people would spend a bit learning from it without throwing it away simply because it's MS bloat.

Re:ZFS vs HFS vs NTFS?

NTFS is one of the few worthwhile things that's ever come out of Redmond. I wish more people would spend a bit learning from it without throwing it away simply because it's MS bloat.

I wish MS would let us. NTFS is worthless if you don't run Windows. And it hinders interoperability with other systems because its implementation and disk layout is secret/patented.

Why, do you think, there is no stable implementation that can write NTFS volumes (other than the MS implementation)?

Contrast this with ZFS which is released under an open source license.

Re:ZFS vs HFS vs NTFS?

NTFS is one of the few worthwhile things that's ever come out of Redmond. I wish more people would spend a bit learning from it without throwing it away simply because it's MS bloat.

I think the negative opinion some people (including me) have of NTFS come not directly because it is from MS, but come from the incompatibility with everything else. I can't (reliably) read/write to it from a Mac, Linux, or Sun. That leaves only people totally in the MS camp able to use it. It may have some nice technical features, but I can't ever see them, so it is a little hard to be impressed or care about them too much.

Re:ZFS vs HFS vs NTFS?

I've never found plain-Jane posix permissions to be all that useful on anything other than the most basic of server environments. ...
What I'd really like to see is both that kind of functionality along with NTFS's really excellent ACL permission system implemented.

I wish you could read more about ZFS before suggesting how you could improve it by adding ACLs. It already supports them!

http://blogs.sun.com/marks/entry/zfs_acls [sun.com]

Re:ZFS vs HFS vs NTFS?

HFS has going for it all the fun stuff we've come to love apple for, such as transparent file customization like icons, labels, meta data, and whatnot through resource forks. I assume that these can be made to work with ZFS by making hidden files.

You assume correctly, since most of that business is taken care of with Bundles. This is why it more or less works on UFS, which is already supported on Mac OS X, and has been for years. Forks & whatnot are really a legacy idea.

What I'd really like to see is both that kind of functionality along with NTFS's really excellent ACL permission system implemented.

That's funny! The HFS+ ACL system is Microsoft's ACL system, much to the chagrin of the Unix community.

:w

It's to support Time Machine

See this Ars Technica article [arstechnica.com] where John Siracusa said back in August:

"For Mac geeks of a certain persuasion, the first mention of a soon-to-be-revealed feature of Leopard during the WWDC keynote set off a mental chain-reaction. That feature was Time Machine, and the name alone was enough to cause one particular phrase to hammer in the mind of many people, including me: "New file system in Leopard!" It was even a bingo square. In fact, it was my personal favorite bingo square, and the one that I most looked forward to marking.

But let's back up a bit. Why should the mere name "Time Machine" scream "new file system" to anyone? And why the excitement about a new file system in the first place? What's wrong with HFS+, Mac OS X's current file system? It's got journaling. It supports arbitrarily extensible metadata. It can even be case-sensitive to satisfy the Unix geeks. Does Mac OS X really need a new file system?

In a word, yes. HFS was a state-of-the-art personal computer file system when it was first released...twenty-one years ago. HFS+ is only eight years old, but it's built on many of the design decisions of HFS. Progress marches on. Today, there are new capabilities that the best modern file systems have, but that HFS+, even with all of its recent additions, does not. Here's a short list.

• Efficient storage and handling of very small files.
• Logical volume management through a pooled storage model.
• Improved data integrity using checksums on all data.
• Snapshots.

So it's about the snapshot ability of ZFS, and that's exactly what will be needed for Time Machine [apple.com].

There's a LOT more to ZFS than snapshots...

Over past months, I've read a lot of people commenting on ZFS who have no idea what it is. What it is, is the next generation of filesystems, not a "tweak" of current fs technology. It just happens to "look like" an ordinary POSIX fs, from a distance (if you ignore the administration/pool stuff...) But inside, it's something new under the Sun, folks.

RAID experts don't grok it, because it does things RAID can't do (end-to-end).

Devotees of ext2fs, reiserfs (yay!), NTFS (LOL!), or HFS+ don't grok it, because none of those filesystems do what ZFS does.

Read about it before you write it off as old wine in a new bottle. To ask the question, "Does OS X need a new filesystem?" is a perfect example of missing the point. Once you've looked at what ZFS really brings to the table, you'll see why it's an inevitable future, sooner or later, and you'll stop looking foolish.

Some links I posted this week: [google.com]

- http://www.osnews.com/story.php/16739/Screenshot-Z FS-in-Leopard [osnews.com] - http://mac4ever.com/news/27485/zettabyte_sur_leopa rd/ [mac4ever.com] (older rumour http://www.osnews.com/story.php?news_id=14473 [osnews.com])

For OS X people wondering why the fuss about ZFS - summaries include: - http://www.sun.com/2004-0914/feature/ [sun.com] - http://www.sun.com/bigadmin/features/articles/zfs_ part1.scalable.html [sun.com]

"Why ZFS for home": - http://uadmin.blogspot.com/2006/05/why-zfs-for-hom e.html [blogspot.com]

"Here are ten reasons why you'll want to reformat all of your systems and use ZFS.": http://www.tech-recipes.com/rx/1446/zfs_ten_reason s_to_reformat_your_ [tech-recipes.com]...

And some more technical explanations from Chief Engineer: - http://blogs.sun.com/bonwick/entry/zfs_end_to_end_ data [sun.com] - http://blogs.sun.com/bonwick/entry/smokin_mirrors [sun.com]

Send/receive useful for .mac sorts of features

I can imagine that using zfs send/receive to export/import pools would be an extremely efficient/safe method of replicating data. Perhaps some sort of ".mac mirror" could work. This would make Time Machine exceptionally useful, and I'd definitely commit extra $ for .mac services (if reasonable) for this.

Re:Just to get it out of the way...

Since we're nitpicking:

The letter "Z" is properly pronounced "Zee" in the USA and Iraq (after 2003)

That would correctly read "The letter "Z" is improperly pronounced "Zee" in the USA and Iraq (after 2003)"

Re:Just to get it out of the way...

Zed's dead, baby.

Re:Just to get it out of the way...

Actually 'Zed' is probably closer to the source from which it comes - the Greek letter 'zeta'

... and "Bed" is closer than "Bee" to "Beta", yet everyone says "Bee". At least the American pronunciation of the alphabet is internally consistent. ;)

Re:What a moron

It's called compatibility.

Wouldn't full NTFS support (or well, support for any FS more in use then ZFS today) make more sense?

Re:What a moron

Wouldn't full NTFS support (or well, support for any FS more in use then ZFS today) make more sense?

Yeah, I mean it's not like NTFS is defined and controlled by an organization renowned for its hostility to other platforms, reluctant to document things in a way that other people can implement them, and scared of interoperability, is it?

Re:What a moron

You both miss the point of HFS+ and ZFS. In Solaris ZFS has not replaced UFS. ZFS is an elegant way to manage large amounts of storage tied together with inexpensive and simple SATA drives. If you have one disk in your Mac, ZFS probably will not be your choice. HFS+ will work very well and be very easy to manage. A file server with 3 or 4 750 GB drives however, might be cut up so that part of the storage is mirrored for safety, limited for certain uses, and spanned over drives for size. For example, 3 750's could be divided into 1 TB unmirrored storage, 250 GB mirrored, a temp area of up to 100GB and the rest (650+ GB depending on temp are usage) held in reserve. In addition ZFS does quite a bit of error checking on the data to avoid any possible corruption during reads. However, it will never replace HFS+ on an iMac for your average user.

...so how does one define "capacity" therein?

I've been wandering about this and am insanely curious: if ZFS really does intelligently copy on write how far can it take it?

for starters, does the FS "know" that i've just clicked "Save As" in my word processor? what about copy and pasting a file back into the same directory to make a local copy? Also? is it just within variations on the same file? if i have a particular setup exe on my system but forget, and download it again to the desktop surely the FS has no initial way of knowing that they are one and the same, does some funky heuristic happen?

basically: does the OS's read/write/copy/delete functionality have to invoke copy-on-write via a FS API or is it built in for every single sector-sized chunk that gets stuffed into the FS?

the next question is the one in my subject: how therefore do you define "capacity"? if i've got a bunch of files that take up 700mb on a ZFS device and try to back up to a (Joliet) CD will i get a message telling me that the CD doesnt have room? i can imagine this scenario being unlikely with optimised binary data (jpegs and mpegs) but if i'm backing up a dev environment with autobackups (main.c,main.c.bak.001,main.c.bak.002,etc.) and manually created and dated directory tree "snapshots" (dev,dev_backup_2006-12-18,dev_backup_2006-12-01,e tc.) then this could probably happen quite easily.