Measuring Fragmentation in HFS+ 417
keyblob8K writes "Amit Singh takes a look at fragmentation in HFS+. The author provides numbers from his experiments on several HFS+ disks, and more interestingly he also provides the program he developed for this purpose. From his own limited testing, Apple's filesystem seems pretty solid in the fragmentation avoidance department. I gave hfsdebug a whirl on my 8-month-old iMac and the disk seems to be in good shape. I don't have much idea about ext2/3 or reiser, but I know that my NTFS disks are way more fragmented than this after similar amount of use."
File allocation Table (Score:4, Interesting)
NTFS is not so bad (Score:5, Interesting)
HFS Filesystem vs. ReiserFS (Score:2, Interesting)
Under HFS+ in Mac OS X Jaguar or Panther, after about a day of having a clean install, fresh partition and format my hard drive starts making clunking noises and the system locks up (without actually freezing) -- then when reboot attempts are made they take aeons.
Under ReiserFS in Gentoo Linux for PPC: never have the problem. Same hard drive. Months of use, never once hear the hard drive being funky. No lockups.
Do I put the blame on HFS? OS X? I just can't figure out this strange problem.
Re:HFS+ defrag source (Score:3, Interesting)
Re:Anonymous (Score:5, Interesting)
Re:HFS Filesystem vs. ReiserFS (Score:2, Interesting)
Seriously - it's likely that gentoo just isn't using the particular sector on the drive that OSX is - perhaps there is a file there that doesn't get accessed regularly or something. In any case, Clunk-Clunk is never ok.
Re:HFS+ defrag source (Score:5, Interesting)
HFS+ has been around since OS 8.5 (?? somewhere in OS 8). So either this is a feature of HFS+ that hasn't been implemented until now, or its a bit of code added to Panther. Or has HFS+ been updated?
Re:HFS+ defrag source (Score:0, Interesting)
Re:HFS+ defrag source (Score:5, Interesting)
You've only defeated the purpose if you re-fragment the file again after opening it. If this isn't the case, the amortized cost (the initial cost of de-fragmentation when opening the first time minus the speed benefits from a file in a single chunk) over the many times the file is read yields a speed bonus, not a speed loss.
A good example is me, installing a program from disk onto my computer. I run the program and it accesses a group of files that have been fragmented when copied to my hard drive. The first time it opens the files it spends a little extra time de-fragmenting them. However, subsequent times that I open the program, these files will load faster.
Re:Anonymous (Score:2, Interesting)
Panther Defrag (Score:5, Interesting)
HPFS (Score:3, Interesting)
But not sure how this are managed in linux filesystems, not just ext2/3 and reiserfs, but also in xfs and jfs.
ReiserFS and fragmentation (Score:2, Interesting)
It's really good on filesystems with a lot of files or on databases.
Who even cares about Fragmentation anymore? (Score:5, Interesting)
Both have 40gig HD's and both have applications installed/uninstalled quite often. My PC feels the worst of this as he gets games installed and uninstalled in addition to the apps.
For example the last time I reinstalled either of these machines was back in january(new year fresh install) and since then my pc has felt the install/uninstal of various games usually ranging from 2-5 gigs each. The Apple has been installed and with the exception of updates, plugins, video codecs and basic small apps that get added/upgraded often has done alright.
Right now Norton System Works on my PC is saying the drive is 4% fragmented. Disk Warrior on my Apple is saying the drive is 2% fragmented.
Conclusion: Fragmentation is no longer an issue for the HOME USER(note how i'm not saying your companies network doesn't need to be concerned), unless there still running a FAT32 partition >. which well they deserve to have there computer explode at that point anyway.
Re:NTFS is not so bad (Score:5, Interesting)
Re:Measuring fragmentation in NTFS (Score:5, Interesting)
C: 5,72 GB Total, 1,97 GB (34%) Free, 4% Fragmented (8% file fragmentation)
D: 40,00 GB Total, 1,00 GB (2%) Free, 41% Fragmented (82% file fragmentation)
E: 66,69 GB Total, 105 MB (0%) Free, 10% Fragmented (21% file fragmentation)
F: 30,00 GB Total, 1,21 GB (4%) Free, 3% Fragmented (7% file fragmentation)
G: 10,00 GB Total, 1,54 GB (15%) Free, 5% Fragmented (9% file fragmentation)
H: 35,03 GB Total, 551 MB (1%) Free, 39% Fragmented (79% file fragmentation)
D ("Dump") and H ("Online") get a lot of throughput, by personal computing standards anyway, E ("Games") doesn't get changed that much, but if it does, a lot of data leaves and comes. Seems like whenever I defrag D or H, they're back to the values above within days. I guess Win XP has a hard time doing the internal on-the-fly defragging of the hard drives that rarely have moer than 1% free space... Guess I should just get a new HD and have some more free space that way - but I bet I'd have that filled up with junk after some weeks, anyway.
That said, I'm not sure how relevant this is for NTFS partitions, anyway. I recall hearing that they aren't affected by fragmentation as much as FAT partitions (which were a nightmare), however I'm not sure if that means they don't fragment that easily (heh) or whether accessing data isn't slowed down as much by any existing fragmentation.
I've also rarely heard anyone talking about fragmentation in the popular Linux file systems, a Unix partisan I know actually thought they didn't fragment full stop, which I don't believe is possible, at least not if you consider situations which might not occur in practice. But then again, I suppose Linux might solve it the same way Apple seems to - I guess I'll know more after a couple of hundred comments on this article.
Defrag = placebo? (Score:3, Interesting)
Re:My stats (Score:1, Interesting)
Re:File allocation Table (Score:2, Interesting)
Re:Defrag = placebo? (Score:5, Interesting)
As a cute side note, I remember having to explain fragmentation to my high school FORTRAN class and teacher back in the '80's. I'd changed schools in my senior year and the new school had just started using the Apple II FORTRAN environment, which happened to be the same as the Aple II Pascal environment that I'd used at the previous school. The file system was incapable of slapping files into whatever blocks happened to be available (I'm not even sure it used blocks. Probably not...) so you would not be able to save your files if it was too fragmented, even if there was enough space available to do so. Ah those were the days...
Defragging XP now... (Score:3, Interesting)
20 minutes later, and it's on 17%. That's pretty damn fragmented, in my opinion.
Re:ReiserFS and fragmentation (Score:1, Interesting)
It's a crock and evenyone knows it. It was only put into the kernel to make him stop his whining. Only a fool uses reiserfs for a thing other than a freak show. A fool and his data are soon parted.
my stats (Score:3, Interesting)
Not bad. That's 8 months of heavy use since my last format.
I gotta bring this to work today and see what that machine's like. My co-worker has been complaining that he doesn't have a defrag utility since he got OSX. I've been telling him that I don't think it matters. Now I can prove it to him.
I remember back in the days of my Powermac 8100/80av, we would leave the 2 800mb drives defragging over the weekend because they had like 75% fragmentation.
Re:Defrag = placebo? - yes and no (Score:4, Interesting)
however, defraging is not the same for every defrag utility. For example, I was working with Avid Audiovision about 5-6 years ago on a TV show, it seems that defraging a drive hosting files created or edited with Audiovision with Speed Disk by Symantec would actually corrupt the entire projects contained on the drive (the biggest mistake and the only serious one I had in my career, I didn't loose my job but my boss did loose his temper, live and learn!), audio file were not readable at all after, it was actually a documented bug of Audiovision and I even think it was affecting every OMF files not just the ones used by Audiovision (not sure about this though), thats what happens when your boss won't let you RTFM. Only Disk Express, some Avid defrager or, later, Techtool could defrag those drives.
On a side note, in the Classic mac (7-9.2), defragmenting your drive was also a way to prevent data corruption, actually its the other way around, not defraging would lead to data corruption. I don't know if its also the case with NTFS, EXT2 et al.
portable fragmenter (Score:3, Interesting)
Result: you have a bunch of large files, all very fragmented, and the free space is very fragmented.
Re:Huh? (Score:5, Interesting)
For the record I also use XP on my laptop. Until everything works perfectly out of the box, ACPI and all, I'm not installing any nix on it.
Microsoft really pisses me off (Score:5, Interesting)
That would be well and good if the problem were otherwise insurmountable. But, it turns out, we've known how to minimize, if not entirely eliminate, filesystem fragmentation for twenty years now - since the introduction of the BSD Fast File System.
It doesn't take expensive (in time, if not in money) tools. All it takes is a moderately clever block allocation algorithm - one that tries to allocate a block close in seek time to the previous one, rather than just picking one at random.
The fundamental insight that the authors of FFS had was that while there may only be one "optimal" block to pick for the next one in a file, there are tens of blocks that are "almost optimal" and hundreds that are "pretty darn good." This is because a filesystem is not a long linear row of storage bins, one after another, as it is treated by many simplistic filesystems. The bins are stacked on top of each other, and beside each other. While the bin right next to you might be "best", the one right next to that, or in another row beside the one you're on, or in another row above or below, is almost as good.
The BSD folk decided to group nearby bins into collections and try to allocate from within collections. This organization is known as "cylinder groups" because of the appearance of the group on the disk as a cylinder. Free blocks are managed within cylinder groups rather than across the whole disk.
It's a trivial concept, but very effective; fragmentation related delays on FFS systems are typically within 10% of optimum.
This kind of effectiveness is, unfortunately, difficult to achieve when the geometry of the disk is unknown -- and with many modern disk systems the actual disk geometry is falsely reported (usually to work around limits or bugs in older controller software). There has been some research into auto-detecting geometry but an acceptable alternative is to simply group some number of adjacent blocks into an allocation cluster. In any case, many modern filesystems do something like this to minimize fragmentation-related latency.
The gist of this is that Microsoft could have dramatically reduced the tendency towards fragmentation of any or all of their filesystems by doing nothing else but dropping in an improved block allocator, and done so with 100% backward compatibility (since there is no change to the on-disk format).
Maybe it was reasonable for them to not bother to so extravagantly waste a few days of their developers' time with MS-DOS and FAT, seeing as they only milked that without significant improvement for eight or nine years, but it's hard to explain the omission when it came to Windows NT. NTFS is a derivative of HPFS which is a derivative of FFS. They had to have known about cylinder group optimizations.
So the fact that, in 2004, we're still seeing problems with filesystem fragmentation absolutely pisses me off. There's no reason for it, and Microsoft in particular ought to be ashamed of themselves. It's ridiculous that I have to go and degragment my WinXP box every few months (which takes like 18 hours) when the FreeBSD box in the basement continues to run like a well-oiled machine despite the fact that it works with small files 24 hours a day, 365 days a year.
Hey Microsoft: You guys have like fifty billion bucks in the bank (well, ok, 46 or 47 billion after all the antitrust suits) and yet you can't even duplicate the efforts of some hippy Berkeleyite some twenty years after the fact? What's up with that?
(I mean "hippy Berkeleyite" in an affectionate way, Kirk. :-)
Re:HFS+ defrag source (Score:3, Interesting)
You can also use UFS.
fragmentation delays are always within 10% of opt (Score:1, Interesting)
And fragmentation is not a bugaboo. It's a fact. When you have random allocation on a volume, it will get fragmented. You can go back after the fact and unfragment it, but doing so in any serious fashion when writing files actually degrades performance due to the extra effort required.
Also note NTFS is not a derivative of HPFS. I cannot speak as to whether HPFS is a derivative of FFS.
HFS+ does nothing to prevent fragmentation except for use super-clusters. That is, when you extend a file, you get more than one block at at time. When you close the file, some of those blocks might be scavenged, or might not. NTFS could do this, but I believe they do not. However, NTFS on servers has an allocation block size of 8K, which isn't far off of the super-cluster size on HFS+ which I believe is 16K.
Finally, note that on modern drives, you can seek all the way across the disk in only about 30-50% more time than you can seek a short seek. Thus keeping your blocks close to the current cylinder but not in it has very limited value. Note that this is not the case on optical (CD/DVD) disks.
Re:Offtopic (Score:3, Interesting)
If we take the extremely generous assumption that foreign terrorists attack the US every 8 years, for you to make an even remotely reality based assessment of our relative security since 9/11 I'd say you need to wait at least 12 years without any attacks.
Granted, the grandparent would have difficulty proving his assertion that we are in much greater danger now, but I'm sure it could be proven that we have many more enemies. That's not exactly comforting.
Re:How to determine fragmentation... (Score:3, Interesting)
The files on the drive had an average size of 200 MB, were downloaded in 1kB increments several files at a time over a period of a week on average per file.
The reason for it failing on defraging (it doesn't say it fails, it just doesn't do much and stops after a while) is because the free space was also so badly fragmented that it couldn't even defragment a single file (it expects there is free space in a single chunk large enough to defragment one file... which is hard since the files are all quite large). The worst fragmented files had all well over 5000 fragments, and none less than 100 orso.
Even after making more than 50% space free on the drive, it was still suffering from fragmented free space and refused to defrag. After a while I decided it was far easier to just copy the whole drive (30 GB) and copy it back... writing the 30 GB back was about 20-30 times faster compared to reading the drive for the initial copy...
On a side note, I've put the exact same stress on a ext3fs disc, and it also fragmented very badly, despite claims that ext3fs doesn't need defragmentation "because it won't fragment your files much". If free space was also very fragmented, I cannot say, but I assume it was as deleting a very fragmented file would result in a very fragmented area of free space.
--Swilver