Disk Fragmentation & OS X: When Does it Become a Problem?

Andrew Bednarz — Tue, 14 Apr 2009 13:49:10 +0000

Disk fragmentation is an old problem that has affected every operating system throughout history. File fragmentation occurs when a single file isn’t located in the same physical location on the disk, but is scattered around. OS X does a great job at minimizing file fragmentation by rewriting files in contiguous space when a file is opened, is under 20MB and contains more than eight fragments. This works quite well to prevent heavy file fragmentation, but what it doesn’t prevent is free space fragmentation.

Free space fragmentation?

Free space fragmentation can become a problem when you start running low on disk space. If you have a small percentage of your disk free — and you frequently use and then free-up disk space with temporary files — you’re likely to have high free space fragmentation. This means the free space is spread all over your hard drive in small amounts here and there — its not all laid out in one contiguous section in one spot.

So what’s the problem with this? Ultimately, this impacts OS X’s ability to keep file fragmentation under control as it can not rewrite large growing files into contiguous empty space, as there is none. Any new large files that are written also get fragmented by the same issue. Just as importantly, when space is fragmented, caching files for the system and for applications also get spread around all over the place. OS X utilizes file caching heavily. The problem is that when the file cache is spread all over the hard disk, it takes a time penalty to access (and also to write) these files. Being able to write and access caches from the same physical area gives a very noticeable speed increase — rebuilding caches after de-fragmenting can thus be advantageous.

How do I tell if my free space is fragmented?

If you suspect you might be suffering from free space fragmentation, there is a free tool you can use to verify your suspicion. Download hsfdebug (the download link is at the end of the page) and copy the file hsfdebug to any directory (such as your home directory, ~/). Open a terminal window, navigate to the directory you copied the file to, and run the following command:

sudo ./hfsdebug -0 | sort -n

This will list all the blocks of free space available, in order from smallest to largest. For example the last 10 lines of output when I ran this was:
10391 0x134e488 0x1350d1e 40.59 MB 10993 0x11004e7 0x1102fd7 42.94 MB 11691 0x152cd74 0x152fb1e 45.67 MB 12229 0x26d1a05 0x26d49c9 47.77 MB 14537 0x2693cdc 0x26975a4 56.79 MB 24090 0x129b388 0x12a11a1 94.10 MB 57704 0x12a9b0c 0x12b7c73 225.41 MB 68490 0x133c5dc 0x134d165 267.54 MB 69680 0x12c82f7 0x12d9326 272.19 MB 237568 0x1d02 0x3bd01 928.00 MB

This tells me that the largest amount of free space I have in one contiguous block is 928 MB. The next biggest is 272.19 MB and its all downhill from there (there were 60,000 lines in total, with 43,000 of them under 100 KB). Since I have a total of 22 GB free disk space, most of that is clearly spread out in tiny blocks. That’s pretty fragmented, which doesn’t surprise me as I deal with a lot of temporary files that are constantly being written and then deleted over a few days.

Of interest, hfsdebug can also tell you the amount of file fragmentation for the curious:
sudo ./hfsdebug -f -t 5

This will take a little while to process, but will tell you the percentage of files that are NOT fragmented:

Out of 1214506 non-zero data forks total, 1209938 (99.624 %) have no fragmentation. Out of 4135 non-zero resource forks total, 4074 (98.525 %) have no fragmentation.

So OS X has been doing a good job at keeping file fragmentation low, as expected.

Why else would I need to defrag?

Another reason to defragment free disk space is to make it possible to create a Boot Camp partition. While OS X can smoothly resize the existing partition and create the new one for Windows, it requires contiguous free space. If this doesn’t exist, you will be told “Can not create partition, please re-install OSX and try again.” This is because Apple does not provide a method to explicitly defrag your hard drive. For the cases it can help (such as creating the Boot Camp partition), Apple recommends just doing a full backup and restore; formatting the hard drive, re-installing the OS and restoring your files. This does indeed do the trick, as it forces every single byte to be rewritten to the disk — however there are a number of third party applications that will do the trick with much less effort.

Defrag Tools

The following applications will provide comprehensive disk defragmenting. I think its important to note that to maximize the benefit of caching performance, a tool such as Onyx should also be used first to clear your System and User caches before you do your defragging. This results in the caches being rebuilt in contiguous space afterwards.

iDefrag — iDefrag is all about defragging and provides a number of methods to do so. For example we could just use the ‘Compact’ algorithm which will put all files at the start of the disk, making our free disk space all at the end with the minimum time spent. iDefrag is £19.95 ($30).

Drive Genius — Drive Genius is a full disk maintenance utility, allowing partitioning and recovery as well as defragmenting. It is claimed that Apple Geniuses at the Genius Bars use this when dealing with problematic hardware. Drive Genius costs $99.

TechTool Pro 5 — TechTool Pro 5 claims to be a complete Mac problem-solving system, that includes disk defragmenting. A cut-down version (TechTool Deluxe) is included with Apple’s Apple Care product, but this does not contain the defragmenting functionality. The Pro version costs $98.

Results?

After a full defrag with iDefrag, re-running hfsdebug gave me an output of only 123 lines, with the following being the last 10:
370 0x1883fd 0x18856e 1.45 MB 377 0x232da2e 0x232dba6 1.47 MB 445 0x232efb1 0x232f16d 1.74 MB 479 0x1670b2 0x167290 1.87 MB 768 0x1d02 0x2001 3.00 MB 8343 0x232abfa 0x232cc90 32.59 MB 63659 0x78756 0x88000 248.67 MB 184298 0x3a202 0x671eb 719.91 MB 1183949 0x881437 0x9a2503 4.52 GB 5014722 0x232fb3d 0x27f7ffe 19.13 GB

So the bulk of my free space is now nicely contiguous and large. My caches are rebuilt and my system is noticeably snappier and more responsive. Milage may vary, depending on exactly how the fragmentation is affecting you to begin with, but free space defragmentation can definitely improve performance in some cases. Importantly, be sure to always have a backup on hand, as any power glitch during any long disk operation may cause damage your file system.