40 Comments

Summary:

Disk fragmentation is an old problem that has affected every operating system throughout history. File fragmentation occurs when a single file isn’t located in the same physical location on the disk, but is scattered around. OS X does a great job at minimizing file fragmentation by […]

idefrag_iconDisk fragmentation is an old problem that has affected every operating system throughout history. File fragmentation occurs when a single file isn’t located in the same physical location on the disk, but is scattered around. OS X does a great job at minimizing file fragmentation by rewriting files in contiguous space when a file is opened, is under 20MB and contains more than eight fragments. This works quite well to prevent heavy file fragmentation, but what it doesn’t prevent is free space fragmentation.

Free space fragmentation?

Free space fragmentation can become a problem when you start running low on disk space. If you have a small percentage of your disk free — and you frequently use and then free-up disk space with temporary files — you’re likely to have high free space fragmentation. This means the free space is spread all over your hard drive in small amounts here and there — its not all laid out in one contiguous section in one spot.

So what’s the problem with this? Ultimately, this impacts OS X’s ability to keep file fragmentation under control as it can not rewrite large growing files into contiguous empty space, as there is none. Any new large files that are written also get fragmented by the same issue. Just as importantly, when space is fragmented, caching files for the system and for applications also get spread around all over the place. OS X utilizes file caching heavily. The problem is that when the file cache is spread all over the hard disk, it takes a time penalty to access (and also to write) these files. Being able to write and access caches from the same physical area gives a very noticeable speed increase — rebuilding caches after de-fragmenting can thus be advantageous.

How do I tell if my free space is fragmented?

If you suspect you might be suffering from free space fragmentation, there is a free tool you can use to verify your suspicion. Download hsfdebug (the download link is at the end of the page) and copy the file hsfdebug to any directory (such as your home directory, ~/). Open a terminal window, navigate to the directory you copied the file to, and run the following command:

sudo ./hfsdebug -0 | sort -n

This will list all the blocks of free space available, in order from smallest to largest. For example the last 10 lines of output when I ran this was:

10391 0x134e488 0x1350d1e 40.59 MB
10993 0x11004e7 0x1102fd7 42.94 MB
11691 0x152cd74 0x152fb1e 45.67 MB
12229 0x26d1a05 0x26d49c9 47.77 MB
14537 0x2693cdc 0x26975a4 56.79 MB
24090 0x129b388 0x12a11a1 94.10 MB
57704 0x12a9b0c 0x12b7c73 225.41 MB
68490 0x133c5dc 0x134d165 267.54 MB
69680 0x12c82f7 0x12d9326 272.19 MB
237568 0x1d02 0x3bd01 928.00 MB

This tells me that the largest amount of free space I have in one contiguous block is 928 MB. The next biggest is 272.19 MB and its all downhill from there (there were 60,000 lines in total, with 43,000 of them under 100 KB). Since I have a total of 22 GB free disk space, most of that is clearly spread out in tiny blocks. That’s pretty fragmented, which doesn’t surprise me as I deal with a lot of temporary files that are constantly being written and then deleted over a few days.

Of interest, hfsdebug can also tell you the amount of file fragmentation for the curious:
sudo ./hfsdebug -f -t 5

This will take a little while to process, but will tell you the percentage of files that are NOT fragmented:

Out of 1214506 non-zero data forks total, 1209938 (99.624 %) have no fragmentation.
Out of 4135 non-zero resource forks total, 4074 (98.525 %) have no fragmentation.

So OS X has been doing a good job at keeping file fragmentation low, as expected.

Why else would I need to defrag?

Another reason to defragment free disk space is to make it possible to create a Boot Camp partition. While OS X can smoothly resize the existing partition and create the new one for Windows, it requires contiguous free space. If this doesn’t exist, you will be told “Can not create partition, please re-install OSX and try again.” This is because Apple does not provide a method to explicitly defrag your hard drive. For the cases it can help (such as creating the Boot Camp partition), Apple recommends just doing a full backup and restore; formatting the hard drive, re-installing the OS and restoring your files. This does indeed do the trick, as it forces every single byte to be rewritten to the disk — however there are a number of third party applications that will do the trick with much less effort.

Defrag Tools

The following applications will provide comprehensive disk defragmenting. I think its important to note that to maximize the benefit of caching performance, a tool such as Onyx should also be used first to clear your System and User caches before you do your defragging. This results in the caches being rebuilt in contiguous space afterwards.

iDefrag — iDefrag is all about defragging and provides a number of methods to do so. For example we could just use the ‘Compact’ algorithm which will put all files at the start of the disk, making our free disk space all at the end with the minimum time spent. iDefrag is £19.95 ($30).

Drive Genius — Drive Genius is a full disk maintenance utility, allowing partitioning and recovery as well as defragmenting. It is claimed that Apple Geniuses at the Genius Bars use this when dealing with problematic hardware. Drive Genius costs $99.

TechTool Pro 5 — TechTool Pro 5 claims to be a complete Mac problem-solving system, that includes disk defragmenting. A cut-down version (TechTool Deluxe) is included with Apple’s Apple Care product, but this does not contain the defragmenting functionality. The Pro version costs $98.

Results?

After a full defrag with iDefrag, re-running hfsdebug gave me an output of only 123 lines, with the following being the last 10:

370 0x1883fd 0x18856e 1.45 MB
377 0x232da2e 0x232dba6 1.47 MB
445 0x232efb1 0x232f16d 1.74 MB
479 0x1670b2 0x167290 1.87 MB
768 0x1d02 0x2001 3.00 MB
8343 0x232abfa 0x232cc90 32.59 MB
63659 0x78756 0x88000 248.67 MB
184298 0x3a202 0x671eb 719.91 MB
1183949 0x881437 0x9a2503 4.52 GB
5014722 0x232fb3d 0x27f7ffe 19.13 GB

So the bulk of my free space is now nicely contiguous and large. My caches are rebuilt and my system is noticeably snappier and more responsive. Milage may vary, depending on exactly how the fragmentation is affecting you to begin with, but free space defragmentation can definitely improve performance in some cases. Importantly, be sure to always have a backup on hand, as any power glitch during any long disk operation may cause damage your file system.

Related research

Subscriber Content

Subscriber content comes from Gigaom Research, bridging the gap between breaking news and long-tail research. Visit any of our reports to learn more and subscribe.

You're subscribed! If you like, you can update your settings

Related stories

  1. BE CAREFUL

    I ran defrag on a drive and it rendered it completely useless. Disk utility could not repair. Had to revert to a cloned copy.

    Share
  2. ZEDenterprizes Tuesday, April 14, 2009

    Being generally fairly cheap (a.k.a. “frugal” ;-)), I was wondering if anyone’s come across any free utilities along the defrag line. I know there were always tons for Windows based systems….probably because it’s been more of a problem for them…

    Share
  3. I have never run into a situation in which I had to degrag my hard drive with OsX. There was one occasion when I experienced a slow down in my system and realize that I had less than 10% of space left in my hard drive. My solution to the problem was very simple. I just back up a lot of my large files (movies mostly) onto an external hard drive to free up the space and the problem corrected itself by magic. This is the beauty of OsX and the realization that it is something window users have to deal with in their normal computing life.

    Share
    1. OSX does get fragmented. Windows more so but most users dont ever have to defrag as vista does it for them. Even if it did not most users would only have to do it every few months to improve disk performance. I have a mac and tried to like the os but could not.

      Share
  4. This was very interesting, that “hfsdebug” tool is useful and easy to use. Unfortunately, it dosn’t fix anything, it just tells you what’s going on inside your Mac. Isn’t there any free defragmentation software out there? If I have to spend a thousand dollars to get it fixed, then I’ll keep my drive slow and dirty. ;)

    Share
  5. I used Drive Genius in the past. However, I hear nightmares of people using this and other products – rendering their drives unusable – so I am avoiding using this or other products again.

    Also, you mention your software hsfdebug link is at the bottom of the article. It isn’t there. If I press work hsfdebug in the middle of this article, the link is broken

    http://www.osxbook.com/software/hfsdebug/

    Credibility of the article here is low.

    Share
  6. @anon: Did you even bother to read the article? There is a link to hfsdebug in the 4th paragraph and we said the download is at the end of that page.

    Share
  7. [...] roundup: Palm Pre release date (jkOneTheRun) HD Cloud: Video trandscoding gets hip (NewTeeVee) When does disk fragmentation on OS X become a problem? (TheAppleBlog) The road to mass market electric vehicles in China (Earth2Tech) [...]

    Share
  8. You can use the standard Disk Utility to eliminate a lot of fragmentation. It works best if you have a lot of free space on a volume.

    Just run Disk Utility, click on your boot drive (not the volume) and then click on the Partition tab. You can then resize your partition on the fly. Make the partition smaller and Disk Utility will move files to accommodate the smaller partition. Keep making the partition smaller until you get an error then start trying smaller changes. I was able to defrag 130 GB on my 320 GB drive this way.

    Using Leopard, I started by reducing the partition by 30 GB and then I kept reducing by 20 GB until I got down to 14 GB free on my boot volume. I stopped then. According to hfsdebug my largest defragged block is 129 GB in size and my disk is 99.949 % fragmentation free. Good enough and free.

    Share
  9. The ‘cheap’ solution would be to use the freeware tool (sorry, forgot its name) to clone your drive. Now reformat your drive. Reinstall everything. It might sound like it will take a long time, but so does defragmenting. And its probably safer to do it this way too.

    Share
  10. You might want to ask yourself: is this a problem at all? I’d say “no”. The gain in performance etc is minimal compared to the time you’re wasting defragmenting.

    Share
    1. I can NOT resize my partition in order to create a 2nd partition (so I can dual boot osx and linux)…I WOULD call that a problem!!!!

      (and i have no external drive, etc. in order to fix this problem a different way)

      Share

Comments have been disabled for this post