40 Comments

Summary:

Disk fragmentation is an old problem that has affected every operating system throughout history. File fragmentation occurs when a single file isn’t located in the same physical location on the disk, but is scattered around. OS X does a great job at minimizing file fragmentation by […]

idefrag_iconDisk fragmentation is an old problem that has affected every operating system throughout history. File fragmentation occurs when a single file isn’t located in the same physical location on the disk, but is scattered around. OS X does a great job at minimizing file fragmentation by rewriting files in contiguous space when a file is opened, is under 20MB and contains more than eight fragments. This works quite well to prevent heavy file fragmentation, but what it doesn’t prevent is free space fragmentation.

Free space fragmentation?

Free space fragmentation can become a problem when you start running low on disk space. If you have a small percentage of your disk free — and you frequently use and then free-up disk space with temporary files — you’re likely to have high free space fragmentation. This means the free space is spread all over your hard drive in small amounts here and there — its not all laid out in one contiguous section in one spot.

So what’s the problem with this? Ultimately, this impacts OS X’s ability to keep file fragmentation under control as it can not rewrite large growing files into contiguous empty space, as there is none. Any new large files that are written also get fragmented by the same issue. Just as importantly, when space is fragmented, caching files for the system and for applications also get spread around all over the place. OS X utilizes file caching heavily. The problem is that when the file cache is spread all over the hard disk, it takes a time penalty to access (and also to write) these files. Being able to write and access caches from the same physical area gives a very noticeable speed increase — rebuilding caches after de-fragmenting can thus be advantageous.

How do I tell if my free space is fragmented?

If you suspect you might be suffering from free space fragmentation, there is a free tool you can use to verify your suspicion. Download hsfdebug (the download link is at the end of the page) and copy the file hsfdebug to any directory (such as your home directory, ~/). Open a terminal window, navigate to the directory you copied the file to, and run the following command:

sudo ./hfsdebug -0 | sort -n

This will list all the blocks of free space available, in order from smallest to largest. For example the last 10 lines of output when I ran this was:

10391 0x134e488 0x1350d1e 40.59 MB
10993 0x11004e7 0x1102fd7 42.94 MB
11691 0x152cd74 0x152fb1e 45.67 MB
12229 0x26d1a05 0x26d49c9 47.77 MB
14537 0x2693cdc 0x26975a4 56.79 MB
24090 0x129b388 0x12a11a1 94.10 MB
57704 0x12a9b0c 0x12b7c73 225.41 MB
68490 0x133c5dc 0x134d165 267.54 MB
69680 0x12c82f7 0x12d9326 272.19 MB
237568 0x1d02 0x3bd01 928.00 MB

This tells me that the largest amount of free space I have in one contiguous block is 928 MB. The next biggest is 272.19 MB and its all downhill from there (there were 60,000 lines in total, with 43,000 of them under 100 KB). Since I have a total of 22 GB free disk space, most of that is clearly spread out in tiny blocks. That’s pretty fragmented, which doesn’t surprise me as I deal with a lot of temporary files that are constantly being written and then deleted over a few days.

Of interest, hfsdebug can also tell you the amount of file fragmentation for the curious:
sudo ./hfsdebug -f -t 5

This will take a little while to process, but will tell you the percentage of files that are NOT fragmented:

Out of 1214506 non-zero data forks total, 1209938 (99.624 %) have no fragmentation.
Out of 4135 non-zero resource forks total, 4074 (98.525 %) have no fragmentation.

So OS X has been doing a good job at keeping file fragmentation low, as expected.

Why else would I need to defrag?

Another reason to defragment free disk space is to make it possible to create a Boot Camp partition. While OS X can smoothly resize the existing partition and create the new one for Windows, it requires contiguous free space. If this doesn’t exist, you will be told “Can not create partition, please re-install OSX and try again.” This is because Apple does not provide a method to explicitly defrag your hard drive. For the cases it can help (such as creating the Boot Camp partition), Apple recommends just doing a full backup and restore; formatting the hard drive, re-installing the OS and restoring your files. This does indeed do the trick, as it forces every single byte to be rewritten to the disk — however there are a number of third party applications that will do the trick with much less effort.

Defrag Tools

The following applications will provide comprehensive disk defragmenting. I think its important to note that to maximize the benefit of caching performance, a tool such as Onyx should also be used first to clear your System and User caches before you do your defragging. This results in the caches being rebuilt in contiguous space afterwards.

iDefrag — iDefrag is all about defragging and provides a number of methods to do so. For example we could just use the ‘Compact’ algorithm which will put all files at the start of the disk, making our free disk space all at the end with the minimum time spent. iDefrag is £19.95 ($30).

Drive Genius — Drive Genius is a full disk maintenance utility, allowing partitioning and recovery as well as defragmenting. It is claimed that Apple Geniuses at the Genius Bars use this when dealing with problematic hardware. Drive Genius costs $99.

TechTool Pro 5 — TechTool Pro 5 claims to be a complete Mac problem-solving system, that includes disk defragmenting. A cut-down version (TechTool Deluxe) is included with Apple’s Apple Care product, but this does not contain the defragmenting functionality. The Pro version costs $98.

Results?

After a full defrag with iDefrag, re-running hfsdebug gave me an output of only 123 lines, with the following being the last 10:

370 0x1883fd 0x18856e 1.45 MB
377 0x232da2e 0x232dba6 1.47 MB
445 0x232efb1 0x232f16d 1.74 MB
479 0x1670b2 0x167290 1.87 MB
768 0x1d02 0x2001 3.00 MB
8343 0x232abfa 0x232cc90 32.59 MB
63659 0x78756 0x88000 248.67 MB
184298 0x3a202 0x671eb 719.91 MB
1183949 0x881437 0x9a2503 4.52 GB
5014722 0x232fb3d 0x27f7ffe 19.13 GB

So the bulk of my free space is now nicely contiguous and large. My caches are rebuilt and my system is noticeably snappier and more responsive. Milage may vary, depending on exactly how the fragmentation is affecting you to begin with, but free space defragmentation can definitely improve performance in some cases. Importantly, be sure to always have a backup on hand, as any power glitch during any long disk operation may cause damage your file system.

  1. BE CAREFUL

    I ran defrag on a drive and it rendered it completely useless. Disk utility could not repair. Had to revert to a cloned copy.

    Share
  2. ZEDenterprizes Tuesday, April 14, 2009

    Being generally fairly cheap (a.k.a. “frugal” ;-)), I was wondering if anyone’s come across any free utilities along the defrag line. I know there were always tons for Windows based systems….probably because it’s been more of a problem for them…

    Share
  3. I have never run into a situation in which I had to degrag my hard drive with OsX. There was one occasion when I experienced a slow down in my system and realize that I had less than 10% of space left in my hard drive. My solution to the problem was very simple. I just back up a lot of my large files (movies mostly) onto an external hard drive to free up the space and the problem corrected itself by magic. This is the beauty of OsX and the realization that it is something window users have to deal with in their normal computing life.

    Share
    1. OSX does get fragmented. Windows more so but most users dont ever have to defrag as vista does it for them. Even if it did not most users would only have to do it every few months to improve disk performance. I have a mac and tried to like the os but could not.

      Share
  4. This was very interesting, that “hfsdebug” tool is useful and easy to use. Unfortunately, it dosn’t fix anything, it just tells you what’s going on inside your Mac. Isn’t there any free defragmentation software out there? If I have to spend a thousand dollars to get it fixed, then I’ll keep my drive slow and dirty. ;)

    Share
  5. I used Drive Genius in the past. However, I hear nightmares of people using this and other products – rendering their drives unusable – so I am avoiding using this or other products again.

    Also, you mention your software hsfdebug link is at the bottom of the article. It isn’t there. If I press work hsfdebug in the middle of this article, the link is broken

    http://www.osxbook.com/software/hfsdebug/

    Credibility of the article here is low.

    Share
  6. @anon: Did you even bother to read the article? There is a link to hfsdebug in the 4th paragraph and we said the download is at the end of that page.

    Share
  7. [...] roundup: Palm Pre release date (jkOneTheRun) HD Cloud: Video trandscoding gets hip (NewTeeVee) When does disk fragmentation on OS X become a problem? (TheAppleBlog) The road to mass market electric vehicles in China (Earth2Tech) [...]

    Share
  8. You can use the standard Disk Utility to eliminate a lot of fragmentation. It works best if you have a lot of free space on a volume.

    Just run Disk Utility, click on your boot drive (not the volume) and then click on the Partition tab. You can then resize your partition on the fly. Make the partition smaller and Disk Utility will move files to accommodate the smaller partition. Keep making the partition smaller until you get an error then start trying smaller changes. I was able to defrag 130 GB on my 320 GB drive this way.

    Using Leopard, I started by reducing the partition by 30 GB and then I kept reducing by 20 GB until I got down to 14 GB free on my boot volume. I stopped then. According to hfsdebug my largest defragged block is 129 GB in size and my disk is 99.949 % fragmentation free. Good enough and free.

    Share
  9. The ‘cheap’ solution would be to use the freeware tool (sorry, forgot its name) to clone your drive. Now reformat your drive. Reinstall everything. It might sound like it will take a long time, but so does defragmenting. And its probably safer to do it this way too.

    Share
  10. You might want to ask yourself: is this a problem at all? I’d say “no”. The gain in performance etc is minimal compared to the time you’re wasting defragmenting.

    Share
    1. I can NOT resize my partition in order to create a 2nd partition (so I can dual boot osx and linux)…I WOULD call that a problem!!!!

      (and i have no external drive, etc. in order to fix this problem a different way)

      Share
  11. Has anyone checked a hard drive contents in both the original and a duplicate created by CarbonCopyCloner or SuperDuper?

    Share
  12. Do not defragment. OSX has enough built-in intelligence to limit defragmentation. Do not waste your time looking for freeware. Do not spend money on a “solution”. The imaginary need for defragmentation is a Windows-oriented behavior that is unnecessary on a Mac. Windows users spend hours tweaking, modifying, configuring in pursuit of a 1/2 of one percent speed increase in their Windows boxen, rather than just getting their real and productive work done. Ricco, answer #9 has it right. New Mac users? Technically oriented users? Users used to doing a lot of maintenance with Windows? I repeat, do NOT waste time defragmenting a drive. The only, only time in which defragging might be needed, is if you are a user with large Photoshop projects, multitrack projects or After Effects/Final Cut video projects. In that case, use a separate drive, perhaps Firewire/USB external. If it is sufficiently large you could parttion it for 2, 3, 4 projects, couple of hundred Gigs each. Do your work, a week, two weeks, whatever, then when you finish and archive footage and/or a finished project, re-initialize the drive or the partition and you start with a brand new blank slate. For ordinary day-in, day-out use on a Mac, do NOT defragment.

    As to #10 Kiwiiano’s question, using CCC as the example. I would guess that a block level copy would yield a copy that is identical to the source drive, but doing a file-oriented copy would probably yield a drive less fragmented than the original, since whole files would be read then written contiguously.

    Share
    1. Claiming that fragmentation (note it’s fragmentation and not defragmentation) is not a problem with OS X is ignorant. Perhaps if you know a bit of CS theory and how any file system works, you would know that fragmentation is inevitable on any file system. You can mitigate some of it (at a performance cost), but you can not completely eliminate it. Depending on your usage patters as well, it may not be a significant problem for you. If you mostly listen to music and browse internet, then don’t worry about it. IF you work with large video files and/or images, then it may be a problem for you.

      Perhaps the following Wikipedia article might help to elucidate the issues:

      http://en.wikipedia.org/wiki/File_system_fragmentation

      In any case, of all the mentioned tools, I find iDefrag the best.

      Share
  13. Perhaps I need to clarify to everyone that I am not recommending defragging as a regular preventative activity. I am recommending it as a troubleshooting step under certain specific conditions only:
    * Low percentage of disk space free
    * Noticable system performance degredation
    * Confirmed *free space* fragmentation (using the tool I linked to

    Under THOSE conditions, defraging can most definitely help.

    OS X handles file fragmentation very well, and there is NO need to defrag for this reason, as I clearly stated in TFA.

    Share
  14. To those that say defragging is unnecessary in OSX, perhaps that is true in your situation, but if you do a large number of file writes and rewrites, and run close to disk capacity, then defragging makes a fairly big difference. I would agree that in general, it is much less necessary than in a windows environment, but that does not mean that it is never helpful, or at times necessary.

    Share
  15. If I understood his/her comment, Anon read some reports on the Internet about Data Rescue causing corruption [a few out of thousands and thousands of users] and decided that it was inherently unsafe.

    Yacko wrote that defragmenting drives on Macs is a waste of time. But I’m not sure he read the article carefully, as it mentions OS X’s ability to defrag files–but not free space. And as the article mentions, sometimes a lack of contiguous free space can cause problems.

    I have used Windows and Mac defrag programs for the last twenty years without any data loss. I use iDefrag a couple times a year typically on my Macs. The speedup is not huge, but noticeable. iDefrag has an excellent interface, is smart about ordering frequently used files in the Hotfiles zone and is frequently updated.

    You’ll have to decide if a modest speed improvement is worth $30. I’ve been using iDefrag for over three years now and consider it a good bargain.

    Bot
    Mac Fanbot and occasional Defragbot

    Share
  16. I’m pretty sure the Leopard Boot Camp Assistant does what Disk Utility does when you shrink the partition size – it’ll move everything down out of an area to make contiguous free space.

    Your other alternative if you really really want to have a complete defrag is to boot from install media, reformat your disk, and restore from Time Machine. (You have a Time Machine backup, right? If not, then buy a USB external disk already, zomg.)

    Not that I recommend doing that unless you really have to either, but it’s certainly a good free option.

    Share
  17. @Thorfinn – When I created my Boot Camp partition, The Boot Camp Assistant tried to do it, but failed and told me to reinstall OS X and try again. I used iDefrag to Compact my disk and after that Boot Camp Assistant created the partition with no problem.

    Share
  18. No mention of Disk Warrior? It’s been a while since I used it, but I always found my computer distinctly snappier with disk-intensive operations after using it to defragment. I never quite understood that since OS X was supposed to auto-defrag, but maybe it had to do with free space fragmentation.

    Share
  19. DiskWarrior doesn’t defray files or free space. It can recreate the volume bitmap (sort of an index to hard drive contents).

    Bot

    Share
  20. Um, I typed “defrag,” not “defray.” Curse you, autocorrect!

    Share
  21. [...] problem for anyone who’s worked with Windows, but what about fragmentation under Mac OS X? The Apple Blog’s Andrew Bednars examines the problem and also takes a look at several solutions on currently available on the [...]

    Share
  22. Hi Andrew,

    You should talk to Diskkeeper. they did some research on MACs.

    http://www.diskeeperblog.com/archives/2009/02/mac_fragmentati.html

    Share
  23. [...] the Disk Fragmentation & OS X: When Does it Become a Problem? blog post on theAppleBlog.com  Andrew Bednarz does the best job of telling you what to do to [...]

    Share
  24. Most people don’t need to defrag their macs. In my experience people who do fall in to two groups: a) people with old macs and small hard disks and b) people that work in design and do a lot of work, creating and destroying files and filling the disk.
    Some may say that that group a) people should buy new macs. That may be true in the United States . In the rest of the world, not.

    Share
  25. [...] Disk Fragmentation & OS X: When Does it Become a Problem? (tags: tools apple performance storage macosx defrag fragmentation disk harddrive mac) [...]

    Share
  26. out of the apps listed above, which is most recommended to defrag? I’m definitely exhibiting slowdowns from file space fragmentation.

    Share
  27. Re: Which app to defrag

    LazyJ, if you don’t already have a disk utility, I’d recommend you consider either Techtool Pro (which also does defragmentation) or DiskWarrior plus iDefrag. iDefrag may be the best because it can do hotfile clustering, which can improve disk performance. I’m not sure if TechTool Pro has that capability.

    Bot

    Share
  28. lazyj – I’ve only tried iDefrag, and it worked really well for me – plus its pretty cheap.

    Share
  29. Should one enable “compact b-tree files” under the COMPACT and METADATA tabs in the preferences?

    Share
  30. Thanks for the hints, especially the tool hsfdebug and the reminder to delete caches before defragging. I chose the cheapest path and did a last TimeMachine backup of my hard drive, rebooted from my Leopard DVD, ran the “restore hard drive” menu option, which formatted my hard drive quickly and copied all my files and the entire system nicely organized back onto the disk. This only took about 90 minutes. My MBP runs smoothly now and I could finally install Boot Camp. I am really wondering why I waited over a year before daring this step.

    Share
  31. lazyj, it sounds like you should do a full disk check before anything else with Disk Utility. You may need something more powerful like Disk Warrior to repair and rebuild your disk before you keep trying to defrag.

    Share
  32. For everyone who says fragmentation is a non-issue on OS/X, you are wrong and I have proven it and comprehensively documented the problem. Small files heal themselves but free-space fragmentation is a very real problem.

    If your free space becomes significantly fragmented then the on-the-fly virtual memory file allocation on OS/X gets into deep trouble. I have recorded a swapfile in over 6,000 fragments. System performance improved noticeably after defragging.

    Picture of the report and links to even more detailed discussion at http://aussiedesignedsoftware.com/blog/?p=94

    Share
  33. Mac: SuperDuper to the Rescue…

    Of late, my MacBookPro has been going slower and slower, to the point of being almost unusable due to severe and continual disk swapping. It seems (despite what various pundits say) that my disk was severely fragmented – despite having close to 30 GB…

    Share
  34. Those saying defraging in OS X is never necessary don’t know what they are talking about. For those of us who delete, rewrite, move, etc files extremely often, like me (programmer and web designer) have to defrag once or twice a year, and there is an extremely big difference in performance when I do so. The ONLY OS I have EVER used that never needed defragged is Linux. Been using Linux for close to 10 years. Been using OS X sense the first release. I also run a computer with Windows, and yes, Windows does need defragged a lot more, but OS X still needs it from time to time.

    Share
  35. *** The main reason Linux never needs defragged is partially because of the SWAP partitioning system it uses. In addition to the system partition, most Linux users also use an additional and separate “SWAP” partition, which I have found helps prevent free-space fragmentation, and most Linux distros, like OS X, defrag fragmented files upon opening to prevent file fragmentation. For more information on Linux SWAP see: http://en.wikipedia.org/wiki/Paging#Linux

    Share

Comments have been disabled for this post