Linux Hard Disk Upgrade: An Experience

Home : Linux resources : Hard Disk Upgrade

The following notes recount my March 2001 experience with upgrading the hard drive on my cheapo Linux box, which was then two years old. This turned into a major project, because it also involved repartitioning the disk and doing a memory upgrade while I was at it. It also turned out to be somewhat harrowing, because, despite precautions, I screwed up badly enough so that the system wouldn't boot up. Twice.

Nevertheless, it was well worth the trouble. The system seems much faster because it pages much less, and seems to page faster when it needs to (though some of that is undoubtedly due to the simultaneous memory upgrade). I also have tons more disk space on which to put stuff; before, I was unable to keep the source files for new packages online for very long. And last but not least, my file space is more usefully divided for the way I do backups.

This document is divided roughly equally into three major sections. The first section gives an overview of what I intended to do, the second consists of the detailed step-by-step instructions I wrote out for myself (in order to ensure that I didn't goof it up), plus a little hindsight, and the final section recounts the lessons I learned the hard way, and includes a postscript of sorts.

Table of contents

1. Linux Hard Disk Upgrade: An Experience
   1. Table of contents
   2. Upgrade strategy
      1. New partitions
   3. Customized upgrade procedure
      1. If I had to do it all over again
   4. Cautionary tales
      1. Never do more than one thing at a time
      2. Count your cylinders carefully
      3. When disk errors are not disk errors: a postscript
      4. Another one bites the dust

Another disadvantage of this setup is that the swap partition is /dev/hda7, at the very end of the drive. As I understand it, this means that it is close to the spindle, where the disk surface moves relatively slowly past the heads. This is the slowest part of the disk to read and write, not only because it takes longer for a sector to be read, but because there are fewer sectors per cylinder, so the space must be spread out over more cylinders, leading to more seeks. Accordingly, I wanted to move the swap partition closer to the start of the disk.

New partitions

I bought a used 30G drive from PCs for Everyone (they now call themselves Thinkmate.com, and planned to divide it up as follows:

partition    size      blocks    what
/dev/hdb1      2G                DOS
/dev/hdb5     20M        7929    /boot
/dev/hdb6    300M                swap
/dev/hdb7      1G       62389    /
/dev/hdb8      3G      195476    /home
/dev/hdb9      5G      329336    /usr/local
/dev/hdb10     2G      622439    /usr
/dev/hdb11     2G       17698    /var
Total:        15.32G

• The partition names are shown as /dev/hdbi, because that's where I thought the new drive would be when installed using the CDROM cable (turns out I was wrong).
• The "DOS" partition was the original boot partition, which I kept for no good reason. I was thinking that it might be useful to have an alternative way of booting the system, but I later turned this into a /shared partition without ever having tried it. (I know now that booting Windows on any but the original hardware on which it was installed is usually difficult to impossible, so it probably wouldn't have worked.) In any case, the original owner of this disk is lucky that I was never curious enough to go ploughing through his old files.
• The block counts are from the actual usage on the original disk, and were computed by doing "du -s" recursively.
• Given the partition contents, the sizes in the second column were chosen to reflect probable future growth patterns.

Notice that I'm still only using slightly more than half the disk . . .

As implemented (according to "fdisk -l"):

Disk /dev/hda: 255 heads, 63 sectors, 3720 cylinders
Units = cylinders of 16065 * 512 bytes

   Device Boot    Start       End    Blocks   Id  System
/dev/hda1   *         1       261   2096451    6  FAT16
/dev/hda2           262      2261  16065000    5  Extended
/dev/hda5           262       264     24066   83  Linux
/dev/hda6           265       525   2096451   83  Linux
/dev/hda7           526       563    305203+  82  Linux swap
/dev/hda8           564       694   1052226   83  Linux
/dev/hda9           695      1086   3148708+  83  Linux
/dev/hda10         1087      1739   5245191   83  Linux
/dev/hda11         1740      2000   2096451   83  Linux
/dev/hda12         2001      2261   2096451   83  Linux

[I have since given /dev/hda6 an ext2 file system, and mounted it as /scratch; this partition is never backed up. I use it for such things as 250MB of Squid cache space, and as a holding area for backup files waiting to be cut to CD. -- rgr, 7-Jul-02.]

Here is how df reports the state of the mounted filesystem partitions on the new disk:

Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/hda8              1018298     54919    910768   6% /
/dev/hda5                23300      7924     14173  36% /boot
/dev/hda12             2028098     30275   1893001   2% /var
/dev/hda11             2028098    622426   1300850  32% /usr
/dev/hda10             5065710    349576   4453875   7% /usr/local
/dev/hda9              3043987    192556   2693996   7% /home

1. Do the standard end-of-month upgrade tasks.
2. Logout and go to single-user mode ("init S").
3. Do the final pre-upgrade system backup.
4. Shutdown, power off ("shutdown -h now").
5. Install the new memory module.
6. Install the new disk as slave using the CDROM slot and cable. It was easiest to take the CDROM out to do this; then I could just put the new disk in the space thus vacated. Cabling and jumpering turned out to be trivial to figure out; both old and new drives were made by Seagate, had almost identical packaging, and they and the CDROM drive had power, jumper, and data connections in the same locations. (Since then, all IDE devices I've ever seen have looked almost identical, especially from behind.) Because I wired it up as the slave on the second IDE port, Linux addressed the new disk as /dev/hdd. In hind sight, since I had removed the CD drive anyway, it would have been easier to install the new disk as the second master (/dev/hdc to Linux), since then I wouldn't have needed to rejumper it when reinstalling as /dev/hda.
7. Reboot into single-user mode ("linux single" at the LILO prompt). [Oops; didn't boot.]
8. Run tripwire in verification mode. Don't bother building a new database; moving partitions will just screw it up anyway.
9. Use cfdisk to build new partitions on /dev/hdd. Since the disk was used, I had to delete some of what was there, then add the ones I wanted. Otherwise, I probably would have needed to supply the -z option (i.e. "cfdisk -z /dev/hdd"), at least to start off.
10. Reboot into single-user mode again. (After saving the new partition structure, cfdisk says the following on exit:

    Reboot the system to ensure the partition table is correctly updated.

    This may be obsolete, as "fdisk -l" showed the new information immediately. But I didn't take chances.)
11. Use "mke2fs -c /dev/hddi" to build each of the Linux file systems. This actually took much longer than copying the files, which only involved reading 1.1GB from one disk and writing it out to the other. "mke2fs -c" reads every sector of the new partition to check it for read errors, so that required reading some 15G of non-data. In hind sight, I didn't need to do "-c"; hard drives have all had built-in hard-error recovery for some time now.
12. Copy relevant directories as follows:

    	mkdir /new/home
    	mount -t ext2 /dev/hdd9 /new/home
    	(cd /home; tar cf - .) | (cd /new/home; tar xBfp -)
    	(cd /home; tar cf - .) | (cd /new/home; tar dBf -)
    

    The first tar line copies, the second one compares. Repeat for all partitions, being careful about the order of mounts due to directory nesting.

    [NB: I had to copy /usr and / differently in order to avoid copying all subdirectories. -- rgr, 17-Mar-01.]

13. Just for the sake of paranoia, make sure the new partitions are still in good order:

    	umount /new/home
    	e2fsck -f /dev/hdd9
    

14. Maybe switch to new /home partition?

    	mv /home /old-home
    	mkdir /home
    	mount -t ext2 /dev/hdd9 /home
           

    This allows a certain amount of testing before punting the old drive. [I don't remember whether I did this or not, but it's kinda silly; I never actually threw the old drive away, so the files are still available. -- rgr, 23-Dec-04.]
15. Use "mkswap -c /dev/hdd7" to make the new swap partition.
16. Modify /new/etc/fstab to reflect the new partitioning, so that Linux on the new drive will mount the right things in the right places when it starts up. For me, this required changing the first three lines to reflect the new partition numbers, and adding four new lines for the four new partitions.

    Old:

        /dev/hda6        /              ext2    defaults        1 1
        /dev/hda5        /boot          ext2    defaults        1 2
        /dev/hda7        swap           swap    defaults        0 0
    

    New:

        /dev/hda8        /              ext2    defaults        1 1
        /dev/hda5        /boot          ext2    defaults        1 2
        /dev/hda12       /var           ext2    defaults        1 2
        /dev/hda11       /usr           ext2    defaults        1 2
        /dev/hda10       /usr/local     ext2    defaults        1 2
        /dev/hda9        /home          ext2    defaults        1 2
        /dev/hda7        swap           swap    defaults        0 0
    

    Note that these are all still described as partitions of /dev/hda, and not of /dev/hdd, because the new drive will be installed as /dev/hda before we need this fstab.
17. Make a new configuration file to tell LILO how to install itself on the new disk structure, and run lilo. (For clarity, "LILO" refers to the LInux LOader as a whole, and "lilo" is the binary that runs under Linux to install LILO. If you are using the GRUB boot loader, you will need to do something different; see the "GRUB(8)" man page. Unfortunately, I don't have specific instructions, but I suspect this will be easier with GRUB.)

    To do this, create a new file, /etc/lilo.conf.new for example, with the following contents, modified appropriately for your configuration. Do not replace your current /new/etc/lilo.conf file, as this will probably only need small changes (see below).

        disk=/dev/hdd bios=0x80   # Tell LILO to treat the second disk as if
                                  # it were the first disk (BIOS ID 0x80).
        boot=/dev/hdd             # Install LILO on second hard disk.
        map=/new/boot/map         # Location of "map file".
        install=/new/boot/boot.b  # File to copy to hard disk's
                                  # boot sector.
        prompt                    # Have LILO show "LILO boot:" prompt.
          timeout = 300
          vga = normal
          read-only
        image=/new/boot/vmlinuz-2.2.17-14  # Location of Linux kernel.
        label=linux               # Label for Linux system.
        root=/dev/hda8            # Location of root partition on the new
                                  # hard disk.  Modify accordingly for
                                  # your system.  Note that you must use
                                  # the name of the future location, once
                                  # the old disk has been removed.
        read-only                 # Mount partition read-only at first, to
                                  # run fsck.
           

    Note: In this configuration file, the disk is referred to as /dev/hdd and the names of files on its boot partition use /new/boot/ because lilo needs to access the disk as it is now to set it up for booting later.

    Once you have this file set up, do "lilo -C /etc/lilo.conf.new to get LILO to install itself on the new disk.

    [This is based on section 8 of the Hard-Disk-Upgrade mini-HOWTO.]

18. At this point, everything of value (that is accessible to Linux) has been copied off the old drive, and the new drive is ready to take its place. This would be a good time to destroy any sensitive information you have on the old disk, such as PGP private key databases, correspondence with friends and lovers, Swiss bank account numbers, etc.
19. Shutdown, power off ("shutdown -h now").
20. Remove old drive, rejumper new drive as master and install in its place, reinstall CDROM as before.
21. With fingers crossed, reboot in single user mode. [Oops again.]
22. Run tripwire to rebuild the database. The tw.config file will need updating due to the partitioning, since tripwire never crosses partition boundaries without explicit instructions. Don't bother verifying; you've already done that, and the new inodes will all be different anyway.
23. Go to multi-user mode ("init 5"). Verify that all servers come up properly.
24. Update the original /etc/lilo.conf to refer to the correct root partition on the new disk, and rerun lilo (but this time you don't need the "-C" argument). Aren't you glad you kept the original?
25. Do full dumps of all partitions. (The contents of /boot haven't changed, but the inodes have, which makes a difference to dump, so it needs to be dumped anyway.)
26. Heave a great sigh of relief.

1. Pick up the installation CDs for the latest version of my preferred distro, possibly at the same time as picking up the new disk.
2. Perform backups as before, but don't bother upgrading OS software. You may want to print a copy of your /etc/fstab for reference, so that you know which partitions were mounted where (see step 4).
3. Re-install the old disk as the slave device on the second IDE controller (/dev/hdd) using the same cable as the CD drive, and install the new disk in its place (master on the first IDE controller, /dev/hda). Don't remove the CD drive; this is their permanent new configuration.
4. Use the installation CDs to partition the new drive and do a fresh OS installation on it. When partitioning, the installer should detect /dev/hdd and give you an opportunity to mount its partitions in the new filesystem. I recommend /mnt/old for the original root partition mount point, and then mount any other partitions in the correct relative place, e.g. /mnt/old/home for the old /home partition. In any case, you should not pick mount points that will receive parts of the new OS (e.g. "/usr").
5. Finish the OS installation normally. You needn't worry too much about getting the configuration right just yet, since you can use the old configs as a starting point after you're done.
6. Copy data from the old drive to the new one, e.g. from /mnt/old/home to /home, as above. You need not move it all if you are planning to keep using the old drive where it is. Be careful not to overwrite any part of the new OS with files from the old one.
7. If you are using dump/restore for system backups, you will need to create new full dumps of the moved partition(s) in their new location(s), as dump won't be able to make new incrementals relative to the old full dumps.

Voila; you are done already, and you didn't even have to do any more reboots than it took for the OS installation. You now have two choices:

1. You can keep the old drive in place, and continue to use it. A second disk drive is an ideal place for temporary storage of backup dumps from the main disk.
2. If you decide to remove the old disk, don't forget to remove the /dev/hdd mount points from /etc/fstab, and ensure that the new OS doesn't need anything on it.

I wrote this page partly out of a need for catharsis, and partly because of the chance that I may be able to save others some amount of aggravation. May you have better luck than I did in avoiding the "stupid rays."

Never do more than one thing at a time

The moral, of course, is to do only one thing at a time, so that you can test one thing at a time. At least I only wasted an hour or two, if you exclude having dinner ruined for being bummed about my broken system.

Count your cylinders carefully

Fortunately, Linux skips the BIOS entirely, and talks directly to the disk, so once it is booted, Linux is not affected by archaic BIOS protocols. This also means that Linux can access much bigger disks than are made these days. Unfortunately, LILO has to use BIOS disk calls in order to pull Linux off of the disk, so BIOS arcana can't be entirely ignored.

I was rudely reminded of this sad state of affairs when I attempted to reboot off of my snazzy new disk . . . and couldn't even get to the LILO prompt; it hung with "LI", and that's all she wrote. Several attempts to reboot after reseating connectors met with the same result (though the BIOS seemed to be finding the drive, so that shouldn't have been an issue). Finally, suspecting I had make a mistake in /etc/lilo.conf, I reverted to the two-disk configuration; fortunately I hadn't actually moved the drives nor replaced the CDROM at that point, so reverting was easy. (If I still hadn't been able to reboot then, I'd have probably pushed the damn thing off a bridge.)

Well, /etc/lilo.conf seemed to be what the HOWTO said it should, so, in desperation, I started hunting through all of the LILO documentation I could find. While ploughing through /usr/doc/lilo-0.21/doc/tech.tex looking for clues, I learned that the incomplete "LI" prompt means that LILO thought it had loaded the secondary boot loader (having printed the "I"), but it failed to start (having not printed a second "L"). That made me suspect the 1024-cylinder problem, though I had thought sure I was "safe" in that regard. Here's what the partition table on the new disk looked like at the time:

Disk /dev/hdd: 255 heads, 63 sectors, 3720 cylinders
Units = cylinders of 16065 * 512 bytes

   Device Boot    Start       End    Blocks   Id  System
/dev/hdd1   *         1       261   2096451    6  FAT16
/dev/hdd2           262      2261  16065000    5  Extended
/dev/hdd5           262       522   2096451    6  FAT16
/dev/hdd6           523       525     24066   83  Linux
. . .

Then, in a flash of inspiration, I realized that these numbers were based on fdisk's fictional notion of disk geometry, using the maximum values of 255 heads and 63 sectors (and still it shows almost four times as many cylinders as DOS could use). If the BIOS had a different notion of geometry, then it might be using cylinder indices that were at least a factor of two greater. Sure enough, the original layout (see above) was reported using 128 heads instead of 255. And, though lilo normally warns if it tries to use files beyond the 1024-cylinder limit, it would have no way of knowing that the BIOS was using different geometry.

So then I did the wrong thing: I moved the partition. Since the boot partition is really small, at least this didn't waste much time. For reference, here's what was involved:

1. Delete /dev/hdd2 (the second FAT16) and /dev/hdd5 (the first-cut boot partition). This makes a single chunk of contiguous free space.
2. Create a new /dev/hdd5 at the start of the new free space. (It got numbered 5 instead of 2 because I created it as a logical partition; /dev/hdd2 had been a physical partition.)
3. Create a new Linux partition that uses up the rest of this chunk of free space.
4. Build a filesystem on /dev/hdd5, and recopy the files from the /dev/hda drive's original boot partition.
5. Rerun lilo to take account of the new location of the boot partition.

But since I had nothing better to do, I poked around in the BIOS disk setup menu, and fortuitously discovered that the geometry of the old drive had been hardwired for the first disk. Once I changed the first disk to "Auto/LBA" (which seemed to have worked when it was the second disk), Linux booted like a charm. Of course, this probably would have worked without moving the partition, but I hope you'll excuse me for not feeling like moving it back to try.

For completeness, I should also note that the Large-Disk mini-HOWTO, which I had read (or at least skimmed) while developing my upgrade procedure, spends a whole chapter on the need for agreement on geometry between LILO and the BIOS. Here is the first paragraph from this chapter (titled "Consequences"):

What does all of this mean? For Linux users only one thing: that they must make sure that LILO and fdisk use the right geometry where `right' is defined for fdisk as the geometry used by the other operating systems on the same disk, and for LILO as the geometry that will enable successful interaction with the BIOS at boot time. (Usually these two coincide.)

The moral for this tale? I would like to say, "Steer clear of ancient operating systems and their obsolete protocols," but this is hardly possible. Perhaps a better one would be "Nobody is completely unaffected by the Wintel monopoly."

When disk errors are not disk errors: a postscript

So I was left with an unbootable system, possibly with flaky hardware -- time for a coffee break and some cool-headed thinking. I grabbed several sheets of paper and a pen (being a computer guy, I don't usually run around with pens or pencils in my pockets), and lit out for our local Carberry's bakery and cafe. I needed to scope out my options, because whatever I did, getting the system back up would probably be painful and time-consuming, and I only wanted to go through it once.

By then it had dawned on me that the kernel version thing couldn't explain all the errors I was seeing, and that the most likely explanation for all the symptoms I had seen -- indeed, the only reasonable one -- was that my disk had Alzheimer's, of a sort. (This was also incorrect, as it turned out, but at least it gave me something concrete to work on.) With some physical and mental space between me and the computer, plus the welcome distraction of coffee and a scone (Carberry's makes awesome maple-oatmeal scones, BTW), I began to plan what I could do to get my data (and my sense of normalcy) back.

The following is a rendering of my notes from that Monday morning session:

Option 1: Rescuscitate

1. swap old drive for new, new for CD
2. boot old drive
3. rescuscitate FS partition(s) on new drive
4. if fail, punt. else, reswap drives, reboot

May have to rebuild root partition;

screwed if so. no; can preload dumps onto old partition.

Option 2: Punt

1. get new HD, Linux, maybe new CD
2. install drive
3. install Linux
4. restore backups from CD, Zip -- mostly functioning at this point.
5. swap broken drive for CD
6. recover latest files from /dev/hdd9, maybe attempt to diagnose.
7. replace CD (maybe new)
8. finish reinstalling opt software (AbiSuite, ntpd, squid, AOLserver, ssh, qmail, netatalk, PostgreSQL, CLC+Araneida+CLSQL, dump, ACL

Cost/benefit	Opt 1	Opt 2
$$	+/0	-
time	+/-	--
upgrade		+
risk	-	+

1. Replaced the new drive (the broken one) with my original 3 gigabyte hard drive, and booted off of that.
2. Pulled the relevant backup files for the new root partition from CD, and put them on the old drive. This was in case I had in fact trashed the root partition.
3. Shut the system down again, and swapped the broken drive for the CD.

At PCs for Everyone, the guy agreed that I had a bad power supply, but they didn't have the 145-watt supply I needed in stock, and in any case it would be another two weeks before they could even look at my disk problem. I must have blanched at the prospect of being without my computer, the "main server" for my life and livelihood, for all of two weeks. After some back-and-forth, we came up with an alternative plan: He could attach a 300-watt supply, even though that was too big to install properly within the enclosure, after which I would be able to continue flogging the disk problem. When a new 145-watt supply came in, he said he'd give me a call, and I could return the old one under their 14-day refund policy and replace it with something that would fit the box. Since this seemed the only path to getting my machine up in any sort of reasonable time, I readily agreed, and after executing the sale, he popped out the old supply and hooked up the new one, leaving it trailing wires out the back of the box. (Even though this was a 3-year-old machine, PCs for Everyone has a lifetime labor warranty.) I left my name and phone number so he could give me a call when the 145-watt units came in, and carefully carried my Dr. Frankenstein PC back to the car.

Once home again, I picked up the rescue process where I had left off late that morning. I moved the new drive back to where the CD-ROM was, and booted off the old drive. To my surprise, the new drive no longer seemed quite so flaky. I was able to use e2fsck to repair the bad superblock, plus a few other minor partition data structure consistency problems I had seen before as a result of unexpected shutdowns, and that was it. (Just to be sure I wouldn't run afoul of version problems, I used the e2fsck on the old drive to check out /dev/hda8 on the new drive, and then used the new drive's e2fsck (which itself lives on /dev/hda8) to recheck /dev/hda8, and then fix the other partitions on the new drive.)

Was that all there was to it? Had all of my "disk problems" been due to a flaky power supply? That would have been the first bit of good luck I had seen all day, so I was disinclined to believe it. I checked each partition with the current e2fsck version, then checked them again to be sure they were still clean, and finally backed up all partitions, so I would have the latest data in case the new disk resumed misbehaving when I booted off of it. (Doing "cross-disk" backups turned out to be tricky -- I had to cut-and-paste entries from the new /usr/local/etc/dumpdates to the old /usr/local/etc/dumpdates so that dump used the correct starting dates, changing the partition names from /dev/hdaX to /dev/hddX as I went. Then, after I was done, I had to move the entries for the backups I had just made to the new drive, undoing the partition renaming.)

Sure enough, it worked. The system booted properly, all services came up without problems, and has been working flawlessly since then. (Well, more or less flawlessly.)

For consistency, perhaps I ought to conclude with a moral that summarizes what I learned from the experience. Certainly, the lesson I learned (relearned, actually) is not to assume I know what the problem is until I've actually fixed it. This is also a good maxim for software, but it goes double for hardware, especially for me, since I have much less hardware troubleshooting experience. But I can't think of anything really pithy and original that conveys this. If you're willing to forego orginality, we could use "It ain't over until the fat lady sings." But I'll understand if you're unwilling.

There's actually slightly more to the tale -- the subsequent de-frankensteinification of my system. This took longer than expected, partly because the guy from PCfE never called, but I would up getting the new power supply and swapping it for the Franko-monster without incident. I also got a new CD-burner at the same time.

So perhaps a good moral would be, "Choose your hardware supplier carefully." Kudos to PCs for Everyone for taking such good care of an old (and not particularly rich) customer.

Another one bites the dust

Once I had the spare system in place, I was in much less of a hurry to get the old system fixed, so I didn't take it in for servicing until about a month later. The previous winter, two older systems at work had failed with what appeared to be the same symptoms, and both turned out to be due to bad processor chips; the CPUs had fried themselves, and one had also taken out its motherboard. Based on this experience, I was expecting a repair bill of at least a hundred dollars, so I was relieved to learn that it was only the stupid power supply (again), and the bill only came to $51.45. But I was lucky they had the right unit; as it was, I just managed to snag the last one that PCs for Everyone had in stock. It seems that they had recently decided to discontinue this model on account of unreliability.

So if the new power supply is similarly short-lived, I can look forward to another such failure in the summer of 2006, at which time I may have to get rid of the machine. But even if so, I will have gotten my money's worth; the system will have lasted more than seven years in that case, and even with the cost of repairs and the monitor thrown in, it works out to about $12/month, which is pretty darn good. My high-speed Internet connection costs four times as much.