How to upgrade hard drives on a Linux file server

 Another in my series of blog posts about "The way I do things that fit my very specific and peculiar set of needs".  I recently upgraded my HDDs in my Linux file server, and found a pattern that worked quite well with minimum disruption.

How I have things set up

I have a headless Linux server with a SSD as the primary drive, and then a few spinning HDDs for bulk media storage.  All the drives are just formatted in ext4.  I do backups to S3 for important files.  Then, every time I upgrade the storage of these drives I copy everything to the new drive, and then remove the old drive from the machine.  That old drive serves as a backup for the bulk media stuff, combined with the plan to just redownload anything I got recently.

I've never bothered with RAID or more complex file systems because they only made sense to me if I had 4 or more drives, which I've never had.  This makes upgrading the drives a simple process.

My /etc/fstab file has a bunch of lines like this:
UUID=9fa3b7dc-3cd7-42c1-93a8-46dcf38da09d  /mnt/media         ext4  defaults  0  2

And then I share those drives using NFS by putting these lines in /etc/exports:

/mnt/media 192.168.1.0/24(rw,sync,no_subtree_check,no_root_squash)

Format the new drive

Put the drive in the machine and power it on.  If the drive is brand new it won't have any partition tables on it, or the UUID needed for the fstab file.  Here's a bunch of commands which are useful for figuring out which drive is which:

sudo blkid
lsblk -f
sudo fdisk -l

Once you know what the path to the drive is (eg, /dev/sdb), you can use fdisk to format the drive.  Be sure you have picked the right drive, because you'll wipe all your data on whatever disk you run fdisk on.  Use m to see the list of fdisk commands. But both F and p are useful to confirm you have the right (empty) disk.  Once you're sure you can run g to create the partition table, and then n to create the new partition.  The defaults should be fine.  Once you are sure you have things right, you can write your changes to the disk with w.

You now need to create the ext4 filesystem on your partition.  Do that with sudo mkfs.ext4 /dev/sdb1 making sure that your use the correct drive letter.  

Mount the new drive

Now run the above commands used to identify the drives again, and hopefully you see a UUID for the new drive.  Copy that UUID down and edit your fstab file with sudo nano /etc/fstab

and add the line for your new drive there, with a temporary mount point, like /mnt/media_new.  Then you need to make sure you create that mount point (sudo mkdir /mnt/media_new).  Then you can mount the drive with sudo mount -av . Now you can poke around and make sure things look right, you probably want to update the owner with sudo chown.

Copy the old data to the new drive

There are many ways you could copy the data, but I like rsync.  One nice thing is that if it gets interrupted it'll pick up where it left off.  Here's the command I came up with after exploring the options for a while: rsync -axHAWXS --info=progress2 /mnt/media/ /mnt/media_new/ The options largely came from here, so you can go there if you want to read what they do.  That took about 12 hours for me to copy about 5 TB.  While it will print out the progress, I found it much better to just ssh in on a new tab and then run df -Th and compare the disk usage of the old and new drives.  After the first pass though, running it again, to make sure there wasn't anything new, only took a couple minutes.

Make the swap

This isn't a foolproof process, you should close anything that is using the disks, and run the rsync command one more time.  Then you can unmount the current drive with sudo umount /dev/sda1.  Now you want to edit your fstab one more time and delete the old drive's line from it, while updating the mount point of the new drive to the one the old drive was using.  Now you can probably get way with just running sudo mount -av again, but I like to just shutdown (sudo shutdown -h now), and physically remove the old drive.

That's it

That's it. If all went according to plan, then everything that used the old drive previously, should now just use the new drive.  I had to restart my remote machines that used the NFS drives before they would connect.

Rome: Decline and Fall?

 https://acoup.blog/2022/02/11/collections-rome-decline-and-fall-part-iii-things/

But this now raises two related questions: first, why did population decline so sharply and second, what was the impact on quality of life that resulted? The old answer to the first question was of course ‘the barbarians killed everyone’ but as we’ve seen, while the fifth century was a violent time, the violent discontinuities were not that extreme. Surely the violence of the period has something to do with some of this declining population, but as noted, the underlying population (with their language and religion) didn’t much change (and the raw number of ‘barbarians’ coming over the frontier was, in demographic terms, fairly small). Most of those Roman cities decayed, rather than being burned. But if the ‘barbarians’ didn’t kill everyone, what did and why did that somehow have a negative impact on the survivors? The answers to these two questions are actually linked in that they depend on the same evidence, so that is where we will go next.

...

If you will permit me an extended metaphor, Rome wasn’t so much demolished by invaders as it was burned down by Roman arsonists who set fire to their own house – and they had been setting those fires since at least 235, long before Adrianople. The emperors of the fourth century (particularly Diocletian and Constantine) may have put out some of the fires by collapsing a wing of the house to smother them, but this can hardly be regarded as improvement, not the least because neither of them did anything to deal with the arsonists (one of which, Constantine, at least, must be reckoned). The emperors of the late fourth and fifth centuries then proceeded to invite people into the house, promising its shelter, if only they would help them light one more fire – and then when the house was burned down and everyone was left on the cold ground, they tried to shift the blame onto the very guests they had invited.