Bed Weight Sensor v3

One of my favorite sensors I'm using with Home Assistant is a home made bed weight sensor.  This lets me know when someone is in bed to do things like dim lights, and stop automations that would announce things.  I also have a DIY burglar alarm set up to detect when a door opens while everyone is in bed.  If you search Amazon for "weight sensor" a lot of listings like this one show up.  That listing includes the strain gauges themselves and also the HX711 board you use to combine 4 of those into a single reading.  If you search around the internet you'll see a lot of tutorials for how to wire those up to measure with an ESP32 (I'm using ESPHome to get that into Home Assistant).  People commonly use 4 gauges, in each of the 4 corners of a platform, but using 2 or 1 is also possible and you can find examples of those as well.

weight gauges and typical wiring

This picture from the above Amazon listing shows how 4 gauges are commonly wired together in what is called a "Wheatstone bridge".

The problems with Bed Weight Sensor v1 and v2

When I first set these up for my bed, I just did a test by putting 2 gauges between the frame and my foam mattress.  That worked ok; it was enough to sense "someone is in bed", but it wasn't very accurate beyond that.

Version 2 used 4 gauges in a row across the bed, with little wooden squares cut out and placed under each, which were then all attached to the frame.  The sensor measured the total weight as a whole, rather than trying to measure the left side of the bed vs the right side, which is another common solution I've seen.

The v2 sensor worked pretty well; it was very accurate at knowing if the bed was empty or not, and was pretty good at knowing how many people were in bed.  However, it didn't do a great job at knowing who was in the bed.  I didn't have many automations that relied on knowing which person was in bed, but there were some.

The sensor also drifted over time, and every few months I had to recalibrate it, which wasn't too much work, but was annoying to have to do.  Frankly, it was amazing it worked at all, considering the gauges were just under the foam mattress, with plenty of the weight on the mattress able to spread out past the gauges and go directly to the frame.

The dream that became Bed Weight Sensor v3

My dream was to build a sensor which put the gauges between the legs of the bed and the floor, so that 100% of the weight of the bed was going through them, and so both sides of the gauge were on a hard surface.  The one problem I had preventing me from doing that was that my bed frame has 12 legs.  While the gauges are cheap enough, I could never find an example for how to wire 12 of them together to get one measurement.  I even asked this question and the advice I got was just to set up 3 sensors, each with 4 gauges, and then just sum their weights in software.  That seemed silly and wasteful to me, but the main problem I had with it was that I had no way to calibrate each of those 3 sensors independently.

To calibrate one of these sensors, you just put two known weights on it, and record the reading each gives you.  Generally you use 0 as one weight, and then some other known weight for the other one, which would be "me" in this case.  So, with 3 of these, one for the left side, one for the right side, and one along the middle, it would be easy enough to get a 0 reading, but how was I supposed to know how much weight was on just the left hand side when I got into bed?  My best hope would be to try to lay across the bed and assume my weight would be evenly spread, but that just didn't seem like a path to success.

Instead, I was sure there was a way to just wire up all 12 gauges to give a single unified weight measurement.  I knew these gauges were just varying their resistance when a strain was applied, and the HX711 was just a high precision ADC that would measure the voltage across the combined resistors and convert that into a digital reading.  So, it seemed pretty clear that the resistors could be wired up to allow the HX711 to read all 12 the same as if they were a group of 4.  Certainly, the specific resistance for a given weight would change, but the calibration step would account for that.

Testing

4 weight gauges set up to measure the change in resistance when weights are put on them

I began just by confirming how these gauges worked with 4 arranged in the common Wheatstone bridge.  I used this StackExchange answer as a reference quite a bit while working on this, and the below image from it shows how one gauge works.

datasheet for a weight gauge

There are 3 wires coming out of each gauge, and there are two resistors inside the gauge.  One resistor is between the white and red wires and its resistance goes up when a weight is applied down on the gauge.  The other resistor is between the red and the black wire, and its resistance goes down as weight is applied down on the gauge.  This means the overall resistance across the entire gauge, from the white wire to the black wire is always the same (2 kΩ for these).  That's why these are often arranged in a Wheatstone bridge, using 4 gauges, each with their black and white wires connected together to form a square, with a voltage applied across 2 of the red wires and then the resulting voltage measured across the other two tells us what the overall reading is.  This image (also from that StackExchange post) shows an example.

example of a Wheatstone bridge with 4 gauges and a weight on 1 of them

This image shows what happens when gauge 4 (G4), on the right hand side has some weight applied to it, which causes one half of it to increase in resistance slightly and the other side of it to decrease slightly.  The other 3 gauges have no weight on them.  Without knowing much about electronics you can trace the path from the positive voltage at G1Red and see that it has less resistance to go through to get to the left hand side (labeled Sig+), and more resistance to get to the right hand side (Sig-).  This means we'd expect the voltage at Sig+ to go up relative to Sig-.  Then tracing up from the bottom, where the negative voltage is applied, we see that the path to Sig- is less resistance, so we'd expect the voltage at that spot to drop further relative to Sig+.  That is how we use a Wheatstone bridge to measure the change in these 4 gauges.

How can we do that for 12 gauges though?  Well if you connect two resistors in parallel you just get a new combined resistor with a lower overall resistance.  There's no reason simply connecting two of these Wheatstone bridges in parallel wouldn't work, you might just lose out on some precision.  I tested this by buying a bunch of strain gauges and wiring up two independent Wheatstone bridges, then connecting them in parallel (so the same points on the red wires from each bridge connected together), and taking a bunch of measurements with different weights.

needlessly complicated data collection

I paid special attention to making sure that the measured voltage was the same however the weight was spread between the two bridges.

the data in question

two Wheatstone bridges, labeled A and E with various weights and the combined voltage

Plotting that data showed exactly what I wanted to see, the voltage scaled linearlly as the weight increased.  It didn't matter which side the weight was applied to, if it was evenly balanced or all on one side or the other, the output voltage was the same.

The next thing to solve for was that I needed some sort of stands for the gauges.  The gauges have rivets sticking out of the bottom of them and the center portion needs to be able to flex slightly, so you have to mount them supported along the outer edge, with the middle unsupported.  This was another thing I didn't consider while making v2 of the bed sensor, and another reason I'm surprised it worked as well as it did.

These gauges are common, so it wasn't hard to find a 3d print model for them.  I used this one, which I slightly modified to remove the screw holes since I wouldn't be using them.  I got a bunch of those printed at my library, and spray painted them and the gauges black so they'd blend in better with my bed frame.

weight gauges in 3d printed trays

Once I had those, I wired up the full system for the first time.  All 12 gauges, on the plastic stands, wired up in 3 parallel Wheatstone bridges, all going to a single HX711 which was connected to an ESP32 running ESPHome configured to measure weight from the HX711.  Each of the 3 bridges had their 4 gauges under the corners of a scrap piece of plywood.  Then I essentially had 3 plywood scales all wired up together reporting a single weight across all of them.

testing the full setup

This worked very well, and it was immediately clear the overall idea would work fine.  The above picture shows 55 lbs total on the system, with 50 of those spread evenly between two Wheatstone bridges and the remaining 5 lbs on the third bridge.  I moved that weight around in different arrangements, and if it was evenly spread or focused on different platforms I still got accurate measurements, generally to within a 1/10th of a pound.  I also left it running overnight and came back the next day to see very little variation.

Putting the bed in Bed Weight Sensor

With the testing done it was now time to install this sensor on my bed.  I planned on making this install a bit more robust and more hidden than v2.  My bed frame has 3 groups of 4 legs each, with one set along the left side, one set along the right side, and one set up the middle.  Thinking about it as rows instead of columns, there's one row along the foot of the bed, two in the middle, and one up at the head.

soldering in the bedroom

This meant each Wheatstone bridge of 4 gauges would have G1 on the leg by the foot, G2 in the middle, G3 above that, and finally G4 up near the head.  Each gauge connected to its neighboring gauge, using either its black or white wire.  If the black wire went down, the white wire went up.  With the exception of G1 and G4, which "wrapped" around and went along the length of the bed from the foot to the head.  That was how each bridge was wired independently, and that gave a red wire coming up each leg.  Then all the similar red wires were connected together, along the rows.  So the red wire from G1 on the left was connect to the red wire from G1 in the middle and G1 on the right.  Each group of 3 red wires was connected to a fourth wire which connected it to a single spot on the HX711.

I don't have any close up pics of the wiring.  You might think it would be helpful, but it's just a confusing mess of wires running all over the bed frame.

I started this process around 4pm, hoping to have it done by dinner time.

At this point, let's stop and do some math.  Each bridge had 4 connections between the black and white wires.  However, only one of those was short enough that the wires from the gauge could be directly connected to each other.  The other 3 needed an extension between them, which doubled the number of connections, giving 7 connections.  For the red wires, these generally also needed to be extended to reach where they all landed in the center of the bed; only the two legs in the center didn't need that.  So this means there were (3 * 7) + 10 = 31 wire to wire solders that needed to be done.  And that only got me a mess of 12 wires in the center of the bed.

Another thing you might want to notice about my bed frame is that it's a dense mesh of bars such that you can't really walk between them, but also can't really walk on them.  This led to me constantly carefully stepping through the bars to get to the center of the frame, absolutely sure I would fall at some point and break my leg.  In retrospect, sliding around under the bed, on my back, might have been a better strategy, but that didn't occur to me until I was done.

The wires on the gauges were very tiny 28 gauge stranded wire, which made them a nightmare to work with.  Once I had all 12 wires at the center of the bed, all I had to do was land them all on some prototype board.  After all the wire to wire soldering I figured that just doing some through hole soldering would be simple, and it probably would have, it it wasn't being done in the center of the bed frame, bending over to butt level.  This is the part that for sure would have been easier to do from underneath.

Anyway, I finally finished up, and had managed to not forget a single heat shrink tube, either on the connections or the legs.  I did a pretty through testing of all the connections with a multimeter, measuring both connectivity, and the resistance between various points.  At this point I had a pretty ingrained sense for what resistance value I would expect from any given two points in the network of wires.  Much to my surprise, everything checked out.  So I powered up the ESP32 and started monitoring the measurements it was reporting.  After a basic calibration, I was delighted to see it was very accurate.  I could put the roll of masking tape on the frame and easily detect that.  I sat on different points along the frame and got very consistent weights for me.

I did have a little bit of cleaning up to do, but I purposely left things not fully done in case I had to make repairs.  I left the heat shrink unshrunk, and didn't tape up the wires as much as I had intended.  I also left the v2 sensor on the bed for now, as a fall back and to collect some parallel data.

the mostly finished v3 sensor with the v2 also still attached with the beige masking tape

I proudly stood and admired a job well done.  It was 2am, my wife was asleep in the guest room, and aside from a very quick cheesesteak break, I had worked continuously on this for 10 hours.

Also visible in the above picture is the old v2 sensor still attached.  All the beige masking tape was either temporary or related to the v2 sensor.

a closeup of a weight gauge, in its stand, with unshrunk heat tubing hiding the wires

This picture shows the nearly finished product, just without the tube shrunk or a bit more black masking tape I used to keep it tight to the leg.  With the overhang of the mattress, the gauges aren't visible at all.  One thing you can see is some electrical tape I used to hold the gauge to the leg.  I expected the gauges to slide on the floor and had some rubber I was prepared to put under them, but actually the leg slipping off them was the bigger problem.  I tried a few things, including hot glue, but electrical tape worked very well, and has held up for months at this point.

Where's the data?

I know you all came here for data, so without further ado.

comparison of v2 and v3 weight sensor data over about a day

This graph shows a comparison of the two sensors over the same day.  Yellow is v2 and blue is v3.  You can see how the yellow graph jumps all over the place, basically anytime someone rolls over.  While it is "clear" there is a pattern of one person, then both, then the other person, you can imagine how hard it would be to attempt to draw a horizontal line where we can say "below this weight is person 1, and above is person 2".  And in fact, the highest point of the yellow graph on the left (my wife) is higher than the lowest point on the right (me).  Now compare that to the blue line representing v3 and it's a stark difference.  The blue line is not perfectly flat, but the slight variation you can see amounts to a pound or two of difference.  The larger square wave pattern you can see (especially visible on the far right end, but also throughout the first phase on the left and in other spots) is about 10 lbs of variation, and is our cat coming and going.  Note the v2 sensor doesn't even acknowledge that.

closeup of v3 weight sensor data over a few hours showing the variation

Here is shorter period of time, from a different day, about 2 months later with no recalibration since the initial set up.  It's a very un-cherry picked period, there are better looking sections, but this is about the worst case I see.  There is a bit of variation there, but accounting for the cat, it's within a few pounds of the same measurement.  After what I had before, I am very happy with this setup.  I would love it if the sensor were accurate enough to monitor our weights over time, but this seem unlikely, even with much more precise gauges, just due to stuff like the headboard touching the wall, or the pillows, blankets, or clothing changing, sometimes during the night.

Either way, I consider this a smashing success, and removed the v2 sensor and taped up the lose wiring after about a week.  At this point the sensor has been working for over 2 months and seems as accurate as the first day.

Investigating an "evil" RJ45 dongle

https://lcamtuf.substack.com/p/investigating-an-evil-rj45-dongle

Apple 'Find My' network can be abused to steal keylogged passwords

I don't usually post random tech news type articles here, but this story is really interesting to me, and I didn't see it when it came out a few months ago.

To summarize:

Apple devices will respond to any seemingly valid bluetooth packet purporting to be from an AirTag and broadcast that out to their network.  If an attacker then says to the network "Hey have you seen my device with ID 1234?" the network will forward those packets along to them.  The key problem is that small amounts of arbitrary data can be inserted into those packets.  The example given the the POC attack is a hardware keylogger which uses the Find My network to transmit recorded keystrokes without using the devices network (which might be detected), or even if the device is totally offline.

To be clear, this has nothing to do with tracking the location of a device, and instead is just taking advantage of Apple's Find My network to transmit messages in places where it might not otherwise be possible.

https://www.bleepingcomputer.com/news/apple/apple-find-my-network-can-be-abused-to-steal-keylogged-passwords/

Writing down (and searching through) every UUID

https://eieio.games/blog/writing-down-every-uuid/

I think the site is great. I can quickly find my favorite UUIDs and star them or browse them all to find one that’s just right.

But having 5,316,911,983,139,663,491,615,228,241,121,378,304 2 possible values made it way harder than it needed to be to write them all down. I’m not sure why the authors of the UUID spec wanted to include so many bits!

So I think the final implementation here is pretty interesting. Let me tell you about it.

This problem had a few major challenges:

• Browsers do not want to render a window that is over a trillion trillion pixels high, so I needed to handle scrolling and rendering on my own 3
• I didn’t want to generate UUIDs in order from first to last. We all know the good UUIDs are in the middle! So I needed a way to generate UUIDs that ensured that I generated them all once.
• Since I was handling scrolling and rendering on my own, ctrl-f didn’t really work for search. I wanted to search through every UUID, not just the ones I could see! So I had to implement that too.

Sir, there's a cat in your mirror dimension

https://lcamtuf.substack.com/p/sir-theres-a-cat-in-your-mirror-dimension

 

Buy, Borrow, Die - Explained

Step 1A. Buy.

This stage of the planning really is that simple. Peter will purchase an asset for $50M. His "basis" in the asset is therefore $50M. Let's assume the asset appreciates at an annual rate of 8 percent. After 10 years, the asset now has a fair market value of $108M and Peter has a "built-in" (or "unrealized") capital gain of $58M.

If Peter sells the asset, it's a "realization" event and he'll be subject to income tax. The asset is a capital asset, and since Peter has owned the asset for more than 1 year, he'd receive long-term capital gain treatment and pay income tax at preferential rates if he sold it. Nevertheless, Peter's long-term capital gain rate would be 20 percent, he'd be subject to the net investment income tax of 3.8 percent, and Peter lives in Quahog which has a 5 percent income tax rate.

So, if Peter were to sell the asset and cash in on his gain, he'd have a total tax liability of around $17M, and his after-tax proceeds would be $91M.

Peter's buddy Joe overheard some of his cop buddies talking about how the ultrawealthy never pay taxes because they implement "buy, borrow, die," and he shares the idea with Peter. Peter decides to look into it.

Step 2A. Borrow.

Peter goes to the big city and hires a private wealth attorney, who connects him with an investment banker at Quahog Sachs. The investment bank might give Peter a loan or line of credit of up to $97M (a "loan to value" ratio of 90 percent) based on several conditions, including that the loan/line of credit is secured by the asset. Now Peter has $97M of cash to use as he pleases, and he's paid no taxes.

Step 3A. Die.

Peter has been living off these asset-backed loans/lines of credits and his asset has continued appreciating in value. Let's say 35 years have passed. With an annual rate of return of 8 percent, the asset now has a fair market value of $740M.

Then Peter dies. When Peter dies, the basis of the asset is "adjusted" to the asset's fair market value on Peter's date of death. In other words, Peter's basis of $50M in the asset is adjusted to $740M.

Peter's estate can now sell the asset tax free, because "gain" is computed by subtracting adjusted basis from the sales proceeds ($740M sales proceeds less $740M adjusted basis equals $0 gain).

Peter's estate can use the cash to pay back the loans/lines of credits. He's paid no income tax and his beneficiaries can now use the cash to buy assets and begin the "buy, borrow, die" cycle themselves.

https://old.reddit.com/r/BuyBorrowDieExplained/comments/1f26rsf/buy_borrow_die_explained/

Using Snapraid for drive redundancy and quasi-backups

Intro

I recently added some hard drives to my home media server, and have finally decided to add some redundancy there.  I've long resisted using RAID, because it only protects against one very specific form of data loss (drive failure), while ignoring things like file corruption, or just user error.  On the other hand, mirroring data to another drive is such a waste of disk space that I could never bring myself to do it, especially for stuff I could ultimately redownload if I needed to.  I'll explain my solution, although if you've read the title you may already have guessed what it is.  First though, I'd like to go over what RAID is and why I don't like it.

What is RAID?

If you know what RAID is, you can skip over this section.  Alternatively you can read this better overview.

RAID stands for Redundant Array of Inexpensive Disks.  The idea being you buy cheaper drives, which may be more likely to fail, but then have some redundancy between them, so that if any do fail you don't lose data.

There are many types of RAID setups each with different numbers, and we'll go over a few of them.  All RAID setups share some things like all your drives are combined to look like one large drive.  There are four criteria that vary between the different RAID types to consider when thinking about different RAID setups:

1.   How many drives can I lose before I lose data?
2.   Space efficiency (how many drives are wasted on redundancy?)
3.   Read performance
4.   Write performance

RAID 1

RAID 1 is the simplest; it is just 2 drives where they each hold a copy of your data.  In RAID 1 if you have two 12 TB drives it would look like you just have one 12 TB drive, but there would be a real time copy of it on the second drive.  If you have more than 2 drives the same concept applies, where half the drives are devoted to mirroring the other drives.

In RAID 1 you can lose half your drives before you lose data, however, if you have more than 2 drives that will depend on which drives fail, since they are paired up.  If you had a 6 drive RAID 1 setup, you could lose data with 2 drive failures, or be fine with 3 failures, depending on which drives fail (but you'd always be safe for 1 drive failure).

Lastly, the read and write performance of RAID 1 is interesting.  While the write speed isn't affected much, the read speed is roughly doubled.  This is because half of each file can be read from each drive.  Which leads us to our next RAID level...

RAID 0

RAID 0 is similar to RAID 1, except only half of each file is stored on each drive.  This means that no single drive has any full file.  The benefit of this is that you get roughly double read and write speed, since you are only reading and writing half the file on any single drive.  RAID 0 also doesn't lose any drives to redundancy.  If you have two 12 TB drives, your RAID 0 system will have 24 TB available.

If you haven't figured it out already, there is one downside to RAID 0.  You will lose all your data if you lose a single drive.  RAID 0 has no redundancy (hence why it's RAID "0"), and in fact, it's worse than not using any RAID, because a single drive failure will destroy all your data on both drives, rather than just one.

RAID 0 isn't typically used by itself, but often combined with RAID 1 to form various permutations of RAID 10.  RAID 10 is just 4 drives in some combination of RAID 1 and 0.  I'm going to skip over giving more details because this background section is already grown much too long.

RAID 5

This is where RAID gets interesting.  In RAID 5 you have 3 or more drives, and you lose exactly 1 of those to parity data.  We haven't mentioned "parity" yet, but the idea is that we can calculate a checksum based on the combined data on the rest of the drives, and that checksum is enough to recover the data if one of those drives fails.

With RAID 5 you lose one drive to redundancy, which means it becomes more efficient as you get more drives (with 2 drives it would be the same efficiency as RAID 1, while having worst performance).  The read performance of RAID 5 is the same as no RAID, but the write speed takes a hit because the parity needs to be calculated in real time for everything you write to the disk.

Another thing to consider is that you can safely handle a single drive failure, but as the number of drives you have goes up, the odds of a second drive failing before you are able to recover the first drive goes up.  Therefore, it's a balancing act of how many drives you "waste" on parity vs how many you have for data.

RAID 6

RAID 6 is just RAID 5, but with two parity drives instead of one.  Everything I said about RAID 5 applies, just with two drives for redundancy vs one.  This makes more sense as you get more drives, but the trade off is losing more usable space.  The key thing here is that you can lose the same number of drives as you have parity drives.  It doesn't matter if the lost drives are data drives or parity, or any combination of them.  If you have 2 parity drives, and you lose 2 drives, you will be able to recover.

This page is a good summary of the RAID levels, including some I didn't talk about.

What is parity?

I want to explain what I mean by "parity" when discussing redundancy, mainly because I think it's a pretty cool concept.  Again, feel free to skip this section if you understand what parity is.

In its simplest form, parity is just combining the bits on the drive using the XOR function.  XOR stands for exclusive OR.  I'm going to restrain myself from explaining XOR in depth, and just say for our purposes, XOR asks the question "Are there an odd number of 1 bits?".  So if you are looking at 3 bits, and they are 1 0 1, then the answer to that question is no (there are two 1s, and two is even).  Therefore XOR(1 0 1) = 0 and then XOR(0 0 1) = 1 because there is one 1, and one is odd.

Now here's the really cool thing about XOR.  If you take the output of XOR and store it along with the inputs, you can remove any one of those inputs and the XOR on the remaining inputs along with the output will equal the missing input.  For example XOR(1 0 1) = 0, if we lose the first bit there, and instead have X 0 1, we can just take the XOR of what we have left, plus the output (which was 0) to get XOR(0 1 0) = 1.  I know that is confusing, but as another example: XOR(1 1 1) = 1 (three 1s, three is odd), so we store 1 1 1 1 (the last bit is the parity bit that we got from the XOR), and then if we lose any of those 1s, we can calculate XOR of what remains and we know XOR(1 1 1) = 1.  Another quick example:
XOR(1 0 0) = 1

Store that combined as 1 0 0 1

Then lose one bit: 1 X 0 1

Calculate XOR of what remains: XOR(1 0 1) = 0

Replace the lost bit above with the 0 we just got: 1 X 0 1 -> 1 0 0 1

I made a spreadsheet to show this with full 8 bit bytes.  This is representing 3 drives (d1 through d3), each with 8 bits of data on them (lettered A through H), and then a parity drive (p1) with the calculated XOR of each column.  If you look at any column and count how many 1s there are (excluding the final row, which is parity), then the final row should have a 1 if there were an odd number of 1s and a 0 if there were an even number of 1s.  So in the below image, column A has two 1s, so the parity bit is 0.  Column C has one 1, and so the parity bit is 1.

The first table in the image shows the three data drives and the parity drive all full.  The middle table shows d2 having failed, highlighted in red, now replaced and empty.  The final table shows d2 after it was replaced by calculating the XOR of the other white rows.  In the first and last table the gray row is the result of XORing the 3 white rows.

This scales up to any number of drives, and we always only lose a single drive to parity.  Here we have 7 data drives and only 1 parity drive, and still can lose any drive and then recover.

You may now be asking how RAID 6 works with 2 simultaneous parity drives, where you can lose any 2 drives and then recover.  The answer is the math becomes much more complex, and I can't explain it, but if you want to read through it, this post does a good job of explaining it.

Why don't I like RAID?

We've covered a lot of ground for me to be able to answer this question.  First off there are some practical concerns I haven't really touched on.  RAID is annoying to set up, and requires identically sized drives.  This is fine for a data center that is going to have many servers each running RAID, where the drives only need to match in a single server, but for the home user, whose storage is going to grow organically, it's very annoying to have to replace all your drives every time you want to increase your storage.

The bigger problem with RAID though, is that it's not a backup.  I mentioned this above, but RAID protects you against your drive failing, and nothing else.  It does not protect against software corruption, user error, or your house burning down.  If you delete a file from a RAID server, and then realize that was a mistake, it's too late, it and the redundant copy are gone.  If a file gets corrupted when you save it, the corruption is also instantly copied to the parity.

What about Unraid?

Unraid is a paid operating system, which aims to make RAID much more user friendly.  Despite its name, Unraid really is just RAID, but with a lot of the annoying parts eliminated.  It handles the problem of having a mix of drive sizes, and allows adding new drives to an existing array.

I strongly considered Unraid for my home server.  The main reasons I didn't go with it though, are:

1. It isn't cheap.  It's either $50/year or $250 for a lifetime license.
2. It requires starting with empty drives.  While you can add a new blank drive to an existing Unraid array, if you are starting out with full drives you will have to buy enough empty drives to start a new empty array, and then move stuff onto them.  This also means all the configuration I have set up on my server (Home Assistant, Plex, etc) would be lost, or at least would require me to migrate it all over, and deal with the downtime while I did that.
3. Finally, Unraid is not any more of a backup than RAID is.  As far as parity goes, Unraid is the same as RAID 5, 6, etc depending on how many parity drives you use.  But if you corrupt a file, you still have no backup of it.

Snapraid to the rescue

I discovered something called Snapraid, and instantly knew it was the best choice for me.  Snapraid is an open source command line tool which calculates parity for any number drives and stores it as a single parity file on another drive.

Snapraid has one main catch, which is both a pro and con: It only runs on demand, meaning that when you make changes, they aren't added to the parity until you actually run the snapraid command to recalculate the parity.  This however, means it serves as a sort-of-backup.  Things like corrupted files or mistakenly deleted files can be recovered, as long as you discover the problem before running the parity check.

Snapraid works best where you have a large amount of media that rarely changes.  This is my exact use case, so it works very well for me.

The drawbacks of Snapraid are:

1. It doesn't combine drives into one filesystem (although it often combined with mergerfs, which does exactly that).
2. It's a command line tool (although really pretty easy to use).
3. Updates only happen on demand, which means anything you write will not be protected until you recalculate the parity.
4. Unintuitively, updating a file will cause other files to be unprotected until you rerun the parity command.  I'll explain this one more later, it's the biggest catch.

The benefits of Snapraid are:

1. It's a free open source command line tool.  I realize I had command line as a drawback above, but it will be a pro or a con depending on your point of view.
2. It can be added to an existing system, very easily.  All you need is enough space for the parity file, which means you need one empty drive, which is at least as large as your largest other drive.
3. You can use any mix of drives, and add and remove them from the array easily.
4. You can use any number of parity drives, to cover whatever level of risk multiple drive failures you're ok with.  Their FAQ includes a good guideline on how many parity drives you should use.
   
5. There is no performance overhead when you write (or read) files.  It's not running at all aside from whenever you schedule it to run.  It only takes a few minutes to run after the first time.
6. It has features to help protect against random errors that can happen in RAM, and for which people often use ECC memory with RAID servers to protect against.  Namely, it can run the parity calculations twice for each file, and has a "scrub" feature where it can double check the existing parity data for a percentage of your total data.  The scrub will also detect corruption that occurs from a failing HDD.
   
7. The fact that it only runs on demand means you can recover from mistakes and corruption as long as you notice before the next run.

I mentioned Snapraid is pretty easy to use.  You set up a .conf file, telling it what your data drives are, and which drive to store parity on.  You can also exclude files or folders and tell it to ignore them, which is a good option for frequently changing data (although obviously consider some other backup strategy for that data).  Once it's set up, you run it with Snapraid sync, and your setup could be as simple as running that via a cronjob every night.

Snapraid AIO Script

There is a nice bash script that makes running Snapraid much easier.  You can read through the readme, but some main features of it are an email report every time it runs, as well as a lot of configuration on when Snapraid should not run.  For example, if it detects a large number of files have changed or been deleted, you can have it email you a warning and not run.  This would give you a chance to recover those files if it turned out they had been deleted by accident.

I'd recommend checking out that script if you do use Snapraid.  From what I can see, the contributors to it are all really cool people.

Biggest gotcha to Snapraid

I mentioned this above, but it's worth reiterating, as it's the biggest potential problem with Snapraid.  If you either delete or modify a file, that file will of course not be protected until you run the snapraid sync command again.  However, other random files, on your other drives will also lose their protection until you run the command again.  To understand why, go up and review those parity diagrams from above.  By modifying a file on drive 1, you are changing the parity calculation for the same spot on drive 2, drive 3, etc.  Modify the bit in column A of d1, and you can no longer recover the bits in column A or d2 or d3.  So, if you delete a 1 MB file on drive 1, you now have a 1 MB hole on drive 2, 3, etc.

There are two ways to mitigate this.  First, don't delete and modify things often.  And when you know things are going to be modified and deleted schedule that to happen shortly before the sync command is scheduled to run (but not so shortly before that it's still running when the sync command starts).  Second, when you need to delete something, you can move it to a snapraid excluded folder.  Then the file still exists, and if a drive fails you can move it back to its prior location and be able to recover everything.  Once you run the sync command snapraid will calculate a new parity, without the excluded files, and you can delete them whenever you want.

Odds and ends

This, predictably, grew to be a quite a large post.  Still there are some random additional thoughts I wanted to include, so here they are.

I've written before how I'm backing up my documents to AWS S3 Glacier.  This is still my strategy for the things I care the most about.  It gives me daily snapshots and is quite cheap. My AWS bill is currently $0.25/month, and I have years of backups I haven't deleted.

I also have a bunch of systems that I want to be able to recover if their root drive fails.  My strategy for these is to mirror some of the folders nightly to a HDD in the server.  I exclude that folder from Snapraid, since they are already a mirror from their primary locations.  I'm using rsync to sync a few important folders from my desktop and the server itself.  I also have a bunch of Pis that I want to be prepared if their SD card fails.  I wrote before about doing that with rsync, but I'm now using this script to make a full image of the entire SD card to a NFS drive.  That image can be mounted and examined.  It works pretty well, although it's not super user friendly.  I also found this script, which might be better, but I haven't tried it.

I discovered the site healthchecks.io which I'm a really big fan of.  It helps you keep track of all your backup scripts to detect if any start to silently fail.  You can create up to 20 checks for free, and for each you get a unique URL, which you then hit as the final step of your backup script and it'll register as a run.  Then you can configure the site with how often those scripts should run and it'll alert you if they miss a check in.  I have a slightly more complex pattern I've been following which pings the site at the start and end of the script, so it can monitor how long they run for, and will be alerted right away if they fail.  You can see an example of that in my backup script here.

Finally, I used the site serverpartdeals.com to buy my additional drives.  They are pretty well regarded around the internet, and they have good prices on recertified drives.  I've never used recertified drives before, but I figure if I'm going to have them protected with parity anyway, I might as well buy more, cheaper, drives and just increase parity if I feel that is too risky.

Youtube TV Channels

https://ytch.xyz/

Replicates the TV experience with channels of always playing Youtube videos.

Here are the themes of the channels:
Channel 1: Science and Technology
Channel 2: Travel and Events
Channel 3: Food
Channel 4: Architecture
Channel 5: Film and Animation
Channel 6: Documentaries
Channel 7: Comedy
Channel 8: Music
Channel 9: Autos and Vehicles
Channel 10: News and Politics
Channel 11: UFC
Channel 12: Podcasts/Interviews/Talk Shows

A wonderful coincidence or an expected connection: why π² ≈ g.

https://roitman.io/blog/91

Let's start by taking a close look at the right side. The value 9.81 is in m/s². But these are far from the only units of measurement. If you express this value in any other units, the magic immediately disappears. So, this is no coincidence—let's dig deeper into the meters and seconds.

What exactly is a "meter," and how could it be related to π? At first glance, not at all. According to Wikipedia, a "meter is the distance light travels in a vacuum during a time interval of 1/299,792,458 seconds." Great, now we have seconds involved—good! But there's still nothing about π.

Wait a minute, why exactly 1/299,792,458? Why not, for example, 1/300? Where did this number come from in the first place? It seems we need to delve into the history of the unit of length itself to understand this better.