I’m relatively new to FreeNAS, which I somehow missed out on for many years. Now that I found it I’ve moved over to using it as much as possible for my storage needs.
Based on a number articles it seems as though it’s easy to add storage any time you like. This is true, BUT there are some HUGE caveats to this. The FreeNAS GUI makes it somewhat hard to make a mistake, but if you think you know what you’re doing you can override these features and actually put your data at risk.
I started out with three drives and created a RAIDZ-1 volume. For those unaware, this means that if any ONE drive fails then all the data will still be safe. To be even more safe, it’s recommended to use a RAIDZ-2 configuration and/or backup your data to yet another location. I decided on the latter, and have backed up my data to a second FreeNAS server (via snapshot replication).
So far I’ve followed generally-accepted good data integrity practices here, but now for my mistake. I decided to buy another drive of the same size, and I wanted to add that to my server to increase the space available. The FreeNAS GUI wouldn’t let me add it with the basic Volume Manager utility, so I went to advanced mode and added it to the ZVOL. What I ended up with was this:
The three original disks (da0-da2) maintain their RAIDZ-1 state, but now a portion of the data is shared with the striped disk da3. The striped disk has NO redundancy.
Note that the cache drive ada0 is a SSD providing a L2ARC to the volume. Cache is usually a single striped drive, and if the data is lost there then no harm will be done to the system.
What wasn’t explained very well (or I failed to read earlier), is that RAIDZ volumes can be added to, but each group of disks, once configured, remain in the configuration that they were initially created with. What I had done was EXPANDED the storage available, but STRIPED my original raidz1-0 array with a single disk. Because the expanded storage is a SINGLE disk, if that one disk fails then the entire volume will be destroyed.
So if any single one of da0-da2 fails then the array continues to operate in a degraded state. But if da3 were to fail, then the entire array goes down! Kind of kills the point of RAID, right?
The reason for this is that RAIDZ is a high-performance datacenter-quality product. Datacenters aren’t usually adding single drives, but rather entire arrays of disks when they need to increase storage. If you decide to use FreeNAS and RAIDZ, then you must keep this in mind. It does take more planning (and money for additional disks), but that’s the tradeoff of using this tool.
So how do I solve it? Well right now da3 needs some redundancy. I ordered another drive of the same size, which will then be a mirror for this disk. Note that the mirror must be added via the console, not through the GUI. There is no GUI-based way to convert a striped disk to a mirrored disk, even though the FreeBSD tools support it.
So then I will have a RAIDZ-1 array + expanded storage of a mirrored array, all sharing data of VOL0. This still has some limitations on redundancy though (assuming disk 5 is named da4):
- If any one of da0-2 fails, then the array is OK.
- If any one of da3-4 fails, then the array is OK.
- If da3 and da4 fail then the array is dead (the mirrored array has completely failed).
- If any two of da0-2 fails, then the array is dead (the raid-z1 array has completely failed).
The likelihood of this happening is low, plus all my data is still backed up on a second server.
Another option (since all my data is backed up) would be to destroy the array, and rebuild it with all 5 disks (da0-3 + the one that will arrive soon) in a RAIDZ-2 configuration. That way if up to TWO of ANY of the disks fails then the array would continue to operate.
I haven’t yet decided if I will do this, but for now adding the mirrored disk fixes my mistake. Don’t do what I did.