I’m very very new to NAS, so apologize for these obvious questions.
I have some 6 old school USB drives (4 x 1TB and 2 x 2TB) with the USB 2 interface (and mostly formatted as HFS+) that i’ve used for backups etc. Though sadly there’s loads of duplication and there’re mostly at capacity (about 7TB total).
Ideally, I would like “reuse” these drives to become a NAS (i have a suitable unused server for this). But after reading this some questions arise.
USB drives can’ be attached/pooled/shared etc without having a btrfs partition (yere yere i know it’s not a real partition in the old school sense), and that said partition needs to be blank/erased. (i.e i cant simple plug my HFS+ USB drive and its magically on the network). Yes I have also read about importing, but that’s just “copying” the data to the NAS drive/share anyways right?
To benefit from a NAS “pool” it would make sense (esp speed) to remove the 3.5" drives and wire the sata connections directly (bypassing the restrictive USB interface that’s slower) AND pair the drives (similar sizes)… though I’m going to lose my “data” and halve my available “space”. (in which case i’m going to need a LARGE intermediary drive to “hold” the data, before “transferring back” to the smaller sized capacity NAS share?).
I take it that pool/share can be expanded? i.e add “pairs of drives” to increase the overall capacity of said network share?
Let’s assume I have a left over intermediary drive after doing the above, could this USB drive become an automated offline spare? (the docs I have read to date suggest so, but it wasn’t newbie friendly or how to do it). Can the offline spare be a format other than Btrfs
Future? what happens if say Btrfs becomes old school (aka another NTFS… yes we still love you NTFS), would one need to copy all the data off again to a different formatted HDD and do this all again?
I’ve also played with TrueNAS… but which is honestly best for me?
How would one “upgrade” the drives? lets say I have 2 x 1TB in RAID, and want to replace with 2 x 2TB in RAID?.. (copy off and back again? oh no please no!)
@set_phasers_to_stun Welcome to the Rockstor community
I’ll have a go at your questions in order hopefully
1a.
Correct for pooled, if you switch “partition” to “format/filesystem”. Rockstor, via he Web-UI, only deals with btrfs which has a raid capability to do the pooling. You can attach any filesystem or device supported by linux (our base OS) which is basically all of them. You can also share those attached devices via the underlying linux. But the Web-UI only guides your through, and understands btrfs. This helps to keep down the complexity of development and user experience as btrfs is the most flexible filesystem/device-manager given it’s capability to change it’s raid level on the fly etc.
1b.
As in the answer to 1a. you can definitely attach pretty much any storage device to Rockstor and have the underlying linux access that data. The ‘import’ term is within the Web-UI only and as you indicate is btrfs only. But linux itself can read/access/mount/copy-from pretty much all filesystem. But you will have to use the command line to do this.
2a.
Definitely. USB is a bad bus for storage devices, especially those associated with redundancy/robustness. It has a habit of resetting and will often take down many devices in one go. Avoid USB for storage at almost all costs. And as you say, speed is often not on it’s side. And you mention USB 2.0 as these devices current interface. That’s basically a bit slow, but doable if that is all you have.
2b.
Btrfs can pair dis-similar sized drives almost without constraint. But if you pair say btrfs raid1 (2 copies across 2 devices irispective of total device count) then the limit is set to the smallest device initially as otherwise who would it maintain the ‘across 2 devices’ bit. But if you had 3 device then it would be limited to the second smallest device as long as either of the other 2 has free space. So in short, as long as there are 2 drives in the pool with sufficient space then a save is possible. But if you are using the non redundant btrfs raid single then the total space is whatever is free on the combination of all pool members. But no redundancy so if any one device dies then the pool is also dead.
A ‘passage’ to migrating to our native btrfs is to setup an initial pool on the NAS after first wiping the initial members and creating the start of your eventual pool. Then copy over the network from say your desktop/laptop where you have attached each of the existing drives in turn. Once each has had it’s contents copied over the network to the initial pool members, you can add that drive (after wiping it) to the NAS pool. And move onto the next drive. Btrfs, in all it’s raid levels, supports on-line ReRaid. You can add or remove device (space permitting in the later) to a live pool. You can also, as part of this ReRaid capability change the raid level on the fly also. These processes can take a long time but the pool remains usable during these modifications, but will have reduced performance as it’s members and support hardware are already busy doing the re-suffle.
Yes, and no. “pairs of drives” are not required here. That is likely a concept from ZFS’s vdev arrangement where folks create virtual devices to make up a pool and maintain some flexibility. Btrfs is far more flexible and you can add or remove any number of drives from a pool within the constraints of a pools minimum count (btrfs raid level) and the data sitting on the pool at the time of course. And as stated before you can ‘live’ change the raid level if required to alter the minimum drive count required. Interestingly this was a design goal of ZFS originally as well but was lost along the way. Our Shares are currently un-constrained and so all shares (btrfs subvolumes) can grow to the size of the Pool (btrfs volume). We don’t yet enforce quotas to constrain this. In short you can start with a single drive using btrfs single raid level. Populate it to say 70% to be safe and add another drive to it. You can, depending on the hardware, data size, time etc just add another drive and stick with single. Bad idea but if it has to happen then so bit it. Or you could also change the raid level while adding the second drive (much longer as auto runs a full balance. This capability may allow you to bootstrap within your limited storage constraints. Then in time as drive count and time permits you can move to a redundant raid level once you have enough drives carrying sufficiently little data to permit this move.
No, not in the sense of a live spare such as “automated” suggests. Btrfs does not yet have hot-spare capability automated. But is is a planned capability. But the drive could sit there attached ready to jump in by hand as it were. Again with the USB proviso, don’t use it for storage if you can help it. Btrfs drive pooling (the device management raid bit) only works within btrfs. If you want pooling at a higher level, say geographically irispective of filesystem then you will need a clustering solution. We plan to add GlusterFS for accomodate this in time. But that is a long term plan for us and we will maintain our btrfs filesystem for the individual ‘bricks’. But XFS is another fs recommended for GlusterFS. And if you are considering such cluster file systems then you would be advised to first be familiar with NAS basics and the base capabilities of the underlying fs choices first. So from your intro this is a step too far at least for now. But it, in turn, can provide hot shares but they would be hot spare systems not drives.
Btrfs is actually a future file-system in it’s own right actually. The current XFS author has stated this actually. Hence it’s current ‘shiny new’ known issues reputation such as “avoid parity raid 5/6” and the like, although it can have differing raid levels for data and metadata that can mostly address such concerns. But to address your question head on: “Change is the only constant”. And btrfs lead developers are currently working on the next ‘format’ of btrfs to address the edges they have found in their exploration of the current format. So it has some life left in it yet. NTFS was never hailed as great I think you will find. Btrfs has a similar but newer heritage to ZFS and is doing what ZFS wanted to do, block pointer re-write. We are in good hands and have some head way yet. Also note that we are in the throws of the first transition of the format in the form of space cache v2 which is the default if you create a pool with backported kernels such as we introduce for ‘special’ purposes in the following how-to:
But don’t do that unless you have to, or you fancy it and understand fully the potential ramifications / risks. Also note, in the same thread, one can update ‘in-place’ an older btrfs space-cache pool to space-cache-v2 via some mount commands and a reboot. We haven’t covered this in our docs yet. I am unsure if the same in-place transition will be offered for the next format change.
ZFS is unarguable more mature: it’s just been developed for many years longer. But it is, and always will be, less flexible due to the removal of the block-pointer re-write capability that they had originally wanted to keep but development and capability pressures caused it to be lost. The very latest version has added some flexibility but it is in a very different and far more constrained area than brfs maintains. On the other hand ZFS has vdevs. I’d quite like them myself they also have an arguably more robust device management, but again there is time for this to develop in btrfs and work is always underway. But their management is their-own. Btrfs shares a lot more device management and low level stuff with other linux filesystems and so has it’s own flexibility/compatibility on that front. So horses for courses but we are btrfs and designed from the ground up with that in mind. Hence our flexibility re ReRaid capability. TrueNAS is likewise but for ZFS. Which is better for you is almost entirely a subjective thing. I used FreeNAS for several years myself, before btrfs and Rockstor, but I prefer the flexibility and clean licensing of btrfs (in kernel) rather than the state of even openZFS. The very next thing you need ‘in-kernel’ bar device access itself (hardware) is data access: i.e. the filesystem.
See previous answers re btrfs flexibility and zfs differences. Btrfs can morph live, while in-play, from any one raid level to any other if the data size, device count, and target pool spec allow this. It’s pretty magic that way and makes it super convenient for such tight corners as yourself and almost all other non enterprise users. But is also servers facebook OK so it has some range . But again to answer head-on, you don’t have to do pairs such as you have likely picked up from the vdev work around common in ZFS. You can add on drive to the original 2 X 1 TB. You can even start out, as mentioned earlier, with a single drive. No redundancy but you will still have checksumming, unless you turn that off of course. But starting with rudundancy is always best if you can. You can then later add another drive and so on. And when you reach say 4 or 5 drives you can convert to raid10 if need be. For large drive counts (i.e. 10+ depending on reliability) where you would be best recommended to adopt a raid with a 2 drive failure capability you should consider say raid6 data raid1c4 metadata. Btrfs can have differing raid levels for each simultaneously. We don’t support this yet but with sufficient knowledge it can and is done with the current Rockstor stable release. But you will need the backported kernel arrangeemnt for that and avoid, for now, some Web-UI stuff as it will re-assert similarity of data & metadata raid levels. But if you can handle the risk (backups) this can always be undone via the command line.
Wow, that was a lot of questions. Our docs have most of these scattered around but thought I’d have a quick go here since you asked. Also we do plan on having a btrfs-primer for such things but the short of it is: it’s not like the other filesystems you may already know :). More of the future than of the past. Plus the partition thing is important by the way as we drop that completely on the data pool members. Btrfs is able to deal with the raw drive. FreeNAS defaults to creating saw partitions on each Pool member to allow for slight drive size variation. Each member must be exactly the same size. This is simply not a thing in btrfs and so we can use drives raw and throw out decades of legacy ‘standards’ that arn’t. Less complexity and more in-house can only be good. That’s how I see it of course.
As are we all with such magic afoot. My next long term newbie adventure (bar regular maintenance and development) is to extend Rockstor to have build-in GlusterFS to avail the next generation of meta magic storage too regular users. We, purposefully, stick to the core capabilities of btrfs to keep our ‘appliance’ simple and mostly applicable. There is tones of stuff btrfs can already do that we don’t cover, but we could if it could be incorporated without the cost of user complexity. This is harder than it seems but the mixed raid thing for data/metadata is next on our list on that front (at least to display it in the Web-UI first), bar the ever ongoing maintenance and development / bug fixing of course.
Also fewer focused questions can help in getting focused answers that can, in-turn, lead to their own discoveries. In short knowing the ReRaid capability in btrfs could have answered a few of your own questions in one:
“Pool Resize/ReRaid”: https://rockstor.com/docs/interface/storage/pools-btrfs.html#poolresize
A convenience feature of btrfs Pool management is the ability to add or remove disks, and change redundancy profiles, while still using the Pool. The persistence of a pool’s accessibility is otherwise known as it’s ‘online’ state. And so these changes are referenced as it’s online capabilities.
Wow @phillxnet! Many thanks for your detailed response. Much appreciated
Might take me a few days to digest this fully.
1a/b USB drives - you can also share those attached devices via the underlying linux
Do you mean that in the case of HFS+/APFS (say) that they can be “shared” through Rockstor, but just not facilitated through the UI per se?
I assume I would need to follow something like this ?
2a. NAS Pooling
2b. Migration
Haters gunna hate… but here goes… my NAS is wireless (TPLink AP in client mode hooking into eero 6 mesh wifi)!! (yes yes i know) … so network copies (and well the NAS in general) will be less performant.
3.Pooling expansion
4.USB offline
Way too much info for this 8-bit brain. I’d imagine I would need to setup some sort of cron/type job to do this say scheduled at midnight using the shell/console instead?.
5.NTFS
Ha, wasn’t saying NTFS was good/bad :-). More reflecting on the good-old-days of Windows NT (life was simpler).
6.ZFS
7.Upgrading drives
Think I just went blue-screen-of-death for a moment.
Let me ask a different way (though you might have kinda answered above). Lets assume Rockstor setup with 2 drives (raid1) (i.e 2 x 1GB HDD). Now lets further assume that the PC cannot facilitate anymore HDD’s (i.e at capacity).
Could you remove 1 x 1Gb drive, and replace with a 2Gb drive. wait for the data/clone/check to finish. Then remove the “other” 1Gb drive, and again wait for it to finish… then just up the original pool to now be 2Gb?
Would the same work if say pool being 4 x 1Gb drives (configured as pairs/RAID1) and you wanted/changed to 2 x 1GB and 2 x 2GB.
I will also take a re-read of Pool Resize/ReRaid, which might very well answer the above.
Apologise that so many q’s in one. They just kinda snowballed when I started writing.
@set_phasers_to_stun Hello again, and your welcome.
The problem we have here is our missing btrfs on-boarding / primer docs. The info is kind of all over the place where it’s needed but we could also do with a small succinct primer to disavou folks of their prior misconceptions base on all other software raid setups. Btrfs does raid at what they call “chunk” level with an eye to the device spread where required. Some data or indeed metadata can be in one raid level while another bit of data or metadata can be in another. This is normal for instance while transiting from one raid level to another. A “chunk” is 1 GB and if for example it’s personality is btrfs raid1 then it will have a partner/twin “chunk” on one other drive: to meet the btrfs raid1 profile/personality of storing on 2 independent (from the newer acronym of raid) devices simultaneously. But if for example a chunk is btrfs single; then there is only a single copy of it: no redundancy. Both single and raid1 pools can have as many drives as you like, but the individual chunk profiles
Yes that’s pretty much it. We are a generic JeOS (Just enought Operating System) variant of openSUSE Leap 15.3. But note that we enforce, and re-enforce on Web-UI re-config, Samba (the network share bit) shares/config. This is not a task for the NAS / linux newbie however. But I just wanted to be clear for all those reading as this is an important element of our make-up. We make stuff easy that is otherwise hard, but we entirely allow the hard command line stuff. However that can, if you know enough to be dangerous :), interfere with the stuff we normally make easy. So very much better to go with the flow and move stuff to the native format. But we in no way block anything via the command line, you may just end up upsetting/confusing the Web-UI if you do this however.
Note that Rockstor’s Web-UI doesn’t support Wifi. But again the command line does. Again this is something of an advanced topic but to get you going you would need to use nmcli (Network Manager Command Line Interface) and potentially install some possibly missing wifi config stuff. Far easier actually to re-locate your NAS so it has an ethernet connection. Or even ‘fake’ the ethernet connection by using a wifi-to-ethernet bridge so Rockstor (openSUSE and NetworkManager) see only ethernet.
That’s over complicating it. I’d just start a wholesale copy via command line and wait till it’s done. Or copy/organise as you go via the client laptop/desktop connection. This will be slower but a lot easier for a newer users of NAS linux and avoid you having to manage mounts etc.
If the PC in this case is the NAS then if you only have 2 slots you will be very limited.
Without going into a degraded mode no. And you want to, at almost all costs, avoid degraded mounts. A degraded mount, is where you are running a pool (btrfs volume) below it’s minimum drive count. Btrfs raid1 needs at least 2 dives (the 2 independant devices to stash the twin copies of the raid1 chunks). So removing a drive, although possible, is highly ill advised and will threaten the data security. Better to start out with a non degraded single larger drive until you can free a partner drive to accompany it so that you can move the entirely of that single pool, and it’s data (which can remember have 1 or more members) over to a more resilient/redundant btrfs raid1. Then as you copy more info over to that pool you can in turn add more and more of the drives to that raid1 pool. It will only ever store 2 copies of each chunk on 2 of the however many pool members at any one time. This way you never ‘break’ a pool only expand a healthy single pool into a raid1 pool when you are able to free up enough drives/space to move them over. It’s a drag but entirely doable. The main risk is when you have the initial single pool before you are able to ReRaid it to be raid1, but it is less risky and less technically challenging than splitting/breaking/degrading a raid1. I think the thing you are missing is that btrfs can start out with a single pool and migrate it in-place/on-line to a raid1 by adding one or more drives as and when. What I’m trying to head-off here is that when breaking a raid1 pool with 2 members each of those members can, theoretically, re-create the entire data set via a degraded mount. But if you accidentally pair them up again you will mostly likely corrupt both copies. To re-add the removed device it would have to be properly wiped first.
Again we have this preconception that raid1 means pairs of devices. That is where our lack of a primer is showing. Btrfs raid1 remember is chunk based raid. The drives are not in pairs (such as is recommended to maintain a modicum of flexibility in ZFS via pairs of drives making vdevs) the 1GB chunks are. Admittedly the chunks are spread across the pool members but those members can number more than 2. I.e. for a 3 drive raid1, chunk A existing on drive id1 and drive id2; but chunk b may exist on drive id1 and drive id3. They both meet the criteria of existing on 2 independent devices but they are on different independent devices (at least their second copy is). For a 4 drive raid one chunk A could be on id1 and id2 while chunk B could be on id3 and id4. Again the criteria of chunk redundancy across 2 devices is met but without overlap of devices.
Interestingly you can’t know/tell without very deep dives which 2 drives a raid1 chunk is stored on. They are just somewhere on the pool. But this does men you can have odd numbers and sizes of pool members even if the raid level is 1 or 10 say.
Hopefully, and feedback on it’s clarity, usefullness would be appreciated. It does highlight how in some settings one has to change the raid level to facilitate greater flexibility regarding reducing the minimum drive count. Again to avoid breaking a pool. The degraded mount state is an emergency state that is to be avoided, not used as a regular tool. It has many pitfalls and puts your data below the normal protections offered by the raid level. In fact it is only at all usable if there is only 1 drive missing below the minimum count. There are exceptions for example with btrfs raid6 but that is not recommended and is read-only by default anyway in our default installer.
. But again to answer head-on, you don’t have to do pairs such as you have likely picked up from the vdev work around common in ZFS. You can add on drive to the original 2 X 1 TB. You can even start out, as mentioned earlier, with a single drive. No redundancy but you will still have checksumming, unless you turn that off of course. But starting with rudundancy is always best if you can. You can then later add another drive and so on. And when you reach say 4 or 5 drives you can convert to raid10 if need be. For large drive counts (i.e. 10+ depending on reliability) where you would be best recommended to adopt a raid with a 2 drive failure capability you should consider say raid6 data raid1c4 metadata. Btrfs can have differing raid levels for each simultaneously. We don’t support this yet but with sufficient knowledge it can and is done with the current Rockstor stable release. But you will need the backported kernel arrangeemnt for that and avoid, for now, some Web-UI stuff as it will re-assert similarity of data & metadata raid levels. But if you can handle the risk (backups) this can always be undone via the command line.
