I’ve read the documentation on share replication, but I still have a few questions.
Would a replication send all created snapshots, too? Can I configure it to do that?
E.g. my plan is to create periodic snapshots and I want all of them replicated to my second system.
I am trying to work out as many details as possible before doing the actual move.
Related question: Can I backup/replicate a snapshot from my laptop /home to rockstor?
@Marenz hello again.
No and No.
Although it does use snapshots itself. The docs need to be improved on this feature and I have some almost finished technical wiki docs to publish on the forum when I get time from when I last worked in that area of the code.
Rockstor’s replication system aims to replicate a share, btrfs subvol, and does it by snapshotting that share and then using btrfs send receive to send that snapshot to the target Rockstor system. On running a second time it takes another snapshot and sends the difference from the first snapshot (from my now fuzzy memory of this) and keeps doing this but will ‘tidy’ up old snapshots so that once it reaches a stable state (after 5 replication events) you will see the final state that it then maintains.
It’s actually a fair complex process and there is some non trivial bits of code to accomplish this.
So I would advise that you setup a small share and let it go through 5 to 6 replication events (if the share is small you could schedule them to be quite often) and take a look at the replication specific snapshot names that are created at each stage. And note that after the very first replication event you will have a strangely named share (actually a snapshot), ignore it, it will be auto removed after the second or 3 subsequent replication event. And don’t delete any of these obviously named shares as that will break the replication and you will have to start it over.
Replication worked when I last check in our latest Stable release and in our last CentOS based ISO release, but is broken in our now deprecated, but otherwise working CentOS testing channel release. It was one of the reasons we never released a stable at the end of the CentOS testing channel. 3.9.1-16 would have been released as 3.9.2 stable and put out to ISO but replication and a few other things were just broken so we continued on with the relevant fixes in the Stable Channel.
I don’t see why not, as long as your /home is btrfs; assuming you are suggesting btrfs send recieve here. You would have to set things up so that the target was understood by Rockstor, ie an existing share and you put snapshots in a location understood by Rockstor. It’s actually going to be quite tricky come to think of it and I’m afraid I won’t have time with all that is required for the move to help much at all with this. But you can always look at the mount points used by Rockstor and setup your send receive accordingly. This will all be command line stuff however, you are not likely going to be able to tie this into the Rockstor to Rockstor replication as it uses it’s messaging system etc. But if you are doing plain btrfs stuff, and your resulting subvols (which includes snapshots) are understood by Rockstor then your should be good.
Would be a fantastic feature to have a Web-UI initiated ‘receive my btrfs /home’ type setup that ended up prompting your to enter stuff on the client and you would then be done. Oh how we can dream, but the priority of that feature is likely to put it into the distant future with our current backlog of issues.
Hope the helps.
I am thinking how exactly the feature of transferring snapshots could look.
One easy way would be that each time a scheduled snapshot is done, it is also replicated. Logic that cleans up old snapshots would need to be extended to do that on the remote machine, too.
Would there be interest in such a feature? I could give this a shot. Or maybe you have a different idea of how this could work…
@Marenz, Re:
If you are at all interested I would encourage you to take a look at the existing code for replication. Personally I think the focus should be on improving that code as it has some outstanding fragility; there are references to this within the code itself from when I was last there. But the mechanism your roughly describe is pretty much that which is used in the replication, sort of.
Try our what we have and take it through it’s full cycle, noting the state of snapshots for the given share at both ends. And make sure to go at least 6 consecutive replications along to reach the stead state (should be on the 5 replication run from memory).
But be warned that this code ares is non trivial and resources many of the other share / snapshot / clone technologies that we employ individually across the rest of the subvol management areas of the Web-UI and associated backend. It’s not for the feint hearted but any help in that ares would be great. And there may be a clever trick we can just ‘bolt on’ to what there is that will significantly aid that area of the code and it’s user utility. But from memory it’s not going to be a quick fix / addition. But could also be pretty rewarding to improve / extend. Also note that testing changes in that code is also very time consuming. I ran sever multi hundred repetitions of the code prior to releasing it last time and this ended up being, quite time consuming. Plus you have to account for overlapping requests and the like, ie a user configured request to duplicate TB’s of data every few minutes is not going to fly. But it does back off from what I remember and looks for a prior still running tasks and the like. Also logging is far from trivial in the case of the replication as there is often zero feedback from the btrfs tools for quite some time. I think, again from memory, either the receiver or the sender are far more chatty that way. You will see this during a large transfere one one side is only able to say a replication task is ongoing. While the other end will give you an near live update of it’s speed (which is often pretty slow by the way).
Would be great to get some more eyes on that code so do dive in if you fancy.
My last pull request on that code, which “mostly” fixed it from it’s prior broken state in our last released CentOS testing channel variant (because of necessary API changes in that testing run) was:
But not that there have been many improvement and some fairly deep changes to disk / pool / and subvol management since then. But it can server at least to given an indicating of the files concerned.
So do look at what we have already as you can then see more of the complexity involved. Plus it would be far more valuable to have this system improved / made more robust than to have additional functionality that may also come with more fragility. A key point here is that btrfs is currently unable to resume a replication that has been interrupted. A non trivial problem to ‘work around’. There are upstream plans however to ‘up’ the btrfs send / receive protocol but they are waiting until other stuff settles before making what would be breaking changes. Probably best in the long run.
Good luck and take you time looking through those code areas as they are non trivial, but quite interesting actually as there is a message passing system in place between the sender and the receiver at the Rockstor level from what I can remember. And if we could get this to be more robust (see my comments in that pr) then we would have quite a powerful system with with to ‘orchestrate’ this otherwise delicate process. And maybe even extend it to manage your proposed functional expansion.
Good luck.