Replication - Failure

dvanremortel · February 17, 2018, 3:37am

Fellow #ROCKERS, your thoughts please.
Two appliances, both registered with each other, both with replication service on and a scheduled event configured. It barely lasts a second then this error.

No IP’s have changed

Any ideas? Thanks in advanced. Dave

[16/Feb/2018 19:25:04] ERROR [smart_manager.replication.sender:74] Id: 06621353-4BB0-41E9-8615-017696E81528-3. unexpected reply(receiver-error) for 06621353-4BB0-41E9-8615-017696E81528-3. extended reply: Failed to validate the source share(FileStorage) on sender(uuid: 06621353-4BB0-41E9-8615-017696E81528 ) Did the ip of the sender change?. Exception: [u’Invalid api end point: https://fs.redbeards.home:443/api/shares/FileStorage’]. Aborting. Exception: unexpected reply(receiver-error) for 06621353-4BB0-41E9-8615-017696E81528-3. extended reply: Failed to validate the source share(FileStorage) on sender(uuid: 06621353-4BB0-41E9-8615-017696E81528 ) Did the ip of the sender change?. Exception: [u’Invalid api end point: https://fs.redbeards.home:443/api/shares/FileStorage’]. Aborting

phillxnet · February 17, 2018, 12:50pm

@dvanremortel Thanks for the report but this has already been fixed as of stable channel release 3.9.2-13 (current stable channel release version is 3.9.2-15):

See GitHub tags:

via the linked issue:

https://github.com/rockstor/rockstor-core/issues/1853

and it’s associated pull request:

github.com/rockstor/rockstor-core

Fix replication regression re share api change. Fixes #1853

rockstor:master ← phillxnet:1853_suspected_replication_regression_re_share_api_change

opened 07:24PM - 24 Jan 18 UTC

phillxnet

+270 -70

Prior function was restore by updating the relevant api urls in the replication …system. But in one, non critical check this was not managed. As a result the non critical but desirable internal sanity check was, for the time being, remarked out and TODOs added to signify required future attention. Also includes a number of replication UI fix ups including table formatting, sort capability, appropriate ordering, and consistency with the rest of the UI. A core mechanic of the replication system was also abstracted such that it could benefit from the existing share / snapshot management system. This alteration made more explicit, in code, a quirk involving the initial subvol transferred and how it requires a special treatment given it's unique nature in the context of the existing share / snapshot / clone / import structures. Summary: - Update internal replication api urls to fit new share by id scheme. - Remove problematic non critical internal check (TODO added for later). - Create new repclone command to improve replication share / snap handling, ie more transactional by using existing structures via api. - Minor improvements were added in the new repclone system, ie sys refresh qgroup transfer, auto mounting etc. - Ensure we import initial btrfs receive quirk subvol with replica flag and add logging to indicate it's quirk nature. - Improve replication tables formatting, text, sort ability / order. - Reduce and mirror (between send and receive) rep-snap-count. - Fix bugs in receive status update, includes utilising the pending state. - Improve user communication re data transferred / rate table cell text. - Remove broken pool link in replication overview table. - Improve debug logging re share / snap import. - Minor steps / TODOs towards future share name/subvol_name seperation. - TODOs towards centralising mount path creation. - Various TODO added for future consideration. N.B. this pr assumes the prior application of the following (currently pending) pull request: "remove immutable flag prior to share delete. Fixes #1882" pr #1883 As that fix also addresses an observed breakage during replication runs and was used (pre-applied) during all testing of this pr's code. Fixes #1853 (if #1883 in pre-applied) @schakrava Ready for review. Testing included several hundred individual replication events (btrfs send receive hosts) with runs up to 150 ish. A real hardware arrangement was also tested where the source machine's (sender) share had more data than could be replicated in the chosen scheduled replication interval (10 mins). Additional send receive pairs were not activated and all data was successfully transferred. Note that with the in pr rep-snap-count change to 2, 3 snapshots are stored both on the sender and against the receivers share counterpart. And that for a replication cycle to reach it's final state, which will in turn there after be maintained by rotation, 5 replication events must be completed: the initial full-send followed by 4 incremental sends. There are still known limitations and bugs in the replication code post pr but these can be addressed more specifically in pending issues to be opened in due course: and depend on review outcome of this pr.

The indicated api URL has the share name in where as we now use the share ids. It was quite a bumpy path to changing the whole project over (pools were also changed similarly) but the replication was, as far as I’m aware, the last regression candidate. Although we still have some code tidy and ‘nice to have’ stuff that needs to be re-asserted.

See also in-progress changes re testing channel releases discussed towards the end of the following thread:

Hope that helps.

dvanremortel · February 17, 2018, 6:42pm

Thank you for the replies. I have check both servers version and it states is it higher then the number indicated. Also I have TWO servers one is registered with production updates the other is a standard non-registered server with testing updates.
the replication job shows as OFF but it was on previously. I have included the versions of both servers and the replication configuration windows for both as well.

Any ideas?

phillxnet · February 17, 2018, 7:34pm

@dvanremortel As per:

Unfortunately the highest testing updates version is currently 3.9.1-16 (which one of your servers indicates) and does not yet have the fix. Currently only the stable channel has this fix and both the sender and the receiver require the fix as they ‘chat’ back and forth and there were fixes on both sides to move to the new share and pool api.

Yes, after a number of failures it will auto disable to avoid spamming logs. If you enable email notifications it will email you each event, including the auto disable.

Also note that to resume a failed replication task it may be necessary to delete the last created snapshot on the sender side created for this purpose (via UI is find), should be named obviously enough. Improvements to be made there robustness wise that I hope to get around to soon, if another dev doesn’t beat me to it of course. Also note that I’m unsure of the state your replication tasks are in as given the required changes in the fix there was no point in testing between current testing and stable variants.

If in doubt delete the current tasks and once you have both sender and receiver ‘up to spec’ re-create and start fresh.

Hope that helps and thanks for helping to support Rockstor development.

dvanremortel · February 17, 2018, 8:38pm

Thank you. When will this capability be in the testing release ? I’m doing a PoC and this was a key feature.

thanks, Dave

phillxnet · February 17, 2018, 11:17pm

@dvanremortel Oops, got my versions number wrong here:

Meant to be 3.9.2-13, sorry. I’ve corrected in my post above.

Probably on the next release I expect, however this is down to project lead @suman and is waiting on the update mechanism / docs that need to be in place first. As referenced earlier.

Hope that helps.