Rockstor - How to do a multiple replicated servers around the globe. Help wanted!

General
1

The salute: Hi there. Roberto0610 here from one corner of the world. This time to ask you about a recipe to to get multiple server with same information replicated in multiple location.

The conflict: I have a team of video editors around the world. At my headquarters I have 3 fully loaded workstation connected to our main 1 Rockstor via 40GB network witch is connected online by a 1GigbDOWN / 100MbisUP. System its a Dell Power Edge R730XD and Has 12x 8TB on RAID 60 (wonder if drives will perform better in a longer run using BTRFS) and uses Rockstor licensed (planning on using 2 similar servers with RS plus 2 smaller for Mex/Ven offices for this approach). Our one system is fully in operations with its 40GB, 10GB and 1GB nics. Part of the problem: I am editing content for multiple photos/video studios throughout the states. And sending chunks of video work and re-work to multiple editors around the globe. 3 Editors are in Mexico, 2 more in Venezuela and 2 more remotely out of the US.

Scenario: In my small town Wenatchee WA. I have 2 counties Douglas, gives me for $78 symmetrical 1GB up/down fiber. But I only have limited space there. While in the Chelan county where I do live an also have my small data center where our One RS server leaves, same place as the video/photo studio We have… Here the ISP only work 1gig UP by 100Mb down for a much much higher price. At the Office in Mexico we have 250Mb/65Mb fiber. At the office in Venezuela we got 150/150mb fiber symmetric too.

Approach: I am willing to collocate One of the R730XD Server at Douglas location to gain on the 1Gb by 1Gb symmetric fiber, because of the speed to receive files from multiple customers in the states. The data is in dozens of video.mp4 per studio is usually 400 to 800GB (Maybe I’ll use NextCloud for syncing? -Suggestions accepted). All this videos are recodring from about 2, 3 or 4, audio recodres ans cuch like FullHD, 120fps, slowmos and more like audios.wav, photos.raw, music.mp3 and slides shows from live recorded events like Weddings and Quinceañeras. All this information We wanted to have it replicated to our headquartes where will be render to generate video proxy files. This files are much smaller (60 to 130GB instead) version that usually will be downloaded smaller servers at editors location in Mexico and Venezuela to use as proxy footage and editing material. We all use Premiere Pro. Once editing is done. We will render/export finals at hour headquared using original media and submit it back to end customer.

Details: I know this many need a lot to go to get it runnin, but I’m all for it and really really need your help on how to do it operational. I just can’t get it done by myself. Also need a solution to remote access, manage and update this remote RS servers as I’m the only one that know something about this behind the scens for all this. What will you recommend me to use for remote accessing this servesr if they are behind the nat some where in the world. I do have public static ip and domains for both fiber locations at our Studio and the small small Douglas office.

Help wanted:

  • Recomendations of RAID 60 vs BTRFS?
  • What will be the bast way to use replication and if it good to use NextCloud with Rockstor as I been having issues NextCloud accessing my NFS shared folder currently?.
  • What should I do for remote management of RS servers if they get behind a NAT?
  • Regarding the multiple users an access what should I do to control un desired people from accessing my our data?
  • We are pro users and won’t mess with each other projects and data. But at some point I will have to prevent users from accessing: How could I create editors accounts and manage that for multiple locations?
  • Will any should recommend a good way for backups? What will you recommend?
2

@roberto0610, mostly one comment from my side on the RAID60 vs. btrfs: if you’re using Rockstor your best option out of the box is RAID10, the RAID5/6 is still not considered production worthy.

On the remote management, I would think that if you have access to the NAT server initially, you could set up some port forwarding and then establish a SSH connection, or OpenVPN server (there’s a rockon for that). Of course, if you the server doesn’t come up for any reason, you will need somebody to locally interfere).

Fundamentally, the groups/users concept on the servers would allow you to restrict who can do what. If you need it to be across multiple servers (without manually creating the same accounts/passwords on every servers, then you could look into an ldap based solution (since Rockstor supports that), though I have not implemented that, but I believe there are some forum entries on here, so we should have some experts.
On the backup side, I let others recommend. I am using combinations of rsync and a windows product called GoodSync which works for me.

2 Likes
3

Thank you for your recommendations. regarding Raid60. I have hardware raid controller for it. I could pass all drives to the RS system to use BTRFS or just use the raid controller with all 12x, 8TB drives to run as Raid10, 5, 6 or 60?.. . Most of the stuff will be reading the video files for editing and I’m planning on using local nvme at on site for all temps, and fast writing stuff.

Will definitely dig into the ldap for more details. Thanks for suggesting. The issue at some ISP in lat-america is that they don’t provide dedicated public IP So I was thinking on setting up the server to look for main server where I do have Public IP and self-managed router.

Regarding data sync I’m testing with next cloud since it has automated syncing app for, linux, mac and windows. I will provide the how to when all this get squared down.

Any recommendation/suggestion is welcome.