Rock-on requests

Fredrik · June 14, 2017, 3:24pm

I can look into a rockon for this but i would think optimal solution would be to have it part of the rockstor core and configuration in UI.

tomo_atlacatl · June 15, 2017, 1:00am

Hi Fredrik,

I agree, this should be part of the core rockstor…I mean, don’t get me wrong, a rockon would be ok for now, but…looking at all the competitors, they all support FTP native. I would hate to create users and directories on one place…just to duplicate the same on a rockon. Seriously, I already reinstalled and removed like 3-4 competitors and keep going back to rockstor, I like it, but the FTP is really holding me back from full deployment.

paulsmyth · June 15, 2017, 9:19am

If FTP is going to be an option it should be part of core. Personally I do everything I can to avoid using FTP and have made devs rewrite their code because FTP is horrendously bad security practice. It’s one thing to risk it on your own LAN but people will port forward this to their routers and use it from outside on public open wifi.

tomo_atlacatl · June 17, 2017, 1:06am

Yeah, I get the part where people will simply read a blog somewhere urging them to open port 21, etc… From the home NAS front, it’ll be nice to have some sort of firewall built in as well…like synology or even nas4free…for now, plain FTP locally will suffice for me

luke · October 29, 2018, 7:42pm

I would suggest including a Rockon that uses Portainer. This is a wicked easy docker GUI run through a web interface and is fairly powerful…

In fact I’ll do that myself soon.

Flox · October 29, 2018, 8:05pm

Hi @luke,

Agreed… there already is a pending PR for a Portainer Rock-on thanks to @alazare619 :

github.com/rockstor/rockon-registry

Portainer

rockstor:master ← alazare619:portainer

opened 02:48PM - 19 Dec 17 UTC

alazare619

+38 -0

Portainer is an advanced docker cluster,swarm, and single node management framew…ork. Its very advanced and extremely lightweight. It integrates perfectly into rockstor i've been using it for a few weeks now without any issue managing the advanced tasks for my rockstor docker node. Note: This is a new commit as I've changed my repo to have multiple branches as I have multiple PR's open now. Previous PR no one was reviewing or commenting for months...

I personally use it and it works very well.

luke · October 31, 2018, 10:22am

Excellent, wish I’d spotted that one earlier. I’ve found creating a share with permissions for docker, then using this with portainer and creating bind volumes in that share works a treat. Nice and simple with no mucking about with permissions.

ArmyHill01 · June 2, 2019, 5:17pm

Reviving an old thread here but I have 2 cents worth mentioning if at least for those googling “Rockstor Docker” or “Rockstor portainer” and the like…

Portainer (or any of the other Docker managers) is an absolute must for anyone with an existing container setup that is looking to move to Rockstor. This is especially so if they’ve already adopted the stack/service model of docker.

I’m still pretty new to docker but I have about 30 containers I’m testing and I’m loving it thus far. Rockstor’s “Rock-ons” honestly put a bad taste in my mouth right off the bat.

I totally get having a click and run solution for those users not interesting in having a complex container configuration but I’d really like to be able to just turn off Rockstor’s docker integration and be allowed to completely decouple Rockstor from Docker. That way I can get the latest and greatest and do my own thing (which very well may be possible, I’m only like 2 days into my Rockstor testing

Not trying to be negative Nancy here as the Rock-ons system is great work. It does add a HUGE feature into Rockstor so many kudo’s for that!

Flox · June 3, 2019, 1:46am

Hi @ArmyHill01, and welcome to the community!

Thanks a lot for sharing your experience and feedback, I personally particularly appreciate and welcome it .

I agree with you that Portainer is a very useful tool for those with an existing setup and needing complete customization. I myself am using it for this reason .

My first reaction was: “yes, you can indeed just turn on the Rock-on service (which starts the docker daemon), and then simply set up your own containers whichever way you want (via portainer, for instance)”. I then wondered whether you would also mean controlling which docker version you install and with which settings, etc… This case would, too, be doable, but I can’t guarantee what would happen if you attempt to turn on the Rock-on service later on. Indeed, turning on the Rock-on service for the first time set the docker.service configuration with some slight customization to better fit with rockstor’s system (and btrfs, mostly). This is the step that might introduce some incompatibility if you were to first install and run your docker version on your own and at a later time decide to turn on the Rock-on service, as your previous configuration may get overridden and the different docker version (if any) may be incompatible with that configuration. Note, however, that this is based on my memory and I’ve never tested that so I actually may be wrong and everything be smooth.

Most importantly, note that Rockstor is in currently transitioning to a rebase onto openSUSE (see updates and rationale in a separate post by @phillxnet and his other posts therein). Under this process, there is a current pull request proposing to simply source the docker.service configuration straight from the docker package itself (and then apply rockstor’s settings–including the user defined ones–on top of that).
https://github.com/rockstor/rockstor-core/pull/2046#issue-280832979

I’m unsure of exactly which docker version will be shipped, but I believe it will simply provided by upstream directly, so either openSUSE Leap 15.1, or Tumbleweed… the latter should be the latest and greatest, I believe. @phillxnet, am I correct with that one?

Yes, that’s personally what helped me at first to get acquainted to docker and its ecosystem. I progressively became more familiar with it and started feeling some limitations that the rock-on system had at the time to get a few more “customized” configurations. I personally do see a very interesting opportunity to provide an easy way for unfamiliar users to set up more complex containers configurations provided some improvements to the current rock-ons framework. On this topic, there has been quite a bit of work done lately and I personally have an upcoming series of rework to implement docker networking into Rockstor. You can read more on this in the issue below and the links therein if interested:
https://github.com/rockstor/rockstor-core/issues/2009#issuecomment-493048687

For this reason, I am particularly looking forward to hearing more about the following:

Was there something in particular that you would like to see or not to see? As you can probably guess from above, I’m interested in getting as much feedback as I can and getting as many suggestions for improvement as possible, so your experience seems to be very helpful for me–pending you’d be willing to share the gist of it.

Welcome again to the community!

ArmyHill01 · June 3, 2019, 3:54pm

Thank you for the thorough feedback hoss & the warm welcome!
I’ll jump right into it then…

This is great news. I still need to review the source now though as to see what config changes you you’re referencing that are specifically for BTRFS. I wouldn’t think the FS would matter on the app layer but again I’m still new-ish to docker.

I’m tracking a few posts but I didn’t find a parent topic ‘see here for latest info about RS’ but maybe I’m blind. Lots of info on the forum to thumb through for this new guy

Honestly, as a power user, I don’t need RS’s layer of abstraction. I have several docker-compose’s for my various needs so there is literally no way for me to switch to Rockons.
Beyond that, I guess I’d make the following statements:

For simple users:

Provide an ‘normal’ & ‘advanced’ container install solution
- Advanced would allow users to simply paste in docker-compose yaml and BOOM, their off.
- Normal would be “like” the current ‘click to install’ system
Allow ability to install container X from repositry/url Y
Change the way container shares are done
- Allow users to select a path, not a share (why force the root of a share… makes no sense)
- I actually have several docker_store_X shares where X corresponds to my various container stacks
- This allows containers to share any needed data all while allowing me to set custom setting on those various shares (i.e. compression/quotas/etc)
- Also need the ability to set the access mode on a containers volumes. Example: An image gallery container should have 2 at least 2 volumes: 1 its data dir with r/w access and 2: the media it’ll show but i’d only want this readonly.

For Power Users:
I started to write a list but really, it was just everything Portainer does. Full stack/swarm support with total control over everything. I’d honestly like to see a setup question in the Rockon’s space that asks, “will you be using your own docker manager?” and if yes, RS disables all docker related stuff and then provides an option for a name and URL of the manager for an ‘ease of use’ shortcut somewhere in the RS UI.
Beyond that I really don’t like the idea of the community having to maintain the yamls needed for Rockons to work.
Why not scaffold all the needed info the run a conatiner in the UI (what image, what repo/url, what ports, what volumes, what networks, what restart policy, etc, etc) and then pass that on to docker?

I’m trying to write this bit in a way as to not come off as a dick but it’s not really working out that well. The gist of what I want to say is, I don’t really see the value in enhancing the whole Rockon system beyond the super simple click and install model simply because any ‘advanced need’ can simply be filled with a docker manager like Portainer/ShipYard/Rancher. I’m a dev so anytime I see someone rewrite something… we’ll you know how that smells…
And Portainer is just too easy to use IMO.

Again, I appreciate your feedback & help here!

Flox · June 4, 2019, 7:24pm

Here’s the current related code:

github.com

rockstor/rockstor-core/blob/1102e2805a9290704ed6f3fd8963dbab37674c5e/src/rockstor/smart_manager/views/docker_service.py#L75-L98


      
          docker_wrapper = '{}bin/docker-wrapper'.format(settings.ROOT_DIR)
          distro_id = distro.id()  # for Leap 15 <--> Tumbleweed moves.
          if distro_id not in KNOWN_DISTRO_IDS:
              distro_id = 'generic'
          # TODO: Consider sourcing /usr/lib/systemd/system/docker.service
          inf = '{}/docker-{}.service'.format(settings.CONFROOT, distro_id)
          outf = '/etc/systemd/system/docker.service'
          with open(inf) as ino, open(outf, 'w') as outo:
              for l in ino.readlines():
                  if re.match('ExecStart=', l) is not None:
                      outo.write('{} {}\n'.format(
                          l.strip().replace(DOCKERD, docker_wrapper, 1),
                          mnt_pt))
                  elif re.match('Type=notify', l) is not None:
                      # Our docker wrapper use need NotifyAccess=all: avoids
                      # "Got notification message from PID ####1, but
                      # reception only permitted for main PID ####2"
                      outo.write(l)
                      outo.write('NotifyAccess=all\n')
                  elif re.match('After=', l) is not None:

This file has been truncated. show original

Note, however, that we have a pending pull request aiming at simplifying this part as mentioned in my previous post. See here for information:
https://github.com/rockstor/rockstor-core/pull/2046#issue-280832979

The few “Rockstor” options that are added are as follows:

github.com

rockstor/rockstor-core/blob/1102e2805a9290704ed6f3fd8963dbab37674c5e/src/rockstor/scripts/docker_wrapper.py#L26-L29


      
          ROCKSTOR_DOCKER_OPTS = [
              '--log-driver=journald',
              '--storage-driver', 'btrfs',
              '--storage-opt', 'btrfs.min_space=1G']

These mostly included one critical fix to docker-ce disabling quotas in BTRFS, which caused some issues in the past (see rock-ons-root host pool quota disabled by docker-ce. Fixes #1872 by phillxnet · Pull Request #1873 · rockstor/rockstor-core · GitHub). @phillxnet will be able to correct me and bring more details to these if needed.

Mmmm… for the most “formalized” source of information regarding the latest update on the Rocsktor’s changes, I would say we can look at the current release notes for the Stable branch, as well as the Github pull requests. @phillxnet always writes quite the detailed notes so they usually have a lot of information in them.

Thanks a lot for spending the time to write these down :-). I actually agree with pretty much everything you listed. Before I add my 2 cents to it, I’ll point you to a rather old but still information thread in this forum, in which you will find similar conversation and ideas. Also, note that what I’m saying only reflects my personal opinion on things, and explains the recent contributions I’ve had in this part of Rockstor.

I personally see the rock-ons framework as a very nice tool to give quick and easy access to what docker containers can offer without necessarily needing to get familiar with it or even the project(s) they will end up using. As you mentioned, a “more advanced user” can simply run Portainer to run her/his own choice of containers with any setting (s)he wants, use docker run from the cli, or even docker-compose up. None of these should interfere with Rockstor’s functions and I personally run several containers like that. From my own experience, I originally wasn’t familiar enough with docker to do any of that, but I was still able to run Owncloud and Emby thanks to the existing rock-ons and it’s how I see their use. I believe the rock-on framework still has a lot of room for improvement to reasonably expand this simplicity to other areas of the docker ecosystem, but–like you said–while trying not to reinvent the wheel. I’m thinking, for instance, about multi-containers rock-ons such as the older owncloud rock-on, which ran its own postgres database in a separate container but from the same owncloud install wizard.

Again, I agree with you on pretty much everything you said . A few points, though:

I apologize in advance if I completely misunderstood this, but would the “Add Storage” feature fit your need here? The related detailed documentation is in the process of being published, but you’ll find a description in the post below. It basically allows to bind the same share to multiple rock-ons in order to make data available to as many rock-ons as desired.

I like this idea very much. I believe the json file defining the rock-on does support the definition of rw or ro (I will need to verify that, however), but I like the idea of being able to customize this on any share post-install.

Thanks again a lot for all your feedback, and I hope I was able to help a little bit.

ArmyHill01 · June 6, 2019, 10:25pm

Thanks again for all the detail man. You and phillxnet provide good details and I appreciate it!

Thank you for all the related code. It’s good to see not too much is in the docker_wrapper.py

That particular thread is closed but I get what you’re putting down.

That’s one thread I missed. Thank you. I’m surprised the thread ended so quickly but ehhh… it was 2015.
As far as the wizard/helper ideas; that’s actually great. I’m thinking of my wife specifically. If you asked her to pick a pool or share… she’ll close the window so I see the target audience better now.

Container Storage
No, we’re not on the same page here.
Why force container volumes to be mapped to a particular share? I shutter at the idea of 79 shares on my NAS
Absolute pathing options would solve this issue for me.Maybe this would be an ‘advanced’ option but I still don’t see the original logic in wanting a share per container. Just seems excessive I guess. Maybe I’m just being anal-retentive.

I really like explicitly stating what the container can do with the data I provide. Some, like Plex, should only read from your media collection while others, say your downloader, naturally need r/w.

I’m still not sold on Btrfs over ZFS so I’m going to be doing some testing on a bare linux OS doing recoveries. I had really hoped to switch to a more linux native FS as BSD is driving me nuts. Either way I have some work ahead of me… thanks again for all the info.

phillxnet · June 7, 2019, 2:50pm

@ArmyHill01 Just chipping in on:

Yes, I keep that thread locked unless I’m updating it (directly after each release) so as to keep it’s content focused on that particular changelog. In the past our ‘changelog threads’ have often attracted general discussion on each feature or change as they arrive which can be quite distracting. And any discussion can simply link to the relevant release post anyway so we only gain by keeping it focused.

Hope that helps at least on that bit.

Flox · June 7, 2019, 9:31pm

Sorry for the delayed answer; I was convinced we had a similar discussion somewhere to which I wanted to link here, but I couldn’t seem to find it… I finally did, however, so I’ll do that below.

Sorry for misunderstanding you the first time, it’s clear to me now. Unfortunately I don’t have an answer to it and agree with you that it can become cluttered and annoying if/when the rock-on collection grows substantially.
As to why forcing a particular share, my initial thought was that there may be the advantage of apps and services separation but it is true that it can be become a disadvantage if the number of shares become too large as a result. I then, however, remembered pieces of several discussions we had in this forum and related Github repositories about similar points (this is the link I was looking for above). After re-reading those, my best attempt at an answer would be that it simply hasn’t been implemented yet. The Github issue below, although slightly misplaced, addresses this topic and includes several paths to resolving and improving this very point.
https://github.com/rockstor/rockon-registry/issues/95#issue-215116599

As you can see in it, several options were discussed, including:

Alleviating the “burden” of creating a share for each by automating the process during the rock-on install wizard.
and / or
Implementing the possibility of specifying a path for the users who need / prefer this option.

As you can see in that issue, I do like these options and am hopeful to see if I can implement them next. That wouldn’t be very soon as I currently still need to finish a few other things, but that is definitively on my list of things I’d like to address.

I believe there are quite a few users here experienced in both so I’m confident you’ll be able to get answers from the community if needed. Good luck with your testing endeavors and enjoy!

Cheers,

phillxnet · March 13, 2020, 5:03pm

@flooiks A very belated welcome to the Rockstor community.

We do support Full Disk LUKS encryption: see our docs section on this:

LUKS Full Disk Encryption

Hope that helps. Thanks for your other ideas here. You might reach more folks if this was not in “Rock-on requests” section which was originally intended as requests for whole rock-ons rather than specific requests for particular rock-ons.

Interesting Idea.

We do accept GitHub pull requests if they are tested on our current 3 distro targets; rockstor (CentOS) and openSUSE Leap15 and Tumbleweed. Although our CentOS base is to be deprecated in the future.

Hope that helps.