Couple of issues: dnsmasq and sendmail

Troubleshooting
1

I’ve got a couple of strange issues presenting on my system recently. I haven’t been able to find anyone else bringing it up anywhere else, so it’s very much a possibility that its something that I’ve unintentionally caused, since I’m not super familiar with Linux, although I’ve been trying to use the cli as much as possible for a couple years now anyway to get familiar with it.

Relevant setup info:

System: Rockstor 3.9.2-57
Kernel: 5.9.10-1.el7.elrepo.x86_64
Intel Celeron 1037U Dual-core 4-BAY NAS with Intel HM77 chipset
Hardware RAID 0 / 1 / 5 / 10
8GB DDR3L 1600 RAM
SATA III
128GB System SSD, 4 x 4TB WD Reds
Dual Bonded/Teamed Gigabit NICs
btrfs: v5.9

Note: I would update to openSUSE, but I haven’t been able to find any easy to follow instructions on how to do it yet… and I don’t know enough about how building from source works to be able to come up with the commands on my own. also - I needed a newer version of btrfs because my data was using RAID1C3 and the version that came with rockstor refused to read it, so that’s why I ended up needing to build that. lol like I said, I can follow instructions…

Anyway, so the first issue is that there were update for dnsmasq, net-snmp and a couple associated net-snmp library packages that were released recently. I’m subscribed to the stable update channel and as a result my system was automatically installing these packages.
Somehow, after it did my system would become unresponsive every time (I primarily access via samba linked to my windows desktop). My very limited troubleshooting skills discovered that the new packages were causing several services to fail right from boot, including network.service, smbd.service, and probably about 4-6 more .services that I don’t recall offhand. The web interface still worked however, and samba would be disabled every time and couldn’t be turned back on unless I used the cli.
This would happen every time I tried to use the server, and the only way I found to fix it was by rolling back the updated packages via the yum history feature. After that it would work fine again - until it decided to install those new packages again. Rinse, repeat.
I eventually got irritated with having to dick around with my server every time I wanted to use it, so I added an exclusion for those packages in the /etc/yum.conf file, which seemed to fix the issue. A bit of a band-aid solution, but it works. Just wanted to bring that to your attention, anyway.

The second issue I haven’t found a solution for, and that’s that my server is endlessly spawning the sendmail and postdrop executables. They are remaining in memory until the server essentially grinds to a halt. The web interface is SO slow (but works, sometimes).
When I ssh into the box and run ps -A I find a list 3/4 of a mile long, mostly “sendmail” and “postdrop” processes, although there is a doubling up of multiple other executables as well (nfsd shows up 8 or 9 times, postgres about 15 times, grep here and there, a bunch of kworker threads and various others). If i issue a pkill sendmail command - and then a similar one for postdrop - all of a sudden my web interface works great and all is well. I tried counting how many there were one time after I had just cleared it about 10 mins before, and it already had 26 new instances of both sendmail and postdrop.

I have tried to de-register or remove every email reporting feature I could find but they still keep showing up. I suspect it has something to do with the netdata rockon, because I’ve been receiving nearly 500 emails/day from netdata saying “cant connect” “connection restored” “entropy low” “disk use 10 mins alarm” etc etc etc.
I don’t care about the so-called issues it’s reporting, but I haven’t been able to figure out how to turn all those “alarms” off anywhere on the netdata UI, and I don’t have enough experience with docker yet to know how to track down the config file within the rockon container… What I do care about is the fact that my processor sits there running at full tick with all these processes that don’t ever seem to die, and the higher than normal system temperatures that result from it. So, I’ve been having endless fun with that the past few weeks anyway… :laughing:

Like I said, its probably something I did to it out of sheer ignorance, but any ideas on how to fix it would be appreciated because I’m not finding any success with it.

2

@koskee Welcome to the Rockstor community.
I can chip in on this one:

It’s not currently available as a downloadable installer but the following repo has directions on building your own installer:

and note that there is also a vagrant/virtual-box orientated method available in a subdirectory of that repo that may be more accessible:
rockstor-installer/vagrant_env at master · rockstor/rockstor-installer · GitHub
And once you have the installer you can do a fresh install, ideally to another system drive with no other drives attached. And then once you are happy it boots you can attache you old data pool members and import the pool:

Import BTRFS Pool: http://rockstor.com/docs/disks.html#import-btrfs-pool

You can also use the config backup-restore between the variants, as long as you download-upload the config file:

Configuration Backup and Restore: http://rockstor.com/docs/config_backup/config_backup.html

But as always it’s best to have a reliable tested backup before doing anything like this.

And as for the new install itself we have the following guide which covers every step of the new install process:

Rockstor’s “Built on openSUSE” installer - Beta release
http://rockstor.com/docs/installer-howto/installer-howto.html

Re:

Not required. The above DIY installer maker pre-intergrates a know good rpm version and you can then update to your preferred updates channel. Or not.

The issue here is that I think the default kernel in openSUSE Leap 15.2, our new base employed in Rockstor 4, also doesn’t yet include this newer raid level. Also Rockstor itself doesn’t yet cater for the > 2 copies of raid1 variant either. I’m actually not sure how it will react either. But we definitely intend to add these newer profiles. A third concern here is that if you btrfs subvolume arrangement is not known / recognised by Rockstor then it will not be recognised by Rockstor. At least in the sense of subvols recognised anyway.

Thanks, yes we were having a number of looming and actaul issues with the CentOS variant, hence the ‘Built on openSUSE’ effort. In the end we couldn’t even build the rpms on CentOS hence our final Stable channel release on that OS being 3.9.2-57, but we are now on 4.0.5 for the Leap 15.2 rpms:

OK, haven’t heard of that one before.

That sounds like a likely candidate. But still shouldn’t happen.

Yes, very annoying.

I would suggest, given your obvious patience/perseverance that you strongly consider the Rockstor 4 DIY installer build route. I know it’s a nuke and pave option. But if you do have issues there then at least we can try and fix them if they originate in the Rockstor code. Plus you will be that bit closer to a newer kernel for your exotic raid levels: which you seem to be managing as-is anyway. Also when we do properly manage those raid levels you can help with our treatment of them.

I say have a go at making the Rockstor 4 installer and if all does not turn out rosy within your patience/perseverance limits then post the exact issue you are experiencing here on the forum, ideally in a new thread. That way the growing number of forum members who have managed to build the installer can chip in and steer you right. Plus it will help us see where our docs are lacking on this procedure as I’d really like for it to be as accessible as possible.

Apologies for offering only a work around here, but our development effort is now sole focused on the ‘Built on openSUSE’ 4 variant and in that regard it would be easier to help you going forward anyway.

Hope that helps.

3 Likes
3

@phillxnet Has linked the resources for building the opensuse Rockstor, and they are pretty straightforward to follow. If you are able to set up a VM somehow, it’s a great way to build the installer - and extremely easy to drop and restart if things don’t go to plan.

However, follow the instructions above and there’s no reason why things shouldn’t go to plan. Give it a shot, and I’m quite happy to chip-in to help your journey. It is quite rewarding to see that installer file pop out the sausage machine :slight_smile:

I cannot comment on the other issues you are having I’m afraid, but I do recommend giving the v4 installer a go.

3 Likes
4

I guess that’s fair. Pretty hard to expect you to backtrack in the codebase too much, and things are bound to eventually break as things get updated.
I guess I’ll have to figure out how to do this VM thing anyway, its probably about time. I’m a bit curious, what exactly is holding back from uploading an installer? Is it like… licensing or something?

I guess the thing that has held me back is when it started talking about running specific versions of linux distros that I barely recognize the name of, and in a VM to boot, I just kind of figured there was a learning curve there that I didn’t have the free time to pursue at the time. Then again, since I’m spending quite a bit of time troubleshooting and limping what I’ve got along, at some point I will have to, I guess. Headache now or headache later, but I’ll be annoyed until I do.

While I understand what a VM is, and have used them (to a pretty limited extent) before, this will be interesting anyway.
Wish me luck…

1 Like
5

@koskee

Re:

That was my thinking really. Plus what you have diagnosed and apparently done already on the CentOS side is far more than would be required to build the new installer; complexity wise.

We did release for both Leap and CentOS for over a years of so. But in the end we simply couldn’t build on CentOS anyway due to a dependency issue involving setup tools I think it was. Anyway all in the past now.

We still have folks installing 3.9.1 from 2017 and applying zero updates. We failed to release 3.9.2 (testing channel 3.9.1-15 or so) as it had broken replication. And by the time replication was fixed the ‘Built on openSUSE’ endeavour had begun as a concept. Inevitably it was a larger than expected endeavour. But we are almost there now. Except we found at the last minute that some AD stuff was flat out broken. This also indicated that we have insufficient testing in place both in-house and in the community. So I’ve back off a little from the release cadence to make sure we release a more proper instance. And the ISO release, as indicated by folks running 3.9.1 still without any updates, marks some what of a statement for many. I’m also very keen to spread the DIY capability of the ISO builds. And unless folks have to do this they often wont ever explore it. Plus we are in testing still and the intention is to only offer paid support for example on the stable releases. All in I’m reluctant to label the final Stable 4 until we are fairly assured it is sound. Especially since it will represent our first release in 3 years and in a very real sense a re-launch for the project as a whole since RedHat announced their removal of btrfs from even the tentative technical preview status. Hence the requirement to re-base on openSUSE. Just as well as it turns out given the more recent news re CentOS.

I hope note :slight_smile: . Whenever you install openSUSE or the Rockstor 4 derivative, you will notice an initial license within the installer. Rockstor’s License here is a straight sed edit of the ‘Marks’
rockstor-installer/config.sh at master · rockstor/rockstor-installer · GitHub
so that we might comply with the openSUSE guidelines for derivatives. Their docs, from my reading, are highly encouraging of derivatives and they go out of their way to help with such things. And as far as we are aware we have done all that is required. But as stated in our developer notes:

And since then we have released our DIY installer recipe which helps us test our installer (which uses kiwi-ng, again from the openSUSE/SuSE folks) prior to releasing the binary variant. We also had a regression issue with kiwi-ng where the resulting installer images were no longer compliant where they had been previously:

https://github.com/OSInside/kiwi/issues/1575

As usual the kiwi-ng folks were super helpful and responsive.

We have also had GitHub interactions already with the prior openSUSE board chairman Richard Brown aka sysrich here:

https://github.com/rockstor/rockstor-core/issues/2024#issuecomment-472803765

and the recent openSUSE board member Neal Gompa (ニール・ゴンパ)
Conan-Kudo (of openSUSE/Fedora fame) advising on our Python 2 to 3 move here:

https://github.com/rockstor/rockstor-core/issues/1877#issuecomment-678662593

We also have, among our forum members and Rockstor installers, a CTO @ SUSE UK. So we are not unknown to the openSUSE/SuSE folks. But as per our recent ‘Built on openSUSE’ installer doc howto:

http://rockstor.com/docs/installer-howto/installer-howto.html

we are keen, obviously, to be a good open source citizen. And our use in fact of the term “Built on openSUSE” here on the forum and in our docs, and the term “Uses openSUSE” within the Web-UI itself comes from the openSUSE guide here:

openSUSE:Trademark guidelines

openSUSE:Trademark guidelines - openSUSE Wiki

subsection:

Distributing openSUSE With Modifications:

https://en.opensuse.org/openSUSE:Trademark_guidelines#Distributing_openSUSE_With_Modifications

Where these two variants are among those suggested.

Me and @Flox have also been actively involved in reporting and testing issues with kiwi-ng during the development of our installer recipe. Plus I’ve made minor upstream contributions to the shellinabox package, all using obvious rockstor related identities. I.e. check out the last (to date) changelog entry here (Tue Mar 03 15:21:00 GMT 2020):

https://build.opensuse.org/package/view_file/shells/shellinabox/shellinabox.changes?expand=1

which was submitted from our official “rockstor” OBS account:
https://build.opensuse.org/request/show/781298

So as stated: I hope we are OK on the license side. Plus I hear from our Ten64 hardware partner, see:
“Rockstor on Ten64 HOW-TO and notes”
https://ten64doc.traverse.com.au/applications/rockstor/
that the openSUSE/SuSE folks are nothing if not helpful, and from my own meagre interactions this has definitely been the case. I just want to have a good show out of the gate so we don’t let the side down :slight_smile: .

But yes, I would really like to soon publicly distribute a pre-built installer but I’m awaiting confirmation that we ‘meet the mark’ re ‘Marks’ use prior to doing this. But currently I’ve had my plate full with some last minute hick-ups concerning background tasks and our use of a long abandoned library to mange these. So I’ve yet to chase up on this side of things.

Oh well, all in good time.

Yes, it’s actually pretty straight forward once you get going. And “Good luck”.

Do remember to take notes of ‘stuff’ that is not clear (read pain points) as I’m super keen on making this admittedly potentially intimidating process as straight forward as possible. We have already made a number of changes to the Readme in that repo so do feel free to make your own pull requests there if you fancy. Or open a forum thread as stated earlier in this thread.

Hope that helps.

1 Like
6

So I’ve been trying to get the VM up and running but I’m encountering this error when I run ‘vagrant up’:

(I’m currently using my windows machine since I can’t even seem to get Virtualbox to install on my linux mint 18 laptop)

C:\rockstor-installer\vagrant_env>vagrant up
The plugin ‘vagrant-vbguest’ is not currently installed.
Bringing machine ‘rockstor-installer’ up with ‘virtualbox’ provider…
==> rockstor-installer: Box ‘opensuse/Leap-15.2.x86_64’ could not be found. Attempting to find and install…
rockstor-installer: Box Provider: virtualbox
rockstor-installer: Box Version: >= 0
==> rockstor-installer: Loading metadata for box ‘opensuse/Leap-15.2.x86_64’
rockstor-installer: URL: HashiCorp Cloud Platform
==> rockstor-installer: Adding box ‘opensuse/Leap-15.2.x86_64’ (v15.2.31.344) for provider: virtualbox
rockstor-installer: Downloading: https://vagrantcloud.com/opensuse/boxes/Leap-15.2.x86_64/versions/15.2.31.344/providers/virtualbox.box
rockstor-installer:
An error occurred while downloading the remote file. The error
message, if any, is reproduced below. Please fix this error and try
again.

SSL certificate problem: self signed certificate in certificate chain

I’m not sure how to install the vbguest plugin, but I’m not sure if its necessary because it isn’t mentioned in the docs at all that I can find. I located a github repo that seems to have the source code for it but with no clear way of installing it necessarily.

I also tried ssh ing into my rockstor box and installing virtualbox there, but it was giving me this error:

yum install VirtualBox-6.1

Loaded plugins: changelog, fastestmirror
Loading mirror speeds from cached hostfile

https://wiki.centos.org/yum-errors

If above article doesn’t help to resolve this issue please use https://bugs.centos.org/.

Resolving Dependencies
→ Running transaction check
—> Package VirtualBox-6.1.x86_64 0:6.1.18_142142_el7-1 will be installed
→ Processing Dependency: libSDL-1.2.so.0()(64bit) for package: VirtualBox-6.1-6.1.18_142142_el7-1.x86_64
→ Processing Dependency: libvpx.so.1()(64bit) for package: VirtualBox-6.1-6.1.18_142142_el7-1.x86_64
→ Running transaction check
—> Package SDL.x86_64 0:1.2.15-17.el7 will be installed
—> Package libvpx.x86_64 0:1.3.0-8.el7 will be installed
→ Finished Dependency Resolution

Dependencies Resolved

===========================================================================================================================================
Package Arch Version Repository Size

Installing:
VirtualBox-6.1 x86_64 6.1.18_142142_el7-1 yum.oracle.com_repo_OracleLinux_OL7_developer_x86_64 97 M
Installing for dependencies:
SDL x86_64 1.2.15-17.el7 base 206 k
libvpx x86_64 1.3.0-8.el7 base 498 k

Transaction Summary

Install 1 Package (+2 Dependent packages)

Total size: 97 M
Installed size: 221 M
Is this ok [y/d/N]: y
Downloading packages:
warning: /var/cache/yum/x86_64/$releasever/yum.oracle.com_repo_OracleLinux_OL7_developer_x86_64/packages/VirtualBox-6.1-6.1.18_142142_el7-1.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID ec551f03: NOKEY

Public key for VirtualBox-6.1-6.1.18_142142_el7-1.x86_64.rpm is not installed

after I went thru the vb install docs to add the repo to yum, which seemed to work other than it won’t let me add the public key. It accepted the key but still reported that it wasn’t installed? I tried adding their repo file to /etc/yum.repos.d/ as well but it didn’t like that either and told me to disable it.

7

Oh nevermind, I think my antivirus was interfering with the connection. It seems to be working now that I’ve tried it again and added an exclusion to the av engine

3 Likes
8

Well alright then, that went surprisingly well despite using windows to build it.

There was a few caveats that while not impossible to figure out, would have been good to know from the start. Things like dependencies that could have been installed all at the same time but instead I had to go through the build process about 4-5 times as it kept failing. I built the entire thing using the default settings (only 2 cores so it took a while), and some other tips that would have made things easier and/or quicker if I had known.

That being said, it seems to have worked perfectly as far as I can tell. Since windows is much more commonly used by the average person, I think a targeted how-to guide on that platform would be pretty beneficial for the project and its overall usership while waiting for the iso version to be posted. Especially because users on that platform are much more likely to be intimidated of the process, and because typically windows installs are basically all the same for the most part since its much less modular of an OS. ie perfect for one size fits all instructions.

I did make a few notes on what was required specifically and as such, I can probably contribute a pretty refined windows install method for you guys when I have a minute to test it out and verify the correct order of steps, best practices, etc.

My question is now, what is the best way to submit a PR? I ask because there are several places (repos, wikis, sourceforge, etc) that I could probably submit it to, but let me know which one(s) you prefer.

1 Like
9

@koskee Thanks for the update & method test. And well done on an apparently successful installer build. You don’t mention testing the resulting installer however?
Re:

That would be great, thanks. Looking forward to it.

I’m thinking in the installer repo itself.

That way it can be maintained there as things change within that repo. As you state it’s likely most will just use the ISO when available so we are still in the land of the DIY here and that is catered for within the repo itself. It keeps it more self contained that way. However we do already have a link out to the docs but that is the using the end result, and our docs are more end user focused. Where as this repo is more for those taking the DIY route.

And given you have presumably used the vagrant / virtual box method, I’m guessing, then the required additions would be to it’s associated README.md file:

That way we improve on what is there rather than forking and creating yet another set of instructions. It might also be a good idea to link from the top README.md to this ‘alternative’ more automated approach.

As for the procedure it’s a generic pull request to that repo so we can review the changes. And if you fancy mulling over the approach you intend to take re the instructions a little more you could open a fresh forum post with your intentions to see what others thing.

As always I’m up for other suggestions but that’s my vote for where the improvements might go. Plus you mention missing dependencies / setup instructions on the MS windows side: these might be automated within the included script. Which would again reinforce that repo as the right place for this to go. I’m keep for that repo to be a one-stop-shop for how to build the ISO and it’s assumed it will likely only be taken on by the more technically curious, as is the act of building/installing your own DIY NAS anyway.

Glad you’ve gotten over the first hurdle and thanks again for the offer to help with improving the method/instructions.

We do have the following doc guide for GitHub contributions but it has yet to receive rockstor-installer specific instructions:
Community Contributions
http://rockstor.com/docs/contribute_section.htmlhttp://rockstor.com/docs/contribute_section.html

Hope that helps and I look forward to your suggested windows based improvements for our windows based options. Bit by bit.

1 Like