AD Service throws an error

[maz (John Masterson)]

Brief description of the problem

I just finished installing the 3.9.0 iso and starting to configure the storage. We are an active directory shop and as such I need the AD integration to work. When I supply a known working domain account and password, the system generates a traceback and an error:

expected string or buffer


Detailed step by step instructions to reproduce the problem

Install system, configure storage, create share, create export. Then, select System->Services-> Active Directory. Click the wrench and fill in the pertinent information. Finally, turn the service on.

Web-UI screenshot

Error Traceback provided on the Web-UI

Traceback (most recent call last): File "/opt/rockstor/src/rockstor/rest_framework_custom/generic_view.py", line 41, in _handle_exception yield File "/opt/rockstor/src/rockstor/smart_manager/views/active_directory.py", line 199, in post smb_config = self._get_config(smbo) File "/opt/rockstor/src/rockstor/smart_manager/views/base_service.py", line 40, in _get_config return json.loads(service.config) File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads return _default_decoder.decode(s) File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) TypeError: expected string or buffer

[maz (John Masterson)]

Just to add some info: we are, on the Linux side, a redhat shop. We also have some Ubuntu systems. All are running sssd successfully. Being a bright boy (debatable), I used the config files for sssd and samba from our existing systems to setup Rockstor sssd. After restarting the daemon, I was able to resolve IDs against the DC’s.

So I went back into the webUI and selected System->Services->Active Directory. Checked and my information is still there; username domain, etc… The service was still listed as off, when I set it to on I get the same error as above.

[phillxnet (Philip Guyton)]

@maz First off; welcome to the Rockstor community forum.

I’m not up much on AD stuff myself (although @Flyer is) but I can hopefully provide a few pointers. Bear in mind that I may have this wrong though but these doc pointers and history may help anyway.

Rockstor currently uses the Winbind approach I believe:

How Rockstor used to work with Active Directory

This is the old implementation which is not valid as of 3.8-9. It’s retained here for reference. For current implementation, please see this post. This is a wikified post documenting the internal implementation details of AD configuration in Rockstor. Please feel free to update/edit this post to make it better. This will serve as our reference document to help developers make Rockstor work flawlessly with AD. ##User story and implementation summary User should be able to add their Rockstor sy…

the above actually states how it used to work as then for a bit the AD system was switched to sssd which I personally was quite exited about:

Rockstor Active Directory(AD) integration implementation

This is a wikified post documenting the implementation of Active Directory integration in Rockstor. It was done differently before version 3.8-9 using winbind as described here. This new approach uses sssd and is a much simpler and recommended one as per RHEL/CentOS documentation. ##User story and implementation summary User should be able to add their Rockstor system to their Active Directory domain controller. We use realmd, which provides and simple way to discover and join Active Directory…

this was of major interest to me and I became much more interested in this area myself. However soon there after (3.8-11.15) I believe this subsystem was reverted back to using windbind, so I again have little interest myself as I think the sssd route is really the one we should be using but as you can see from @suman’s comment upon having to revert back there is a will to return to the sssd approach:

Dev log for upcoming 3.8-12 [Latest: 3.8-11.18]

3.8-11.15 is now available! This is a special one where we fixed two issues that have been painful to many users. AD integration has been broken in a few different ways with our sssd approach. It looks great in theory and I am open to bringing it back again, but many users ran into issues including Rockstor dropping off the domain, id mapping issues and windows ACLs. So, we are bringing back the winbind approach. We’ve tested it quite a bit and encourage everyone to do the same. It has bee…

So ultimately at the time it was the best decision from a pragmatic point of view but I am myself only interested in working on sssd in this area so I was hoping at one point to be able to help with bringing back the sssd approach (possibly along side the winbind approach as an either/or type thing); it was quite a nice patch set as well (by @suman) as I remember it. However I have had my hands full with other Rockstor issues so haven’t gotten around to even looking at it.

Hope that helps and really @Flyer, and of course @suman, are the folks on the dev side for AD functionality.

Your reported error looks like we have a parsing bug of some sorts however. But I’m not going to be able to look into this myself for a little while I’m afraid. Hopefully the above and other contributions to this thread may well help get you sorted until what ever has gone wrong on the Rockstor side is pinned down and resolved.

[Flyer (Mirko Arena)]

Pitching in:
try to configure AD Service, switch on/off samba service, switch on AD

Let us know, but probably we’ll have to check code over this

File "/opt/rockstor/src/rockstor/smart_manager/views/active_directory.py", line 199, in post
smb_config = self.getconfig(smbo)
File "/opt/rockstor/src/rockstor/smart_manager/views/base_service.py", line 40, in getconfig
return json.loads(service.config)

Mirko

[maz (John Masterson)]

Thank you for pitching in! That seems to have resolved the errors/traceback. I just got in and wanted to let you know that initializing the smb.conf file seems to have allowed the system to join the domain. I will be doing more testing and will update this ticket with status. Once I have finished, I will resolve/close this ticket.

Thanks,

John

[Flyer (Mirko Arena)]

Thanks @maz,
adding a bookmark to this thread, that missing smb.conf is a regression:
we should have a smb.conf with AD only too, going to open a Github issue

Mirko

[maz (John Masterson)]

OK Rockers. I have not had much time to work on the Rockstor AD integration using sssd. The boss has given me time from the day to day to work on Rockstor; he sees the value in a solid open source NAS. So I hope to making some progress. I have setup my github account and have forked the core. I have a VM setup and running and have been familiarizing myself with the workflow.

So, is there a decoder ring for how all the pieces fit together; full disclosure, I am not a python coder. I can hack at it, but do not have much experience with python. So if I could get some help with the flow of the logic around the Active Directory integration. How do I flip RS back to using SSSD? How are the config files populated? Are there templates for the config files?

Thanks,
John

[phillxnet (Philip Guyton)]

@maz Hello again.

@maz

The boss has given me time from the day to day to work on Rockstor; he sees the value in a solid open source NAS.

Good news.

@maz

… is there a decoder ring …

Nice term but unfortunately not, at least as of yet. I believe @suman has some diagrams in the works but don’t think they are as of yet release ready.

@maz

… but do not have much experience with python.

I wouldn’t worry about that, python is very friendly and mostly the project’s complexity comes from it’s scope and the django elements. I would say that we are a bit weak on code comments however so as you work stuff out a nice addition is to add comments as you go, this has been a more recent and ongoing effort so not a whole lot of coverage just yet: but getting there.

@maz

How do I flip RS back to using SSSD?

I would say that this is not something we can do, given that this is exactly what @suman did and then undid in the referenced release notes repeated for ease of reference:

Dev log for upcoming 3.8-12 [Latest: 3.8-11.18]

3.8-11.15 is now available! This is a special one where we fixed two issues that have been painful to many users. AD integration has been broken in a few different ways with our sssd approach. It looks great in theory and I am open to bringing it back again, but many users ran into issues including Rockstor dropping off the domain, id mapping issues and windows ACLs. So, we are bringing back the winbind approach. We’ve tested it quite a bit and encourage everyone to do the same. It has bee…

The end result of the original ‘flip’ was that many users just can’t manage with sssd and had compatibility issues. From my understanding the two systems have a slightly different bias with sssd having a more independent approach where it relied more on trusted domain relationship with existing domains where as windbind was happier / more compatible in some ways. I’m afraid I just don’t know enough to be more helpful here but that was my understanding after having a little look. It may well be that sssd has come on far enough since then to risk another change but given it seems to be quite a popular feature I think the way to go would be to enact the two systems in a mutually exclusive fashion. This is going to be more difficult but I think will lead to the best overall compatibility and choice. And hopefully avoid the breakage we encountered last time the flip approach was attempted.

So the approach I was going to take was to re-introduce @suman prior work but with the addition of an initial check to see if an existing winbind service was active and warn that the two implementations are incompatible and so refuse to enable one when the other was enabled. This is a little messy but I don’t see a way around it.

@maz

So if I could get some help with the flow of the logic around the Active Directory integration.

I would say a re-read of the above referenced wifi entries (remembering that they are now reversed again) and to study what @suman did last time, along with the what was then re-instated.

So the following announcement and pull requests plus any others you can find that touch the same files (git history) are going to be the key:
Note that this was a long time ago now and much has changed but the origin and development of the prior sssd approach should be usefull:

Dev log for upcoming 3.8-9

I finally managed to improve the Active Directory integration code. The new approach is looking much better and detailed here. So, 3.8-8.07 update is now available. I’d greatly appreciate if AD users out there give it a spin. Here’s the issue for this work.

and it’s related pull request:

860 ad fix

Note worthy in that pr is the following commit: “remove deprecated winbind service. #860”

There was then a significant improvement announced a little while later in the 3.8-12 dev log:

Dev log for upcoming 3.8-12 [Latest: 3.8-11.18]

It’s been a bit slower development pace last few days, I spent way too much time than I wished on fixing up AD integration issues. Thanks to users for reporting issues and @JBelthoff especially for providing deeper insights into a winbind based setup. Some of you are aware, I am sure, that Rockstor uses the newer SSS based approach summarized here. I almost gave in and went back to winbind, but I was able to fill in some gaps and everything seems to be working in my environment with 2012 R2 AD s…

which referenced the following pr:

1024 server role

ie look to the files changed tab in that pr and the commit messages.

But then shortly there after Rockstor reverted to windbind as covered earlier ie:

Dev log for upcoming 3.8-12 [Latest: 3.8-11.18]

3.8-11.15 is now available! This is a special one where we fixed two issues that have been painful to many users. AD integration has been broken in a few different ways with our sssd approach. It looks great in theory and I am open to bringing it back again, but many users ran into issues including Rockstor dropping off the domain, id mapping issues and windows ACLs. So, we are bringing back the winbind approach. We’ve tested it quite a bit and encourage everyone to do the same. It has bee…

and it’s associated pr:

1122 ad

Be sure to read all the commit messages as that will give you a good idea of how it all works and why.
Ie the following is particularly relevant:
https://github.com/rockstor/rockstor-core/pull/1174/commits/27fd688d500ba42717e2db210aad93d0ae119ce9

@maz

How are the config files populated?

By programmatic in-line editing (mostly) ie a services config is used to replace an existing config entry if found or added otherwise, varies slightly form service to servcie.

@maz

Are there templates for the config files?

Sometimes, ie some services start with a template and then edit it while others just attempt to edit what ever they find there in the first place. Some times Rockstor machine edits are put at the of a file but often this is not viable and inline edits are required to enact a config change without duplicate entries for the same item.

Hope that helps.

I would say that this is a slightly monster task to start with but if your game then great. It might be more encouraging to start with something smaller to ‘warm up’ ie picking something from the issues list:

Issues · rockstor/rockstor-core

Or anything you find otherwise which is a-miss.

And I’m assuming you are already familiar with the Developers section of the Contributing to Rockstor - Overview docs.

Good luck, have fun, and @Flyer and @suman are your main contacts here as I’ve done very little in this area of the code my self.

See how you get on and always ask question when needed. Might be best to start a new thread and link back to this one for context.

Hope that helps, at least with pointing out the genesis, development, and hopefully temporary deprecation of a prior attempt to enact sssd based active directory capability.