[Beta] Can't change mgmt. IP-address (openSuse Leap Linux: 5.3.18-lp152.66-default / 4.0.4-0)

MainrtNr5 · March 14, 2021, 11:23am

When setting up Rockstor a couple of days ago the system got the IP-address 192.168.0.163 through DHCP. After installation I changed the IP-address through the web GUI to 192.168.0.80.

I could then access the system through this IP-address but started using the hostname (Rockstor) instead as that was easier, both for web access and for SMB access.

Today I decided to troubleshoot the issues with my Rock-Ons not using the correct config shares and realized that I no longer could access the system through the 192.168.0.80 IP-address. When checking the network configuration it had reverted to 192.168.0.163, despite still being set to Manual config method.

I changed the IP-address to 192.168.0.80 and saved but to no use, it’s still using the .163 address.

Changing back to Auto (DHCP) it kept the .163 address as expected. I changed back to Manual and set the .80 address but still to no use. The config still shows .163 and I can’t access the system on .80.

Only after rebooting did the IP-address change.

Perhaps this is by design but there’s nothing to indicate this when updating the network config.

phillxnet · March 14, 2021, 8:09pm

@MainrtNr5 Hello again and thanks for the report. Much appreciated.

Of note here is that we don’t present any Web-UI on port 80 (http) only 443 (https). So it may be there was some superstitious learning going on here :). If you could double check your findings while taking care to use only https and come-up with a repeatable easy reproducer then this could form the basis for a bug report issue on GitHub as this should all work as intended. And no a reboot shouldn’t be required.

Hope that helps, and thanks again for testing and reporting all this.

MainrtNr5 · March 15, 2021, 5:17pm

Hi Philip.

I think you misread my post. I never mentioned port 80, I did however abbreviate the 192.168.0.80 address to .80 at times.

phillxnet · March 15, 2021, 7:22pm

@MainrtNr5 Ops sorry about that. Still; hopefully the rest still stands as relatively sane.

Cheers.

MainrtNr5 · March 22, 2021, 7:43pm

I put together a short video where I reproduce the bug. I added on-screen info to describe what I’m doing, and why. Let me know if you need more in the form of logs or similar.

Rockstor network troubleshooting - YouTube

Flox · March 23, 2021, 12:33pm

Hi @MainrtNr5,

Thanks for the video… that is definitely a clear way to describe the issue!

I quickly tried to reproduce the first problem your video shows and unfortunately couldn’t reproduce it… On a fresh VM (running Rockstor-4.0.6), here’s what I did:

Edit Wired connection 1, the only one in this VM, and thus the currently active connection.
Change DHCP to manual
Edit IP address from 192.168.1.35 to 192.168.1.135
Clicking the “Submit” button works and gets me back to the Network page. On your video, this hangs with the blue loading arrows.
I could thus confirm that the changes were taken into account on the command line. This is where my lack of knowledge in NetworkManager shows, unfortunately, so excuse any inaccuracy. As this is the currently active connection, even though Rockstor does puts the connection down then up after this change, it is not actually in use as there’s a device with the old settings still at play. I could see this when looking into the settings themselves:
First, to identify the connection:

rockdev:~ # nmcli con show
NAME                UUID                                  TYPE      DEVICE
Wired connection 1  7590d252-8a23-40a1-ab87-442a48ca00f3  ethernet  eth0

Then, let’s look at the settings for this connection (I simplified the output for clarity):

rockdev:~ # nmcli c show 7590d252-8a23-40a1-ab87-442a48ca00f3
connection.id:                          Wired connection 1
connection.uuid:                        7590d252-8a23-40a1-ab87-442a48ca00f3
connection.stable-id:                   --
connection.type:                        802-3-ethernet
connection.interface-name:              eth0
(...)
ipv4.method:                            manual
ipv4.dns:                               192.168.1.1
ipv4.dns-search:                        --
ipv4.dns-options:                       --
ipv4.dns-priority:                      0
ipv4.addresses:                         192.168.1.135/24
ipv4.gateway:                           192.168.1.1
(...)
GENERAL.NAME:                           Wired connection 1
GENERAL.UUID:                           7590d252-8a23-40a1-ab87-442a48ca00f3
GENERAL.DEVICES:                        eth0
GENERAL.IP-IFACE:                       eth0
GENERAL.STATE:                          activated
GENERAL.DEFAULT:                        yes
GENERAL.DEFAULT6:                       no
GENERAL.SPEC-OBJECT:                    --
GENERAL.VPN:                            no
GENERAL.DBUS-PATH:                      /org/freedesktop/NetworkManager/ActiveConnection/3
GENERAL.CON-PATH:                       /org/freedesktop/NetworkManager/Settings/3
GENERAL.ZONE:                           --
GENERAL.MASTER-PATH:                    --
IP4.ADDRESS[1]:                         192.168.1.35/24
IP4.GATEWAY:                            192.168.1.1
IP4.ROUTE[1]:                           dst = 0.0.0.0/0, nh = 192.168.1.1, mt = 100
IP4.ROUTE[2]:                           dst = 192.168.1.0/24, nh = 0.0.0.0, mt = 100
IP4.DNS[1]:                             192.168.1.1
DHCP4.OPTION[1]:                        dhcp_lease_time = 86400
DHCP4.OPTION[2]:                        domain_name_servers = 192.168.1.1
DHCP4.OPTION[3]:                        expiry = 1616587172
DHCP4.OPTION[4]:                        ip_address = 192.168.1.35
DHCP4.OPTION[5]:                        requested_broadcast_address = 1
DHCP4.OPTION[6]:                        requested_domain_name = 1
DHCP4.OPTION[7]:                        requested_domain_name_servers = 1
DHCP4.OPTION[8]:                        requested_domain_search = 1
DHCP4.OPTION[9]:                        requested_host_name = 1
DHCP4.OPTION[10]:                       requested_interface_mtu = 1
DHCP4.OPTION[11]:                       requested_ms_classless_static_routes = 1
DHCP4.OPTION[12]:                       requested_nis_domain = 1
DHCP4.OPTION[13]:                       requested_nis_servers = 1
DHCP4.OPTION[14]:                       requested_ntp_servers = 1
DHCP4.OPTION[15]:                       requested_rfc3442_classless_static_routes = 1
DHCP4.OPTION[16]:                       requested_root_path = 1
DHCP4.OPTION[17]:                       requested_routers = 1
DHCP4.OPTION[18]:                       requested_static_routes = 1
DHCP4.OPTION[19]:                       requested_subnet_mask = 1
DHCP4.OPTION[20]:                       requested_time_offset = 1
DHCP4.OPTION[21]:                       requested_wpad = 1
DHCP4.OPTION[22]:                       routers = 192.168.1.1
DHCP4.OPTION[23]:                       subnet_mask = 255.255.255.0

As you can see, the settings are indeed taken:

ipv4.addresses:                         192.168.1.135/24

But there’s the active device keeping the old settings in:

IP4.ADDRESS[1]:                         192.168.1.35/24

After rebooting the machine, Rockstor is available on 192.168.1.135 and not 192.168.1.35, as expected.

I’ll try to see if I can dig further and understand why I can’t reproduce what you have in your video, but wanted to share this in case it rings a bell for our NetworkManager experts here.

Hooverdan · March 23, 2021, 3:34pm

Not that I really know whether I am looking at the right piece of code, but for the nmcli, shouldn’t the option reapply be used to make use of a changed interface (e.g. changed IP address) so the old interface config is discarded and the changed one applied? When looking here, I only see the reloading of connections (e.g. for newly defined interfaces):

github.com

rockstor/rockstor-core/blob/eeb0084a2ea6f6bdb1cdbe35428aa5021ddadc72/src/rockstor/system/network.py#L306




def toggle_connection(uuid, switch):
    return run_command([NMCLI, "c", switch, uuid])




def delete_connection(uuid):
    if valid_connection(uuid):
        return run_command([NMCLI, "c", "delete", uuid])




def reload_connection(uuid):
    return run_command([NMCLI, "c", "reload", uuid])




def new_connection_helper(
    name, ipaddr, gateway, dns_servers, search_domains, mtu=DEFAULT_MTU
):
    manual = False
    if ipaddr is not None and len(ipaddr.strip()) > 0:
        manual = True
        run_command([NMCLI, "c", "mod", name, "ipv4.addresses", ipaddr])

and I couldn’t see any mention of the device reapply switch after a connection change within the network.py area that would represent a call like
ncmli device reapply <ifname>
which would remove the need to reboot, and also avoid a “sticky” interface, if I am not mistaken.

MainrtNr5 · March 23, 2021, 5:56pm

Maybe it matters that I had already changed to manual and rebooted the server before performing the test, where you changed from DHCP to manual and then performed the test directly afterwards?

Also, it might be because you’re not running any Rock-Ons.

The reason for me suspecting this is that I’ve had a somewhat rare bug where I click on “Edit” for the wired connection and get the form for editing the Docker connection. I can’t reliably reproduce that bug though so I haven’t reported it yet.

Flox · March 23, 2021, 11:06pm

You are, don’t be too modest .

Yes, I’m wondering that too after looking at what is happening under the hood exactly. Briefly, when editing a connection, we delete the previous one, and create a new one with the new settings, and then reload it. For reference:

When editing an existing connection, we trigger the following:

github.com

rockstor/rockstor-core/blob/eeb0084a2ea6f6bdb1cdbe35428aa5021ddadc72/src/rockstor/storageadmin/views/network.py#L449-L454


if nco.ctype == "ethernet":
    device = nco.networkdevice_set.first().name
    self._delete_connection(nco)
    sysnet.new_ethernet_connection(
        nco.name, device, ipaddr, gateway, dns_servers, search_domains, mtu
    )

As we can see above, we:

step 1: get the device name of the current connection, to be used in step 3 below
step 2: delete the current connection

github.com

rockstor/rockstor-core/blob/eeb0084a2ea6f6bdb1cdbe35428aa5021ddadc72/src/rockstor/storageadmin/views/network.py#L538-L542


def _delete_connection(nco):
    for mnco in nco.networkconnection_set.all():
        sysnet.delete_connection(mnco.uuid)
    sysnet.delete_connection(nco.uuid)
    nco.delete()

step 3: we create a new connection with the newly-updated settings:

github.com

rockstor/rockstor-core/blob/eeb0084a2ea6f6bdb1cdbe35428aa5021ddadc72/src/rockstor/system/network.py#L331-L346


def new_ethernet_connection(
    name,
    ifname,
    ipaddr=None,
    gateway=None,
    dns_servers=None,
    search_domains=None,
    mtu=DEFAULT_MTU,
):
    run_command(
        [NMCLI, "c", "add", "type", "ethernet", "con-name", name, "ifname", ifname]
    )
    new_connection_helper(name, ipaddr, gateway, dns_servers, search_domains, mtu=mtu)
    # @todo: probably better to get the uuid and reload with it instead of
    # name.
    reload_connection(name)

It’s in the new_connection_helper() function that we modify the different settings (IP address, DHPC, etc…) using nmcli con mod ...:

github.com

rockstor/rockstor-core/blob/eeb0084a2ea6f6bdb1cdbe35428aa5021ddadc72/src/rockstor/system/network.py#L310-L328


def new_connection_helper(
    name, ipaddr, gateway, dns_servers, search_domains, mtu=DEFAULT_MTU
):
    manual = False
    if ipaddr is not None and len(ipaddr.strip()) > 0:
        manual = True
        run_command([NMCLI, "c", "mod", name, "ipv4.addresses", ipaddr])
    if gateway is not None and len(gateway.strip()) > 0:
        run_command([NMCLI, "c", "mod", name, "ipv4.gateway", gateway])
    if manual:
        run_command([NMCLI, "c", "mod", name, "ipv4.method", "manual"])


    if dns_servers is not None and len(dns_servers.strip()) > 0:
        run_command([NMCLI, "c", "mod", name, "ipv4.dns", dns_servers])
    if search_domains is not None and len(search_domains.strip()) > 0:
        run_command([NMCLI, "c", "mod", name, "ipv4.dns-search", search_domains])
    if mtu != DEFAULT_MTU:
        # no need to set it if it's default value
        run_command([NMCLI, "c", "mod", name, "802-3-ethernet.mtu", mtu])

@Hooverdan, it does seem like we could finalize the procedure with a nmcli d reapply <ifname> indeed, but it would need to be tested and verify it actually fix that need for a reboot.

Nevertheless, I don’t think this is the root cause of @MainrtNr5’s issue here so I’ll try to see what I can find about that.

That’s what I was thinking as well. I will need to set up a proper testing and debugging of that part of the code to see what is happening here.

Flox · March 23, 2021, 11:38pm

As usual, I might have spoken too soon… I still need to inspect what is really happening here in the background, but I could indeed reproduce @MainrtNr5’s issue (@MainrtNr5: you were right: with the connection already set to manual, I could reproduce what you see). After submitting the IP address change, it does hang (understandably, as that is the connection used for the webUI), and the DHCP-provided address was back in use. After a nmcli device reapply eth0, the new IP address set in the form was up and functional.
We still need to make sure this is actually what the root cause is, so that we can make sure it is the proper fix, but at least we’re making progress thanks to both of you!

MainrtNr5 · March 24, 2021, 4:58pm

No need to prioritize this over other work items, it’s not a deal breaker for me and I don’t want you to spend time on this just because I posted it to the forum when there might be other more pressing issues to deal with.