@raju_ga153
mhm, I can’t identify anything related to the Rockon problem you’re having. There is an error related to the replication service (have you enabled/tried the replication service between 2 Rockstor instances recently?) and an error related to writing a network connection definition to the database. I don’t think either of these two is causing your Rockon POST failure.
I assume, the drive where you installed the Rockstor package is mounted in read/write mode and not, for some reason, (e.g. lack of space) mounted as read-only?
I hope @Flox might have some other suggestion on what to check?
I have only one disk(500GB) and its been used as both ROOT and Home. And used Home as root for rockons-root. You can refer snapshots in below. You can confirm me that if it has any issue.
I am seeing issue with “docker” network connection and IPv6 tables. Pls cross check same.
On the docker network connection, I am not sure what you are referring to about the IPv6 tables. Looking at the screenshot, I have the same attributes.
Usually, the recommendation is to separate OS and everything else (also to make restores/reinstalls more stable) onto separate drives. I am not sure that, while not recommended, this is causing your Rockon refresh problem. Your physical disk shows only a low usage. However, you would want to create a separate share that is only for the Rockon service, and not intermingled with any other files. So, single disk comment from earlier aside, you want to create a separate share that’s representing the Rockon root.
I am seeing following info from Kernel(dmesg). Does this helps to debug my issue?
Aug 01 18:50:15 RazNAS kernel: r8169 0000:02:00.0: Direct firmware load for rtl_nic/rtl8168e-3.fw failed with error -2
Aug 01 18:50:15 RazNAS kernel: r8169 0000:02:00.0: Unable to load firmware rtl_nic/rtl8168e-3.fw (-2)
Aug 01 18:50:15 RazNAS kernel: RTL8211E Gigabit Ethernet r8169-0-200:00: attached PHY driver (mii_bus:phy_addr=r8169-0-200:00, irq=MAC)
Aug 01 18:50:15 RazNAS kernel: r8169 0000:02:00.0 eth0: Link is Down
Aug 01 18:50:18 RazNAS kernel: r8169 0000:02:00.0 eth0: Link is Up - 1Gbps/Full - flow control off
Aug 01 18:50:18 RazNAS kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Aug 01 18:50:18 RazNAS kernel: NET: Registered PF_PACKET protocol family
Aug 01 18:50:49 RazNAS kernel: device-mapper: uevent: version 1.0.3
Aug 01 18:50:49 RazNAS kernel: device-mapper: ioctl: 4.45.0-ioctl (2021-03-22) initialised: dm-devel@redhat.com
Aug 01 18:50:51 RazNAS kernel: BTRFS info (device sdb): use no compression
Aug 01 18:50:51 RazNAS kernel: BTRFS info (device sdb): disk space caching is enabled
Aug 01 18:50:51 RazNAS kernel: BTRFS info (device sdb): has skinny extents
Aug 01 18:50:55 RazNAS kernel: audit: type=1400 audit(1690896055.953:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="docker-default" pid=13652 comm="apparmor_parser"
Aug 01 18:50:56 RazNAS kernel: bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
Aug 01 18:50:56 RazNAS kernel: Bridge firewalling registered
Aug 01 18:50:56 RazNAS kernel: bpfilter: Loaded bpfilter_umh pid 13657
Aug 01 18:50:56 RazNAS unknown: Started bpfilter
Aug 01 18:50:56 RazNAS kernel: Initializing XFRM netlink socket
Not sure that this will address the Rockon issue, however it seems that your board is using Realtek hardware for the network device, which are not included in the Rockstor install at this time.
Based on this:
you can try to install them from the OpenSUSE repo using: zypper in kernel-firmware-realtek
or follow @Flox instructions further down in the thread to get the very latest ones, however that is a little bit more involved.
By following given solution, I am able to resolve issue with Realtek Network device. But still no luck on Rock-Ons.
I am also seeing below issue from Kernel(dmesg).
Aug 01 21:29:12 RazNAS kernel: audit: type=1400 audit(1690905552.226:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="docker-default" pid=13537 comm="apparmor_parser"
Aug 01 21:29:12 RazNAS kernel: bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
Aug 01 21:29:12 RazNAS kernel: Bridge firewalling registered
Aug 01 21:29:12 RazNAS kernel: bpfilter: Loaded bpfilter_umh pid 13543
Aug 01 21:29:12 RazNAS unknown: Started bpfilter
Aug 01 21:29:12 RazNAS kernel: Initializing XFRM netlink socket
I am fresh out of ideas right now, will continue to think about it.
I hope, @Flox might have a few minutes to provide some additional pointers on what to look for, but I know he’s very busy right now.
@raju_ga153 in the terminal, can you try curl https://rockstor.com/rockons/root.json
to see whether you’re getting back a list of the Rockons in json format? Just want to make sure that you don’t (for any weird reason) have connectivity issue. If you do get the list, then at least we have confirmed that the connection “works”.
ok, again out of ideas from my side. I assume, somewhere here, something goes wrong and does not return a clear error/exception message
@phillxnet, @Flox would this suggest temporarily turning django to debug mode to get better insights on where this might go wrong (e.g. during the save to the database)?
Thank you so much @Hooverdan for all.your time and help there! This is a rather curious one indeed.
Could be worth it, indeed, good idea! I’m not sure it’ll show anything else directly related to the update of Rock-Ons available itself, unfortunately, because I believe we have a wide range of error catching in place already in this area but it’s worth trying indeed. Ital may also reveal a related issue but in a completely different part of the code.
A quick question: you never had a successful update of the list for Rock-Ons available on this machine, correct? If yes, it may be helpful to have a look at the state of the storageadmin_rockons table in the database. I wonder if the process fails half way during the update, or before that. If the former, you would see some entries there; if the latter, you wouldn’t see anything.
I’m currently traveling for work so I can’t test for sure but I’ll see if I can setup something real quick to let you know how to do that. It would basically constitute of running a psql command.
RazNAS:~ # cd /opt/rockstor/
RazNAS:/opt/rockstor # poetry run debug-mode ON
DEBUG flag is now set to True
RazNAS:/opt/rockstor # clear
RazNAS:/opt/rockstor # tail -f /opt/rockstor/var/log/rockstor.log
File “/opt/rockstor/.venv/lib/python2.7/site-packages/django/db/models/sql/compiler.py”, line 1204, in execute_sql
cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)
File “/opt/rockstor/.venv/lib/python2.7/site-packages/django/db/models/sql/compiler.py”, line 899, in execute_sql
raise original_exception
OperationalError: deadlock detected
DETAIL: Process 13502 waits for ShareLock on transaction 1853; blocked by process 13467.
Process 13467 waits for ShareLock on transaction 1856; blocked by process 13502.
HINT: See server log for query details.
CONTEXT: while updating tuple (0,27) in relation “storageadmin_pool”
I don’t see anything obvious here.
This part is your “page refresh” action
This portion is the “Update” event after you pressed the button.
I tried the commands with my working system yesterday and saw the above messages as well during those actions, but in your case there also doesn’t seem be resulting in an obvious error.
The deadlock message on top of your post I did not see, but I suspect it’s not related to your issue since the context seems to be around the storageadmin_pool table (though it’s also something on the database, so I could be wrong):
I’m curious about that database deadlock… If it’s a continuous deadlock, that could explain it… Although it would most likely have a lot more repercussions throughout the UI.
Would this deadlock message still show up if you try the tail command again now?
It does seem that the error appears before any Rock-On info is fetched so that helps.
