Disc Activity Dashboard widget is not working

oivanenko · March 31, 2018, 12:07pm

Hi, all!

Disc Activity Dashboard widget is not working in my Rockstor instance. What should I do to fix it?

phillxnet · March 31, 2018, 12:55pm

@oivanenko Welcome to the Rockstor community.

That’s a funny one, and haven’t seen it before (except on mdraid devices). Could you post a pic of your Disks page as that may help to see the nature of your disks. Could you also give us more info on the nature of your setup, ie virtualisation or not etc. Type of hardware how the disks are attached etc.
Also has the Disk Activity widget ever worked for you?

My initial guess is that your particular setup (raid hw etc) doesn’t present disk statistics in /proc/diskstats as that is where those statistics are gathered from ie:

github.com

rockstor/rockstor-core/blob/master/src/rockstor/smart_manager/data_collector.py#L565-L578


def disk_stats(prev_stats):


    disks_stats = []
    # invoke body of disk_stats with empty cur_stats
    stats_file_path = '/proc/diskstats'
    cur_stats = {}
    interval = 1
    # Build a list of our db's disk names, now in by-id type format.
    disks = [d.name for d in Disk.objects.all()]
    # /proc/diskstats has lines of the following form:
    #  8      64 sde 1034 0 9136 702 0 0 0 0 0 548 702
    #  8      65 sde1 336 0 2688 223 0 0 0 0 0 223 223
    with open(stats_file_path) as stats_file:
        for line in stats_file.readlines():

And the Disk Activity widget just displays what is gathered form there.

By way of further diagnosis, what does the following command output when run as root:

cat /proc/diskstats

You should see a series of lines such as are indicated in the code snippet comments above, one line for each device. If there is no output from that command then we have our culprit and there is little we can do from there other than change the nature of how your disks are presented / attached.

Thanks for the report and it would be good to identify the reason behind this. I have seen mdraid ‘breaking’ this widget but you make no mention of using this.

Hope that helps and with a little more info we should be able to know why this is happening.

oivanenko · March 31, 2018, 2:25pm

Hi, Philip!

Thank you for quick response.

My Disks page

My setup is very usual, I have 4 Western Digital SATA disks and old PC.

Motherboard: Asus P5KPL
CPU: Intel Pentium Dual CPU E2180
Memory: 4x1GB DDR2
Video: NVidia GT218 [GEForce 8400 GS Rev. 3]
Ethernet: TP-LINK TG-3269

WDC_WD30EFRX size: 3000.6GB
WDC_WD30EARx size: 3000.6GB
WDC_WD20EARS size: 2000.4GB
WDC_WD20EARX size: 2000.4GB

Yes, it was, but after one of updates (it was year or two ago, sorry for late message ) it was stop working.

Sorry, I’m not familiar with “mdraid”. I believe that it is not present in my setup.

If I can give you some logs, just say the word.

phillxnet · March 31, 2018, 3:19pm

@oivanenko Thanks for the info.

Yes, agreed; all looks to be fairly normal.

OK, so there might be a clue there. The Dashboard was re-written by @Flyer around about that time so it may just be that you have ‘old’ or stale client side javascript code still in your browser cache. Although it should have auto cleared by now so bit of a long shot; but have you tried doing a browser page refresh when viewing the dashboard. That’s all I can think of for the time being, That and disabling and re-enabling that widget in the hope that it refreshes some how. Everything else looks to be OK.

oivanenko · March 31, 2018, 3:42pm

Honestly, I don’t think so. I have many PC at home and many different browsers on its. And everywhere this widget does’nt working. And of course in that period I have reinstalled browsers and clearing their cache.

Unfortunately it is still not working after that.

Can I provide some logs or remote control for my PC for diagnostics?

oivanenko · March 31, 2018, 3:58pm

I don’t understand how to get github.com code reference, but if you go to https://github.com/rockstor/rockstor-core/blob/1c3135ab105394c50350aef8fab97814c636cfcb/src/rockstor/smart_manager/data_collector.py#L578-L587 you’ll see the lines:

            for line in stats_file.readlines():
                fields = line.split()
                # As the /proc/diskstats lines contain transient type names
                # we need to convert those to our by-id db names.
                byid_name = self.byid_disk_map[fields[2]]
                if byid_name not in disks:
                    # the disk name in this line is not one in our db so
                    # ignore it and move to the next line.
                    continue
                cur_stats[byid_name] = fields[3:]

Where I can check that condition? Where is that “our db”?

the disk name in this line is not one in our db so
ignore it and move to the next line.

phillxnet · March 31, 2018, 4:37pm

@oivanenko So I take it a browser refresh didn’t work for you then?

It is a puzzle this one.

The lines you references can be displayed within the forum by just pasting the link you provide on a line of it’s own with no leading spaces, the forum then does the rest and the results should appear in the preview window to the right.

The comments indicate disk names as stored in the db but they display as expected in the Disk page, hence me asking for a pic of that page. If they didn’t translate then the disk page would be full of big red warnings. And the byid_disk_map for the dashboard is generated on connection by the following code:

github.com

rockstor/rockstor-core/blob/master/src/rockstor/smart_manager/data_collector.py#L552-L556


def on_connect(self, sid, environ):


    self.byid_disk_map = get_byid_name_map()
    self.switch = True
    self.spawn(self.send_top_disks, sid)

which in turn calls via an import:

github.com

rockstor/rockstor-core/blob/master/src/rockstor/smart_manager/data_collector.py#L45


from gevent.subprocess import Popen, PIPE  # noqa E402
from os import path  # noqa E402
from sys import getsizeof  # noqa E402
from glob import glob  # noqa E402


from system.pinmanager import (has_pincard, username_to_uid,  # noqa E402
email_notification_enabled, reset_random_pins,
generate_otp)


from django.conf import settings  # noqa E402
from system.osi import uptime, kernel_info, get_byid_name_map  # noqa E402
from datetime import (datetime, timedelta)  # noqa E402
import time  # noqa E402
from django.utils.timezone import utc  # noqa E402
from storageadmin.models import Disk, Pool  # noqa E402
from smart_manager.models import Service  # noqa E402
from system.services import service_status  # noqa E402
from cli.api_wrapper import APIWrapper  # noqa E402
from system.pkg_mgmt import (update_check, yum_check)  # noqa E402
import logging  # noqa E402
logger = logging.getLogger(__name__)

the following code in system.osi:

github.com

rockstor/rockstor-core/blob/master/src/rockstor/system/osi.py#L1599-L1617


def get_byid_name_map():
"""Simple wrapper around 'ls -l /dev/disk/by-id' which returns a current
mapping of all attached by-id device names to their sdX counterparts. When
multiple by-id names are found for the same sdX device then the longest is
preferred, or when equal in length then the first listed is used. Intended
as a light weight helper for the Dashboard disk activity widget or other
non critical components. For critical components use only:
get_dev_byid_name() and get_devname() as they contain sanity checks and
validation mechanisms and are intended to have more repeatable behaviour
but only work on a single device at a time.  A single call to this method
can provide all current by-id device names mapped to their sdX counterparts
with the latter being the index.
:return: dictionary indexed (keyed) by sdX type names with associated by-id
type names as the values, or an empty dictionary if a non zero return code
was encountered by run_command or no by-id type names were encountered.
"""
byid_name_map = {}
out, err, rc = run_command([LS, '-l', '/dev/disk/by-id'],
                           throw=True)

So you see it just runs

ls -l /dev/disk/by-id

and creates a dictionary (map) of the drive names (smoke and mirrors). Simply to rapidly translate the temp sdx type names indicated in the /proc/diskstats output to the more stable Web-UI and Rockstor internal use by-id type names that are in our db. But for speed and to keep things light we don’t grab them from the db for at least this part of the dashboard.

As an additional thought, have you rebooted since you put that update in place, I see you have a long uptime. And if you don’t fancy that you could try:

systemctl restart rockstor

As that particular drive name mapping code is only used for this purpose as far as I can remember and it may be that it hasn’t been picked up on your system and so an older version of the pyc file associated with it is still in play. It’s rare but can happen. This could be checked by:

ls -la /opt/rockstor/src/rockstor/system/osi.*
-rw-r--r--. 1 root root 100171 Mar 31 16:31 /opt/rockstor/src/rockstor/system/osi.py
-rw-r--r--. 1 root root  60900 Mar 31 16:41 /opt/rockstor/src/rockstor/system/osi.pyc

Where in the above the data of the ‘compiled’ python is ‘akin’ to that of it’s source counterpart.

Incidentally your installed versions of this code are to be found in the following locations:

/opt/rockstor/src/rockstor/smart_manager/data_collector.py
/opt/rockstor/src/rockstor/system/osi.py

In case you fancy having a play. Though you might want to make a backup of the files there prior to making any changes (ie in /root): and note you will likely require that restart command to see the effect of any changes you make.

Let us know how you get on, it may be that we have an anomaly where the fast path map maker is not tallying with the names as displayed on the Disk page, but I really don’t think that will make any differences as they are really just for show, ie translate sdX to at least one by-id equivalent. If they don’t match the canonical db entries it really doesn’t matter as long as they are recognisably similar. As the comments in that last link indicate there can be more than one by-id name.

See how a rockstor service restart goes, or a reboot if that’s possible as we haven’t had this reported previously and many people use those drives.

Hope that helps and do take a look at those code references as we try to make the code as readable as possible with comments where needed hopefully.

phillxnet · March 31, 2018, 4:59pm

Ops, sorry looks like I slipped up there. We do check if the Disk is actually in the DB via the:

# Build a list of our db's disk names, now in by-id type format.
disks = [d.name for d in Disk.objects.all()]

And as you pointed out:

So if may well be we have an anomaly where by-id_name from our quick map is not lining up with the chosen db entry in your case. It could help to paste the output of the command used to build the quick name map:

ls -l /dev/disk/by-id

Then we can pin down if that is what’s happening. Would be a great fine that. And the fix would be to line up the outputs of def get_byid_name_map() and the ‘proper’ canonical counterparts of get_dev_byid_name() and get_devname() which have their own comments on how they select their names: so that shouldn’t be too difficult. Although I would only change the behaviour of the quick map get_byid_name_map() as that should only break that dashboard widget where as the others will break everything else if changed unfavourably.

Sorry for the double take here but just in the middle of preparing a large pr so a little distracted.

If you manage to track this one down then please see our Community Contributions doc and specifically the Developers section. Best prove a restart / reboot doesn’t address this first though.

oivanenko · March 31, 2018, 5:27pm

Such a big text is a challenge for Non-English speaker like me, sorry, but I will try

Bullseye!

So, I should remove “older version of the pyc file” somehow or what? How can I do this right and safe?

See output of “ls -l /dev/disk/by-id” on screenshot above too.

Let’s try to solve “old pyc” case and will see. I’ll try my best, but I’m not python programmer at all, sorry. I have some skills in html, jscript, perl and sql, but it was years ago

phillxnet · March 31, 2018, 5:55pm

This doesn’t actually look like it’s the problem.
I.e. the osi.py had data Oct 22:32 and it’s ‘compiled’ counterpart osi.pyc is a few minutes later as Oct 22 22:36 (ie 4 mins later) so this looks to be OK. Ie the installed osi.py was installed and a few minutes later was run for the first time and it’s osi.pyc variant was created. They are auto created but old ones can hang around some times.

So the fact that the .pyc is later (by 4 mins) than it’s .py counterpart shows that this is not the issue.

Se we are back to a difference in the output between:

get_byid_name_map()

and the canonical:

get_dev_byid_name()

Your screen shot of Disks page shows for example the disk with serial ending 440:
also has a “wwn” type entry in by-id (both symbolic links point to /dev/sda) but this entry is shorter as expected and why we have:

github.com

rockstor/rockstor-core/blob/master/src/rockstor/system/osi.py#L1601-L1603


mapping of all attached by-id device names to their sdX counterparts. When
multiple by-id names are found for the same sdX device then the longest is
preferred, or when equal in length then the first listed is used. Intended

So unless get_byid_name_map() is not doing what it states then I’m not sure now of the cause as the disks page indicates the contents of the db and our map is supposedly also using this longest name so they should align. Need more debugging I think.

How did the browser page refresh / cache clearance go, I’m assuming nothing doing on that front.

I’m going to have to leave this one myself for now but thanks for the report and I’ll try and take another look when I can.

Currently it’s looking like get_byid_name_map() is misbehaving but we don’t have any other reports of this and given your ls -l output all looks to be as expected.

I wouldn’t let that put you off. Tons of js in Rockstor and if you can perl / sql you can python.

Thanks again for engaging on this one and I will have a think.

Anyone else have any ideas on this?

phillxnet · March 31, 2018, 6:18pm

@oivanenko Just a quick one on this:

If you expand the widget are the drive names listed the same as those on the Disk page? If in fact any drive names are listed (in the drop down). There will only be one partition (-part3) listed for the system drive, if at all of course.

Cheers.

oivanenko · March 31, 2018, 6:24pm

I’m afraid all the disks are visible …

phillxnet · March 31, 2018, 6:37pm

@oivanenko Ok cheers.

Just to help try and narrow this down really.

Let us know if you manage to track this down and thanks again for the report and engaging with trying to sort it.

Hopefully someone else can jump in, bit tricky without a reproducer really as non of my systems here show this behaviour.

@Flyer is there something ‘special’ about this widget that could cause it to blank like this, ie could this just be a surface issue for the graphics in this widget? Note that all graphics components are missing but only in this widget. I.e. was it browser sensitive?

oivanenko · March 31, 2018, 7:01pm

I’m not sure that I correctly understand the expression “to track this down”, but if it mean that I need run some commands or send some info or even give you remote access to my rockstor – yes, for sure.

phillxnet · March 31, 2018, 7:59pm

@oivanenko
Apologies. Just meant that if you find any more out about this issue then let the forum know. I’ll post here if I find/think of anything else myself. Can’t look at this issue any more at the moment though as have a little backlog, and it doesn’t appear to affect others currently. Hopefully we will have more reports that we can cross references in the mean time.

Thanks for your patience and willingness to help, I just don’t see what could be causing it at the moment. But others here may well be able to help.

Sorry we couldn’t get this sorted right away. Everything just looks as it should at your end so we will have to wait for other reports that can be cross referenced.

Does anyone else experience this with their Disk Activity widget?

oivanenko · March 31, 2018, 8:19pm

Philip, thank you for your help. If I find something new I’ll let you know. I’ll be wait for good news too.

Flyer · April 1, 2018, 7:51am

HI @oivanenko and @phillxnet :),
I think this is more on client side rather then backend ( if it was a backend issue we probably won’t have that disks list)
@oivanenko can you check your browser console? I think we’ll find some “nice” red errors

M.

Edit with guess: while enlarging images I noticed a disk having a colon ( : ) inside name…probably that is breaking JS code

phillxnet · April 1, 2018, 12:18pm

@Flyer Thanks for helping out on this one, I agree it looks more like client side and good point on disk list.

Pretty sure this is not an issue as I have a number of SanDisk Extreme usb 3.0 system drives here and they also have a colon in the name, as per @oivanenko’s SanDisk Ultra Fit, keen observation however

Pic of working Disk Activity widget with “:” in name (3.9.1-16):

oivanenko · April 1, 2018, 8:18pm

Hi, Mirko!

Chrome 65.0.3325.181 64-bit shows next error only when I try to resize Disk Activity widget:

storageadmin.js:20667 Uncaught TypeError: Cannot read property ‘resize’ of null
at child.resize (storageadmin.js:20667)
at HTMLDivElement.dispatch (jquery-1.9.1.min.js:3)
at HTMLDivElement.v.handle (jquery-1.9.1.min.js:3)

Firefox ESR 52.7.3 (32-bit) shows next error when I try to resize Disk Activity widget

TypeError: _this.TopDisksChart is null[Подробнее] storageadmin.js:20667:9
DiskUtilizationWidget<.resize https://192.168.1.2/static/storageadmin/js/storageadmin.js:20667:9
bound self-hosted:915:17
b.event.dispatch https://192.168.1.2/static/js/lib/jquery-1.9.1.min.js:3:28285
b.event.add/v.handle https://192.168.1.2/static/js/lib/jquery-1.9.1.min.js:3:25025

Flyer · April 2, 2018, 7:37am

Hi @oivanenko,
thx for your console logs!
that TopDisksChartData null means disks widget not initialized probably because of missing data/issues with data so we have to move back on data coming from backend

M.