Retrieving serial # from NVMe device fails

f_l_a · July 1, 2016, 2:07am

Release

3.8.14

Issue

Logging

CommandException: Error running a command. cmd = ['/usr/sbin/smartctl', '--info', '/dev/nvme0n1']. rc = 1. stdout = ['smartctl 6.2 2013-07-26 r3841 [x86_64-linux-4.6.0-1.el7.elrepo.x86_64] (local build)', 'Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org', '', '/dev/nvme0n1: Unable to detect device type', 'Please specify device type with the -d option.', '', 'Use smartctl -h to get a usage summary', '', '']. stderr = [''] [01/Jul/2016 17:51:29] ERROR [storageadmin.views.disk:228] Error running a command. cmd = ['/usr/sbin/smartctl', '--info', '/dev/sde']. rc = 1. stdout = ['smartctl 6.2 2013-07-26 r3841 [x86_64-linux-4.6.0-1.el7.elrepo.x86_64] (local build)', 'Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org', '', '/dev/sde: Unknown USB bridge [0x0781:0x5581 (0x100)]', 'Please specify device type with the -d option.', '', 'Use smartctl -h to get a usage summary', '', '']. stderr = [''] Traceback (most recent call last): File "/opt/rockstor/src/rockstor/storageadmin/views/disk.py", line 226, in _update_disk_state do.name, do.smart_options) File "/opt/rockstor/src/rockstor/system/smart.py", line 311, in available [SMART, '--info'] + get_dev_options(device, custom_options)) File "/opt/rockstor/src/rockstor/system/osi.py", line 98, in run_command raise CommandException(cmd, out, err, rc)

Output of misc tools

lsblk

[root@storageserver ~]# lsblk -P -o NAME,MODEL,SERIAL,SIZE,TRAN,VENDOR,HCTL,TYPE,FSTYPE,LABEL,UUID NAME="sda" MODEL="WDC WD6002FFWX-6" SERIAL="NCGVL7AV" SIZE="5.5T" TRAN="sata" VENDOR="ATA " HCTL="6:0:0:0" TYPE="disk" FSTYPE="btrfs" LABEL="repo_pool" UUID="8bc44902-6437-47cd-b0f4-13c733de3f17" NAME="sdb" MODEL="WDC WD6002FFWX-6" SERIAL="K1G9Z23B" SIZE="5.5T" TRAN="sata" VENDOR="ATA " HCTL="7:0:0:0" TYPE="disk" FSTYPE="btrfs" LABEL="repo_pool" UUID="8bc44902-6437-47cd-b0f4-13c733de3f17" NAME="sdc" MODEL="WDC WD6002FFWX-6" SERIAL="NCGVP81V" SIZE="5.5T" TRAN="sata" VENDOR="ATA " HCTL="8:0:0:0" TYPE="disk" FSTYPE="btrfs" LABEL="media_pool" UUID="3bb70b4d-7456-4ee3-8511-bb14e67dcb2d" NAME="sdd" MODEL="WDC WD6002FFWX-6" SERIAL="K1GA0KYB" SIZE="5.5T" TRAN="sata" VENDOR="ATA " HCTL="9:0:0:0" TYPE="disk" FSTYPE="btrfs" LABEL="media_pool" UUID="3bb70b4d-7456-4ee3-8511-bb14e67dcb2d" NAME="sde" MODEL="Ultra " SERIAL="4C531001440716117184" SIZE="14.3G" TRAN="usb" VENDOR="SanDisk " HCTL="0:0:0:0" TYPE="disk" FSTYPE="" LABEL="" UUID="" NAME="sde1" MODEL="" SERIAL="" SIZE="500M" TRAN="" VENDOR="" HCTL="" TYPE="part" FSTYPE="ext4" LABEL="" UUID="2bfd7134-283f-4873-b9ae-6542fc32888e" NAME="sde2" MODEL="" SERIAL="" SIZE="1.4G" TRAN="" VENDOR="" HCTL="" TYPE="part" FSTYPE="swap" LABEL="" UUID="5fceda40-4d33-475a-a97c-79bc08cad2e7" NAME="sde3" MODEL="" SERIAL="" SIZE="12.4G" TRAN="" VENDOR="" HCTL="" TYPE="part" FSTYPE="btrfs" LABEL="rockstor_storageserver" UUID="5febd275-69b1-4d9f-9a17-2774c096e223" NAME="nvme0n1" MODEL="SAMSUNG MZVPV256HDGL-00000 " SERIAL="" SIZE="238.5G" TRAN="" VENDOR="" HCTL="" TYPE="disk" FSTYPE="" LABEL="" UUID=""

lspci
[root@storageserver ~]# lspci -v -s 07:00.0 07:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd Device a802 (rev 01) (prog-if 02 [NVM Express]) Subsystem: Samsung Electronics Co Ltd Device a801 Flags: bus master, fast devsel, latency 0, IRQ 19 Memory at de700000 (64-bit, non-prefetchable) [size=16K] I/O ports at c000 [size=256] Capabilities: [40] Power Management version 3 Capabilities: [50] MSI: Enable- Count=1/8 Maskable- 64bit+ Capabilities: [70] Express Endpoint, MSI 00 Capabilities: [b0] MSI-X: Enable+ Count=9 Masked- Capabilities: [100] Advanced Error Reporting Capabilities: [148] Device Serial Number 00-00-00-00-00-00-00-00 Capabilities: [158] Power Budgeting <?> Capabilities: [168] #19 Capabilities: [188] Latency Tolerance Reporting Capabilities: [190] L1 PM Substates Kernel driver in use: nvme

udevadm
(for /dev/sdx i see serials being retrieved)
[root@storageserver ~]# udevadm info --query=property --name nvme0 DEVNAME=/dev/nvme0 DEVPATH=/devices/pci0000:00/0000:00:1c.7/0000:07:00.0/nvme/nvme0 MAJOR=240 MINOR=0 SUBSYSTEM=nvme

[root@storageserver ~]# udevadm info --name nvme0n1 P: /devices/pci0000:00/0000:00:1c.7/0000:07:00.0/nvme/nvme0/nvme0n1 N: nvme0n1 E: DEVNAME=/dev/nvme0n1 E: DEVPATH=/devices/pci0000:00/0000:00:1c.7/0000:07:00.0/nvme/nvme0/nvme0n1 E: DEVTYPE=disk E: MAJOR=259 E: MINOR=0 E: SUBSYSTEM=block E: TAGS=:systemd: E: USEC_INITIALIZED=748316

Workaround?

Checking with tools from package nvme-cli was a success:
[root@storageserver ~]# nvme list Node SN Model Version Namespace Usage Format FW Rev /dev/nvme0n1 S1XWNYAH502774 SAMSUNG MZVPV256HDGL-00000 1.1 1 0.00 B / 256.06 GB 512 B + 0 B BXW7300Q

[root@storageserver ~]# nvme id-ctrl /dev/nvme0n1 NVME Identify Controller: vid : 0x144d ssvid : 0x144d sn : S1XWNYAH502774 mn : SAMSUNG MZVPV256HDGL-00000 fr : BXW7300Q rab : 2 ieee : 002538 cmic : 0 mdts : 5 cntlid : 1 ver : 0 rtd3r : 0 rtd3e : 0 oaes : 0 oacs : 0x7 acl : 7 aerl : 3 frmw : 0x6 lpa : 0x1 elpe : 63 npss : 4 avscc : 0x1 apsta : 0x1 wctemp : 0 cctemp : 0 mtfa : 0 hmpre : 0 hmmin : 0 tnvmcap : 0 unvmcap : 0 rpmbs : 0 sqes : 0x66 cqes : 0x44 nn : 1 oncs : 0x1f fuses : 0 fna : 0 vwc : 0x1 awun : 255 awupf : 0 nvscc : 1 acwu : 0 sgls : 0 ps 0 : mp:9.00W operational enlat:5 exlat:5 rrt:0 rrl:0 rwt:0 rwl:0 idle_power:- active_power:- ps 1 : mp:4.60W operational enlat:30 exlat:30 rrt:1 rrl:1 rwt:1 rwl:1 idle_power:- active_power:- ps 2 : mp:3.80W operational enlat:100 exlat:100 rrt:2 rrl:2 rwt:2 rwl:2 idle_power:- active_power:- ps 3 : mp:0.0700W non-operational enlat:500 exlat:5000 rrt:3 rrl:3 rwt:3 rwl:3 idle_power:- active_power:- ps 4 : mp:0.0050W non-operational enlat:2000 exlat:22000 rrt:4 rrl:4 rwt:4 rwl:4 idle_power:- active_power:-

Conclusion

So… extracting the S/N for devices managed by nvme driver by means of nvme tools?

phillxnet · July 1, 2016, 6:00pm

@f_l_a Thanks for posting this, it’s a nice find, however for the time being can a serial not be assigned via a udev rule of sorts. @snafu in the following on going forum thread also has nvme issues (on top of serial attribution), linking to that thread to hopefully tie together current nvme problems.

f_l_a · July 1, 2016, 6:46pm

@phillxnet Thanks for checking. Sure, can research howto implement this via udev; i’m not going to (re-)install RS on this device shortly (running on 16GB USB3 SanDisk drive in the meantime) anyway. So, yes, collect 'them all ;D

(may need to introduce different transports to select backend tools like smartctl according to device type? Like pcix/nvme; iscsi? Current types filtered for are usb, sata, scsi i think)

phillxnet · July 1, 2016, 6:57pm

Yes that’s a good idea and there are pending code changes in the pipeline to make such thing easier to identify from the device name (hopefully).

Bit by bit.

f_l_a · July 1, 2016, 7:32pm

The smartmontools devs are btw aware of missing device type support and implemented initial support just a couple of months ago. Seems smartctl v6.5 should bring us partly relief; RS currently defaults to v6.2:

EDIT:
Nothing newer than 6.2-4 in the official repos, rpmfind/pbone.net both list 6.5-1 packages for a handful of distros, of these FC ones have some chance to be compatible; but as expected dependencies won’t be satisfied :

[root@storageserver smartmontools]# rpm2cpio smartmontools-6.5-1.fc24.x86_64.rpm | cpio -idmv
./etc/smartmontools
./etc/smartmontools/smartd.conf
./etc/smartmontools/smartd_warning.d
./etc/smartmontools/smartd_warning.sh
./etc/sysconfig/smartmontools
./usr/lib/systemd/system/smartd.service
./usr/libexec/smartmontools
./usr/libexec/smartmontools/smartdnotify
./usr/sbin/smartctl
./usr/sbin/smartd
./usr/sbin/update-smart-drivedb
./usr/share/doc/smartmontools
./usr/share/doc/smartmontools/AUTHORS
./usr/share/doc/smartmontools/COPYING
./usr/share/doc/smartmontools/ChangeLog
./usr/share/doc/smartmontools/INSTALL
./usr/share/doc/smartmontools/NEWS
./usr/share/doc/smartmontools/README
./usr/share/doc/smartmontools/TODO
./usr/share/doc/smartmontools/examplescripts
./usr/share/doc/smartmontools/examplescripts/Example1
./usr/share/doc/smartmontools/examplescripts/Example2
./usr/share/doc/smartmontools/examplescripts/Example3
./usr/share/doc/smartmontools/examplescripts/Example4
./usr/share/doc/smartmontools/examplescripts/Example5
./usr/share/doc/smartmontools/examplescripts/Example6
./usr/share/doc/smartmontools/examplescripts/README
./usr/share/doc/smartmontools/smartd.conf
./usr/share/man/man5/smartd.conf.5.gz
./usr/share/man/man8/smartctl.8.gz
./usr/share/man/man8/smartd.8.gz
./usr/share/man/man8/update-smart-drivedb.8.gz
./usr/share/smartmontools
./usr/share/smartmontools/drivedb.h
3379 blocks
[root@storageserver smartmontools]# ll
total 456
drwxr-xr-x 1 root root 44 Jul 1 22:21 etc
-rw-r–r-- 1 root falcone 465686 Jul 1 22:18 smartmontools-6.5-1.fc24.x86_64.rpm
drwxr-xr-x 1 root root 38 Jul 1 22:21 usr
[root@storageserver smartmontools]# ./usr/sbin/smartctl
./usr/sbin/smartctl: /lib64/libstdc++.so.6: version GLIBCXX_3.4.20' not found (required by ./usr/sbin/smartctl) ./usr/sbin/smartctl: /lib64/libstdc++.so.6: versionCXXABI_1.3.9’ not found (required by ./usr/sbin/smartctl)
./usr/sbin/smartctl: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21’ not found (required by ./usr/sbin/smartctl)
[root@storageserver smartmontools]#

…i’m stopping here for now.

EDIT2 - Package built for FC22 complains the least, no older distro version supported:

./usr/sbin/smartctl: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20’ not found (required by ./usr/sbin/smartctl)

f_l_a · July 4, 2016, 7:08pm

Is RS integrating custom-built packages when needed, or do you rely for base packages solely on CentOS?
I’m thinking about setting up a build VM, but when you will integrate your own packages anyway…

phillxnet · July 4, 2016, 7:14pm

@f_l_a I seem to have forgotten the intended link so putting in here, different main problem but nvme related still. I have in turn linked back to this thread as your serial endeavours / discoveries may well help.

phillxnet · July 4, 2016, 7:16pm

@f_l_a I think we have a small number of custom packages, not sure exactly which (@suman) but yes mostly just upstream CentOS 7 and elrepo. I think AFP may be custom build but not sure.

phillxnet · July 4, 2016, 7:40pm

@f_l_a Just as a heads up, as I know you have now re-installed on a non nvme device now, but as of Rockstor 3.8-14.02 testing channel updates all smart calls on nvme devices and devices that have no serial should now be blocked. So at least those smartctl --info messages should no longer occur on these devices. We can easily remove this once the support in in place in smartmontools.

Not the main problem you encountered but at least it’s something.

Also if you fancy trying to patch in serial retrieval for nvme for the time being, prior to udev and co in CentOS getting updated the place to do it may well be in the get_disk_serial() function in src/rockstor/system/osi.py:

github.com

rockstor/rockstor-core/blob/master/src/rockstor/system/osi.py#L820-L839


      
                  config["dns_servers"] = l.split("= ")[1]
              elif re.match("DHCP4.OPTION.*:subnet_mask = .+", l) is not None:
                  config["netmask"] = l.split("= ")[1]
              elif re.match("IP4.GATEWAY:.+", l) is not None:
                  config["gateway"] = l.split(":")[1]
          
          
elif config["method"] == "manual":
              # manual
              if re.match("IP4.ADDRESS", l) is not None:
                  kv_split = l.split(":")
                  if len(kv_split) > 1:
                      vsplit = kv_split[1].split("/")
                  if len(vsplit) > 0:
                      config["ipaddr"] = vsplit[0]
                  if len(vsplit) > 1:
                      config["netmask"] = convert_netmask(vsplit[1])
              elif re.match("ipv4.dns:.+", l) is not None:
                  config["dns_servers"] = l.split(":")[1]
              elif re.match("ipv4.gateway:.+", l) is not None:
                  config["gateway"] = l.split(":")[1]

It is called whenever lsblk can’t retrieve a serial and maybe soon via:

it may be used by default.

There is already a patch in place, as indicated by the “Additional personality” comment, to deal with md devices as as special case so maybe an additional ‘personality’ could be added to deal with nvme serial retrieval via the method you indicated.

Just a thought.

f_l_a · July 8, 2016, 4:13pm

Ok Phil. Thought about combing tru the code in order to find potential places of, err, “interest”. So it is actually very nice of you to basically guide me straight tru! Will see if i come to it; having to swap my server chassis as well as to revive a tablet this weekend. While being glad that the server runs smoothly for over a month now… never touch, and so.

phillxnet · July 15, 2016, 5:52pm

@f_l_a Just a notification that we have a new technical manual entry in the form of a wiki post which may be of interest to you Device management in Rockstor. Hope this helps with seeing the role the serial of a device plays in Rockstor. Just adding here due to your experiments / contributions nvme serial wise.

Hope that helps.

f_l_a · July 16, 2016, 9:29pm

A detailed post, much appreciated.

phillxnet · July 23, 2016, 11:52am

@f_l_a Just a quick notification that as of Rockstor version 3.8-14.07 testing channel updates we have improved nvme compatibility for the system disk. Noting here as you were instrumental in bringing this to the for, as you were one of the first reporters of issues around nvme and Rockstor. I know you have now re-installed on different hw for system disk and I think this is as well as there is still a little more to be done, ie why null serials entered the db with nvme devices but just to let you know. Oh and of course the in ability of Rockstor ‘as is’ to extract serials numbers from these devices; as you kindly researched. But just in case you were interested an in relation to the following:

in the related thread that I see you have also engaged with:

the original poster @snafu developed a set of working udev rules that not only attributes the required serial numbers but also sets up the now required by-id names for a single nvme device used as the system disk.

I still thing we may have a bug here prior to udev settings being applied as our db may still get a null serial which is no good but we will have to circle back around to that another time I think as it may just have been down to ‘in development’ udev rules. Not sure really. But the work around for the resulting db state is also available in the above thread.

Thanks for helping to encourage the nvme related fixes thus far and your examination of their serial issues.

f_l_a · July 24, 2016, 8:00pm

Thanks Phil. This will come very handy; kudos for your & the teams’ efforts. But i haven’t re-installed yet - Still have to move my server components to a replacement chassis first, incl. re-cabling etc, and will want to wait until 3.8.15 final is released.
This weekend was mostly spend trying to fix & integrate a 2nd hand UTM appliance in my network, and also re-configuring / re-cabling said network…