Deadlock deteckted when importing existing pool

Troubleshooting
1

Hello,

I am struggling to import an existing pool.
After a while, the Web-UI reports
Failed to import any pool on this device(sdc). Error: deadlock detected DETAIL: Process 12461 waits for ShareLock on transaction 13988; blocked by process 12547. Process 12547 waits for ShareLock on transaction 13973; blocked by process 12461. HINT: See server log for query details.

Some information:

Label: ‘MainPool’ uuid: a8af3832-48c7-4568-861f-e80380dd7e0b
Total devices 3 FS bytes used 6.78TiB
devid 1 size 2.73TiB used 2.27TiB path /dev/sde
devid 2 size 2.73TiB used 2.27TiB path /dev/sdc
devid 3 size 2.73TiB used 2.27TiB path /dev/sdd

Thanks to Docker, there are quite some subvolumes on this drive:

[root@rockstor log]# btrfs subvolume list /mnt/test1
ID 3096 gen 564539 top level 5 path Backups
ID 3098 gen 564936 top level 5 path Daten
ID 3099 gen 564916 top level 5 path Dokumente
ID 3101 gen 564828 top level 5 path Fotos
ID 3102 gen 564390 top level 5 path HomeVideo
ID 3107 gen 564390 top level 5 path Musik
ID 3108 gen 564450 top level 5 path Video
ID 22971 gen 564417 top level 5 path RockOn
ID 22972 gen 564417 top level 5 path PlexConfig
ID 23130 gen 564936 top level 5 path Docker
ID 23860 gen 564417 top level 23130 path Docker/storage/btrfs/subvolumes/978b0fee5c405109be80f7be732bbd1157392c0b8e2257b13e78b5f1c819a52a
ID 23861 gen 564417 top level 23130 path Docker/storage/btrfs/subvolumes/8e0aa5f79abe873cb3baa5f315327ec957822ef920c07c3dbc2f294a1e9463e5
ID 23862 gen 564417 top level 23130 path Docker/storage/btrfs/subvolumes/1bc8d72513b390c8466f3dbea2becbc7b4eb9de810ed51c6512b9fe00df59aef
ID 23863 gen 564417 top level 23130 path Docker/storage/btrfs/subvolumes/f38588d4d01f7e667fdfcabd0d2c0740b224a5b7b828211c9afeff082fc08c92
ID 23864 gen 564417 top level 23130 path Docker/storage/btrfs/subvolumes/5330d7e97082dca00c1bccaf1b172bfe3a753073bd0da16ae081c0040cc7d639
ID 23865 gen 564417 top level 23130 path Docker/storage/btrfs/subvolumes/8dde79df7e567543588f4c8b0159e1b1d23afdcc9e059fd710f3f284cf3ad2cb
ID 23866 gen 564417 top level 23130 path Docker/storage/btrfs/subvolumes/9e1ea34a70c02895d384685a8f0c1236a9527f3a54429600f927600918cedfc7
ID 23867 gen 564417 top level 23130 path Docker/storage/btrfs/subvolumes/5aecbb0c285c0762c6e5c3efc50f8729880d744f14d95441a811f9e05c645988
ID 23868 gen 564417 top level 23130 path Docker/storage/btrfs/subvolumes/f3d0833492c96cf8cddb7c17ff7a2a07e93a08b5c051bf8b336dfc731fe9700a
ID 23869 gen 564417 top level 23130 path Docker/storage/btrfs/subvolumes/f25fb2c46a5cba7a2b3f96f1fde356a555bb8009efd3464146fb2c309728f4e2
ID 23870 gen 564417 top level 23130 path Docker/storage/btrfs/subvolumes/6f41bbecb8a6fe7685f804b7c5d3154c0b09cb21471b2c3f8f4326d411276bb5
ID 23871 gen 564417 top level 23130 path Docker/storage/btrfs/subvolumes/59474ca4ed6c68f1b9a20e9f73de8954fedb37d3a90f54cdd1445dc3653537d3
ID 23872 gen 564417 top level 23130 path Docker/storage/btrfs/subvolumes/195c88231a80802b4752ae6a34efd8be021a9334f305153c407734311fa0e9b0
ID 23873 gen 564417 top level 23130 path Docker/storage/btrfs/subvolumes/2181fb7190a545556df8a34c3001d6d4fc2f125df6aa79fb61ae6263a9a98a92
ID 23874 gen 564417 top level 23130 path Docker/storage/btrfs/subvolumes/0cf14593b8a71134f48a6215cb8ee11f70fc699badfad4d277e1858993c71631
ID 23876 gen 564418 top level 23130 path Docker/storage/btrfs/subvolumes/92ec6d044cb3e39ae00500126a88c9ac342678d0591675f2231daafbf0877778
ID 23878 gen 564418 top level 23130 path Docker/storage/btrfs/subvolumes/5a52a1d23303d12dc74119b5587bc4aa8b19b741d7a4fa253f24024c60692780
ID 23879 gen 564418 top level 23130 path Docker/storage/btrfs/subvolumes/2ef91804894a296102a9ce1e38bca8c00d2703c414166f59e08541ce8542ab1e
ID 23880 gen 564418 top level 23130 path Docker/storage/btrfs/subvolumes/f80999a1f330b7e680bdf16ae900455abe50121d7c69ba2b5e4fc1fc10e3cc70
ID 23881 gen 564418 top level 23130 path Docker/storage/btrfs/subvolumes/6cc0fc2a5ee3bc506e9a0c41eb25431b4d163b1e407c927ee3ac18169d53cc1a
ID 23890 gen 564418 top level 23130 path Docker/storage/btrfs/subvolumes/6227b6b8580ed194155df9fa3ad97aa870af1405278f7bfcf53d851d1e92672b
ID 23891 gen 564418 top level 23130 path Docker/storage/btrfs/subvolumes/77f3be1cdfd236adec9fa055c3f3a67ec4bf0e84d559bbbcbeed7b3b1931f945
ID 23892 gen 564418 top level 23130 path Docker/storage/btrfs/subvolumes/b5013af453ed30fb88b5653d6f5d4d446adf373f92fc569f5581754a50042cd8
ID 23893 gen 564418 top level 23130 path Docker/storage/btrfs/subvolumes/9d0a60b3fc3bfac520687cfb309796edf28e5832089e9d0733bf878f03d7fbde
ID 23896 gen 564418 top level 23130 path Docker/storage/btrfs/subvolumes/3230102aa42cc9b1fb67bbc210d32141d2bb6fdf4fac2b974c5651090ce3f8db
ID 23897 gen 564418 top level 23130 path Docker/storage/btrfs/subvolumes/84f1253e3996f344fb5f7c577c0d6d666eff34677b4986048c2211186816e817-init
ID 23898 gen 564418 top level 23130 path Docker/storage/btrfs/subvolumes/c40fe59aa44a5e7685681eecb225af542424c08a9467ca2fa153206df683d572-init
ID 23899 gen 564418 top level 23130 path Docker/storage/btrfs/subvolumes/84f1253e3996f344fb5f7c577c0d6d666eff34677b4986048c2211186816e817
ID 23900 gen 564418 top level 23130 path Docker/storage/btrfs/subvolumes/c40fe59aa44a5e7685681eecb225af542424c08a9467ca2fa153206df683d572
ID 23901 gen 564418 top level 23130 path Docker/storage/btrfs/subvolumes/53f4d02b59e83262bd8346888ff043394643f14a281f672c806329769ce21b60-init
ID 23902 gen 564418 top level 23130 path Docker/storage/btrfs/subvolumes/53f4d02b59e83262bd8346888ff043394643f14a281f672c806329769ce21b60
ID 23905 gen 564418 top level 23130 path Docker/storage/btrfs/subvolumes/2a0b105db03aa1adf11209f1289086dacdb65b630f8b4fd96da4f25b3f9202fe
ID 23908 gen 564418 top level 23130 path Docker/storage/btrfs/subvolumes/9a200def196d497dc472f69d97e4bafda686881b1bd990dbbbf36aae3cc9b843
ID 23911 gen 564418 top level 23130 path Docker/storage/btrfs/subvolumes/1c6eb33115b02635f6f3b99649b9260ad949aa2b2dd2ddf6393d78b3ec03188e
ID 23914 gen 564418 top level 23130 path Docker/storage/btrfs/subvolumes/b8bd0cd216328acac1bfb8c76f3114ea8125541a5107d009207358895deed165
ID 23917 gen 564418 top level 23130 path Docker/storage/btrfs/subvolumes/c1d802caf7cbeb141d551034be2cdea421f45ffb92043aa147893c8be001b1cf
ID 23920 gen 564418 top level 23130 path Docker/storage/btrfs/subvolumes/daad25024289474233b7825537a827f4098413c9d684b6a4a976757d8ab7fabc
ID 23921 gen 564418 top level 23130 path Docker/storage/btrfs/subvolumes/d74d4ee8b3242476aa130e8c445c6010f05fc5f870115627038fc08276663c62-init
ID 23922 gen 564418 top level 23130 path Docker/storage/btrfs/subvolumes/d74d4ee8b3242476aa130e8c445c6010f05fc5f870115627038fc08276663c62
ID 23925 gen 564418 top level 23130 path Docker/storage/btrfs/subvolumes/9c53af3f0e9df986e1c76170e7d7a933b59fa21edc94cde343814ea0cb7e73f9
ID 23928 gen 564418 top level 23130 path Docker/storage/btrfs/subvolumes/380640c16dbd97b9dc9d6e83592ec8cc706d9f63decf566bf09c122bbedbc8ae
ID 23931 gen 564418 top level 23130 path Docker/storage/btrfs/subvolumes/bd692530f1b8727292fa660c1a2fcbd34410277ba98f797781253ad34584b318
ID 23934 gen 564418 top level 23130 path Docker/storage/btrfs/subvolumes/e29683eab6e931f6989406aeebde78bc05562023664ed0ebcb714f9db029dae8
ID 23937 gen 564418 top level 23130 path Docker/storage/btrfs/subvolumes/6d0aae68947a621c6bb774d75fd21c9efc75aa2618e92249bf4250338405169c
ID 23940 gen 564418 top level 23130 path Docker/storage/btrfs/subvolumes/0af90b37e5648a57c0f5a9eb34c9e6b74013bf27731f45980075670873e56f08
ID 23943 gen 564418 top level 23130 path Docker/storage/btrfs/subvolumes/b64053bb87a7a9b963673c3efd8034d4915edb826596b322007a99b283cf39c0
ID 23944 gen 564418 top level 23130 path Docker/storage/btrfs/subvolumes/2b2ee21f9edb8f2c9dd1d53337e3aa717deb3b1513f8f55e5f49bfeca3ce0806-init
ID 23945 gen 564418 top level 23130 path Docker/storage/btrfs/subvolumes/2b2ee21f9edb8f2c9dd1d53337e3aa717deb3b1513f8f55e5f49bfeca3ce0806
ID 23946 gen 564418 top level 23130 path Docker/storage/btrfs/subvolumes/8f68c1a030790e5c4f702b8085c2fb555f53d1c04f87d63f8b7daa32754e6742-init
ID 23947 gen 564418 top level 23130 path Docker/storage/btrfs/subvolumes/8f68c1a030790e5c4f702b8085c2fb555f53d1c04f87d63f8b7daa32754e6742
ID 23948 gen 564418 top level 23130 path Docker/storage/btrfs/subvolumes/eadc5c97085f242109fcdec2b276c836843500ebced294350fb208fd96bbf672-init
ID 23949 gen 564418 top level 23130 path Docker/storage/btrfs/subvolumes/eadc5c97085f242109fcdec2b276c836843500ebced294350fb208fd96bbf672
ID 23950 gen 564418 top level 23130 path Docker/storage/btrfs/subvolumes/44ce3d179cd90425e203338031807f3d37d5c0fb7c63873138df9cd2143efa18-init
ID 23951 gen 564418 top level 23130 path Docker/storage/btrfs/subvolumes/44ce3d179cd90425e203338031807f3d37d5c0fb7c63873138df9cd2143efa18
ID 23954 gen 564419 top level 23130 path Docker/storage/btrfs/subvolumes/803cf45639119d365bd07f220385118fad228c0691fd17f46812e5952f3fdda0
ID 23957 gen 564419 top level 23130 path Docker/storage/btrfs/subvolumes/a3a7dd71f1c9e5a0333be3bb2aacb4473bf29b2f31086a1aabd055ead546d663
ID 23960 gen 564419 top level 23130 path Docker/storage/btrfs/subvolumes/d579bc62c749eddbb0d69593b6e025cee0a75411307c6723ebd0605d0f36eca7
ID 23963 gen 564419 top level 23130 path Docker/storage/btrfs/subvolumes/362d79cb1d4abac0b5612441104f924cc3cfc28c59135ddb75b0635709c0ff2a
ID 23966 gen 564419 top level 23130 path Docker/storage/btrfs/subvolumes/90424ad26c9bb72aa21bb63a9819a91a0c8467533ce5c67dd02b92da5ec855ac
ID 23969 gen 564419 top level 23130 path Docker/storage/btrfs/subvolumes/037d79171c0dcdfde87d733f2af01a03a89bdac6cc150604c0620b53d0732fd4
ID 23972 gen 564419 top level 23130 path Docker/storage/btrfs/subvolumes/a76faa5dc869a43e43aa079297ab0d6464f896fac1936a422ca95de32464255a
ID 23975 gen 564419 top level 23130 path Docker/storage/btrfs/subvolumes/31bdc7280165c646a0e466f5ebb7aaaf53464042a2de5aa3c8939fbccc2cc2cd

But apart from that, I see nothing special with this pool.

Can someone help?

Greetings,
Hendrik

[17/Feb/2016 19:28:12] ERROR [storageadmin.util:46] request path: /api/disks/sdc/btrfs-disk-import method: POST data: <QueryDict: {}>
[17/Feb/2016 19:28:12] ERROR [storageadmin.util:47] exception: Failed to import any pool on this device(sdc). Error: deadlock detected
DETAIL: Process 12461 waits for ShareLock on transaction 13988; blocked by process 12547.
Process 12547 waits for ShareLock on transaction 13973; blocked by process 12461.
HINT: See server log for query details.
Traceback (most recent call last):
File “/opt/rockstor/src/rockstor/storageadmin/views/disk.py”, line 274, in _btrfs_disk_import
do.save()
File “/opt/rockstor/eggs/Django-1.6.11-py2.7.egg/django/db/models/base.py”, line 545, in save
force_update=force_update, update_fields=update_fields)
File “/opt/rockstor/eggs/Django-1.6.11-py2.7.egg/django/db/models/base.py”, line 573, in save_base
updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
File “/opt/rockstor/eggs/Django-1.6.11-py2.7.egg/django/db/models/base.py”, line 635, in _save_table
forced_update)
File “/opt/rockstor/eggs/Django-1.6.11-py2.7.egg/django/db/models/base.py”, line 679, in _do_update
return filtered._update(values) > 0
File “/opt/rockstor/eggs/Django-1.6.11-py2.7.egg/django/db/models/query.py”, line 510, in _update
return query.get_compiler(self.db).execute_sql(None)
File “/opt/rockstor/eggs/Django-1.6.11-py2.7.egg/django/db/models/sql/compiler.py”, line 980, in execute_sql
cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)
File “/opt/rockstor/eggs/Django-1.6.11-py2.7.egg/django/db/models/sql/compiler.py”, line 786, in execute_sql
cursor.execute(sql, params)
File “/opt/rockstor/eggs/Django-1.6.11-py2.7.egg/django/db/backends/util.py”, line 53, in execute
return self.cursor.execute(sql, params)
File “/opt/rockstor/eggs/Django-1.6.11-py2.7.egg/django/db/utils.py”, line 99, in exit
six.reraise(dj_exc_type, dj_exc_value, traceback)
File “/opt/rockstor/eggs/Django-1.6.11-py2.7.egg/django/db/backends/util.py”, line 53, in execute
return self.cursor.execute(sql, params)
OperationalError: deadlock detected
DETAIL: Process 12461 waits for ShareLock on transaction 13988; blocked by process 12547.
Process 12547 waits for ShareLock on transaction 13973; blocked by process 12461.
HINT: See server log for query details.

2

In fact, after deleting all the Docker-Subvolumes, I get

Failed to import any pool on this device(sdc). Error: ‘unicode’ object has no attribute ‘disk_set’

So, I would think that there is something wrong when importing an existing pool with many (maybe with nested) subvolumes. On top, there seems to be one more error (the one two lines above) and which I already reported here Warning! Disk serial number is not legitimate or unique

Greetings,
Hendrik

3

This is definitely buggy behavior. Thanks for sharing the traceback. Here’s the issue for it:

4

I have deleted all the Docker-Subvolumes, so I cannot test your fix on the Deadlock anymore.
But the “unicode” one, I still have, and it’s a blocker currently to me. I’d be very happy to test a fix or workaround if you can think of one.

Greetings,
Hendrik

5

Cool. I take it you are running latest code. There should be an update later today. stay tuned and thanks for testing and reporting!

6

Great. I will test your fix and let you know.

7

Brilliant it works!
Thanks!

8

Hello,
I have similar problem with lastest installation of Rockstor 3.8.16-1 (Linux 4.8.7-1.el7.elrepo.x86_64).
I did try to import pool from previous instalation via GUI but I get deadlock:
Failed to import any pool on this device(ata-ST8000AS0002-1NA17Z_Z840J0MS). Error: deadlock detected DETAIL: Process 12779 waits for ShareLock on transaction 3757; blocked by process 12817. Process 12817 waits for ShareLock on transaction 3756; blocked by process 12779. HINT: See server log for query details.

Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/storageadmin/views/disk.py", line 379, in _btrfs_disk_import
    do.save()
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/base.py", line 734, in save
    force_update=force_update, update_fields=update_fields)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/base.py", line 762, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/base.py", line 827, in _save_table
    forced_update)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/base.py", line 877, in _do_update
    return filtered._update(values) > 0
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/query.py", line 580, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/sql/compiler.py", line 1062, in execute_sql
    cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/sql/compiler.py", line 840, in execute_sql
    cursor.execute(sql, params)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/utils.py", line 98, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
OperationalError: deadlock detected
DETAIL:  Process 12779 waits for ShareLock on transaction 3757; blocked by process 12817.
Process 12817 waits for ShareLock on transaction 3756; blocked by process 12779.
HINT:  See server log for query details.

How do you solve this problem? As I understand it occured in old versions since this tread is year old, so it should be the problem now.

Thanks for help,
Pumadace