/ partition full, no GUI; best way to free up space?

Thad · April 18, 2020, 5:00am

Brief description of the problem

My / partition is full and I’m unable to access the web GUI.

Detailed step by step instructions to reproduce the problem

When I attempt to connect to the web GUI, I get an error.
I’m able to log in via SSH, but it appears that my / partition is full; df -h shows 100% disk usage, 0 available.

Web-UI screenshot

“Unable to connect” error message.

Error Traceback provided on the Web-UI

No web UI.

I’m seeking recommendations for how I should free up space. I suspect most of the disk usage is from BTRFS snapshots. Here’s the output of btrfs subvolume list /:

ID 257 gen 3569726 top level 5 path home
ID 258 gen 8011343 top level 5 path root
ID 260 gen 370754 top level 258 path var/lib/machines
ID 294 gen 260378 top level 5 path .snapshots/home/home_201701010400
ID 295 gen 262585 top level 5 path .snapshots/root/root_201701020400
ID 296 gen 269288 top level 5 path .snapshots/home/home_201701050442
ID 297 gen 271491 top level 5 path .snapshots/root/root_201701060442
ID 298 gen 347507 top level 5 path .snapshots/home/home_201702090342
ID 299 gen 349949 top level 5 path .snapshots/root/root_201702100342
ID 300 gen 364539 top level 5 path .snapshots/home/home_201702160342
ID 301 gen 367350 top level 5 path .snapshots/root/root_201702170342
ID 305 gen 396684 top level 5 path .snapshots/root/root-monthly_201703020300
ID 318 gen 737736 top level 5 path .snapshots/root/root-monthly_201708020300
ID 333 gen 852589 top level 5 path .snapshots/root/root-weekly_201709290342
ID 335 gen 858759 top level 5 path .snapshots/root/root-monthly_201710020300
ID 337 gen 894993 top level 5 path .snapshots/root/root-weekly_201710200342
ID 339 gen 909135 top level 5 path .snapshots/root/root-weekly_201710270342
ID 341 gen 919340 top level 5 path .snapshots/home/home-monthly_201711010300
ID 342 gen 921417 top level 5 path .snapshots/root/root-monthly_201711020300
ID 344 gen 923574 top level 5 path .snapshots/root/root-weekly_201711030342
ID 347 gen 2341422 top level 5 path .snapshots/home/snapshot-home
ID 355 gen 2450843 top level 5 path .snapshots/home/home-monthly_201912010300
ID 360 gen 2506157 top level 5 path .snapshots/home/home-monthly_202001010300
ID 366 gen 2563809 top level 5 path .snapshots/home/home-monthly_202002010300
ID 370 gen 3381339 top level 5 path .snapshots/home/home-monthly_202003010300
ID 371 gen 3388134 top level 5 path .snapshots/home/home-weekly_202003050342
ID 372 gen 3400647 top level 5 path .snapshots/home/home-weekly_202003120342
ID 373 gen 3413067 top level 5 path .snapshots/home/home-weekly_202003190342
ID 374 gen 3421442 top level 5 path .snapshots/home/home-weekly_202003260342
ID 375 gen 3569725 top level 5 path .snapshots/home/home-monthly_202004010300

and btrfs qgroup show /:

qgroupid         rfer         excl 
--------         ----         ---- 
0/5             0.00B        0.00B 
0/257        16.00KiB     16.00KiB 
0/258         3.69GiB      2.67GiB 
0/260        16.00KiB     16.00KiB 
0/294        16.00KiB     16.00KiB 
0/295         2.30GiB    253.43MiB 
0/296        16.00KiB     16.00KiB 
0/297         2.17GiB    125.86MiB 
0/298        16.00KiB     16.00KiB 
0/299         2.18GiB    206.22MiB 
0/300        16.00KiB     16.00KiB 
0/301         2.39GiB    135.72MiB 
0/305         2.44GiB    307.67MiB 
0/318         3.18GiB    765.55MiB 
0/333         3.19GiB    172.34MiB 
0/335         3.15GiB     96.93MiB 
0/337         3.21GiB    122.94MiB 
0/339         3.20GiB    121.91MiB 
0/341        16.00KiB     16.00KiB 
0/342         3.20GiB      2.89MiB 
0/344         3.21GiB     19.97MiB 
0/347        16.00KiB     16.00KiB 
0/355        16.00KiB     16.00KiB 
0/360        16.00KiB     16.00KiB 
0/366        16.00KiB     16.00KiB 
0/370        16.00KiB     16.00KiB 
0/371        16.00KiB     16.00KiB 
0/372        16.00KiB     16.00KiB 
0/373        16.00KiB     16.00KiB 
0/374        16.00KiB     16.00KiB 
0/375        16.00KiB     16.00KiB 
2015/1      240.00KiB    240.00KiB 
2015/2        7.90GiB      6.88GiB

I’m figuring I can safely get rid of any of those snapshots with “2017” in the filename, but I’m not great with command-line btrfs. I know the command should look something like btrfs subvolume delete .snapshots/root/root_201701020400 but I’m not sure where those subvolumes are actually located. IIRC I have to unmount the current snapshot and switch to a higher level to access all of them, but I don’t remember how to do that and I’m not even sure if I’m barking up the right tree – should I be focusing on deleting snapshots at all, or is there somewhere else I should be looking into freeing up space?

Any help is appreciated. Thanks.

phillxnet · April 18, 2020, 9:57am

@Thad Hello again.
Re:

This may well be the case, only the snag with copy on write (which includes delete) file systems need a little space to do a delete. A tad more space than is currently taken by the file you are going to delete as it goes as it also need to update the the meta data which is also copy on write. So trying an entire entire snapshot first may well fail as the pool is likely not going to have that much ‘emergency’ breathing room left for such an operation. And a further wrinkle is that there are snapshots of the root itself. So just deleting a file on root won’t remove the space that file occupies as there are likely other snapshots of the file. So you are really just updating a bit of meta-data which will just end taking a little extra space for the meta-data update.

So completely full pools with snapshots are a tricky one. Could you post the output of the following:

btrfs fi show

and

btrfs fi usage /

There is a forum thread somewhere where we tackled this one, bit by bit, and ended up with success. I’ll see if I can find it but it was probably a couple of years ago now. We ended up doing some fine grained balances to retrieve just enough space to start deleting stuff.

If you can delete small things first, logs say if they are not included in a snapshot then that may get you some wiggle room.

Can’t search forum now directly but we have tackled this one before. Although I don’t think in that case there were snapshots of the root.

Lest have a look at those command outputs to see what the emergency reserve space is at.

Seems like the way to go possibly, if there is space. All Rockstor initiated mounts are under:

/mnt2/pool-name

and each subvol, if mounted, has it’s own mount point:

/mnt2/share-name

Apologies for the brevity here, pulled in a number of directions.

and the:

mount

command should show you all current mounts
Hope that helps.

Thad · April 18, 2020, 6:11pm

[root@rockstor ~]# btrfs fi show
Label: 'rockstor_rockstor'  uuid: 433e8267-0e21-42a4-ac64-348d6577e7b8
        Total devices 1 FS bytes used 10.62GiB
        devid    1 size 12.66GiB used 12.66GiB path /dev/sda3

Label: 'rockons'  uuid: bce9564a-9433-4577-bf9f-7bc4f617d70c
        Total devices 1 FS bytes used 6.40GiB
        devid    1 size 111.79GiB used 78.04GiB path /dev/sdb

Label: 'raid'  uuid: e8d4563a-accc-4c67-bb3e-2a05efaf0709
        Total devices 2 FS bytes used 710.17GiB
        devid    1 size 931.51GiB used 712.03GiB path /dev/sdc
        devid    2 size 931.51GiB used 712.03GiB path /dev/sdd

[root@rockstor ~]# btrfs fi usage /
Overall:
    Device size:                  12.66GiB
    Device allocated:             12.66GiB
    Device unallocated:              0.00B
    Device missing:                  0.00B
    Used:                         10.91GiB
    Free (estimated):             20.00KiB      (min: 20.00KiB)
    Data ratio:                       1.00
    Metadata ratio:                   1.99
    Global reserve:               49.72MiB      (used: 0.00B)

Data,single: Size:10.34GiB, Used:10.34GiB
   /dev/sda3      10.34GiB

Metadata,single: Size:8.00MiB, Used:0.00B
   /dev/sda3       8.00MiB

Metadata,DUP: Size:1.12GiB, Used:292.86MiB
   /dev/sda3       2.25GiB

System,single: Size:4.00MiB, Used:0.00B
   /dev/sda3       4.00MiB

System,DUP: Size:32.00MiB, Used:16.00KiB
   /dev/sda3      64.00MiB

Unallocated:
   /dev/sda3         0.00B

And I’m not seeing any snapshots on /mnt2/home or /mnt2/root:

[root@rockstor ~]# ls -al /mnt2/home/
total 0
drwxr-xr-x 1 root root   0 Sep  3  2016 .
drwxr-xr-x 1 root root 174 Oct 12  2019 ..
[root@rockstor ~]# ls -al /mnt2/root
total 0
drwxr-xr-x 1 root root   0 Sep  3  2016 .
drwxr-xr-x 1 root root 174 Oct 12  2019 ..
[root@rockstor ~]# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,size=5847968k,nr_inodes=1461992,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/sda3 on / type btrfs (rw,noatime,space_cache,subvolid=258,subvol=/root)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=23,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14962)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
tmpfs on /tmp type tmpfs (rw)
/dev/sda3 on /home type btrfs (rw,noatime,space_cache,subvolid=257,subvol=/home)
/dev/sda1 on /boot type ext4 (rw,noatime,data=ordered)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=1172108k,mode=700)

phillxnet · April 19, 2020, 12:48pm

@Thad One bit of good news is that your 'emergency reserve on this pool looks to be currently unused.

So you may still have a little wiggle root to start deleting stuff.

And from your mount command:

There are only those 2 subvol mounts on the system pool on /dev/sda3, i.e. the rockstor_rockstor pool, this relates to:

Our CentOS variant doesn’t have boot to snapshot (like our openSUSE variant in testing) so you are always mounting the ‘master’ version of the /root subvol as the ‘/’ mount point. And given those snapshots don’t look to be mounted you may well just be able to delete them like so:

The following uses an example for scheduled snapshots from a newer version of Rockstor which quite a while ago removed the ‘surfacing’ of the ‘root’ subvolume within the Web-UI as it was quite confusing given there is also within a typical linux root filesystem the /root subdirectory and we ended up with a very subtle bug which took quite a while to discover where stuff actually worked as intended but ‘by accident’ of this name ‘collision’. Anyway in short I’ll use an example of home snapshots via scheduler as that’s easier and quicker for me to reproduce as I tend to only have newer instances of the code running:

N.B. if one has de-selected the default snapshot creation option of “Make snapshots visible” then these subvolumes (snapshots are subvolumes, as are clones and shares actually) will not be mounted; meaning they will not show up in the output of the “mount” command with no options run as root.
So with the following default and hidden (not mounted) snapshots scheduled:

The following command:

btrfs subvolume list -s -p /

Will show the snapshots (-s) and with (-p) a display of their parent id, just to be a little clearer, as follows:

btrfs subvolume list -s -p /
ID 314 gen 46530 cgen 46523 parent 5 top level 5 otime 2020-04-19 13:00:01 path .snapshots/home/home-snap-default_202004191300
ID 315 gen 46530 cgen 46524 parent 5 top level 5 otime 2020-04-19 13:00:01 path .snapshots/home/home-snap-not-visible_202004191300
ID 316 gen 46535 cgen 46535 parent 5 top level 5 otime 2020-04-19 13:05:01 path .snapshots/home/home-snap-default_202004191305
ID 317 gen 46536 cgen 46536 parent 5 top level 5 otime 2020-04-19 13:05:01 path .snapshots/home/home-snap-not-visible_202004191305
ID 318 gen 46546 cgen 46546 parent 5 top level 5 otime 2020-04-19 13:10:02 path .snapshots/home/home-snap-default_202004191310
ID 319 gen 46547 cgen 46547 parent 5 top level 5 otime 2020-04-19 13:10:02 path .snapshots/home/home-snap-not-visible_202004191310
ID 320 gen 46558 cgen 46558 parent 5 top level 5 otime 2020-04-19 13:15:02 path .snapshots/home/home-snap-not-visible_202004191315
ID 321 gen 46559 cgen 46559 parent 5 top level 5 otime 2020-04-19 13:15:02 path .snapshots/home/home-snap-default_202004191315
ID 322 gen 46569 cgen 46569 parent 5 top level 5 otime 2020-04-19 13:20:02 path .snapshots/home/home-snap-default_202004191320
ID 323 gen 46570 cgen 46570 parent 5 top level 5 otime 2020-04-19 13:20:02 path .snapshots/home/home-snap-not-visible_202004191320

They are located at the top level of the parent pool. The root itself “/” is a subvolume mount of the system pool which is typically named rockstor_rockstor or ROOT in out openSUSE variants. With the latter naming inherited from the JeOS variants of openSUSE. So we need to access the top level of this pool. In Rockstor’s typical boot up this is as follows:

/dev/sda3 on /mnt2/rockstor_rockstor type btrfs (rw,relatime,space_cache,subvolid=5,subvol=/)

Another wrinkle here for your situation is that you mount command fails to show this. Likely due to the current root failure situation. So you will have to mount the pool to get access to it’s subvolumes. As the system’s root is running from within 2 subvolumes named root and home. Rockstor’s systemd scripts normally look after this initial top level system pool mount but have failed likely do too the lack of root breathing room. So to get this top level mount you could do:

systemctl stop rockstor rockstor-pre rockstor-bootstrap

Just to be sure they are not running or trying to run. Then:

mount /dev/disk/by-label/rockstor_rockstor /mnt2/rockstor_rockstor

and you should then have a default mount of the top level of the pool:

mount | grep rockstor_rockstor
/dev/sda3 on /mnt2/rockstor_rockstor type btrfs (rw,relatime,space_cache,subvolid=5,subvol=/)

The same mount command, for context, within Rockstor’s code is as follows:

github.com

rockstor/rockstor-core/blob/a2b5d45a8949a737c27fb74c483ccaa847b47136/src/rockstor/fs/btrfs.py#L455-L491


      
          def mount_root(pool):
              """
              Mounts a given pool at the default mount root (usually /mnt2/) using the
              pool.name as the final path entry. Ie pool.name = test-pool will be mounted
              at /mnt2/test-pool. Any mount options held in pool.mnt_options will be
              added to the mount command via the -o option as will a compress =
              pool.compression entry.
              N.B. Initially the mount target is defined by /dev/disk/by-label/pool.name,
              if this fails then an attempt to mount by each member of
              /dev/disk/by-id/pool.disk_set.all() but only if there are any members.
              If this second method also fails then an exception is raised, currently all
              but the last failed mount by device name is logged. If no disk members were
              reported by pool.disk_set.count() a separate Exception is raised.
              :param pool: pool object
              :return: either the relevant mount point or an Exception which either
              indicates 'no disks in pool' or 'Unknown Reason'
              """
              root_pool_mnt = DEFAULT_MNT_DIR + pool.name
              if pool.is_mounted:
                  return root_pool_mnt

This file has been truncated. show original

And so to delete a snapshots created as detailed previously from the command line we need to access the top level of the pool normally mounted at /mnt2/rockstor_rockstor (CentOS variant). Hence we can ‘see’ all our subvolumes, mounted or otherwise via:

ls -la /mnt2/rockstor_rockstor/.snapshots/home/
drwx--x--x. 1 root root 492 Apr 19 12:50 home-snap-default_202004191255
drwx--x--x. 1 root root 492 Apr 19 12:55 home-snap-default_202004191300
drwx--x--x. 1 root root 492 Apr 19 13:00 home-snap-default_202004191305
drwx--x--x. 1 root root 492 Apr 19 13:05 home-snap-default_202004191310
drwx--x--x. 1 root root 492 Apr 19 13:10 home-snap-default_202004191315
drwx--x--x. 1 root root 492 Apr 19 12:50 home-snap-not-visible_202004191255
drwx--x--x. 1 root root 492 Apr 19 12:55 home-snap-not-visible_202004191300
drwx--x--x. 1 root root 492 Apr 19 13:00 home-snap-not-visible_202004191305
drwx--x--x. 1 root root 492 Apr 19 13:05 home-snap-not-visible_202004191310
drwx--x--x. 1 root root 492 Apr 19 13:10 home-snap-not-visible_202004191315

and delete one via:

btrfs subvolume delete /mnt2/rockstor_rockstor/.snapshots/home/home-snap-not-visible_202004191300
Delete subvolume (no-commit): '/mnt2/rockstor_rockstor/.snapshots/home/home-snap-not-visible_202004191300'

This is what Rockstor normally does in the following code extract:

github.com

rockstor/rockstor-core/blob/master/src/rockstor/fs/btrfs.py#L903-L911


      
          def share_id(pool, share_name):
              """
              Returns the subvolume id: becomes the share's / snapshots's qgroup.
              @todo: this should be part of add_share -- btrfs create should atomically
              Works by iterating over the output of btrfs subvolume list, received from
              subvol_list_helper() looking for a match in share_name. If found the same
              line is parsed for the ID, example line in above command output:
              'ID 257 gen 13616 top level 5 path rock-ons-root'
              consequent subvol_id return value:

So it occurs to me now that this is possibly what you were referencing, in part, with the following:

And in your case there is no top level pool mount as your system is poorly (out of breathing room) and so the rockstor initscript most likely failed to do this top level overall pool mount.

You may have to delete a little at a time until more breathing room emerges. Also remember about the logs. As once you have removed possibly all snapshots of root you may end up having to remove some old log files also to free up more room i.e.:

ls -la /var/log/messages*
ls -la /opt/rockstor/var/log/*

And keep in mind that snapshot creation on btrfs is pretty much instant, but snapshot removal is very expensive so depending on your root drive speed it make take a little time.

Hope that helps and let us know how you get on.

Thad · April 19, 2020, 8:14pm

Thanks, that did it; I cleared out 4.5GB of snapshots from 2017 and after a reboot I’ve got the GUI up and running again.

I’ve got some other issues I’m troubleshooting right now, but since they don’t appear to be caused by a full disk, I’ll start a new thread if I’m unable to resolve them.

Thanks again for your help.

phillxnet · April 19, 2020, 9:15pm

@Thad Thanks for the update and glad your now sorted.

And yes, focused threads are best. Especially when referencing back to them in the future.

Also remember to include your Rockstor version when reporting issues as that can help to see if they relate to older versions. Current latest release version is 3.9.2-56 in the Stable Channel for CentOS and Testing Channel for our pending ‘Built on openSUSE’ variant.

You can double check your running version via:

yum info rockstor

Cheers.