Disk error trying to delete readonly file system

Stevek · September 13, 2024, 9:58pm

[Please complete the below template with details of the problem reported on your Web-UI. Be as detailed as possible. Community members, including developers, shall try and help. Thanks for your time in reporting this issue! We recommend purchasing commercial support for expedited support directly from the developers.]

Brief description of the problem

Trying to cleanup everything around a failed replication

Detailed step by step instructions to reproduce the problem

[write here]

Web-UI screenshot

[Drag and drop the image here]

Error Traceback provided on the Web-UI


        Traceback (most recent call last):

File “/opt/rockstor/src/rockstor/rest_framework_custom/generic_view.py”, line 40, in _handle_exception

yield

File “/opt/rockstor/src/rockstor/storageadmin/views/pool.py”, line 793, in delete

remove_share(so.pool, so.subvol_name, so.pqgroup, force=force)

File “/opt/rockstor/src/rockstor/fs/btrfs.py”, line 1185, in remove_share

toggle_path_rw(subvol_mnt_pt, rw=True)

File “/opt/rockstor/src/rockstor/system/osi.py”, line 545, in toggle_path_rw

return run_command([CHATTR, attr, path])

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File “/opt/rockstor/src/rockstor/system/osi.py”, line 289, in run_command

raise CommandException(cmd, out, err, rc)

system.exceptions.CommandException: Error running a command. cmd = /usr/bin/chattr -i /mnt2/Data-Raid/.snapshots/c0450590-f2b9-de11-8a1b-90e6ba7a072b_testing/testing_21_replication_1. rc = 1. stdout = [‘’]. stderr = [‘/usr/bin/chattr: Read-only file system while setting flags on /mnt2/Data-Raid/.snapshots/c0450590-f2b9-de11-8a1b-90e6ba7a072b_testing/testing_21_replication_1’, ‘’]

Hooverdan · September 13, 2024, 10:27pm

I believe you will have to change the snapshots you want to delete to read-write.

Stevek · September 13, 2024, 10:50pm

Vault2:/mnt2/Data-Raid/.snapshots # rm -d -R xxxxxx
rm: cannot remove ‘xxxxxx/testing_21_replication_1/1980/01/Green Light.gif’: Read-only file system
rm: cannot remove ‘xxxxxx/testing_21_replication_1/1980/01/Green Light.png’: Read-only file system
rm: cannot remove ‘xxxxxx/testing_21_replication_1/1980/01/Red Light blur.gif’: Read-only file system
rm: cannot remove ‘xxxxxx/testing_21_replication_1/1980/01/Red Light.gif’: Read-only file system
rm: cannot remove ‘xxxxxx/testing_21_replication_1/1980/01/Red Light.png’: Read-only file system
rm: cannot remove ‘xxxxxx/testing_21_replication_1/1980 - Copy/01/Green Light.gif’: Read-only file system
rm: cannot remove ‘xxxxxx/testing_21_replication_1/1980 - Copy/01/Green Light.png’: Read-only file system
rm: cannot remove ‘xxxxxx/testing_21_replication_1/1980 - Copy/01/Red Light blur.gif’: Read-only file system
rm: cannot remove ‘xxxxxx/testing_21_replication_1/1980 - Copy/01/Red Light.gif’: Read-only file system
rm: cannot remove ‘xxxxxx/testing_21_replication_1/1980 - Copy/01/Red Light.png’: Read-only file system
Vault2:/mnt2/Data-Raid/.snapshots # rm -d -R -f xxxxxx
rm: cannot remove ‘xxxxxx/testing_21_replication_1/1980/01/Green Light.gif’: Read-only file system
rm: cannot remove ‘xxxxxx/testing_21_replication_1/1980/01/Green Light.png’: Read-only file system
rm: cannot remove ‘xxxxxx/testing_21_replication_1/1980/01/Red Light blur.gif’: Read-only file system
rm: cannot remove ‘xxxxxx/testing_21_replication_1/1980/01/Red Light.gif’: Read-only file system
rm: cannot remove ‘xxxxxx/testing_21_replication_1/1980/01/Red Light.png’: Read-only file system
rm: cannot remove ‘xxxxxx/testing_21_replication_1/1980 - Copy/01/Green Light.gif’: Read-only file system
rm: cannot remove ‘xxxxxx/testing_21_replication_1/1980 - Copy/01/Green Light.png’: Read-only file system
rm: cannot remove ‘xxxxxx/testing_21_replication_1/1980 - Copy/01/Red Light blur.gif’: Read-only file system
rm: cannot remove ‘xxxxxx/testing_21_replication_1/1980 - Copy/01/Red Light.gif’: Read-only file system
rm: cannot remove ‘xxxxxx/testing_21_replication_1/1980 - Copy/01/Red Light.png’: Read-only file system
Vault2:/mnt2/Data-Raid/.snapshots # rm -R -f xxxxxx
rm: cannot remove ‘xxxxxx/testing_21_replication_1/1980/01/Green Light.gif’: Read-only file system
rm: cannot remove ‘xxxxxx/testing_21_replication_1/1980/01/Green Light.png’: Read-only file system
rm: cannot remove ‘xxxxxx/testing_21_replication_1/1980/01/Red Light blur.gif’: Read-only file system
rm: cannot remove ‘xxxxxx/testing_21_replication_1/1980/01/Red Light.gif’: Read-only file system
rm: cannot remove ‘xxxxxx/testing_21_replication_1/1980/01/Red Light.png’: Read-only file system
rm: cannot remove ‘xxxxxx/testing_21_replication_1/1980 - Copy/01/Green Light.gif’: Read-only file system
rm: cannot remove ‘xxxxxx/testing_21_replication_1/1980 - Copy/01/Green Light.png’: Read-only file system
rm: cannot remove ‘xxxxxx/testing_21_replication_1/1980 - Copy/01/Red Light blur.gif’: Read-only file system
rm: cannot remove ‘xxxxxx/testing_21_replication_1/1980 - Copy/01/Red Light.gif’: Read-only file system
rm: cannot remove ‘xxxxxx/testing_21_replication_1/1980 - Copy/01/Red Light.png’: Read-only file system
Vault2:/mnt2/Data-Raid/.snapshots # ls -AlF
total 0
drwxr-xr-x 1 root root 48 Sep 13 18:15 c0450590-f2b9-de11-8a1b-90e6ba7a072b_testing/
drwxrwxrwx 1 root root 48 Sep 13 14:24 xxxxxx/
Vault2:/mnt2/Data-Raid/.snapshots # cd …
Vault2:/mnt2/Data-Raid # ls -AlF
total 0
drwxr-xr-x 1 root root 100 Sep 13 18:15 .snapshots/
drwxr-xr-x 1 root root 0 Sep 13 18:15 c0450590-f2b9-de11-8a1b-90e6ba7a072b_testing/
Vault2:/mnt2/Data-Raid #

MOUNT CMD everything is rw

Vault2:/mnt2/Data-Raid # mount
/dev/sdf4 on / type btrfs (rw,noatime,ssd,discard=async,space_cache,subvolid=258,subvol=/@/.snapshots/1/snapshot)
devtmpfs on /dev type devtmpfs (rw,nosuid,size=4096k,nr_inodes=1519496,mode=755,inode64)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /run type tmpfs (rw,nosuid,nodev,size=2449704k,nr_inodes=819200,mode=755,inode64)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=32,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2803)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,nosuid,nodev,relatime,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
/dev/sdf4 on /.snapshots type btrfs (rw,noatime,ssd,discard=async,space_cache,subvolid=257,subvol=/@/.snapshots)
/dev/sdf4 on /boot/grub2/i386-pc type btrfs (rw,noatime,ssd,discard=async,space_cache,subvolid=266,subvol=/@/boot/grub2/i386-pc)
/dev/sdf4 on /home type btrfs (rw,noatime,ssd,discard=async,space_cache,subvolid=259,subvol=/@/home)
/dev/sdf4 on /boot/grub2/x86_64-efi type btrfs (rw,noatime,ssd,discard=async,space_cache,subvolid=267,subvol=/@/boot/grub2/x86_64-efi)
/dev/sdf4 on /opt type btrfs (rw,noatime,ssd,discard=async,space_cache,subvolid=260,subvol=/@/opt)
/dev/sdf4 on /root type btrfs (rw,noatime,ssd,discard=async,space_cache,subvolid=261,subvol=/@/root)
/dev/sdf4 on /srv type btrfs (rw,noatime,ssd,discard=async,space_cache,subvolid=262,subvol=/@/srv)
/dev/sdf4 on /var type btrfs (rw,noatime,ssd,discard=async,space_cache,subvolid=264,subvol=/@/var)
/dev/sdf4 on /tmp type btrfs (rw,noatime,ssd,discard=async,space_cache,subvolid=263,subvol=/@/tmp)
/dev/sdf4 on /usr/local type btrfs (rw,noatime,ssd,discard=async,space_cache,subvolid=265,subvol=/@/usr/local)
/dev/sdf2 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
/dev/sdc on /mnt2/Data-Raid type btrfs (rw,relatime,space_cache=v2,subvolid=5,subvol=/)
/dev/sda on /mnt2/Data-2 type btrfs (rw,relatime,space_cache=v2,subvolid=5,subvol=/)
/dev/sdd on /mnt2/Data-1 type btrfs (rw,relatime,space_cache=v2,subvolid=5,subvol=/)
/dev/sdf4 on /mnt2/ROCK-ONS/btrfs type btrfs (rw,noatime,ssd,discard=async,space_cache,subvolid=258,subvol=/@/.snapshots/1/snapshot)
tracefs on /sys/kernel/debug/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
/dev/sdc on /mnt2/c0450590-f2b9-de11-8a1b-90e6ba7a072b_testing type btrfs (rw,relatime,space_cache=v2,subvolid=260,subvol=/c0450590-f2b9-de11-8a1b-90e6ba7a072b_testing)
/dev/sdc on /mnt2/.snapshots/c0450590-f2b9-de11-8a1b-90e6ba7a072b_testing/testing_25_replication_1 type btrfs (rw,relatime,space_cache=v2,subvolid=261,subvol=/.snapshots/c0450590-f2b9-de11-8a1b-90e6ba7a072b_testing/testing_25_replication_1)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=1224852k,nr_inodes=306213,mode=700,inode64)

This is all happening because of replication failing and I still have not been able to fix that. It fails differently every time.

Hooverdan · September 15, 2024, 6:26pm

Did you use the btrfs property to make the snapshot read/write? For example:

btrfs property set -ts /path/to/snapshot ro false

Stevek · September 15, 2024, 6:42pm

No I did not. I’m a newbe and did not know about that. I rebuilt my system. Had 3 good replications of a test shat but none of them put any data in the recieving share. I them rebooted the receiving system and Now I’m getting the below error and it looks like the same problem cropped up.

[15/Sep/2024 14:08:04] INFO [storageadmin.views.snapshot:61] Supplanting share (c0450590-f2b9-de11-8a1b-90e6ba7a072b_testing) with snapshot (.snapshots/c0450590-f2b9-de11-8a1b-90e6ba7a072b_testing/testing_28_replication_1).
[15/Sep/2024 14:08:04] ERROR [storageadmin.util:44] Exception: Error running a command. cmd = /usr/sbin/btrfs property set /mnt2/Data-Raid/.snapshots/c0450590-f2b9-de11-8a1b-90e6ba7a072b_testing/testing_28_replication_1 ro false. rc = 1. stdout = [‘’]. stderr = [‘ERROR: cannot flip ro->rw with received_uuid set, use force if you really want that’, ‘’]
Traceback (most recent call last):
File “/opt/rockstor/src/rockstor/storageadmin/views/clone_helpers.py”, line 94, in create_repclone
set_property(snap_path, “ro”, “false”, mount=False)
File “/opt/rockstor/src/rockstor/fs/btrfs.py”, line 2314, in set_property
return run_command(cmd)
^^^^^^^^^^^^^^^^
File “/opt/rockstor/src/rockstor/system/osi.py”, line 289, in run_command
raise CommandException(cmd, out, err, rc)
system.exceptions.CommandException: Error running a command. cmd = /usr/sbin/btrfs property set /mnt2/Data-Raid/.snapshots/c0450590-f2b9-de11-8a1b-90e6ba7a072b_testing/testing_28_replication_1 ro false. rc = 1. stdout = [‘’]. stderr = [‘ERROR: cannot flip ro->rw with received_uuid set, use force if you really want that’, ‘’]
[15/Sep/2024 14:08:04] ERROR [smart_manager.replication.receiver:100] b’Failed to promote the oldest Snapshot to Share.‘. Exception: 500 Server Error: Internal Server Error for url: http://127.0.0.1:8000/api/shares/6/snapshots/testing_28_replication_1/repclone
[15/Sep/2024 14:11:04] ERROR [smart_manager.replication.receiver:100] b’Failed to promote the oldest Snapshot to Share.’. Exception: Share matching query does not exist.
[15/Sep/2024 14:13:03] ERROR [smart_manager.replication.receiver:100] b’Failed to promote the oldest Snapshot to Share.'. Exception: Share matching query does not exist.

Tried…
Vault2:/mnt2/Data-Raid/.snapshots # btrfs property set -ts /mnt2/Data-Raid/.snapshots ro false
ERROR: Could not get subvolume flags: Invalid argument
Vault2:/mnt2/Data-Raid/.snapshots # ls
c0450590-f2b9-de11-8a1b-90e6ba7a072b_testing
Vault2:/mnt2/Data-Raid/.snapshots # btrfs property set -ts /mnt2/Data-Raid/.snapshots/c0450590-f2b9-de11-8a1b-90e6ba7a072b_testing ro false
ERROR: Could not get subvolume flags: Invalid argument