From NTFS to BTRFS

BTRFS
1

Hello,

I would like to learn a bit about the stored dates of files in BTRFS.

More specifically; currently all my media files are being added to Kodi database (mysql centralized) based on their “Modified Date” under Windows NTFS file system. Also; I can easily handle this date with various tools so as to have more control over my database/file histroy.

I plan to move all of my data (around 4TB) from NTFS (mounted via ntfs-3g) to BTRFS in Rockstor; I will be using either cp or rsync command from command line.

So I am wondering about the following:

  1. Which is the date in BTRFS that will be populated with NTFS “Modified Date”?
  2. Can the dates in BTRFS be manipulated?
  3. Do you have any recommendation/preference/consultation between the two commands (cp or rsync) for the task in hand?

Regards.

2

The modified date (by default) will be the date of the last modification to the file - this is a fairly obvious fact.
While the modification times are supported by BTRFS, they’re performed with standard linux syscalls, so any utility capable of setting the modified time in linux will quite capably set the time on BTRFS as well.

You can’t set the modification time of a file with either rsync or cp, however you can preserve them.

With rsync, you need to specify -t, from the RSYNC manual:

-t, --times                 preserve modification times

With cp, you can specify -p, which will preserve mode, ownership and timestamps, or the full preserve command, with a specific option to preserve (timestamps):

-p same as --preserve=mode,ownership,timestamps

With regards to recommendations, most of this will be personal, and situational.
Will the ntfs disk be mounted on the local system or a remote system?

For a remote system, I would suggest a tarpipe over netcat over both of the above options, as in my experience the thhroughput is MUCH higher.

On the other hand, for a local system, it depends on the file sizes involved.
I would still tarpipe (but without netcat) large amounts of small files (such as JPEGS, anything <~5Mb) as tar is very good at handling large numbers of inodes
However, I would straight copy larger files, as fewer system resources are required for managing the data transfer than either rsync or tar.

1 Like
3

Hello @Haioken and thanks for the answer.

I am asking about dates because Kodi uses them in order to populate a field called “Date Added” in the database. I am just unsure on which exact date will be targeting under BTRFS. I have already asked the guys over at Kodi forums. And from what I read the only date that CANNOT be manipulated is the creation date (ctime). I am guessing when copying from NTFS this is being populated with the respective “Creation Date” fro NTFS; right?

The ntfs disk will be mounted on the local system and my data has both large and small files in the same folders; so I cannot differentiate and treat them differently. I will have to use one command for ALL and that is why I am at a dilemma between cp and rsync.
TBH I am however leaning towards cp.

4

@Sky12016,

I’m pretty well versed in Kodi, having argued with this ‘feature’ before.

The ‘date added’ field in kodi can be set to use any of the following items via the advancedsettings.xml file:

  • modified time of the file (not creation time, although it uses ctime in the event that mtime is invalid, see below note on ctime),
  • Date at which the item was scanned into the library using the advancedsettings.xml file.
  • Newest date between ctime and mtime

In linux, the ctime flag of a file is the time at which the inode was last updated (change time, not created time), which in the case of a copy would be NOW.
It will NOT reflect the created time on the NTFS FS, it will (initially) reflect the time at which the file was created on the linux system. I say initially, as it will update to the latest modification or property change time of the file.
The difference between ctime and mtime is primarily that ctime will include permission, ownership and attribute changes, not just content changes.

Note that copy, rsync and tar are all capable of preserving modified timestamps.

There is another flag in some *nix filesystems, called btime (birth time), however this isn’t supported by some filesystems (and most tools) so it can safely be ignored.

With regards to your copy, I would advise against rsync for this situation, overheads on checksumming etc will likely bottleneck on local transfers.

cp will be quite capable of handling what you’re trying to do, however I think a local tarpipe will probably get a better throughput, but without knowing exactly how your content is distributed on your source FS, I can’t be too sure.

The proviso here is that neither a tarpipe or standard copy will have a reliable recovery technique if the copy dies part way through. A tarpipe can be managed for recovery of a broken copy using exclude lists, but this would be tiresome.

A cp can sort of recover, with the flags (from the man page):

-n, --no-clobber
    do not overwrite an existing file (overrides a previous -i option)
-u, --update
    copy only when the SOURCE file is newer than the destination file or when the destination file is missing

Note however that if you used either of these flags, if the last file was partially copied (which is quite likely), that broken file would not be overwritten with the good file on the source.

rsync is the safest option out of those presented, but also likely to be the slowest.

I hope I’ve provided you the necessary information to make an informed decision to suit your particular needs.

5

@Haioken
Thank you for your analytical response.
Regarding dates/times and kodi what is there to argue about? It’s built to work like that. End of story.

About the copy process. I would like (ideally) to execute three separate copy functions from NTFS to BTRFS.

  • The first will take all my movies. They are all in separate folders along with .srt subs, .jpg pics, .nfo files and subfolders with extra pics for actors. So basically typical movie folders with metadata.
  • The second will be my tv shows with the same arrangement (files-folders-metadata) with extra subfolders for seasons of course.
  • The third will be my personal files’ backup which is completely variant content with pdf, doc, xls, jpg, img, ppt. txt, png, bin, iso -you name it-.

Do you think I should be using different commands among them?

And it kinda worries me that you said there is high probability for partial copies. Is this so common even for copy operations in local systems (not over lan)?

6

@Sky12016

My apologies, a bit of confusion there, the high probability of partials is in the event of failure.
IE: IF the copy process is interrupted, the last file in that chain will be partially copied. Linux will typically not clean this up for you like Windows will.

Also regarding Kodi’s management of date added, I had the issue that my nzb downloader was preserving mtime from rar archives when extracting, which screwed my Kodi database royally. I discovered I could resolve this by changing Kodi’s behaviour in the advancedsettings.xml to use the date the file was actually scanned into the database, rather than relying on the mtime property.

Personally, I would use different copy types, I would tarpipe the movies / tv shows, and straight cp the smaller content.
Could you describe your folder structure a little more? Are your subs, fanart and nfo files in the same dir as the video files?
If so, this makes things a little more complex, as you’ll need to seperate your copies by filespec rather than location. Ex:

cd /mnt/<ntfs_vol>
tar cf - *.mkv *.avi *.mp4 *.m4v | (cd /mnt2/<btrfs_mediavol> && tar xBf -)

Unfortunately tar will not preserve mtime. Note that this should not affect your Kodi install regardless as the mtime value is only used as the initial anchor timestamp upon entry into the database.

The easiest way is guaranteed to be a straight cp, you simply may not have the highest possible throughput if you choose to go that way.

7

@Haioken

My folders contain both video files and all metadata jpg, nfo etc so it is virtually impossible to separate them and treat them differently.

With regard to Kodi I am always adding stuff based on Modified date on windows and would like to keep doing that in Rockstor since I can change the mtime manually.

I will cross fingers and hope that my copy will not be interrupted. And I will go with the cp command. Speed should not be an issue as I plan on leaving it working overnight(s).

8

Hey @Haioken, I have another one.
My rockstor nas will be running headless and I will be managing it remotely through my dekstop pc’s browser and via Putty.
When a command line operation takes place (such as cp or rsync) do I need to be connected until the operation ends?
Please mention any difference between browser connection and Putty.

Thanks.

9

Hi @Sky12016,

Under normal circumstances you will need to be connected for the entire duration of the copy.
This includes via the web based shell in the Rockstor UI or when connected via SSH.

The reason for this is that when you start copying (regardless of the command used), the process is started under your TTY, which closes at the termination of your session.

To allow these process to continue after your session terminates, you need to run them under a terminal multiplexer such as screen or tmux.

I’m most familiar with screen, so we’ll use that.
Note than screen is not installed by default, install it with:

yum install -y screen

You can start a named screen session with:

screen -S copying_loads_of_stuff

You can then use CTRL-a to send commands directly to screen, and then d to detach.
When you’re detached, you can logout, terminate your session etc without incident.
You can then use screen -r to re-attach later if checking progress, etc.

screen -r copying_loads_of_stuff

Screen has a steep learning curve, so here’s an example that should be adaptable for what you want to do:

[ haioken@system1 ~ ]# ssh root@system2
[ root@system2 ~ ]# screen -S sleepSession
[ root@system2 ~ ]# sleep 2000

(ctrl-A d)

[detached from 31996.sleepSession]
[ root@system2 ~ ]# screen -ls
There is a screen on:
        31996.sleepSession     (Detached)
1 Socket in /var/run/screen/S-root.

[ root@system2 ~ ]# exit
[ haioken@system1 ~ ]# ssh root@system2
[ root@system2 ~ ]#  screen -ls
There is a screen on:
        31996.sleepSession     (Detached)
1 Socket in /var/run/screen/S-root.

[ root@system2 ~ ]# screen -r sleepSession
[ root@system2 ~ ]# sleep 2000    
^C
[ root@system2 ~ ]# exit
[screen is terminating]
[ root@system2 ~]# screen -ls
No Sockets found in /var/run/screen/S-root.

[ root@system2 ~]# exit
[ haioken@system1 ~ ]#

In order, from the above example, I have:

  • connected to system2
  • created a screen session called ‘sleepSession’
  • Started a sleep command
  • Detached from the screen
  • shown the screen session still running
  • disconnected from system2 entirely
  • Reconnected to system2
  • shown the screen session still running
  • Reattached to the screen session (which displays what was previously on that screen, hence why you see “sleep 2000” again)
  • Terminated the sleep command
  • exited the shell open in the screen session (thereby terminating the screen session)
  • shown no screens running anymore
  • disconnected from system2

I hope this helps!

3 Likes
10

@Haioken I can’t thank you enough. Great stuff!

I will experiment with both having a browser open overnight (from windows desktop) as well as with the screen command for performing copy operations with sample data.

With regard to the previous subject we were discussing:
Say I have a share (folder) over samba and I am viewing it from Windows.
When i change the modified time from windows; the change will be reflected in mtime under Rockstor (CentOS) correct?
I am asking because I am quite often performing this kind of operations on media files with windows’ software for many files concurrently, and I guess the same would be much more challenging via command line.

Cheers.

11

Honestly, I have no idea, I’ve never tried it. I would assume this is the case, but I have no hard data on the subject.
I also have no programs for manually altering the modification date on windows systems.

I would however advise that you should probably familiarize yourself with the linux CLI for this if you’re going to use a linux based NAS.
You can for example set the mtime timestamps of multiple files using touch, see Section 4 of the linked tutorial.

Combining this with Linux’ find can give you powerful methods of managing file timestamps.

12

Thanks @Haioken. I will take a closer look into those commands.

However; I wish that mtime holds any changes that take place in windows because it will make things much easier.