Intro
I have been using Microsoft Robocopy for years, as it is an easy (by now even a built-in) way to do incremental backups under Windows.
Until recently, I used to backup my data from an internal NTFS hard drive to an external NTFS hard drive. As of late, I’m the proud owner of a DS213+ NAS, which runs some Linux base OS and Ext4 hard drives.
As I’m still looking for the perfect backup/versioning client (rsync on windows?!), I thought to stick to Robocopy in the meantime. Unfortunately, my backup scripts, which have done a splendid job of incrementally backing up my data to an external hard drive for years, now do a full backup to the NAS every time.
As it turns out, there is not only one reason for this behavior, but two:
- Timestamp
- File size
Here are the solutions solving these issues (at least for me), as well as some additional hints to using Robocopy.
1. Timestamp
At first, Robocopy kept telling me NEWER or OLDER for each file (even though the file didn’t change), resulting in copying the file instead of skipping it.
Solution:
First, make sure that both the NAS and the client PC have the same system time (use an NTP server, for example).
If the problem still persists, a good solution is to make Robocopy use FAT file times (/FFT).
This results in a 2-second-granularity, that is a file is only declared NEWER or OLDER, when it there is a difference of more than two seconds between the source and the destination file. If this option is not set, a nanosecond-granularity is used. Obviously, Samba’s time granularity is not as precise, and therefore the time stamps hardly ever match.
2. File size
If your incremental works by now, skip the rest of the article.
As for me, after solving the above problem, the incremental backups still didn’t work.
Robocopy kept telling me CHANGED for most files. Out of the frying pan into the fire!
What does CHANGED mean? The answer can be found here:
The source and destination files have identical time stamps but
different file sizes. The file is copied; to skip this file, use /XC.
Skipping all files with different sizes? No, that’s some dangerous idea when backing up data. So what now?
But why do they have a different size at all? Thats some file on the client PC:
And that’s the same file after transferring to the NAS:
Tthe attentive observer might have recognized that the size on the disk is different.
The reason for this can be found in different block sizes used in NAS and Client. I was wondering first, because I set up both NTFS and Ext4 with a Block size of 4K.
However, the Samba server has a default block size of 1k! So setting up the Samba with an explicit block size that matches the one of your client PC solves this issue.
How?
SSH to your (Synology) NAS.
vi /usr/syno/etc/smb.conf
Press i.
Enter this line bellow the [global] tag (use the block size of your host file system, e.g. 4K = 4×1024=4096)
block size = 4096
Press ESC, then enter :wq and press ENTER.
Restart the samba server by
/usr/syno/etc/rc.d/S80samba.sh restart
That solved my problems and I can now do incremental backups again.
Until I finally have set up perfect rsync for windows solution 🙂
Alternative solution for 1. and 2.
There is, however, an alternative to the solutions for 1. and 2.:
Use the Archive bit. Each file has an archive bit. Everytime you change the file, the bit is set. This behavior can be utilized by Robocopy. Using the /m switch makes Robocopy reset the archive bit on each source file and skips all files whose archive bit is set. That is, it copies only files that changed since the last backup. No need for caring about nasty time stamps or stupid file sizes.
There is one drawback, however. When you want to make a full backup, or you backup your data to several devices, you must not use the /m switch or your backups will be incomplete.
Additional Hints
While I’m on it: Here’s the Robocopy options I use. See Robocopy | SS64.com for complete reference.
robocopy "<source>" "<dest>" /MIR /V /NP /TEE /LOG:"%~f0.log" /Z /R:10 /W:10 /FFT /DCOPY:T
- /MIR – MIRror a directory tree, that is, copy all subfolders and purge extra files on the destination
- /V – Verbose output log, showing skipped files
- /NP – No Progress. Don’t flood log file with % copied for each file.
- /TEE – Output to console window, as well as the log file.
- /LOG:”%~f0.log” – Output status to file <name-of-batch-file>.log, overwrite existing one.
- /Z – Copy files in restartable mode (survive network glitch)
- /R:10 – Number of Retries on failed copies – default is 1 million. Reducing the number is helpful when trying to copy locked files.
- /W:10 – Wait time between retries – default is 30 seconds. Reducing the number is helpful when trying to copy locked files.
- /FFT – Assume FAT File Times, 2-second date/time granularity. See above.
- /DCOPY:T – Copy Directory Timestamps. Why is this not default? You definitely want to keep your directory timestamps!
IT affinity! 