I have been using Microsoft Robocopy for years, as it is an easy (by now even a built-in) way to do incremental backups under Windows.
Until recently, I used to backup my data from an internal NTFS hard drive to an external NTFS hard drive. As of late, I’m the proud owner of a DS213+ NAS, which runs some Linux base OS and Ext4 hard drives.
As I’m still looking for the perfect backup/versioning client (rsync on windows?!), I thought to stick to Robocopy in the meantime. Unfortunately, my backup scripts, which have done a splendid job of incrementally backing up my data to an external hard drive for years, now do a full backup to the NAS every time.
As it turns out, there is not only one reason for this behavior, but two:
- File size
Here are the solutions solving these issues (at least for me), as well as some additional hints to using Robocopy.
At first, Robocopy kept telling me NEWER or OLDER for each file (even though the file didn’t change), resulting in copying the file instead of skipping it.
First, make sure that both the NAS and the client PC have the same system time (use an NTP server, for example).
If the problem still persists, a good solution is to make Robocopy use FAT file times (/FFT).
This results in a 2-second-granularity, that is a file is only declared NEWER or OLDER, when it there is a difference of more than two seconds between the source and the destination file. If this option is not set, a nanosecond-granularity is used. Obviously, Samba’s time granularity is not as precise, and therefore the time stamps hardly ever match.
2. File size
If your incremental works by now, skip the rest of the article.
As for me, after solving the above problem, the incremental backups still didn’t work.
Robocopy kept telling me CHANGED for most files. Out of the frying pan into the fire!
What does CHANGED mean? The answer can be found here:
The source and destination files have identical time stamps but
different file sizes. The file is copied; to skip this file, use /XC.
Skipping all files with different sizes? No, that’s some dangerous idea when backing up data. So what now?
But why do they have a different size at all? Thats some file on the client PC:
And that’s the same file after transferring to the NAS:
Tthe attentive observer might have recognized that the size on the disk is different.
The reason for this can be found in different block sizes used in NAS and Client. I was wondering first, because I set up both NTFS and Ext4 with a Block size of 4K.
However, the Samba server has a default block size of 1k! So setting up the Samba with an explicit block size that matches the one of your client PC solves this issue.
SSH to your (Synology) NAS.
Enter this line bellow the [global] tag (use the block size of your host file system, e.g. 4K = 4×1024=4096)
block size = 4096
Press ESC, then enter :wq and press ENTER.
Restart the samba server by
That solved my problems and I can now do incremental backups again.
Until I finally have set up perfect rsync for windows solution 🙂
Alternative solution for 1. and 2.
There is, however, an alternative to the solutions for 1. and 2.:
Use the Archive bit. Each file has an archive bit. Everytime you change the file, the bit is set. This behavior can be utilized by Robocopy. Using the /m switch makes Robocopy reset the archive bit on each source file and skips all files whose archive bit is set. That is, it copies only files that changed since the last backup. No need for caring about nasty time stamps or stupid file sizes.
There is one drawback, however. When you want to make a full backup, or you backup your data to several devices, you must not use the /m switch or your backups will be incomplete.
While I’m on it: Here’s the Robocopy options I use. See Robocopy | SS64.com for complete reference.
robocopy "<source>" "<dest>" /MIR /V /NP /TEE /LOG:"%~f0.log" /Z /R:10 /W:10 /FFT /DCOPY:T
- /MIR – MIRror a directory tree, that is, copy all subfolders and purge extra files on the destination
- /V – Verbose output log, showing skipped files
- /NP – No Progress. Don’t flood log file with % copied for each file.
- /TEE – Output to console window, as well as the log file.
- /LOG:”%~f0.log” – Output status to file <name-of-batch-file>.log, overwrite existing one.
- /Z – Copy files in restartable mode (survive network glitch)
- /R:10 – Number of Retries on failed copies – default is 1 million. Reducing the number is helpful when trying to copy locked files.
- /W:10 – Wait time between retries – default is 30 seconds. Reducing the number is helpful when trying to copy locked files.
- /FFT – Assume FAT File Times, 2-second date/time granularity. See above.
- /DCOPY:T – Copy Directory Timestamps. Why is this not default? You definitely want to keep your directory timestamps!
7 thoughts on “Microsoft Robocopy vs Linux NAS: Robocopy Pitfalls”
This is the most cogent article on this topic I’ve seen. Especially compared to the noise in the synology forums. Thank you for the deep dive insight.
I’ve tried /fft, /dst, and now your block size trick and I still don’t get incremental backups.
Changing the samba block size per your post did correct the “size on disk” differences. Win!
This is only happening to me with the Windows 8 version of robocopy. If I use the Windows 2003 Resource Kit version (in a win8 OS) incremental backups work as expected (without /FFT). SO FRUSTRATING! But hey, it’s a workaround.
To be completely honest, my solution described above stopped working after updating the DS. So, until just now, I sticked to using the archive bit, as described as above
Right now, I’m doing a “real” incremental backup using Windows Server 2003 Resource Kit Tools: http://www.microsoft.com/en-us/download/details.aspx?id=17657
I never thought about using the “old” robocopy. It’s absolutely weird – but it actually works! Thanks a mil for pointing that out!
Still looking for a good backup solution for Windows, something like Time Machine for Mac or rsync for Unix.
Just didn’t find the time to investigate this any further.
Any ideas anyone?
Informative, thank you.
I just discovered a program called vbackup. I haven’t scripted around it yet but versioning sounds awesome.
Thanks for the hint. I might have a look at it.
I moved most of my jobs to rsync until now. With the Windows Subsystem Linux (WSL) at hand, it’s easy to use the same tools on all plattforms.
Thank you so much for the info!!!
I had the same issue with robocopy and backup on NAS under Linux OS. Your advices with /FFT and setup a same system time between PC and NAS solved this issue.
Thanks for your feedback! Glad my post was helpful.