Synology: Backup and restore encrypted folders

This post quickly introduces encrypted folders and backing them up on a Synology NAS. It focuses on how to restore those backups, as this is not straightforward.

Encrypting shared folders

Creating an encrypted folder on a Synology NAS can be done easily, as described in detail by Synology here. Note: Don’t store you password on the NAS (Mount automatically on startup option), because this will render encryption useless! You don’t write the password to your computer on the display, do you?

Versioned backups – only for unencrypted folders!

Synology offers the Time Machine package that can be used to create different versions of stored file. However, this does still not work with encrypted folders (as of version 1.2-2300). Why Synology, why?

A simple backup solution for encrypted folders

Fortunately, there is an alternative that provides at least rudimentary ways for backing up data: The Backup and Restore package.

Backup and Restore icon

Backup and Restore

Once the encrypted folder is mounted, Backup and Restore can be used to create a local backup for all or some of the folders contained within the encrypted folder. Backups can be create for example on a daily, weekly or monthly basis. Unfortunately, it’s not possible to keep several versions of a backup – that’s what Time Machine would be for ūüė¶

Note: The Maximum number of kept versions only relates to the NAS configuration, not to the data!

Restoring encrypted backups – almost impossible?

By now, our data gets backed up regularly by the NAS. But how to restore a file, in case of emergency? There is the Restore tab within the Backup and Restore package. For encrypted backups, you can only use it to restore all or nothing. You can’t even choose where to restore the data to. That is, if you want to restore a single file, your only option is to overwrite all of your productive data. In other words: This is useless.

Restoring encrypted backups – a comfortable workaround

There is a workaround, however, that will make the backups accessible like any other shared folder:

  1. On the web interface of your NAS: Create a new shared folder, use the same password as for the encrypted folder that should be backed up. Let’s call it myBackup.
    Use the Read only permission within the Privileges setup in order to protected your backups.
  2. Unmount the new folder
  3. SSH to your NAS
  4. Delete the container you just created. For example:
    rm -r /volume1/@myBackup@
    
  5. Create a link to the backup that is named just like the container. For example:
    ln -s /volume1/backup/folder/@folder@ /volume1/@myBackup@
    

    Where backup is the shared folder where the backup was written to by the Backup and Restore package and folder is the name of the folder within backup that was set up in Backup and Restore.

  6. Go back to the web interface and mount the folder using the password of the encrypted container.

That’s it. You can now access the backup like any other folder (SMB/CIFS, NFS, FTP, …)

Restoring encrypted backups – mount backup on separate system

As an alternative, you could also mount the encrypted folder on any other Linux system.

Synology uses EcrytpFS to encrypt shared folders. Those can be mounted on a separate Linux system, as described here:

This is useful for remote backups or when your Synology Diskstation should be damaged but the hard disks still work.

Advertisements

Automatically downloading/backing up/dumping/exporting databases from remote hosts via the web

The problem

You operate a database-backed website (e.g. Drupal) where you can’t access cron jobs, cgi, perl and outgoing connections. So one idea to back up your database on a regular basis (which is always a good idea) is to download SQL dumps via a web-based administration tool (such as the backup and migrate plugin for drupal). Unfortunately, these kinds of downloads cannot simply be automated on the shell by using curl or wget, because they require a bit of javascript, for example to outsmart the php timeout.

The solution

Use a headless browser (that is, a browser without graphical user interface) to automate the task. It fetches the desired page, logs in, (virtually) clicks the download button and downloads the dump file.

It should be a command line tool, in order to run it as cron job from a some server (e.g. a NAS).

Personally, I liked the idea of PhantomJS, but it was not available for my Synology DS213+ PowerPC and I didn’t like the idea of building it from source.

So my plan B was to write a small Java program (remoteDbDumper)  that uses the HtmlUnit framework (our headless browser).

How to use

  1. Install drupal plugin backup and migrate.
  2. Download and extract remoteDbDumper.
  3. Start it from the shell.
    remoteDbDumper -u <username> -p <password> -o <output dir> <url to backup and migrate>

    Note that output dir must be an existing directory

    1. Linux example:
      ./remoteDbDumper.sh -u user -p topsecret -o ~/backup https://ho.st/drupal/?q=admin/config/system/backup_migrate
      
    2. Windows example
      remoteDbDumper.bat -u user -p topsecret -o "%HOMEPATH%\backup" https://ho.st/drupal/?q=admin/config/system/backup_migrate
      
  4. Use the scheduling mechanism of your choice to call remoteDbDumper regularly, creating backups.

Example (Synology)

Just a short exemplary scenario on how to use remoteDbDumper on a Synology Diskstation (running DSM 4.2) to regularly back up a drupal database.

  1. (if Java is not installed) install Java:
    If available for your Diskstation, use the Java Manager package. Otherwise, you could use a third party Java package (that’s what I had to do).
  2. Download, extract and copy remoteDbDumper to the NAS (e.g. to \\diskstation\public\, which corresponds to /volume1/public/)
  3. SSH to the NAS and check if it works
    /volume1/public/remoteDbDumper-1.0/remoteDbDumper.sh -u user -p topsecret -o /volume1/someUser/ https://ho.st/drupal/?q=admin/config/system/backup_migrate
    
  4. (optional) Wrap the command line call in a shell script, e.g.
    BASEDIR=$(dirname $0)
    $BASEDIR/remoteDbDumper-1.0/remoteDbDumper.sh -u user -p topsecret -o $1 https://ho.st/drupal/?q=admin/config/system/backup_migrate
    
  5. Either use the web frontend  or the crontab to schedule the back up.
    1. Web frontend:
      Go to http://diskstation:5000, (or whatever combination of host name and port you’re using)
      login as admin,
      click on Control Panel | Task Scheduler.
      Then click on Create | User-defined Script.
      Enter a task name, choose a user (preferably not root), set up a schedule (e.g. every sunday at 8 p.m.).
      Finally enter the path to remoteDbDumpe or the script (4.) respectively. For the example above, the path would look like this:

      /volume1/public/dumpDb.sh /volume1/public/
      
    2. If you insist to do it on foot, here’s what to enter in crontab:
      vi /etc/crontab
      
      #minute hour    mday    month   wday    who              command
      0       20      *       *       0       enterUserHere    /volume1/public/dumpDb.sh /volume1/public/
      
    3. Set a maker in your calender for the next scheduled run, to check if it worked.

Future tasks

At the current state remoteDbDumper can only backup drupal databases. Fair enough.

However, with just a little more effort it would be possible to extend remoteDbDumper to support addition web-based database administration tools, such as  mysqldumper, phpMyBackupPro, phpMyAdmin or phpPhAdmin.

In order to do so, just fork the repo on github and implement the interface DbDump.

Microsoft Robocopy vs Linux NAS: Robocopy Pitfalls

Intro

I have been using Microsoft Robocopy for years, as it is an easy (by now even a built-in) way to do incremental backups under Windows.

Until recently, I used to backup my data from an internal NTFS hard drive to an external NTFS hard drive. As of late, I’m the proud owner of a DS213+ NAS, which runs some Linux base OS and Ext4 hard drives.

As I’m still looking for the perfect backup/versioning client (rsync on windows?!), I thought to stick to Robocopy in the meantime. Unfortunately, my backup scripts, which have done a splendid job of incrementally backing up my data to an external hard drive for years, now do a full backup to the NAS every time.

As it turns out, there is not only one reason for this behavior, but two:

  1. Timestamp
  2. File size

Here are the solutions solving these issues (at least for me), as well as some additional hints to using Robocopy.

1. Timestamp

At first, Robocopy kept telling me NEWER or OLDER for each file (even though the file didn’t change), resulting in copying the file instead of skipping it.

Solution:

First, make sure that both the NAS and the client PC have the same system time (use an NTP server, for example).

If the problem still persists, a good solution is to make Robocopy use FAT file times (/FFT).

This results in a 2-second-granularity, that is a file is only declared NEWER or OLDER, when it there is a difference of more than two seconds between the source and the destination file. If this option is not set, a nanosecond-granularity is used. Obviously, Samba’s time granularity is not as precise, and therefore the time stamps hardly ever match.

2. File size

If your incremental works by now, skip the rest of the article.

As for me, after solving the above problem, the incremental backups still didn’t work.

Robocopy kept telling me CHANGED for most files. Out of the frying pan into the fire!

What does CHANGED mean? The answer can be found here:

The source and destination files have identical time stamps but
different file sizes. The file is copied; to skip this file, use /XC.

Skipping all files with different sizes? No, that’s some dangerous idea when backing up data. So what now?

But why do they have a different size at all? Thats some file on the client PC:

SomeFile on the Client PC

SomeFile on the Client PC

And that’s the same file after transferring to the NAS:

SomeFile on the NAS

SomeFile on the NAS

Tthe attentive observer might have recognized that the size on the disk is different.

The reason for this can be found in different block sizes used in NAS and Client. I was wondering first, because I set up both NTFS and Ext4 with a Block size of 4K.

However, the Samba server has a default block size of 1k! So setting up the Samba with an explicit block size that matches the one of your client PC solves this issue.

How?

SSH to your (Synology) NAS.

 vi /usr/syno/etc/smb.conf

Press i.

Enter this line bellow the [global] tag (use the block size of your host file system, e.g. 4K = 4×1024=4096)

         block size = 4096

Press ESC, then enter :wq and press ENTER.

Restart the samba server by

/usr/syno/etc/rc.d/S80samba.sh restart

That solved my problems and I can now do incremental backups again.
Until I finally have set up perfect rsync for windows solution ūüôā

Alternative solution for 1. and 2.

There is, however, an alternative to the solutions for 1. and 2.:

Use the Archive bit. Each file has an archive bit. Everytime you change the file, the bit is set. This behavior can be utilized by Robocopy. Using the /m switch makes Robocopy reset the archive bit on each source file and skips all files whose archive bit is set. That is, it copies only files that changed since the last backup. No need for caring about nasty time stamps or stupid file sizes.

There is one drawback, however. When you want to make a full backup, or you backup your data to several devices, you must not use the /m switch or your backups will be incomplete.

Additional Hints

While I’m on it: Here’s the Robocopy options I use. See Robocopy | SS64.com for complete reference.

robocopy "<source>" "<dest>" /MIR /V /NP /TEE /LOG:"%~f0.log" /Z /R:10 /W:10 /FFT /DCOPY:T
  • /MIR – MIRror a directory tree, that is, copy all subfolders and purge extra files on the destination
  • /V – Verbose output log, showing skipped files
  • /NP – No Progress. Don’t flood log file with % copied for each file.
  • /TEE – Output to console window, as well as the log file.
  • /LOG:”%~f0.log” – Output status to file <name-of-batch-file>.log, overwrite existing one.
  • /Z – Copy files in restartable mode (survive network glitch)
  • /R:10 – Number of Retries on failed copies – default is 1 million. Reducing the number is helpful when trying to copy locked files.
  • /W:10 – Wait time between retries – default is 30 seconds. Reducing the number is helpful when trying to copy locked files.
  • /FFT – Assume FAT File Times, 2-second date/time granularity. See above.
  • /DCOPY:T – Copy Directory Timestamps. Why is this not default? You definitely want to keep your directory timestamps!