Automatically downloading/backing up/dumping/exporting databases from remote hosts via the web

The problem

You operate a database-backed website (e.g. Drupal) where you can’t access cron jobs, cgi, perl and outgoing connections. So one idea to back up your database on a regular basis (which is always a good idea) is to download SQL dumps via a web-based administration tool (such as the backup and migrate plugin for drupal). Unfortunately, these kinds of downloads cannot simply be automated on the shell by using curl or wget, because they require a bit of javascript, for example to outsmart the php timeout.

The solution

Use a headless browser (that is, a browser without graphical user interface) to automate the task. It fetches the desired page, logs in, (virtually) clicks the download button and downloads the dump file.

It should be a command line tool, in order to run it as cron job from a some server (e.g. a NAS).

Personally, I liked the idea of PhantomJS, but it was not available for my Synology DS213+ PowerPC and I didn’t like the idea of building it from source.

So my plan B was to write a small Java program (remoteDbDumper)  that uses the HtmlUnit framework (our headless browser).

How to use

  1. Install drupal plugin backup and migrate.
  2. Download and extract remoteDbDumper.
  3. Start it from the shell.
    remoteDbDumper -u <username> -p <password> -o <output dir> <url to backup and migrate>

    Note that output dir must be an existing directory

    1. Linux example:
      ./remoteDbDumper.sh -u user -p topsecret -o ~/backup https://ho.st/drupal/?q=admin/config/system/backup_migrate
      
    2. Windows example
      remoteDbDumper.bat -u user -p topsecret -o "%HOMEPATH%\backup" https://ho.st/drupal/?q=admin/config/system/backup_migrate
      
  4. Use the scheduling mechanism of your choice to call remoteDbDumper regularly, creating backups.

Example (Synology)

Just a short exemplary scenario on how to use remoteDbDumper on a Synology Diskstation (running DSM 4.2) to regularly back up a drupal database.

  1. (if Java is not installed) install Java:
    If available for your Diskstation, use the Java Manager package. Otherwise, you could use a third party Java package (that’s what I had to do).
  2. Download, extract and copy remoteDbDumper to the NAS (e.g. to \\diskstation\public\, which corresponds to /volume1/public/)
  3. SSH to the NAS and check if it works
    /volume1/public/remoteDbDumper-1.0/remoteDbDumper.sh -u user -p topsecret -o /volume1/someUser/ https://ho.st/drupal/?q=admin/config/system/backup_migrate
    
  4. (optional) Wrap the command line call in a shell script, e.g.
    BASEDIR=$(dirname $0)
    $BASEDIR/remoteDbDumper-1.0/remoteDbDumper.sh -u user -p topsecret -o $1 https://ho.st/drupal/?q=admin/config/system/backup_migrate
    
  5. Either use the web frontend  or the crontab to schedule the back up.
    1. Web frontend:
      Go to http://diskstation:5000, (or whatever combination of host name and port you’re using)
      login as admin,
      click on Control Panel | Task Scheduler.
      Then click on Create | User-defined Script.
      Enter a task name, choose a user (preferably not root), set up a schedule (e.g. every sunday at 8 p.m.).
      Finally enter the path to remoteDbDumpe or the script (4.) respectively. For the example above, the path would look like this:

      /volume1/public/dumpDb.sh /volume1/public/
      
    2. If you insist to do it on foot, here’s what to enter in crontab:
      vi /etc/crontab
      
      #minute hour    mday    month   wday    who              command
      0       20      *       *       0       enterUserHere    /volume1/public/dumpDb.sh /volume1/public/
      
    3. Set a maker in your calender for the next scheduled run, to check if it worked.

Future tasks

At the current state remoteDbDumper can only backup drupal databases. Fair enough.

However, with just a little more effort it would be possible to extend remoteDbDumper to support addition web-based database administration tools, such as  mysqldumper, phpMyBackupPro, phpMyAdmin or phpPhAdmin.

In order to do so, just fork the repo on github and implement the interface DbDump.

JSF: Displaying FacesMessages during render response phase

The problem

A controller runs into an error situation during the last phase of the JSF lifecycle (the RENDER_RESPONSE phase). In order to let the user know that something went wrong, a FacesMessage is added. However, the faces message is not shown on the client. There may or may not be a warning on the log which says “FacesMessage(s) have been enqueued, but may not have been displayed“.

The cause

In the render response phase the component tree is traversed and each component is rendered (that is, HTML is generated). When an error occurs after the message component finished rendering, it is not possible to add another message to it.

Some solutions/workarounds

  1. Place the message component at the end of the component tree (that is, the page or template). That way, it’s most likely that the message widget has not been rendered at the time the content is rendered (which is when the error occurs and the FacesMessage is added). The FacesMessages are processed later, when the message widget is rendered.
    However, in most cases you don’t want to display your message at the end of the page. This can be solved with at least these two approaches:

    1. Use an overlay (for example PrimeFace’s growl widget – <p:growl>), whose position on the rendered page is “independent” of its position in the component tree.
    2. Move the widget via CSS positioning (which is a bit of an ugly workaround).
  2. Try to validate the data in an earlier phase. For example, you could check the data before rendering via an event:
    <f:event type="preRenderView" listener="#{bean.initData}"/>
    

    Or (if you’re using Seam), you could use Seam page actions,  as shown in this thread.

  3. Save the messages in the session and trigger a server render (which can be done with ICEfaces). This can be realized with a PhaseListener as described here.

Which one to use?

This depends on your use case and the component framework you’re using.

Solution #3 should always be your last measure, as it is really heavyweight and it requires additional HTTP requests.

#2 cannot be used in conjunction with some components and requires changing the UI logic and increases complexity of your code.

Which leaves #1, which is the most pragmatic solution. However, it can’t be used when your FacesMessage should be displayed on a position within the page (e.g. on the top of the page).

An example

When rendering data using PrimeFace’s DataTable with lazy loading (p:dataTable) the data is fetched from database via the LazyDataModel at rendering time (render repsonse phase). Once the data is fetched it might occur that some data is missing and the user should be informed about this fact. In order to do so, a FacesMessage would be the way to go. However, as described above, the message never gets rendered.

As far as I know, fetching the data cannot be brought forward to an earlier phase.

As the growl widget was already used within the application as the way of displaying information to the user, the solution was as easy to move the growl widgets to the end of the layout template. And voilà, the messages suddenly show up!

This solution was really easy to realize, but finding out about it took a painful amount of time. So hopefully, this post may speed up the process for anyone stuck with the same problem 🙂