Continuous Deployment – Automatic Backup Script

A few words about Continuous Deployment

Continuous Deployment is the deployment or release of code to Production as soon as it is ready. (…) The automated process is key because it should be able to be performed by anyone in a matter of minutes (preferably by the press of a button).
Once you have moved to a Continuous Deployment process, you will have to have several pieces of automation in place. You must automate your Continuous Integration Build Server and Continuous Delivery to Staging, as well as have the ability to automatically deploy to Production.
Read full blog post

…we’re on the way

Because every manual step implies a risk for failure, our goal is to minimize such risks as well as our amount of work by automating our processes of software delivery. In several of our applications we already use a deployment script for automatic deployment of Tomcat applications.
For some applications we even use a continuous deployment script triggered by crontab (e.g. every hour) to check if Nexus has a new version of the application and if so to fetch and deploy it automatically.
However the backup process was still manual – although you always perform the same steps. This is a perfect opportunity for automation. And a perfect opportunity for me to leave the Java world for a while and to learn more about shell scripting.
The typical backup steps before a deployment are:

• saving war file resp. information about the current deployed version
• saving database data in a dump
• saving directories and/or files, e.g. generated error reports

This blog post explains the configuration and the most important steps of the backup script. You can find the entire backup script and the configuration files on Github.

Configuration for mysqldump

The mysqldump client is the usual command line tool to dump a database for backup or for transferring the database data. The created dump contains SQL statements to create and/or to populate table(s). To be able to execute mysqldump in an automatic process with nobody around to enter the password, you have to provide the access data in a file. Create a .ini-style file named .my.cnf providing user and password information. Later in the script, mysqldump will read this configuration and will automatically use the supplied user name and password.
If you have only one database to be dumped, following lines are enough:

[mysqldump]
 user = myUser
 password = xxx

If you have more than one database to be dumped needing different access data, you have to specify an access data section for every database.
For example, if you want the databases “foo” and “bar” to be dumped, you’ll need two sections with the corresponding database name as suffix: [mysqldumpfoo] and [mysqldumpbar].

[mysqldumpfoo]
 user = fooUser
 password = xxx
[mysqldumpbar]
 user = barUser
 password = xxx

Caution: In this case there must not be a [mysqldump] section since the more specific sections with database name as suffix would be ignored.
If you want to prevent other users from reading your .my.cnf, change the file permission so that only the owner has full read and write permissions.

chmod 600 ~/.my.cnf

Configuration for Backup script

The backup script will depend on a configuration file containing details for the backup process:

• the absolute path of the deployed webapp
• the absolute path of the parent directory where the backups should be stored
• the names of the databases to be dumped
• the absolute path of the mysqldump config file with access data
• if the deployed war file itself or if only the information about the current deployed version should be saved
• if there are files and/or directories that should be moved or copied to the backup directory

# absolute path of the deployed webapp
WEBAPP_PATH="$HOME/tomcat/webapps/ROOT"
# absolute path where the backups should be stored
BACKUP_PATH="$HOME/backup"
# names of the databases to be dumped
# separate with whitespace
DB_NAME="db1 db2 db3"
# absolute path of the mysql config file with user data
# for the above specified databases
MYSQL_CONF="$HOME/.my.cnf"
# decide if current deployed war should be saved in backup dir
# or if only the information about the version that is deployed
# should be saved
#
# 0: save war only if the current deployed version is a snapshot
# 1: always save war
#
# if you specify nothing or something that is not 0 or 1,
# current deployed war file won't be saved
WAR=1
# specifiy folders and/or files (absolute path!)
# that should be copied resp. moved into backup dir
# separate with whitespace
CONTENT_TO_BE_COPIED="$HOME/file $HOME/folder"
CONTENT_TO_BE_MOVED="$HOME/folder/file $HOME/folder/*"

The backup script

Now let’s have a look at the actual backup script and the most important steps of it.

Get configuration file

You can call the script with the option -c to specify which configuration file should be used.

./backup.sh -c config/custom.conf

If you call the script without the -c option, the script tries to get a file named backup.conf in the directory the script is located.

# default config file
CONF_FILE="$(dirname "$0")/backup.conf"
# a colon after the option means that the option
# expects an argument
while getopts "hc:" OPTION; do
case "$OPTION" in
c)
CONF_FILE=$OPTARG
;;
esac
done

Read and validate configuration file

The script makes sure that the configuration file exists and contains all the required parameters. Otherwise it will fail with an appropriate error message.

# variable is empty
if [ -z "$CONF_FILE" ]; then
  echo "CONF_FILE not set"
  exit 1
fi
# no file found for the given variable
if [ ! -f "$CONF_FILE" ]; then
  echo "No configuration file found"
  exit 1
fi
# read in variables of conf file
. "$CONF_FILE"
# make sure that conf file contains all required variables
if [ -z "$WEBAPP_PATH" ] || [ -z "$BACKUP_PATH" ] || [ -z "$DB_NAME" ] || [ -z "$MYSQL_CONF" ]; then
  echo "Configuration file seems to be incorrect: required variables missing. Please check your config file: $(readlink -f "$CONF_FILE")"
  exit 1
fi
# make sure that WEBAPP_PATH exists
if [ ! -d "$WEBAPP_PATH" ]; then
  echo "The given webapp path doesn't exist. Please check your config file: $(readlink -f "$CONF_FILE")"
  exit 1
fi
# make sure that mysql config file exists
if [ ! -f "$MYSQL_CONF" ]; then
  echo "The given mysql config file doesn't exist. Please check your config file: $(readlink -f "$CONF_FILE")"
  exit 1
fi

Creating the backup directories

The script checks if the specified parent backup directory already exists – and will create it if it doesn’t. For every processed backup a time stamped folder (with the pattern yyyy-MM-dd_hh-mm) is created in this directory containing the backup data. Allowed is only one backup per minute. If you try to run the script and there is already a backup folder for the current minute, the script will fail with an error message.

CURRENT_DATE="$(date +%Y-%m-%d_%H-%M)"
FULL_BACKUP_PATH="$BACKUP_PATH/$CURRENT_DATE"
# check if backup dir exists, if not create it
if [ ! -d "$BACKUP_PATH" ]; then
  mkdir "$BACKUP_PATH"
  echo "Create parent backup directory $(readlink -f "$BACKUP_PATH")"
fi
# check if there is already a dir for current date, if not create it
if [ ! -d "$FULL_BACKUP_PATH" ]; then
  mkdir "$FULL_BACKUP_PATH"
  echo "Create backup directory $(readlink -f "$FULL_BACKUP_PATH")"
else
  echo "$(readlink -f "$FULL_BACKUP_PATH") already exists"
  echo "Wait until $(readlink -f "$BACKUP_PATH/$(date -d '+1min' +%Y-%m-%d_%H-%M)") can be created"
  exit 1
fi

Save the information about the current deployed version

You find the information about the current deployed version in the pom.properties:

 ~/tomcat/webapps/ROOT/META-INF/maven/org.synyx/foo/pom.properties

It looks like this:

#Generated by Maven
#Sat Mar 30 02:03:52 CET 2013
version=1.1.10-SNAPSHOT
groupId=org.synyx
artifactId=foo

This .properties file contains all the relevant information we need if we’d like to rollback to the backup state. The Nexus deploy script takes the group id, artifact id and version as parameters:

./nexusdeploy.sh -i org.synyx:foo:1.1.10-SNAPSHOT -w ROOT

So this properties file is copied to the backup directory.

PROPS_NAME="pom.properties"
APP_PROPS="$(find "$WEBAPP_PATH" -name "$PROPS_NAME")"
# more than one file matching the criterias found
if [ ! $(find "$WEBAPP_PATH" -name "$PROPS_NAME" | wc -l) -eq 1 ]; then
  echo "No or more than one $PROPS_NAME was found under the webapp path $(readlink -f "$WEBAPP_PATH")"
  exit 1
fi
# copy the information about the current deployed version
cp "$APP_PROPS" "$FULL_BACKUP_PATH"

Save the war file itself if configured

We remember, in the configuration file there are three possibilities what to do with the war file during the backup process:

• always save the war file
• never save the war file
• save the war file only if the current deployed version is a snapshot

The first two states are self-explanatory: either the war file is copied to the backup directory or not. If the war file should be only saved if it is a snapshot, the pom.properties file has to be read in and the variable $version has to be checked if it contains the string ‘SNAPSHOT’.

# variable isn't empty
if [ ! -z "$WAR" ]; then
SAVE_WAR=0
# decide if war should be saved
# if conf = save war only if current deployed version is a snapshot
if [ "$WAR" -eq 0 ]; then
# check if current deployed version is a snapshot
# read in properties
. "$APP_PROPS"
# $version has information about current deployed version
case "$version" in
        *SNAPSHOT)
                SAVE_WAR=1
                ;;
esac
# if conf = save war always
elif [ "$WAR" -eq 1 ]; then
  SAVE_WAR=1
fi
if [ "$SAVE_WAR" -eq 1 ]; then
  echo "Backup war file: "
  cp -v "$WEBAPP_PATH"/../*.war "$FULL_BACKUP_PATH"
fi
fi

Save the database data

For every element in $DB_NAME a dump is created.
Due to the existing .my.cnf with access data information, mysqldump can be called without the parameters user and password. However it needs the following information:

• the path to the config file with the access data (--defaults-file)
• the suffix (database name) which access data section of the config file should be used (--defaults-group-suffix)
• the database name to know which database should be dumped

If something goes wrong during database dump (e.g. access is denied), the error is redirected to /dev/null because the script has its own error handling. It checks if $? is 0 or not. $? gives the exit status of the last executed command. A status not 0 is an error code, i.e. the last command wasn’t successful.

for db in $DB_NAME
do
  # create dump and hide mysqldump errors due to own error handling
  mysqldump --defaults-file="$MYSQL_CONF" --defaults-group-suffix="$db" "$db" > "$FULL_BACKUP_PATH/$db-$CURRENT_DATE.dump.sql" 2>/dev/null
  # creating dump was successful, so return value is 0
  if [ "$?" -eq 0 ]; then
    echo "Create dump for database $db: $(readlink -f "$FULL_BACKUP_PATH/$db-$CURRENT_DATE.dump.sql")"
  # something went wrong while trying to create dump (e.g. access denied), so return value is not 0 but any other number
  else
    echo "Problems encountered while trying to create dump for database $db"
  fi
done

Save specified files and/or directories

To move the specified files and/or directories to the backup directory, following steps are performed:

• make sure that the variable $CONTENT_TO_BE_MOVED isn’t empty
• execute the move operation for each element
• if an element doesn’t exist, print an error message

# make sure that variable $CONTENT_TO_BE_MOVED isn't empty
if [ ! -z "$CONTENT_TO_BE_MOVED" ]; then
for i in $CONTENT_TO_BE_MOVED
do
  # if the given content is a dir or a file, process else throw an error
  if [ -d "$i" ] || [ -f "$i" ]; then
    echo "Move $(readlink -f "$i") into backup dir"
    mv "$i" "$FULL_BACKUP_PATH"
  else
    echo "Can't move $i: doesn't exist";
  fi
done
fi

These steps are similar for the files/directories to be copied, only the action (copy) and the parameter ($CONTENT_TO_BE_COPIED) are different.

Don’t forget to make your script executable 😉

chmod +x backup.sh

Automation

Now it’s no longer required to do the backup manually. 🙂 And even better: if this script is integrated into our existing automatic deployment script, every time a deployment is performed, a backup will be performed first.

Links

Github Project: automated-backup
Link to opening quote: Continuous Delivery vs. Continuous Deployment vs. Continuous-Integration