Maintaining an offline backup

Last modified by Discovery on Thu, June 24, 2010 15:26
Source|Old Revisions  

If you have a development server then you may want to set it up to be a mirror of your live server. To do this you will need a copy of your web root and the database, configured to work with the domain name of your development server.

Maintaining a file backup

If you run more than one site on your live host then you may have lots of web root folders to backup/archive. Therefore you may want to clone /var/www/ instead of just /var/www/html/magento.

Naturally you will want your files to be as fresh as possible - ‘rsync’ is the tool of choice. The instructions at http://www.howtoforge.com for setting this up are straightforward and easy enough to follow:

http://www.howtoforge.com/mirroring_with_rsync

If on CentOS/RHEL/Fedora you will want to ‘yum install rsync’ on both live and development servers.

Now, on both the live and the development server, create a new user. If you have a graphical interface use that and make sure the primary group of your new ‘rsync’ user is ‘apache’. If you do not have a graphical interface, then use the instructions above or install something like ‘webmin’.

Now, as root, still on the development server:

mkdir /root/rsync
ssh-keygen -t dsa -b 1024 -f /root/rsync/mirror-rsync-key 

Press enter when asked for a passkey so that the keyset used for the login doesn’t need human interaction.

Now login to the live server with your new ‘rsync’ user login. Create a new ssh key facility:

mkdir ~/.ssh
chmod 700 ~/.ssh
cd ~/.ssh
touch authorized_keys
chmod 600 authorized_keys

Now to the end of the authorized_keys you will need to add the key. Open the (blank) file and cut and paste the following, putting in the name or fqdn of your development server for “from”:

command="/home/someuser/rsync/checkrsync",from="mirror.example.com",no-port-forwarding,no-X11-forwarding,no-pty

In the line after that cut and paste /root/rsync/mirror-rsync-key.pub from the terminal you have kept open on the development server. Now maximize your terminal window and make sure no line-breaks have been cut and pasted across. Should this be the case, delete them so the key is on one very long line.

Save the authorized_keys file and now create the script that will be used for performing the rsync. Still on the server:

mkdir ~/rsync
vi ~/rsync/checkrsync

Now enter:

#!/bin/sh

case "$SSH_ORIGINAL_COMMAND" in
        *&*)
                echo "Rejected"
                ;;
        *(*)
                echo "Rejected"
                ;;
        *{*)
                echo "Rejected"
                ;;
        *;*)
                echo "Rejected"
                ;;
        *<*)
                echo "Rejected"
                ;;
        *`*)
                echo "Rejected"
                ;;
        rsync --server*)
                $SSH_ORIGINAL_COMMAND
                ;;
        *)
                echo "Rejected"
                ;;
esac

Make the file executable:

chmod 700 ~/rsync/checkrsync

You should now have everything in place to test the rsync. Before doing this you may want to move whatever files you have on your development server in the web root. If these files are valuable to you then park them somewhere else. Now to test the clone, on the development server, as root:

/usr/bin/rsync -azq --delete --exclude=**/html/var/cache --exclude=**/html/var/tmp --exclude=**/html/var/session -e "ssh -i /root/rsync/mirror-rsync-key" rsync@liveserver.com:/var/www/ /var/www/

This should now copy lots of files, depending on your shop size and connection speed. You may want to let this run through, in the first instance. If all has gone well with the public/private keys then you should not be prompted for a password.

Still as root, on the development server type ‘crontab -e’ and add a new entry for the rsync. You can set the rsync to run as often as needed. In this example that is every half hour:

*/30 * * * * /usr/bin/rsync -azq --delete --exclude=**/html/var/cache --exclude=**/html/var/tmp --exclude=**/html/var/session -e "ssh -i /root/rsync/mirror-rsync-key" rsync@liveserver.com:/var/www/ /var/www/

Note that selected files are not backed up with ‘rsync’ - these include the cache, session and tmp files.

Now the development server will have a copy of every file you want on the live server, automatically backed up.

What you may want to do now is to add a backup on rotation, so that a complete set of the backed up files get archived on a daily basis. You can run a simple cron job on your server to do this and create a big tarball monster. Alternatively you can use rsnapshot, to take a clever, incremental backup. On the development server:

yum install rsnapshot

Now edit vi /etc/rsnapshot.conf and add:

backup  /var/www/               localhost/

You may also want to review the backups that will be taken. The defaults should do you fine. You may also want to disable the logging:

#logfile        /var/log/rsnapshot

Now, as root, on the development server, add some crontab entries with ‘crontab -e’:

0 */4 * * *       /usr/local/bin/rsnapshot hourly
30 23 * * *       /usr/local/bin/rsnapshot daily

Now the development server should have a mirror of all of the files on the live server, and have them all backed up, on rotation so that nothing ever need to get lost again through accidental deletion.

More information on retrieving backed up files:

http://www.rsnapshot.org/

Maintaining a database backup

To make your development server a live clone of the live server you will need the database as well as the files. It is up to you as to how often you make a mysql backup, end of day has suited businesses traditionally, you could make it hourly though.

On the live server, logged in as your regular user, test the line needed to backup the db:

mkdir /home/MAGENTOUSER/mysql_backup

mysqldump -h localhost -u MAGENTODBUSER -pPASSWORD MAGENTODB | gzip > /home/MAGENTOUSER/mysql_backup/latest.sql.gz

If that works then place it in the crontab to be done automatically. ‘crontab -e’ and add:

0 3 * * * /usr/bin/mysqldump -h localhost -u MAGENTODBUSER -pPASSWORD MAGENTODB | gzip > /home/MAGENTOUSER/mysql_backups/latest.sql.gz

That is for three in the morning. Getting this file back to the development server and up and running is the next challenge.

With rsync in the crontab mentioned earlier you can retrieve this file. Add a new entry to the crontab on the development server:

* 4 * * * /usr/bin/rsync -azq -e "ssh -i /root/rsync/mirror-rsync-key" rsync@liveserver.com:/home/MAGENTOUSER/mysql_backups/ /home/MAGENTOUSER/mysql_backups/

That should do the job at 4 in the morning, giving your live server a whole hour to get the mysqldump done - hopefully it will take seconds to minutes and not need the hour.

This database can now be read straight in to your development server. The full process is that you will have to drop the existing backup, import the new data, then set the URL‘s in the database to point to the development server, not the live server. In the ~/mysql_backups directory create a file to run before the import, this one creates the new database after dropping the old one. Call it ‘do_first.sql’:

drop database MAGENTODB;
create database MAGENTODB;

Create one to be called after the import. This one is custom to your shop and URL situation, call it ‘do_after.sql’, e.g:

UPDATE `MAGENTODB`.`core_config_data` SET `value` = 'http://livestore.com/' WHERE `core_config_data`.`value` = 'http://developmenturl.com/';

If you have lots of URL‘s, i.e. a multi-shop setup, then you will need more lines to correct the URL‘s to point to the development server, not the live. Same goes for the https entries.

Now create a script to uncompress the database and call the SQL scripts. Call it ‘update_mysql.sh’:

#!/bin/sh
DAY=`/bin/date +%Y%m%d`
cp /home/MAGENTOUSER/mysql_backups/latest.sql.gz /home/MAGENTOUSER/mysql_backups/$DAY.sql.gz
/usr/bin/gunzip /home/MAGENTOUSER/mysql_backups/latest.sql.gz
mysql -h localhost -u root -pMYSQLROOTPASS MAGENTODB < /home/MAGENTOUSER/mysql_backups/do_first.sql
mysql -h localhost -u root -pMYSQLROOTPASS MAGENTODB < /home/MAGENTOUSER/mysql_backups/latest.sql
mysql -h localhost -u root -pMYSQLROOTPASS MAGENTODB < /home/MAGENTOUSER/mysql_backups/do_after.sql
rm /home/MAGENTOUSER/mysql_backups/latest.sql

chmod 700 the script to make it executable. Give it a test. If it all works then add a crontab job:

* 5 * * * /bin/sh /home/MAGENTOUSER/mysql_backups/update_mysql.sh

The complete replication should now be done at 5 a.m. with the database file from the live server archived by date on the development server. Check that the development server actually works. You will need to have development URL‘s in the httpd.conf file that match up to what you have in the mysql database. To test these you may need to create/edit /etc/hosts files on your development workstations. Once setup though, you have a fully working backup server with no more intervention required - unless you need to restore a database or a rsnapshot of the file system.




 

Magento 2 GitHub Repository

Magento Job Board - Some sort of tag line goes here

Latest Posts| View all Jobs