Monday, March 11, 2013

Backups, Backups! Using unison.

I've had a few conversations with folks about backups.  In particular, how you REALLY NEED TO HAVE THEM.  You know how in climbing, people say "never climb unroped higher than you are willing to fall"...  Well, the corollary in computing is "never work longer between backups than you'd want to repeat".  For me that's no more than a day, tops.  Sometimes an hour.

Even backups that are provided by your institution are not completely safe, and it's a good idea to roll your own, that way you will know they are working.  This is illustrated by a very short story from grad school...

We were all working on a Solaris cluster, with several of us having linux workstations as well.  All of our data was centralized on a RAID disk array.  It was RAID, so it was safe, right?  Anyway, the department had a tape backup that ran every night, so everything was cool.  Weeeellll.....

One morning we came in to find that a) the amazing RAID array had suffered a catastrophic failure, and b) the tape backup hadn't been working for months.  Our IT person resigned soon after this, but many people lost a great deal of work.  Fortunately for me, I'd been making backups of my stuff to multiple locations, none of which were affected by this double-failure.  And I'd convinced many of the ice group to do the same, so many of them were unaffected.

I currently use two tools for backups; rsync and unison.  In addition, I have several scripts configured to automate parts of the process.  All of this can be tunneled through SSH, so in order to automate it, you need to configure passwordless ssh.  But no need to go there yet.  We'll start with using unison, as that gives great bang for the buck.  Maybe if there's interest I'll post about rsync another time.

First, it's important to note that my computing environment is simultaneously more complex and simpler than that of most users.  It's more complex because I have several computers on which I actively work, but it's more simple because I have ALL THE SAME FILES on every one of them.  How?  By using unison.

So- I use a hub-and-spoke topology for my primary working computers, and use unison to synchronize files between them, and a tree topology for backing up.

LETS GET STARTED
Sorry for the long preamble.  The first thing to do is choose what you want to back up.  Do not waste your backup space (we're going to make multiple copies) filling it with your music collection, your movies, pictures of your dog, or other things that you can easily recover from elsewhere (this includes large datasets that you can re-download from online archives).  SO that means don't just back up your home directory.  Choose.

What's most important are things that take your time, not computer time, so things like code, papers, presentations, but not things like model output that could be regenerated easily.  You will find that this makes your backup take a lot less space than you'd otherwise think.

When you know what you want to back up, install unison on your machine, and start configuring.  Here's the text of my config file (stored in ~/.unison/hawleymbp.prf); and I'll annotate to show what are the important points:


# this is the contents of the file .unison/hawleymbp.prf (and
#  others that I have modeled after it):

# the text-only interface (old skool)

  ui=text

# trying to get rid of properties issues

  perms = 0
  rsrc = false
# where unison is on the remote server (it's in a nonstandard 
# location on firn; do not use this on isbrae)
#  servercmd=/opt/local/bin/unison

This is the most important piece of the configuration; where you are syncronizing!  The first root should be your local machine, and the second should be where you want to sync to; isbrae is a good choice.  Note the syntax of the second root; that is what you need to tunnel unison through ssh.  

# Roots of the synchronization
   root = /Users/bo/
   root = ssh://bo@firn.dartmouth.edu//Users/bo/

Now the directories within the roots you've just listed, that you want to sync.  So these are all directories just below my home folder.  
# Paths to synchronize 
     path = local 
     path = spri_local 
     path = dartmouth_local 
     path = STUFF 
     path = Desktop 
     path = bin

If you have anything big and easily replaceable in your directories that you're interested in backing up, well, that's a tactical error in directory construction, but you can use the ignore directive to keep them from being synchronized (I also use the ignore to keep from sync'ing the auto-backup file that emacs creates)
# paths and names to ignore:

     ignore = Name *~
     ignore = Name .*~
     ignore = Name .DS_Store
#     ignore = Name SCENARIO
#     ignore = Name data/ICESAT
#     ignore = Name data/ASIRAS

Use these last options with caution, as they can get you in trouble by "automatically" resolving conflicting file changes (if you changed a file in both places).
# Stuff for speed of use.
#      prefer = ssh://bo@firn.dartmouth.edu//Users/bo/
#      batch = true
      auto = true

And that's that.  Place the file in a directory named ~/.unison/ (in your home folder- note that the leading dot makes the directory invisible unless you use "ls -a" to see it).  The file should have a name that you don't mind typing (in this example mine is hawleymbp.prf), and the extension ".prf".

Then, at the command line, type:

HawleyMBP:~ bo$ unison -ui text hawleymbp

The first time you run it, it will mention that it's got to check through everything to make an index file, and that's just fine.  After that, you'll see it scroll through a lot of things as it scans, and then it will present you with something that looks like:

Reconciling changes

local          firn.dart...       
deleted  ---->            Desktop/McMurdoStationGuide.pdf  
new file ---->            Desktop/hawleymbp.prf  
changed  ---->            Desktop/progect_files/GTD.org  
changed  ---->            dartmouth_local/papers/asiras_accum_svalbard/asiras_accum_svalbard.aux  
changed  ---->            dartmouth_local/papers/asiras_accum_svalbard/asiras_accum_svalbard.fff  
changed  ---->            dartmouth_local/papers/asiras_accum_svalbard/asiras_accum_svalbard.lof  
changed  ---->            dartmouth_local/papers/asiras_accum_svalbard/asiras_accum_svalbard.log  
changed  ---->            dartmouth_local/papers/asiras_accum_svalbard/asiras_accum_svalbard.pdf  
changed  ---->            dartmouth_local/papers/asiras_accum_svalbard/asiras_accum_svalbard.tex  

Proceed with propagating updates? [] 

We're seeing that things have only changed on the local machine (which is what I'd expect, as I've been working on this machine only recently).  If anything had changed on firn.dartmouth.edu, we would see arrows going the other direction.  If there was a conflict, arrows would go in both directions, and you would be prompted to choose the copy to propagate.  

At this point, you have the option of hitting ctrl-c and leaving everything as it was.  Or you can type "y" and the changes will be propagated.  Perhaps obviously it's a good idea to create a simple test directory with a couple of files to practice with, before you do this with your own stuff.  

Once you start the process of propagating the changes, unison will keep you posted as to what it's doing and how long it's likely to continue doing it:

[END] Updating file dartmouth_local/papers/asiras_accum_svalbard/asiras_accum_svalbard_preprint_style.pdf
 81%  00:59 ETA

And when it finishes, it'll let you know how things went (this way you don't have to sit there and watch it every time):

UNISON 2.27.57 finished propagating changes at 15:29:53 on 11 Mar 2013


Saving synchronizer state
Synchronization complete  (17 items transferred, 0 skipped, 0 failures)
HawleyMBP:~ bo$ 

So here we see that all 17 items transferred successfully, no failures.  It would skip a file if there were a conflict and you'd chosen the "batch" option (which doesn't ask you if you want to proceed).  Note that unison makes all file copies atomic which means that it keeps the original file in a third place until the updated copy finishes transferring; if there's an error in the transfer, it puts the original back and notes that no copy happened at all, so the file will be updated the next time unison is run.  

Ok, that's enough for now.  Hopefully that got you synchronized.  My advice is to play around with a directory that you create solely for learning how this works to start with, find out how you can trip yourself up by modifying a file in two different places at once, and how to get out of it.  And then get it working on your important files.  Note that all you have to do is get it up onto isbrae, and isbrae will propagate it to several other backup locations- and as a former boss once told me, you need 3 backups:  one because you're stupid, one because equipment gets #%$*&'d up, and one, well, just because.  

And if it seems like this whole process will take too long to set up, consider how much time I just spent typing all of this out!  And be assured that once you get it configured, your backups will be so fast and easy it'll just be a part of your daily workflow.

Happy sync'ing!