So, you know how to ssh and scp things with the best of them, but you're tired of having to always put in your password. Or maybe you want to set up an automated backup or some such thing, and can't always be there to enter the password...
You want to set up passwordless ssh for your machine. I'll take you through the steps to do it, but first a bit of theory.
HOW IT WORKS:
When your computer (say, your mac laptop) initiates an ssh connection to isbrae, it identifies itself, and identifies the isbrae user you intend to use (see the post logging in to isbrae for how you specify the username). It then has a few ways to attempt authentication- the one it defaults to if you haven't set anything up is to ask for your password. But first it checks to see if your user has set up an 'authorized keys' file. If so, it uses information in that file to determine whether your computer/user combination is authorized- if it is, no need for a password. We're going to set this up, through a process known as public key encryption, which essentially generates two keys- a 'private' key that only you and your computer have and have access to, and a 'public' key that you can install on any remote computer that uses this protocol- think of it as creating a personal key and padlock, you wouldn't want to let the key loose, but no worries if someone has your padlock... One key (no pun intended) part of this is that it is virtually impossible to hack your passphrase using only the public key- google 'public key encryption' for more detail.
So- you need to do 3 things:
1) Generate your public and private keys
2) Place each key in its appropriate place
3) Log in- with newer versions of OSX this prompts the mac 'keychain' utility to store the key passphrase for you, which allows you to skip the password.
Ok, on to 1):
in your terminal, start by issuing the command 'ssh-keygen':
ll-phoenix:~ bo$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/Users/bo/.ssh/id_rsa): ./id_rsa
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in ./id_rsa.
Your public key has been saved in ./id_rsa.pub.
The key fingerprint is:
43:c9:2e:5a:12:5b:71:9c:d5:4c:ee:60:07:2e:b0:d9 bo@ll-phoenix.wireless.dartmouth.edu
The key's randomart image is:
+--[ RSA 2048]----+
| o..oo+. |
| Ooo oo |
| . + E + o |
| + o o + |
| o o S . |
| + . . |
| . |
| |
| |
+-----------------+
ll-phoenix:~ bo$
As you may see from the exchange, you need to give the key a filename, and then enter a passphrase. This can be anything, but I'd suggest using one you know and remember well, so as not to have to remember too many things! For the filename, it will default to .ssh/id_rsa if you don't override. Notice that I did indeed override, primarily because I already have a file in .ssh called id_rsa and I didn't want to overwrite it. You will now see two new files:
ll-phoenix:~ bo$ ls -l id_rsa*
-rw------- 1 bo Glaciology 1766 Aug 6 08:50 id_rsa
-rw-r--r-- 1 bo Glaciology 418 Aug 6 08:50 id_rsa.pub
ll-phoenix:~ bo$
The one with the .pub suffix is, perhaps obviously, your public key. For the heck of it, lets see what it looks like:
ll-phoenix:~ bo$ head id_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDbs68yGNOHoJvt9v3XZ+cCMO1ea7kfh41MTGaqw9KeCGqt+H6gXO2NdMSpF67FsplSbCl5rD/PAI9zNU3IyNKf4uUFzDoaeLliRkgljHmI+Cd9Q5sWliBYxcyZ2L/1+gBGGI3YI+Saout2WZGNEvA3Elo4zvlv59QvfqQSFMHpArOMU6j7ufkenClcFeDvqMPz9zuAoQeTCwuIQH6s91QUA5xmG7EQ19hkJn7B5K3IKHjMh0tm9/FWdF4uSweYslbckHM8ilHJDOAjqijjwNcQdJNknU425WcytGItYF+SQ9ZEapqnMl5edlm3s+clUn7Im9RADRd22ylvhBA+VB6j bo@ll-phoenix.wireless.dartmouth.edu
ll-phoenix:~ bo$
Just some 'random' text, but it is important.
Now, lets move to step 2), putting the files in the right place. You will have a directory (a hidden one) called '.ssh' in both your local account and your account on isbrae. Your private key goes in your local .ssh directory, and your public key goes in your remote (on isbrae) .ssh directory:
ll-phoenix:~ bo$ mv id_rsa .ssh/
ll-phoenix:~ bo$ scp id_rsa.pub bo@isbrae:~/
Then log in to isbrae and make the move:
ll-phoenix:~ bo$ ssh -X isbrae
Last login: Tue Aug 5 15:36:04 2014 from ll-phoenix.wireless.dartmouth.edu
[bo@isbrae ~]$ mv id_rsa.pub .ssh/authorized_keys
Note that in addition to moving id_rsa.pub into the .ssh directory, I have also renamed it 'authorized_keys'. This is important. You can have multiple authorized keys, and they all go into the same file. If this is your first setup, then use the command I demonstrate above, but if you already have one machine set up for passwordless ssh, you will have to copy the text of the new id_rsa.pub key into the existing 'authorized_keys' file. Exactly how to do this is left as an exercise for the reader- there are a hundred ways to skin that cat (last word could be a clue).
Step 3) close everything out, quit your terminal, and then restart the terminal, and ssh as you would normally do. If all goes well, you should be prompted with a dialog box to save your password for the key, and say yes. Then, no more password entry!
IMPORTANT- A FINAL WORD ON SECURITY:
If you set up passwordless ssh into isbrae, this means that the security of isbrae is only as good as the security on your local machine- ie if someone gets into your account on your local machine, they will get into isbrae. For this reason, if you set up passwordless ssh, you NEED to secure your machine- no default login on startup, it should require a username and password, and also on waking up from sleep or screensaver you need to have it require password. This is good practice anyway, but extra important given the front door you have just opened to isbrae.
Showing posts with label networking. Show all posts
Showing posts with label networking. Show all posts
Wednesday, August 6, 2014
Monday, March 11, 2013
Backups, Backups! Using unison.
I've had a few conversations with folks about backups. In particular, how you REALLY NEED TO HAVE THEM. You know how in climbing, people say "never climb unroped higher than you are willing to fall"... Well, the corollary in computing is "never work longer between backups than you'd want to repeat". For me that's no more than a day, tops. Sometimes an hour.
Even backups that are provided by your institution are not completely safe, and it's a good idea to roll your own, that way you will know they are working. This is illustrated by a very short story from grad school...
We were all working on a Solaris cluster, with several of us having linux workstations as well. All of our data was centralized on a RAID disk array. It was RAID, so it was safe, right? Anyway, the department had a tape backup that ran every night, so everything was cool. Weeeellll.....
One morning we came in to find that a) the amazing RAID array had suffered a catastrophic failure, and b) the tape backup hadn't been working for months. Our IT person resigned soon after this, but many people lost a great deal of work. Fortunately for me, I'd been making backups of my stuff to multiple locations, none of which were affected by this double-failure. And I'd convinced many of the ice group to do the same, so many of them were unaffected.
I currently use two tools for backups; rsync and unison. In addition, I have several scripts configured to automate parts of the process. All of this can be tunneled through SSH, so in order to automate it, you need to configure passwordless ssh. But no need to go there yet. We'll start with using unison, as that gives great bang for the buck. Maybe if there's interest I'll post about rsync another time.
First, it's important to note that my computing environment is simultaneously more complex and simpler than that of most users. It's more complex because I have several computers on which I actively work, but it's more simple because I have ALL THE SAME FILES on every one of them. How? By using unison.
So- I use a hub-and-spoke topology for my primary working computers, and use unison to synchronize files between them, and a tree topology for backing up.
LETS GET STARTED
Sorry for the long preamble. The first thing to do is choose what you want to back up. Do not waste your backup space (we're going to make multiple copies) filling it with your music collection, your movies, pictures of your dog, or other things that you can easily recover from elsewhere (this includes large datasets that you can re-download from online archives). SO that means don't just back up your home directory. Choose.
What's most important are things that take your time, not computer time, so things like code, papers, presentations, but not things like model output that could be regenerated easily. You will find that this makes your backup take a lot less space than you'd otherwise think.
When you know what you want to back up, install unison on your machine, and start configuring. Here's the text of my config file (stored in ~/.unison/hawleymbp.prf); and I'll annotate to show what are the important points:
# this is the contents of the file .unison/hawleymbp.prf (and
# others that I have modeled after it):
# the text-only interface (old skool)
ui=text
# trying to get rid of properties issues
perms = 0
rsrc = false
# where unison is on the remote server (it's in a nonstandard
# location on firn; do not use this on isbrae)
# servercmd=/opt/local/bin/unison
This is the most important piece of the configuration; where you are syncronizing! The first root should be your local machine, and the second should be where you want to sync to; isbrae is a good choice. Note the syntax of the second root; that is what you need to tunnel unison through ssh.
# Roots of the synchronization
root = /Users/bo/
root = ssh://bo@firn.dartmouth.edu//Users/bo/
Now the directories within the roots you've just listed, that you want to sync. So these are all directories just below my home folder.
# Paths to synchronize
path = local
Even backups that are provided by your institution are not completely safe, and it's a good idea to roll your own, that way you will know they are working. This is illustrated by a very short story from grad school...
We were all working on a Solaris cluster, with several of us having linux workstations as well. All of our data was centralized on a RAID disk array. It was RAID, so it was safe, right? Anyway, the department had a tape backup that ran every night, so everything was cool. Weeeellll.....
One morning we came in to find that a) the amazing RAID array had suffered a catastrophic failure, and b) the tape backup hadn't been working for months. Our IT person resigned soon after this, but many people lost a great deal of work. Fortunately for me, I'd been making backups of my stuff to multiple locations, none of which were affected by this double-failure. And I'd convinced many of the ice group to do the same, so many of them were unaffected.
I currently use two tools for backups; rsync and unison. In addition, I have several scripts configured to automate parts of the process. All of this can be tunneled through SSH, so in order to automate it, you need to configure passwordless ssh. But no need to go there yet. We'll start with using unison, as that gives great bang for the buck. Maybe if there's interest I'll post about rsync another time.
First, it's important to note that my computing environment is simultaneously more complex and simpler than that of most users. It's more complex because I have several computers on which I actively work, but it's more simple because I have ALL THE SAME FILES on every one of them. How? By using unison.
So- I use a hub-and-spoke topology for my primary working computers, and use unison to synchronize files between them, and a tree topology for backing up.
LETS GET STARTED
Sorry for the long preamble. The first thing to do is choose what you want to back up. Do not waste your backup space (we're going to make multiple copies) filling it with your music collection, your movies, pictures of your dog, or other things that you can easily recover from elsewhere (this includes large datasets that you can re-download from online archives). SO that means don't just back up your home directory. Choose.
What's most important are things that take your time, not computer time, so things like code, papers, presentations, but not things like model output that could be regenerated easily. You will find that this makes your backup take a lot less space than you'd otherwise think.
When you know what you want to back up, install unison on your machine, and start configuring. Here's the text of my config file (stored in ~/.unison/hawleymbp.prf); and I'll annotate to show what are the important points:
# this is the contents of the file .unison/hawleymbp.prf (and
# others that I have modeled after it):
# the text-only interface (old skool)
ui=text
# trying to get rid of properties issues
perms = 0
rsrc = false
# where unison is on the remote server (it's in a nonstandard
# location on firn; do not use this on isbrae)
# servercmd=/opt/local/bin/unison
This is the most important piece of the configuration; where you are syncronizing! The first root should be your local machine, and the second should be where you want to sync to; isbrae is a good choice. Note the syntax of the second root; that is what you need to tunnel unison through ssh.
# Roots of the synchronization
root = /Users/bo/
root = ssh://bo@firn.dartmouth.edu//Users/bo/
Now the directories within the roots you've just listed, that you want to sync. So these are all directories just below my home folder.
# Paths to synchronize
path = local
path = spri_local
path = dartmouth_local
path = STUFF
path = Desktop
path = bin
If you have anything big and easily replaceable in your directories that you're interested in backing up, well, that's a tactical error in directory construction, but you can use the ignore directive to keep them from being synchronized (I also use the ignore to keep from sync'ing the auto-backup file that emacs creates)
# paths and names to ignore:
ignore = Name *~
ignore = Name .*~
ignore = Name .DS_Store
# ignore = Name SCENARIO
# ignore = Name data/ICESAT
# ignore = Name data/ASIRAS
Use these last options with caution, as they can get you in trouble by "automatically" resolving conflicting file changes (if you changed a file in both places).
# Stuff for speed of use.
# prefer = ssh://bo@firn.dartmouth.edu//Users/bo/
# batch = true
auto = true
And that's that. Place the file in a directory named ~/.unison/ (in your home folder- note that the leading dot makes the directory invisible unless you use "ls -a" to see it). The file should have a name that you don't mind typing (in this example mine is hawleymbp.prf), and the extension ".prf".
Then, at the command line, type:
HawleyMBP:~ bo$ unison -ui text hawleymbp
The first time you run it, it will mention that it's got to check through everything to make an index file, and that's just fine. After that, you'll see it scroll through a lot of things as it scans, and then it will present you with something that looks like:
Reconciling changes
local firn.dart...
deleted ----> Desktop/McMurdoStationGuide.pdf
new file ----> Desktop/hawleymbp.prf
changed ----> Desktop/progect_files/GTD.org
changed ----> dartmouth_local/papers/asiras_accum_svalbard/asiras_accum_svalbard.aux
changed ----> dartmouth_local/papers/asiras_accum_svalbard/asiras_accum_svalbard.fff
changed ----> dartmouth_local/papers/asiras_accum_svalbard/asiras_accum_svalbard.lof
changed ----> dartmouth_local/papers/asiras_accum_svalbard/asiras_accum_svalbard.log
changed ----> dartmouth_local/papers/asiras_accum_svalbard/asiras_accum_svalbard.pdf
changed ----> dartmouth_local/papers/asiras_accum_svalbard/asiras_accum_svalbard.tex
Proceed with propagating updates? []
We're seeing that things have only changed on the local machine (which is what I'd expect, as I've been working on this machine only recently). If anything had changed on firn.dartmouth.edu, we would see arrows going the other direction. If there was a conflict, arrows would go in both directions, and you would be prompted to choose the copy to propagate.
At this point, you have the option of hitting ctrl-c and leaving everything as it was. Or you can type "y" and the changes will be propagated. Perhaps obviously it's a good idea to create a simple test directory with a couple of files to practice with, before you do this with your own stuff.
Once you start the process of propagating the changes, unison will keep you posted as to what it's doing and how long it's likely to continue doing it:
[END] Updating file dartmouth_local/papers/asiras_accum_svalbard/asiras_accum_svalbard_preprint_style.pdf
81% 00:59 ETA
And when it finishes, it'll let you know how things went (this way you don't have to sit there and watch it every time):
UNISON 2.27.57 finished propagating changes at 15:29:53 on 11 Mar 2013
Saving synchronizer state
Synchronization complete (17 items transferred, 0 skipped, 0 failures)
HawleyMBP:~ bo$
So here we see that all 17 items transferred successfully, no failures. It would skip a file if there were a conflict and you'd chosen the "batch" option (which doesn't ask you if you want to proceed). Note that unison makes all file copies atomic which means that it keeps the original file in a third place until the updated copy finishes transferring; if there's an error in the transfer, it puts the original back and notes that no copy happened at all, so the file will be updated the next time unison is run.
Ok, that's enough for now. Hopefully that got you synchronized. My advice is to play around with a directory that you create solely for learning how this works to start with, find out how you can trip yourself up by modifying a file in two different places at once, and how to get out of it. And then get it working on your important files. Note that all you have to do is get it up onto isbrae, and isbrae will propagate it to several other backup locations- and as a former boss once told me, you need 3 backups: one because you're stupid, one because equipment gets #%$*&'d up, and one, well, just because.
And if it seems like this whole process will take too long to set up, consider how much time I just spent typing all of this out! And be assured that once you get it configured, your backups will be so fast and easy it'll just be a part of your daily workflow.
Happy sync'ing!
Wednesday, May 16, 2012
Logging into isbrae
So, someone's told you to find something on isbrae. Huh?
isbrae is our group linux server- it's got many CPU cores, and more importantly, many Tb of data storage space. So it's a good place to store our data, and if you want to, since it runs matlab, python, and has most of the popular compilers and many graphical visualization packages you can do all of your computing there as well.
But first you have to get there. From a mac, open the "terminal" application, which is under Applications -> Utilities. From Windows, you can use a freeware program called "Putty".
Once you open the terminal, you can type your command to ssh or "secure shell" into isbrae:
HawleyMBP:~ bo$ ssh bo@isbrae.dartmouth.edu
Last login: Tue May 15 09:57:34 2012 from hawleymbp.kiewit.dartmouth.edu
[bo@isbrae ~]$
Note that in this case, my username on isbrae is 'bo' so you will need to replace this command with "ssh your-username@isbrae.dartmouth.edu" You will be asked for your password, which you will have set when you got your account.
If you want to run programs on isbrae that contain graphical output (like matlab running in graphics mode, for example), you need to "pipe" the graphical display from isbrae to your local machine. This can be done easily by using the "-X" switch on ssh:
HawleyMBP:~ bo$ ssh -X bo@isbrae.dartmouth.edu
Last login: Wed May 16 10:36:46 2012 from c-66-31-143-92.hsd1.nh.comcast.net
[bo@isbrae ~]$
It's got to be a capital X, not lowercase, because lowercase means explicitly "do _not_ forward graphics".
Note that you need to have "X-windows" installed on your local machine. For macs, this comes with the developer tools and extras and may not have been installed by default, but should be on your install CD. For windows, you will need an "X emulator", and will need more help than this blog post can provide...
isbrae is our group linux server- it's got many CPU cores, and more importantly, many Tb of data storage space. So it's a good place to store our data, and if you want to, since it runs matlab, python, and has most of the popular compilers and many graphical visualization packages you can do all of your computing there as well.
But first you have to get there. From a mac, open the "terminal" application, which is under Applications -> Utilities. From Windows, you can use a freeware program called "Putty".
Once you open the terminal, you can type your command to ssh or "secure shell" into isbrae:
HawleyMBP:~ bo$ ssh bo@isbrae.dartmouth.edu
Last login: Tue May 15 09:57:34 2012 from hawleymbp.kiewit.dartmouth.edu
[bo@isbrae ~]$
Note that in this case, my username on isbrae is 'bo' so you will need to replace this command with "ssh your-username@isbrae.dartmouth.edu" You will be asked for your password, which you will have set when you got your account.
If you want to run programs on isbrae that contain graphical output (like matlab running in graphics mode, for example), you need to "pipe" the graphical display from isbrae to your local machine. This can be done easily by using the "-X" switch on ssh:
HawleyMBP:~ bo$ ssh -X bo@isbrae.dartmouth.edu
Last login: Wed May 16 10:36:46 2012 from c-66-31-143-92.hsd1.nh.comcast.net
[bo@isbrae ~]$
It's got to be a capital X, not lowercase, because lowercase means explicitly "do _not_ forward graphics".
Note that you need to have "X-windows" installed on your local machine. For macs, this comes with the developer tools and extras and may not have been installed by default, but should be on your install CD. For windows, you will need an "X emulator", and will need more help than this blog post can provide...
Graphical SCP to and from /bigtmp/
So, although I prefer to use the command line, I realize that for some, pointing and clicking is an easier way to do things. It is for these folks that this post is written.
SCP is simply a protocol for copying files securely from one networked machine to another. The command line is to my mind the simplest way to execute this protocol, but there are other ways as well.
For those who wish to use a point-and-click interface, there's fugu for the mac:
http://rsug.itd.umich.edu/software/fugu/
or FileZilla for mac, windows, or linux:
http://filezilla-project.org/
Since I don't personally use either of these much (though I have used them occasionally in the past), I won't go into a tutorial here- but as they are both open source projects, google will surely help if you need guidance. Happy clicking!
SCP is simply a protocol for copying files securely from one networked machine to another. The command line is to my mind the simplest way to execute this protocol, but there are other ways as well.
For those who wish to use a point-and-click interface, there's fugu for the mac:
http://rsug.itd.umich.edu/software/fugu/
or FileZilla for mac, windows, or linux:
http://filezilla-project.org/
Since I don't personally use either of these much (though I have used them occasionally in the past), I won't go into a tutorial here- but as they are both open source projects, google will surely help if you need guidance. Happy clicking!
Tuesday, May 15, 2012
Moving things to and from /bigtmp/
/bigtmp/ is our temporary swap directory on Isbrae. There are lots of ways to get thigns to and from /bigtmp, depending on what platform you use, where you are (whether you are hard-wired to the Fairchild subnet), and your personal preference.
First off, take a look at bigtmp by logging into isbrae, to see what's there: Open a terminal (I'm on HawleyMBP, my macbook pro) and at your prompt:
HawleyMBP:~ bo$ ssh bo@isbrae
Last login: Mon May 14 17:38:03 2012 from hawleymbp.kiewit.dartmouth.edu
[bo@isbrae ~]$ ls /bigtmp/
[bo@isbrae ~]$
Looks like there's nothing there now- which is not surprising, since bigtmp is wiped clean every weekend (otherwise it wouldn't be very tmp, would it?).
So lets find a file we'd like to put up onto isbrae. I'll use a matlab file I created for Blythe:
HawleyMBP:~ bo$ ls -lh for_blythe.mat
-rw-r--r-- 1 bo glaciology 458M May 15 09:53 for_blythe.mat
HawleyMBP:~ bo$
Note that this file would never be possible to email, as it's 485Mb!
So, to copy is up to isbrae, I use "secure copy" or scp:
HawleyMBP:~ bo$ scp for_blythe.mat bo@isbrae:/bigtmp/
for_blythe.mat 100% 458MB 11.5MB/s 00:40
HawleyMBP:~ bo$
And then log in and check that the permissions are correct:
HawleyMBP:~ bo$ ssh isbrae
Last login: Tue May 15 09:57:20 2012 from hawleymbp.kiewit.dartmouth.edu
[bo@isbrae ~]$ ls -lh /bigtmp/
total 459M
-rw-r--r-- 1 bo glaciology 459M May 15 09:55 for_blythe.mat
[bo@isbrae ~]$
and those last two 'r's in the permissions line show that both the workgroup (glaciology) and in fact any user on isbrae has read permissions. So you can send your colleague the path, and he or she should be able to pick it up! Next time- doing it in another way!
First off, take a look at bigtmp by logging into isbrae, to see what's there: Open a terminal (I'm on HawleyMBP, my macbook pro) and at your prompt:
Last login: Mon May 14 17:38:03 2012 from hawleymbp.kiewit.dartmouth.edu
[bo@isbrae ~]$ ls /bigtmp/
[bo@isbrae ~]$
Looks like there's nothing there now- which is not surprising, since bigtmp is wiped clean every weekend (otherwise it wouldn't be very tmp, would it?).
So lets find a file we'd like to put up onto isbrae. I'll use a matlab file I created for Blythe:
-rw-r--r-- 1 bo glaciology 458M May 15 09:53 for_blythe.mat
HawleyMBP:~ bo$
Note that this file would never be possible to email, as it's 485Mb!
So, to copy is up to isbrae, I use "secure copy" or scp:
for_blythe.mat 100% 458MB 11.5MB/s 00:40
HawleyMBP:~ bo$
And then log in and check that the permissions are correct:
HawleyMBP:~ bo$ ssh isbrae
Last login: Tue May 15 09:57:20 2012 from hawleymbp.kiewit.dartmouth.edu
[bo@isbrae ~]$ ls -lh /bigtmp/
total 459M
-rw-r--r-- 1 bo glaciology 459M May 15 09:55 for_blythe.mat
[bo@isbrae ~]$
and those last two 'r's in the permissions line show that both the workgroup (glaciology) and in fact any user on isbrae has read permissions. So you can send your colleague the path, and he or she should be able to pick it up! Next time- doing it in another way!
Subscribe to:
Posts (Atom)