Saturday, February 18, 2017

Getting kicked off Isbrae while on SSH connection? FIX.

This has come up a couple of times, but I'd never had the symptoms so I didn't know how to fix it:  You're logged into isbrae, or any other ssh sever for that matter.  You have something running, scrolling output up your screen, so that you can monitor it, but it's running for hours.  Days.  But then you get cut off with an error like Write failed: Broken Pipe.  Love it.

Hopefully you are running the job in the background and monitoring the output via a log file, using tail -f.  In that case, you only have to reconnect to start looking at the log file and watching your process again.  But what if you had been in an 'interactive' matlab session or something, or you weren't being savvy enough to run your process 'headless'?  your process gets cut off and is either terminated or becomes a 'zombie' process, still running but with no terminal to control it.  Might as well kill it…

How do you avoid this, though, you ask?  Well.  It's about the server knowing there's someone still paying attention on the client side (you).  To do this, you need to have your (ssh) client send a little 'ping' or 'heartbeat' to make the server remember that you're there.  Usually when you are typing this is done automatically, but when you're looking at a bunch of output scrolling up the screen, there are no key presses to be logged, so the server may think you're not paying attention…  so you need to give it a little something extra.  Credit to google, Stack Exchange, and for the answer- which comes fundamentally from here.

The gist of it is this.  You need to add a config file to your ssh setup.  To do this, open a text editor of your choice, and create a new file in the directory ~/.ssh (yes there's a leading dot) called 'config'.

In that file, place the following contents:

Host *
  ServerAliveInterval 30
  ServerAliveCountMax 5

What you are saying is essentially:  

"ok, do this for any host to which I connect:  Every 30 seconds, send a packet saying 'hey, I'm here, are you there?'.  If you send out more than 5 of these without hearing back from the server, the line's been cut so hang up"

One of the effects of this is that every 30 seconds your ssh sends something to the host to say 'hey' and
the host gets it…  and doesn't hang up on you.  Try it if you're having this problem!

Thanks for reading!


Thursday, December 3, 2015

Where stuff is on isbrae, and general guidelines

So, isbrae is our shared server machine.  Among other things it has a large disk array on which we can store data, and is backed up each night to an off-site (over in the Math department) machine (called epica).  If you're logged into isbrae, you probably have access to some of these shared files.

As a user of shared space, you should be aware of and adhere to a few guidelines on where things should go.

One principle I request is do not mix data and code.  Data is large, after we collect it does not change, and thus does not take person-time to generate.  Code, on the other hand, takes lots of person time but is small, therefore easy to back up in multiple places (see my post on using unison to do backups- unison is awesome).

Another possibly obvious one is do not store intermediate files (ie partially processed data) unless you have to.  The way to avoid this is to do your data processing programmatically- I generally start with raw data, and in one or more matlab/python/shell scripts, get to publication-ready figures.  This not only avoids clutter and wasted disk space, but also puts the documentation of how you got from data to figure right in the code, where you can easily find it later.  Another way to put this is to avoid data clutter.

If you want to hack around and don't want to have to worry about cleaning up- ie you need some temporary space, use /bigtmp/.  It will be cleaned up automatically for you each week.  More on that in a minute.

Ok- that's it for general guidelines- here's where to keep things:
1) your home directory: you can of course store things in your home directory- but this is by default not readable by anyone else.  I don't have any restrictions on what you put in there, ie you can put your own personal stuff in there, but please nothing illegal, and you should know that Ed and I can use administrative privileges to look anywhere.  Note that your home directory is on a disk with finite space, so if you have a lot of stuff, please don't store it there.

2) /bigtmp/: This is a space that is great for passing files back and forth between users (see my posts on moving things using the command line and using a gui for this), or for using as scratch space- I often set my output to /bigtmp/ when I am developing a workflow, so that the stuff I produce won't hang around.  THIS GETS WIPED EVERY WEEKEND- that's the point of the 'tmp' part.

3) /code/: everyone can have space to store code in the /code directory (I may have to make you a directory there, just ask).  This is a great place to put code especially that you might want to share with others- just direct them to your code directory!

4) /data1, /data2, /data3, /data4: these are the main (really big) drive array, divided into 4 partitions.  If you want to put something here, talk to me about where it should go and I can give guidelines.  In general right now, /data1 and /data2 are in active use for multiple projects and still have space available.  /data3 is all SIRAL data, and is completely full (a problem I need to solve), and /data4 is currently unused.

5) /doc: this is a place where you can put manuals, or other types of documentation that might be of interest to the general user base.  Let me know if you want to put something here. 

Wednesday, August 6, 2014

Passwordless SSH to isbrae from your mac

So, you know how to ssh and scp things with the best of them, but you're tired of having to always put in your password.  Or maybe you want to set up an automated backup or some such thing, and can't always be there to enter the password...

You want to set up passwordless ssh for your machine.  I'll take you through the steps to do it, but first a bit of theory.

When your computer (say, your mac laptop) initiates an ssh connection to isbrae, it identifies itself, and identifies the isbrae user you intend to use (see the post logging in to isbrae for how you specify the username).  It then has a few ways to attempt authentication- the one it defaults to if you haven't set anything up is to ask for your password.  But first it checks to see if your user has set up an 'authorized keys' file.  If so, it uses information in that file to determine whether your computer/user combination is authorized- if it is, no need for a password.  We're going to set this up, through a process known as public key encryption, which essentially generates two keys- a 'private' key that only you and your computer have and have access to, and a 'public' key that you can install on any remote computer that uses this protocol- think of it as creating a personal key and padlock, you wouldn't want to let the key loose, but no worries if someone has your padlock...  One key (no pun intended) part of this is that it is virtually impossible to hack your passphrase using only the public key- google 'public key encryption' for more detail. 

So- you need to do 3 things:
1) Generate your public and private keys
2) Place each key in its appropriate place
3) Log in- with newer versions of OSX this prompts the mac 'keychain' utility to store the key passphrase for you, which allows you to skip the password. 

Ok, on to 1):

in your terminal, start by issuing the command 'ssh-keygen':

ll-phoenix:~ bo$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/Users/bo/.ssh/id_rsa): ./id_rsa
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in ./id_rsa.
Your public key has been saved in ./
The key fingerprint is:
The key's randomart image is:
+--[ RSA 2048]----+
|      o..oo+.    |
|       Ooo oo    |
|    . + E + o    |
|     + o o +     |
|    o o S   .    |
|     + . .       |
|    .            |
|                 |
|                 |
ll-phoenix:~ bo$

As you may see from the exchange, you need to give the key a filename, and then enter a passphrase.  This can be anything, but I'd suggest using one you know and remember well, so as not to have to remember too many things!  For the filename, it will default to .ssh/id_rsa if you don't override.  Notice that I did indeed override, primarily because I already have a file in .ssh called id_rsa and I didn't want to overwrite it.  You will now see two new files:

ll-phoenix:~ bo$ ls -l id_rsa*
-rw-------  1 bo  Glaciology  1766 Aug  6 08:50 id_rsa
-rw-r--r--  1 bo  Glaciology   418 Aug  6 08:50
ll-phoenix:~ bo$

The one with the .pub suffix is, perhaps obviously, your public key.  For the heck of it, lets see what it looks like:

ll-phoenix:~ bo$ head
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDbs68yGNOHoJvt9v3XZ+cCMO1ea7kfh41MTGaqw9KeCGqt+H6gXO2NdMSpF67FsplSbCl5rD/PAI9zNU3IyNKf4uUFzDoaeLliRkgljHmI+Cd9Q5sWliBYxcyZ2L/1+gBGGI3YI+Saout2WZGNEvA3Elo4zvlv59QvfqQSFMHpArOMU6j7ufkenClcFeDvqMPz9zuAoQeTCwuIQH6s91QUA5xmG7EQ19hkJn7B5K3IKHjMh0tm9/FWdF4uSweYslbckHM8ilHJDOAjqijjwNcQdJNknU425WcytGItYF+SQ9ZEapqnMl5edlm3s+clUn7Im9RADRd22ylvhBA+VB6j
ll-phoenix:~ bo$ 

Just some 'random' text, but it is important.  

Now, lets move to step 2), putting the files in the right place.  You will have a directory (a hidden one) called '.ssh' in both your local account and your account on isbrae.  Your private key goes in your local .ssh directory, and your public key goes in your remote (on isbrae) .ssh directory:

ll-phoenix:~ bo$ mv id_rsa .ssh/

ll-phoenix:~ bo$ scp bo@isbrae:~/

Then log in to isbrae and make the move:
ll-phoenix:~ bo$ ssh -X isbrae
Last login: Tue Aug  5 15:36:04 2014 from
[bo@isbrae ~]$ mv .ssh/authorized_keys

Note that in addition to moving into the .ssh directory, I have also renamed it 'authorized_keys'.  This is important.  You can have multiple authorized keys, and they all go into the same file.  If this is your first setup, then use the command I demonstrate above, but if you already have one machine set up for passwordless ssh, you will have to copy the text of the new key into the existing 'authorized_keys' file.  Exactly how to do this is left as an exercise for the reader- there are a hundred ways to skin that cat (last word could be a clue). 

Step 3) close everything out, quit your terminal, and then restart the terminal, and ssh as you would normally do.  If all goes well, you should be prompted with a dialog box to save your password for the key, and say yes.  Then, no more password entry!

If you set up passwordless ssh into isbrae, this means that the security of isbrae is only as good as the security on your local machine- ie if someone gets into your account on your local machine, they will get into isbrae.  For this reason, if you set up passwordless ssh, you NEED to secure your machine- no default login on startup, it should require a username and password, and also on waking up from sleep or screensaver you need to have it require  password.  This is good practice anyway, but extra important given the front door you have just opened to isbrae. 

Tuesday, July 16, 2013

Digital lab book using pmwiki on isbrae

So, documentation is of course very important.  In our work, we don't always have a lab book, as much of our stuff happens at the computer....

But we can use that computer for logging what we do- it's really useful.  To that end, I've copied Dan Breton's workflow using pmwiki, an open source package for making wiki pages.  This makes it pretty straightforward to just use the web page for a notebook.  

To get yourself started with pmwiki on isbrae:

First you need an account.  Once you've done that, you should have a public_html folder under your home directory:

[bo@isbrae public_html]$ pwd
[bo@isbrae public_html]$ 

If you haven't already got one, make that first.  Then, copy the wiki-skeleton set of files that Dan set up in his public_html directory:

[bo@isbrae ~]$ cp -r /home/dbreton/public_html/wiki_skeleton ~/public_html/hawley_wiki

(substitute your favorite name for 'hawley_wiki' above)

Now, you need to momentarily make a quick opening for pmwiki to access the directory, so exectute

[bo@isbrae public_html]$ chmod 2777 hawley_wiki/

And then open your favorite browser on your desktop/laptop/whatever computer and navigate to:<username>/<wiki_name>/pmwiki.php

(where you replace <username> and <wiki_name> with your username and the wikiname you chose earlier, respectively).

Now that pmwiki has been initialized, you can close up the permissions again:

[bo@isbrae public_html]$ chmod 755 hawley_wiki/

Your wiki page is now active on the link to which you just navigated.  

And then you can start wiki'ing!  See Dan's wiki at for more documentation on how to work with pmwiki and the special things Dan's set up to help make our lives easier.  Also see the pmwiki site and all the support pages.  It's not that hard once you get it up and running!

Hope this helps!

Monday, March 11, 2013

Backups, Backups! Using unison.

I've had a few conversations with folks about backups.  In particular, how you REALLY NEED TO HAVE THEM.  You know how in climbing, people say "never climb unroped higher than you are willing to fall"...  Well, the corollary in computing is "never work longer between backups than you'd want to repeat".  For me that's no more than a day, tops.  Sometimes an hour.

Even backups that are provided by your institution are not completely safe, and it's a good idea to roll your own, that way you will know they are working.  This is illustrated by a very short story from grad school...

We were all working on a Solaris cluster, with several of us having linux workstations as well.  All of our data was centralized on a RAID disk array.  It was RAID, so it was safe, right?  Anyway, the department had a tape backup that ran every night, so everything was cool.  Weeeellll.....

One morning we came in to find that a) the amazing RAID array had suffered a catastrophic failure, and b) the tape backup hadn't been working for months.  Our IT person resigned soon after this, but many people lost a great deal of work.  Fortunately for me, I'd been making backups of my stuff to multiple locations, none of which were affected by this double-failure.  And I'd convinced many of the ice group to do the same, so many of them were unaffected.

I currently use two tools for backups; rsync and unison.  In addition, I have several scripts configured to automate parts of the process.  All of this can be tunneled through SSH, so in order to automate it, you need to configure passwordless ssh.  But no need to go there yet.  We'll start with using unison, as that gives great bang for the buck.  Maybe if there's interest I'll post about rsync another time.

First, it's important to note that my computing environment is simultaneously more complex and simpler than that of most users.  It's more complex because I have several computers on which I actively work, but it's more simple because I have ALL THE SAME FILES on every one of them.  How?  By using unison.

So- I use a hub-and-spoke topology for my primary working computers, and use unison to synchronize files between them, and a tree topology for backing up.

Sorry for the long preamble.  The first thing to do is choose what you want to back up.  Do not waste your backup space (we're going to make multiple copies) filling it with your music collection, your movies, pictures of your dog, or other things that you can easily recover from elsewhere (this includes large datasets that you can re-download from online archives).  SO that means don't just back up your home directory.  Choose.

What's most important are things that take your time, not computer time, so things like code, papers, presentations, but not things like model output that could be regenerated easily.  You will find that this makes your backup take a lot less space than you'd otherwise think.

When you know what you want to back up, install unison on your machine, and start configuring.  Here's the text of my config file (stored in ~/.unison/hawleymbp.prf); and I'll annotate to show what are the important points:

# this is the contents of the file .unison/hawleymbp.prf (and
#  others that I have modeled after it):

# the text-only interface (old skool)


# trying to get rid of properties issues

  perms = 0
  rsrc = false
# where unison is on the remote server (it's in a nonstandard 
# location on firn; do not use this on isbrae)
#  servercmd=/opt/local/bin/unison

This is the most important piece of the configuration; where you are syncronizing!  The first root should be your local machine, and the second should be where you want to sync to; isbrae is a good choice.  Note the syntax of the second root; that is what you need to tunnel unison through ssh.  

# Roots of the synchronization
   root = /Users/bo/
   root = ssh://

Now the directories within the roots you've just listed, that you want to sync.  So these are all directories just below my home folder.  
# Paths to synchronize 
     path = local 
     path = spri_local 
     path = dartmouth_local 
     path = STUFF 
     path = Desktop 
     path = bin

If you have anything big and easily replaceable in your directories that you're interested in backing up, well, that's a tactical error in directory construction, but you can use the ignore directive to keep them from being synchronized (I also use the ignore to keep from sync'ing the auto-backup file that emacs creates)
# paths and names to ignore:

     ignore = Name *~
     ignore = Name .*~
     ignore = Name .DS_Store
#     ignore = Name SCENARIO
#     ignore = Name data/ICESAT
#     ignore = Name data/ASIRAS

Use these last options with caution, as they can get you in trouble by "automatically" resolving conflicting file changes (if you changed a file in both places).
# Stuff for speed of use.
#      prefer = ssh://
#      batch = true
      auto = true

And that's that.  Place the file in a directory named ~/.unison/ (in your home folder- note that the leading dot makes the directory invisible unless you use "ls -a" to see it).  The file should have a name that you don't mind typing (in this example mine is hawleymbp.prf), and the extension ".prf".

Then, at the command line, type:

HawleyMBP:~ bo$ unison -ui text hawleymbp

The first time you run it, it will mention that it's got to check through everything to make an index file, and that's just fine.  After that, you'll see it scroll through a lot of things as it scans, and then it will present you with something that looks like:

Reconciling changes

local          firn.dart...       
deleted  ---->            Desktop/McMurdoStationGuide.pdf  
new file ---->            Desktop/hawleymbp.prf  
changed  ---->            Desktop/progect_files/  
changed  ---->            dartmouth_local/papers/asiras_accum_svalbard/asiras_accum_svalbard.aux  
changed  ---->            dartmouth_local/papers/asiras_accum_svalbard/asiras_accum_svalbard.fff  
changed  ---->            dartmouth_local/papers/asiras_accum_svalbard/asiras_accum_svalbard.lof  
changed  ---->            dartmouth_local/papers/asiras_accum_svalbard/asiras_accum_svalbard.log  
changed  ---->            dartmouth_local/papers/asiras_accum_svalbard/asiras_accum_svalbard.pdf  
changed  ---->            dartmouth_local/papers/asiras_accum_svalbard/asiras_accum_svalbard.tex  

Proceed with propagating updates? [] 

We're seeing that things have only changed on the local machine (which is what I'd expect, as I've been working on this machine only recently).  If anything had changed on, we would see arrows going the other direction.  If there was a conflict, arrows would go in both directions, and you would be prompted to choose the copy to propagate.  

At this point, you have the option of hitting ctrl-c and leaving everything as it was.  Or you can type "y" and the changes will be propagated.  Perhaps obviously it's a good idea to create a simple test directory with a couple of files to practice with, before you do this with your own stuff.  

Once you start the process of propagating the changes, unison will keep you posted as to what it's doing and how long it's likely to continue doing it:

[END] Updating file dartmouth_local/papers/asiras_accum_svalbard/asiras_accum_svalbard_preprint_style.pdf
 81%  00:59 ETA

And when it finishes, it'll let you know how things went (this way you don't have to sit there and watch it every time):

UNISON 2.27.57 finished propagating changes at 15:29:53 on 11 Mar 2013

Saving synchronizer state
Synchronization complete  (17 items transferred, 0 skipped, 0 failures)
HawleyMBP:~ bo$ 

So here we see that all 17 items transferred successfully, no failures.  It would skip a file if there were a conflict and you'd chosen the "batch" option (which doesn't ask you if you want to proceed).  Note that unison makes all file copies atomic which means that it keeps the original file in a third place until the updated copy finishes transferring; if there's an error in the transfer, it puts the original back and notes that no copy happened at all, so the file will be updated the next time unison is run.  

Ok, that's enough for now.  Hopefully that got you synchronized.  My advice is to play around with a directory that you create solely for learning how this works to start with, find out how you can trip yourself up by modifying a file in two different places at once, and how to get out of it.  And then get it working on your important files.  Note that all you have to do is get it up onto isbrae, and isbrae will propagate it to several other backup locations- and as a former boss once told me, you need 3 backups:  one because you're stupid, one because equipment gets #%$*&'d up, and one, well, just because.  

And if it seems like this whole process will take too long to set up, consider how much time I just spent typing all of this out!  And be assured that once you get it configured, your backups will be so fast and easy it'll just be a part of your daily workflow.

Happy sync'ing!

Wednesday, May 23, 2012

Permissions on Isbrae

So, another thing that always comes back to haunt people (myself included) is permissions.  You copied something to /bigtmp (see other posts on that topic), and told someone where to find it, but they came back and said they couldn't copy it, read it, or whatever.  The problem is probably permissions.

Unix (linux and Mac os X are unix variants) file permissions work on three levels:  there are separate permissions for the owner of the file, the group to whom the owner belongs, and the entire world, which means everyone else on the system.  The owner of a file is the one who sets these permissions.

To see the permissions on a file or directory, you need to add the "-l" flag to the common "ls" command for listing files (for "long listing"):

[bo@isbrae ~]$ ls -l /bigtmp/
total 193860
-rw------- 1 bo    glaciology   2720463 May 23 13:26 katie_bos_radar.mat
-rw-r--r-- 1 bo    glaciology      1573 May 23 13:26 katie_density_core.txt
-rw------- 1 bo    glaciology      6385 May 23 13:14 summit_optimization.m

drwxr-xr-x 2 bo    glaciology      4096 May 23 14:30 temp_dir
-rwxrwxrwx 1 bo    glaciology         0 May 23 14:32 all_permissions_file

Here we see a few files I copied to /bigtmp for Laura to check out.  Each line is for a file.  The first thing you see in the line is 10 characters, which contain the file type and permissions.  The first character determines file type- it's a '-' for a regular file, and 'd' for a directory.  There are other characters you might see in that spot, but don't worry about that for now.  Next there are three sets of three characters, indicating the file permissions for, again, the owner, group, and world.  Directly after these permissions characters the owner is listed (in this case me, 'bo'), the group, the file size in bytes, date and time last modified, and finally the file name.

So lets look at what's here.  I've created an empty file (file size 0 bytes) called "all_permissions_file" to illustrate the permissions possibilities.  Each of the three groups gets to read, write, or execute (rwx) this file.  In general, it's not a good idea to give anyone write permissions on a file unless you are sure you want to, because it'd make it easy for people to accidentally delete your work.  So note that for all the other files and directories in this example, the owner is the only one with write permission.

Now take a look at the directory "temp_dir".  All users have read and execute permission, but only the owner has write permission.  This means that only the owner can put things in this directory, and only the owner can rename or delete the directory.  Execute permissions in this case are important, because though others have read permission, to list the contents of a directory requires execute permission.

Next look at a file that is all set to be shared, "katie_density_core.txt".  This file has read permissions for everyone, and write permissions for the owner.  So now Laura can copy this file to her space, giving her ownership over it and thus control.  

There are two files, though, that will thwart any efforts to view, copy, move, delete, rename, or otherwise mess with them, "katie_bos_radar.mat" and "summit_optimization.m".  Only the owner has any permissions over them.  Note that as owner you can even withdraw these permissions from yourself, though it would make the file less useful (one exception is that if you want to keep yourself from accidentally deleting a file you could take away your own write permissions).  But now I, as owner, need to change the permissions so that Laura can copy these files too.  To do this you use the command "chmod" (think "change mode").  There's a lot to this command, and I'm only going to give you the simplest way to use it- give all users read permissions, using the "a+r" flag:

[bo@isbrae ~]$ chmod a+r /bigtmp/katie_bos_radar.mat 
[bo@isbrae ~]$ chmod a+r /bigtmp/summit_optimization.m 

Now look to see that everything is as it should be, 

[bo@isbrae ~]$ ls -l /bigtmp/
total 193860
-rw-r--r-- 1 bo    glaciology   2720463 May 23 13:26 katie_bos_radar.mat
-rw-r--r-- 1 bo    glaciology      1573 May 23 13:26 katie_density_core.txt
-rw-r--r-- 1 bo    glaciology      6385 May 23 13:14 summit_optimization.m

drwxr-xr-x 2 bo    glaciology      4096 May 23 14:30 temp_dir

-rwxrwxrwx 1 bo    glaciology         0 May 23 14:32 all_permissions_file

[bo@isbrae ~]$ 

And you can now safely tell your colleague that the file is available.  Note that if there is a problem with permissions, only the owner can fix it, although as the "root" user, a system administrator can also come in and fix things as well.

Hope this helps!

Wednesday, May 16, 2012

Logging into isbrae

So, someone's told you to find something on isbrae.  Huh?

isbrae is our group linux server- it's got many CPU cores, and more importantly, many Tb of data storage space.  So it's a good place to store our data, and if you want to, since it runs matlab, python, and has most of the popular compilers and many graphical visualization packages you can do all of your computing there as well. 

But first you have to get there.  From a mac, open the "terminal" application, which is under Applications -> Utilities.  From Windows, you can use a freeware program called "Putty". 

Once you open the terminal, you can type your command to ssh or "secure shell" into isbrae:

HawleyMBP:~ bo$ ssh
Last login: Tue May 15 09:57:34 2012 from
[bo@isbrae ~]$

Note that in this case, my username on isbrae is 'bo' so you will need to replace this command with "ssh"  You will be asked for your password, which you will have set when you got your account. 

If you want to run programs on isbrae that contain graphical output (like matlab running in graphics mode, for example), you need to "pipe" the graphical display from isbrae to your local machine.  This can be done easily by using the "-X" switch on ssh:

HawleyMBP:~ bo$ ssh -X
Last login: Wed May 16 10:36:46 2012 from
[bo@isbrae ~]$

It's got to be a capital X, not lowercase, because lowercase means explicitly "do _not_ forward graphics". 

Note that you need to have "X-windows" installed on your local machine.  For macs, this comes with the developer tools and extras and may not have been installed by default, but should be on your install CD.  For windows, you will need an "X emulator", and will need more help than this blog post can provide...