Saturday, February 18, 2017

Getting kicked off Isbrae while on SSH connection? FIX.

This has come up a couple of times, but I'd never had the symptoms so I didn't know how to fix it:  You're logged into isbrae, or any other ssh sever for that matter.  You have something running, scrolling output up your screen, so that you can monitor it, but it's running for hours.  Days.  But then you get cut off with an error like Write failed: Broken Pipe.  Love it.

Hopefully you are running the job in the background and monitoring the output via a log file, using tail -f.  In that case, you only have to reconnect to start looking at the log file and watching your process again.  But what if you had been in an 'interactive' matlab session or something, or you weren't being savvy enough to run your process 'headless'?  your process gets cut off and is either terminated or becomes a 'zombie' process, still running but with no terminal to control it.  Might as well kill it…

How do you avoid this, though, you ask?  Well.  It's about the server knowing there's someone still paying attention on the client side (you).  To do this, you need to have your (ssh) client send a little 'ping' or 'heartbeat' to make the server remember that you're there.  Usually when you are typing this is done automatically, but when you're looking at a bunch of output scrolling up the screen, there are no key presses to be logged, so the server may think you're not paying attention…  so you need to give it a little something extra.  Credit to google, Stack Exchange, and askubuntu.com for the answer- which comes fundamentally from here.

The gist of it is this.  You need to add a config file to your ssh setup.  To do this, open a text editor of your choice, and create a new file in the directory ~/.ssh (yes there's a leading dot) called 'config'.

In that file, place the following contents:

Host *
  ServerAliveInterval 30
  ServerAliveCountMax 5

What you are saying is essentially:  

"ok, do this for any host to which I connect:  Every 30 seconds, send a packet saying 'hey, I'm here, are you there?'.  If you send out more than 5 of these without hearing back from the server, the line's been cut so hang up"

One of the effects of this is that every 30 seconds your ssh sends something to the host to say 'hey' and
the host gets it…  and doesn't hang up on you.  Try it if you're having this problem!

Thanks for reading!

--bob