Working on remote servers for bioinformatics analysis

You can easily access this web address using rclone. You could set up a new remote in your rclone config to point to http://sysg1.cs.yale.edu, but, since you will only be using this once, to get your data, it makes more sense to just specify the remote on the command line. This can be done by passing rclone the URL address via the --http-url option, and then, after that, telling it what protocol to use by adding :http: to the command. Here is what you would use to list the directories available at the sequencing center URL:

# here is the command
% rclone lsd --http-url http://sysg1.cs.yale.edu:3010/5lnO9bs3zfa8LOhESfsYfq3Dc/061719/ :http:

# and here is the output
          -1 1969-12-31 16:00:00        -1 sjg73_fqs
          -1 1969-12-31 16:00:00        -1 sjg73_supernova_fqs

Aha! There are two directories that might hold our sequencing data. I wonder what is in those diretories? The rclone tree command is the perfect way to drill down into those diretories and look at their contents:

% rclone tree --http-url http://sysg1.cs.yale.edu:3010/5lnO9bs3zfa8LOhESfsYfq3Dc/061719/ :http:
/
├── sjg73_fqs
│   ├── AW_F1
│   │   ├── AW_F1_S2_L001_I1_001.fastq.gz
│   │   ├── AW_F1_S2_L001_R1_001.fastq.gz
│   │   └── AW_F1_S2_L001_R2_001.fastq.gz
│   ├── AW_M1
│   │   ├── AW_M1_S3_L001_I1_001.fastq.gz
│   │   ├── AW_M1_S3_L001_R1_001.fastq.gz
│   │   └── AW_M1_S3_L001_R2_001.fastq.gz
│   └── ESP_A1
│       ├── ESP_A1_S1_L001_I1_001.fastq.gz
│       ├── ESP_A1_S1_L001_R1_001.fastq.gz
│       └── ESP_A1_S1_L001_R2_001.fastq.gz
└── sjg73_supernova_fqs
    ├── AW_F1
    │   ├── AW_F1_S2_L001_I1_001.fastq.gz
    │   ├── AW_F1_S2_L001_R1_001.fastq.gz
    │   └── AW_F1_S2_L001_R2_001.fastq.gz
    ├── AW_M1
    │   ├── AW_M1_S3_L001_I1_001.fastq.gz
    │   ├── AW_M1_S3_L001_R1_001.fastq.gz
    │   └── AW_M1_S3_L001_R2_001.fastq.gz
    └── ESP_A1
        ├── ESP_A1_S1_L001_I1_001.fastq.gz
        ├── ESP_A1_S1_L001_R1_001.fastq.gz
        └── ESP_A1_S1_L001_R2_001.fastq.gz

8 directories, 18 files

Whoa! That is pretty cool!. From this output we see that there are subdirectories named AW_F1 and AW_M1 that hold the files that we want. And, of course, the ESP_A1 samples must belong to someone else. It would be great if we could just download the files we wanted, excluding the ones in the ESP_A1 directories. It turns out that there is! rclone has an --exclude option to exclude paths that match certain patterns (see Section 7.2.4.3, above). We can experiment by giving rclone copy the --dry-run command to see which files will be transferred. If we don’t do any filtering, we see this when we try to dry-run copy the directories to our local directory Alewife/fastqs:

% rclone copy --dry-run --http-url http://sysg1.cs.yale.edu:3010/5lnO9bs3zfa8LOhESfsYfq3Dc/061719/ :http: Alewife/fastqs/
2019/07/11 10:33:43 NOTICE: sjg73_fqs/ESP_A1/ESP_A1_S1_L001_I1_001.fastq.gz: Not copying as --dry-run
2019/07/11 10:33:43 NOTICE: sjg73_fqs/ESP_A1/ESP_A1_S1_L001_R1_001.fastq.gz: Not copying as --dry-run
2019/07/11 10:33:43 NOTICE: sjg73_fqs/ESP_A1/ESP_A1_S1_L001_R2_001.fastq.gz: Not copying as --dry-run
2019/07/11 10:33:43 NOTICE: sjg73_supernova_fqs/AW_M1/AW_M1_S3_L001_I1_001.fastq.gz: Not copying as --dry-run
2019/07/11 10:33:43 NOTICE: sjg73_supernova_fqs/AW_M1/AW_M1_S3_L001_R1_001.fastq.gz: Not copying as --dry-run
2019/07/11 10:33:43 NOTICE: sjg73_supernova_fqs/AW_M1/AW_M1_S3_L001_R2_001.fastq.gz: Not copying as --dry-run
2019/07/11 10:33:43 NOTICE: sjg73_supernova_fqs/AW_F1/AW_F1_S2_L001_I1_001.fastq.gz: Not copying as --dry-run
2019/07/11 10:33:43 NOTICE: sjg73_supernova_fqs/AW_F1/AW_F1_S2_L001_R1_001.fastq.gz: Not copying as --dry-run
2019/07/11 10:33:43 NOTICE: sjg73_supernova_fqs/AW_F1/AW_F1_S2_L001_R2_001.fastq.gz: Not copying as --dry-run
2019/07/11 10:33:43 NOTICE: sjg73_supernova_fqs/ESP_A1/ESP_A1_S1_L001_I1_001.fastq.gz: Not copying as --dry-run
2019/07/11 10:33:43 NOTICE: sjg73_supernova_fqs/ESP_A1/ESP_A1_S1_L001_R1_001.fastq.gz: Not copying as --dry-run
2019/07/11 10:33:43 NOTICE: sjg73_supernova_fqs/ESP_A1/ESP_A1_S1_L001_R2_001.fastq.gz: Not copying as --dry-run
2019/07/11 10:33:43 NOTICE: sjg73_fqs/AW_F1/AW_F1_S2_L001_I1_001.fastq.gz: Not copying as --dry-run
2019/07/11 10:33:43 NOTICE: sjg73_fqs/AW_F1/AW_F1_S2_L001_R1_001.fastq.gz: Not copying as --dry-run
2019/07/11 10:33:43 NOTICE: sjg73_fqs/AW_F1/AW_F1_S2_L001_R2_001.fastq.gz: Not copying as --dry-run
2019/07/11 10:33:43 NOTICE: sjg73_fqs/AW_M1/AW_M1_S3_L001_I1_001.fastq.gz: Not copying as --dry-run
2019/07/11 10:33:43 NOTICE: sjg73_fqs/AW_M1/AW_M1_S3_L001_R1_001.fastq.gz: Not copying as --dry-run
2019/07/11 10:33:43 NOTICE: sjg73_fqs/AW_M1/AW_M1_S3_L001_R2_001.fastq.gz: Not copying as --dry-run

Since we do not want to copy the ESP_A1 files we see if we can exclude them:

% rclone copy --exclude */ESP_A1/* --dry-run --http-url http://sysg1.cs.yale.edu:3010/5lnO9bs3zfa8LOhESfsYfq3Dc/061719/ :http: Alewife/fastqs/
2019/07/11 10:37:22 NOTICE: sjg73_fqs/AW_F1/AW_F1_S2_L001_I1_001.fastq.gz: Not copying as --dry-run
2019/07/11 10:37:22 NOTICE: sjg73_fqs/AW_F1/AW_F1_S2_L001_R2_001.fastq.gz: Not copying as --dry-run
2019/07/11 10:37:22 NOTICE: sjg73_fqs/AW_F1/AW_F1_S2_L001_R1_001.fastq.gz: Not copying as --dry-run
2019/07/11 10:37:22 NOTICE: sjg73_fqs/AW_M1/AW_M1_S3_L001_I1_001.fastq.gz: Not copying as --dry-run
2019/07/11 10:37:22 NOTICE: sjg73_fqs/AW_M1/AW_M1_S3_L001_R2_001.fastq.gz: Not copying as --dry-run
2019/07/11 10:37:22 NOTICE: sjg73_fqs/AW_M1/AW_M1_S3_L001_R1_001.fastq.gz: Not copying as --dry-run
2019/07/11 10:37:22 NOTICE: sjg73_supernova_fqs/AW_F1/AW_F1_S2_L001_I1_001.fastq.gz: Not copying as --dry-run
2019/07/11 10:37:22 NOTICE: sjg73_supernova_fqs/AW_F1/AW_F1_S2_L001_R2_001.fastq.gz: Not copying as --dry-run
2019/07/11 10:37:22 NOTICE: sjg73_supernova_fqs/AW_F1/AW_F1_S2_L001_R1_001.fastq.gz: Not copying as --dry-run
2019/07/11 10:37:22 NOTICE: sjg73_supernova_fqs/AW_M1/AW_M1_S3_L001_I1_001.fastq.gz: Not copying as --dry-run
2019/07/11 10:37:22 NOTICE: sjg73_supernova_fqs/AW_M1/AW_M1_S3_L001_R1_001.fastq.gz: Not copying as --dry-run
2019/07/11 10:37:22 NOTICE: sjg73_supernova_fqs/AW_M1/AW_M1_S3_L001_R2_001.fastq.gz: Not copying as --dry-run

Booyah! That gets us just what we want. So, then we remove the --dry-run option, and maybe add -v -P to give us verbose output and progress information, and copy all of our files:

% rclone copy --exclude */ESP_A1/* -v -P  --http-url http://sysg1.cs.yale.edu:3010/5lnO9bs3zfa8LOhESfsYfq3Dc/061719/ :http: Alewife/fastqs/

7.3.1 Step 1: Set up your SSH config file to automatically apply port forwarding to your connections to your server

The first thing we will do is something that can especially useful if you have several servers you connect to, and you would like to have shorter names for accessing them: set up an “alias” to them in your SSH config file. Here we show how to set up such an alias to the SUMMIT cluster at Boulder.

To do so, edit your ~/.ssh/config file by adding the following lines to it:

Host summit
   HostName login11.rc.colorado.edu
   RemoteForward 52XXX localhost:52698

But change the XXX in 52XXX to three digits of your choice. This is important so you aren’t trying to access the same port as another user at the same time. For example, use the digits of your birthday, like 52122 if you were born on January 22, etc. This step connects port number 52XXX on the server to the local port 52698 on your laptop.

Note that the login node you choose could be login12.rc.colorado.edu rather than login11.rc.colorado.edu, or even some other number if you routinely login to a specific login node (for example to use tmux…see Section 7.4).

Once you have done that, you should logout of SUMMIT, and then, when you log back in in the future you can use

ssh [email protected]@summit

instead of, for example,

ssh [email protected]@login11.rc.colorado.edu

and, when it logs you in, it will enable the port forwarding.

7.3.2 Download SublimeText and add the RemoteSubl package to it

If you haven’t already downloaded SublimeText, you should do that and you should experiment with using it. It is an outstanding text editor. You can try it for free for a long time (indefinitely, it seems), but if you find you like it, then officially buying a license for it is is a good idea.

Once you have it installed, you must install the RemoteSubl plugin. This is done with SublimeText’s package control system. The steps to do this are:

1. Hit Shift-Command-P on a mac. On windows I think it is Shift-Windows-P. When you do this, sublime text will open a little text window on your screen. Type into that window: Package Control: Install Package. You don’t have to type very much of that phrase before you see the whole phrase as a possible completion below where you are typing. When you see the full phrase, use the arrow keys to select that phrase and hit Return. If you don’t see Package Control: Install Package you might see Install Package Control. Select that and install Package Control. After that, Package Control: Install Package should show up.
2. This should give you another text box. Start typing RemoteSubl into that window, until you see it in the possible completions. Select it from the completions (with your arrow keys, for example) and then hit return.

7.3.3 Download the rmate shell script to your server and put it on your PATH

On the server, we need the rmate command to send the contents of a file you wish to edit to the remote port that will get forwarded to the local port on your laptop. This command is a shell script that you can download with wget.

As we talked about in an earlier section, you should have a directory, ~/bin that is in your PATH, because you should have a line in your ~/.bashrc file like:

export PATH=$PATH:~/bin

If this is the case, then simply cd-ing to ~/bin on SUMMIT and running the commands:

wget https://raw.githubusercontent.com/aurora/rmate/master/rmate
chmod u+x rmate

should get you rmate and make it executable.

7.3.4 Using rmate

If everything has gone according to plan, then, if you have logged into summit using the summit alias (i.e. using ssh username@summit), then you should be able to edit any file on your server with:

rmate -p 52XXX path/to/file

where path/to/file represents the path to whatever file you want to edit, and 52XXX is actually the number of the remote port you are forwarding from, as set up in your ~/.ssh/config file. For example, in keeping with the example above, this would be:

rmate -p 52122 ~/.bashrc

to edit the .bashrc file on your remote server, or,

rmate -p 52122 my_script.R 

etc.

Note that, by default, rmate will open a new tab in the currently active SublimeText window. If you want to open the file in its own window, you can use:

rmate -p 52XXX -n path/to/file

The -n option forces the opening of a new SublimeText window.

If you want to edit multiple files at once, for example, all the files in a scripts folder on your remote machine, you can do like this:

rmate -p 52XXX -n scripts/*

If you have a lot of files in that folder, then keeping track of them in SublimeText can be made a lot easier if you choose View->Side Bar->Show Open Files from the SublimeText menu options. This will show the names of all the open files in a side bar to the left.

Because it can be a hassle to remember your remote port number and type it each time you use rmate, you can set up an alias in the ~/.bashrc file on your server by adding the line:

alias subl='rmate -p 52XXX ' 

Then, on the command line, you can simply use

subl path/to/file

# or

subl -n path/to/file

# or

subl -n path_to_dir/*.R

# etc.

and open remote files on your local SublimeText.

When you have edited the file, save the change in SublimeText and then close the window. It is that easy.

Note that if you lose connection to the server (for example you close your laptop and it goes to sleep), then you will get a message telling you that SublimeText is no longer connected to any files on the server, and you will have to reconnect them if you want to edit them.

7.4.1 An analogy for how tmux works

Imagine that the first time you log in to your remote server you also have the option of speaking on the phone to a super efficient IT guy who has a desk in the server room. This dude never takes a break, but sits at his desk 24/7. He probably has mustard stains on his dingy white T-shirt from eating ham sandwiches non-stop while he works super hard. This guy is Tmux.

When you first speak to this guy after logging in, you have to preface your commands with tmux (as in, “Hey Tmux!”). He is there to help you manage different terminal windows with different bash shells or processes going on in them. In fact, you can think of it this way: you can ask him to set up a terminal (i.e., like a monitor), right there on his desk, and then create a bunch of windows on that terminal for you—each one with its own bash shell—without having to do a separate login for each one. He has created all those windows, but you still get to use them. It is like he has a miracle-mirroring device that lets you operate the windows that are on the terminal he set up for you on his desk.

When you are done working on all those windows, you can tell Tmux that you want to detach from the special terminal he set up for you at the server. In response he says, “Cool!” and shuts down his miracle-mirroring device, so you no longer see those different windows. However, he does not shut down the terminal on his desk that he set up for you. That terminal stays on, and any of your processes happening on it keep chugging away…even after you logout from the server entirely, throw the lid down on your laptop, have drinks with your friends at Social, downtown, watch an episode of Parks and Rec, and then get a good night’s sleep.

All through the night, Tmux is munching ham sandwiches and keeping an eye on that terminal he set up for you. When you log back onto the server in the morning, you can say “Hey Tmux! I want to attach back to that terminal you set up for me.” He says, “No problem!”, turns his miracle-mirroring device back on, and in an instant you have all of the windows on that terminal back on your laptop with all the processes still running in them—in all the same working directories—just as you left it all (except that if you were running jobs in those windows, some of those jobs might already be done!).

Not only that, but, further, if you are working on the server when a local thunderstorm fries the motherboard on your laptop, you can get a new laptop, log back into the server and ask Tmux to reconnect you to that terminal and get back to all of those windows and jobs, etc. as if you didn’t get zapped. The same goes for the case of a backhoe operator accidentally digging up the fiber optic cable in your yard. Your network connection can go down completely. But, when you get it up and running again, you can say “Hey Tmux! Hook me up!” and he’ll say, “No problem!” and reconnect you to all those windows you had open on the server.

Finally, when you are done with all the windows and jobs on the terminal that Tmux set up for you, you can ask him to kill it, and he will shut it down, unplug it, and, straight out of Office Space, chuck it out the window. But he will gladly install a new one if you want to start another session with him.

That dude is super helpful!

7.4.2 First steps with tmux

The first thing you want to do to make sure Tmux is ready to help you is to simply type:

% which tmux

This should return something like:

/usr/bin/tmux

If, instead, you get a response like tmux: Command not found. then tmux is apparently not installed on your remote server, so you will have to install it yourself, or beg your sysadmin to do so (we will cover that in a later chapter). If you are working on the Summit supercomptuer in Colorado or on Hummingbird at UCSC, then tmux is installed already. (As of Feb 16, 2020, tmux was not installed on the Sedna cluster at the NWFSC, but I will request that it be installed.)

In the analogy, above, we talked about Tmux setting up a terminal in the server room. In tmux parlance, such a “terminal” is called a session. In order to be able to tell Tmux that you want to reconnect to a session, you will always want to name your sessions so you will request a new session with this syntax:

% tmux new -s froggies

You can think of the -s as being short for “session.” So it is basically a short way of saying, “Hey Tmux, give me a new session named froggies.” That creates a new session called froggies, and you can imagine we’ve named it that because we will use it for work on a frog genomics project.

The effect of this is like Tmux firing up a new terminal in his server room, making a window on it for you, starting a new bash shell in that window, and then giving you control of this new terminal. In other words, it is sort of like he has opened a new shell window on a terminal for you, and is letting you see and use it on your computer at the same time.

One very cool thing about this is that you just got a new bash shell without having to login with your password and two-factor authentication again. That much is cool in itself, but is only the beginning.

The new window that you get looks a little different. For one thing, it has a section, one line tall, that is green (by default) on the bottom. In our case, on the left side it gives the name of the session (in square brackets) and then the name of the current window within that session. On the right side you see the hostname (the name of the remote computer you are working on) in quotes, followed by the date and time. The contents in that green band will look something like:

[froggies] 0:bash*                                               "login11" 20:02 15-Feb-20

This little line of information is the sweet sauce that will let you find your way around all the new windows that tmux can spawn for you.

Now, imagine that we want to use this window in our froggies session, to look at some frog data we have. Accordingly, we might navigate to the directory where those data live and look at the data with head and less, etc. That is all great, until we realize that we also want to edit some scripts that we wrote for processing our froggy data. These scripts might be in a directory far removed from the data directory we are currently in, and we don’t really want to keep navigating back and forth between those two directories within a single bash shell. Clearly, we would like to have two windows that we could switch between: one for inspecting our data, and the other for editing our scripts.

We are in luck! We can do this with tmux. However, now that we are safely working in a session that tmux started for us, we no longer have to shout “Hey Tmux!” Rather we can just “ring a little bell” to get his attention. In the default tmux configuration, you do that by pressing <cntrl>-b from anywhere within a tmux window. This is easy to remember because it is like a “b” for the “bell” that we ring to get our faithful servant’s attention. <cntrl>-b is known as the “prefix” sequence that starts all requests to tmux from within a session.

The first thing that we are going to do is ask tmux to let us assign a more descriptive, name—data to be specific—to the current window. We do this with

<cntrl>-b ,

(That’s right! It’s a control-b and then a comma. tmux likes to get by on a minimum number of keystrokes.) When you do that, the green band at the bottom of the window changes color and tells you that you can rename the current window. We simply use our keyboard to change the name to “data”. That was super easy!

Now, to make a new window with a new bash shell that we can use for writing scripts we do <cntrl>-b c. Try it! That gives you a new window within the froggies session and switches your focus to it. It is as if Tmux (in his mustard-stained shirt) has created a new window on the froggies terminal, brought it to the front, and shared it with you. The left side of the green tmux status bar at the bottom of the screen now says:

[froggies] 0:data- 1:bash*

Holy Moly! This is telling you that the froggies session has two windows in it: the first numbered 0 and named data, and the second numbered 1 and named bash. The - at the end of 0:data- is telling you that data is the window you were previously focused on, but that now you are currently focused on the window with the * after its name: 1:bash*.

So, the name bash is not as informative as it could be. Since we will be using this new window for editing scripts, let’s rename it to edit. You can do that with <cntrl>-b ,. Do it!

OK! Now, if you have been paying attention, you probably realize that tmux has given us two windows (with two different bash shells) in this session called froggies. Not only that but it has associated a single-digit number with each window. If you are all about keyboard shortcuts, then you probably have already imagined that tmux will let you switch between these two windows with <cntrl>-b plus a digit (0 or 1 in this case). Play with that. Do <cntrl>-b 0 and <cntrl>-b 1 and feel the power!

Now, for fun, imagine that we want to have another window and a bash shell for launching jobs. Make a new window, name it launch, and then switch between those three windows.

Finally. When you are done with all that, you tell Tmux to detach from this session by typing:

<cntrl>-b d

(The d is for “detach”). This should kick you back to the shell from which you first shouted “Hey Tmux!” by issuing the tmux a -t froggies command. So, you can’t see the windows of your froggies session any longer, but do not despair! Those windows are still on the monitor Tmux set up for you, casting an eerie glow on his mustard stained shirt.

If you want to get back in the driver’s seat with all of those windows, you simply need to tell Tmux that you want to be attached again via his miracle-mirroring device. Since we are no longer in a tmux window, we don’t use our <cntrl-b> bell to get Tmux’s attention. We have to shout:

% tmux attach -t froggies

The -t flag stands for “target.” The froggies session is the target of our attach request. Note that if you don’t like typing that much, you can shorten this to:

% tmux a -t froggies

Of course, sometimes, when you log back onto the server, you won’t remember the name of the tmux session(s) you started. Use this command to list them all:

% tmux ls

The ls here stands for “list-sessions.” This can be particularly useful if you actually have multiple sessions. For example, suppose you are a poly-taxa genomicist, with projects not only on a frog species, but also on a fish and a bird species. You might have a separate session for each of those, so that when you issue tmux ls the result could look something like:

% tmux ls
birdies: 4 windows (created Sun Feb 16 07:23:30 2020) [203x59]
fishies: 2 windows (created Sun Feb 16 07:23:55 2020) [203x59]
froggies: 3 windows (created Sun Feb 16 07:22:36 2020) [203x59]

That is enough to remind you of which session you might wish to reattach to.

Finally, if you are all done with a tmux session, and you have detached from it, then from your shell prompt (not within a tmux session) you can do, for example:

tmux kill-session -t  birdies

to kill the session. There are other ways to kill sessions while you are in them, but that is not so much needed.

Table 7.1 reviews the minimal set of tmux commands just described. Though there is much more that can be done with tmux, those commands will get you started.

7.4.3 Further steps with tmux

The previous section merely scratched the surface of what is possible with tmux.
Indeed, that is the case with this section. But here I just want to leave you with a taste for how to configure tmux to your liking, and also with the ability to create different panes within a window within a session. You guessed it! A pane is made by splitting a window (which is itself a part of a session) into two different sections, each one running its own bash shell.

Before we start making panes, we set some configurations that make the establishment of panes more intuitive (by using keystrokes that are easier to remember) and others that make it easier to quickly adjust the size of the panes. So, first, add these lines to ~/.tmux.conf:

# splitting panes
bind \ split-window -h -c '#{pane_current_path}'
bind - split-window -v -c '#{pane_current_path}'

# easily resize panes with <C-b> + one of j, k, h, l
bind-key j resize-pane -D 10
bind-key k resize-pane -U 10
bind-key h resize-pane -L 10
bind-key l resize-pane -R 10

Once you have updated ~/.tmux.conf you need to reload that configuration file in tmux. So, from within a tmux session, you do <cntrl>-b :. This let’s you type a tmux command in the lower left (where the cursor has become active). Type source-file ~/.tmux.conf

The comments show what each line is intended to do, and you can see that the configuration “language” for tmux is relatively unintimidating. In plain language, these configurations are saying that, after this configuration is made active, <cntrl>-b / will split a window (or a pane), vertically, in to two panes. (Note that this is easy to remember because on an American keyboard, the \ and the |, share a key. The latter looks like a vertical separator, and would thus be a good key stroke to split a screen vertically, but why force ourselves to hit the shift key as well?). Likewise, <cntrl>-b - will split a window (or a pane) into two panes.

What do we mean by splitting a window into multiple panes? A picture is worth a thousand words. Figure 7.7 shows a tmux window with four panes. The two vertical ones on the left show a yaml file and a shell script being edited in vim, and the remaining two house shells for looking at files in two different directories.