Logging into GMU (ORC) Computing

Overview

Teaching: 15 min
Exercises: 5 min

Questions

• Why do I need to login to another computer?

• How do I login to GMU computers?

Objectives

• Explain why it is necessary to login to another computer

• Explain what computer systems exist at GMU to log into

• Explain how to login to the GMU computers from a student’s computer

Why do I need to login to another computer instead of using my own computer?

In climate science most of our data analysis work is primarily done on either: 1. Workstations (individual computers with multiple CPUs, large internal memory and access to large data storage) 2. Clusters (groups of dozens to ~100 connected workstations, often called “nodes”), or 3. Supercomupters (very large clusters with hundreds or thousands of cores)

Such high-powered computers are almost always running a version of the Unix or Linux operating systems. These types of computers have more computational capability, more memory, and much more disk (or hard drive) space than a personal computer. We need this to handle the very large datasets used in climate data analysis.

Our research computers at GMU already contain many of the datasets we will work with. These datasets are too large for your home computer and would take weeks to months to download over the Internet. The Center for Ocean-Land-Atmosphere Studies (COLA) in the Department of Atmospheric, Oceanic, and Earth Sciences (AOES) has petabytes (i.e., thousands of terabytes, millions of gigabytes) of climate data from models and observations.

Linux/Unix computers use a command line interface that allow us to automate our data processing tasks much faster than a graphical user interface (GUI) that you may be familiar with in Windows or Mac OS.

GMU Computing - ORC and the clusters

The GMU Office of Research Computing maintains two separate clusters:

1. The HOPPER Cluster is ORC’s primary compute cluster, which provides support for both batch jobs via Slurm and interactive computing. The cluster includes: 74 compute nodes each with 192 GB of memory, and two GPU nodes each with 1 TB of memory.
2. The ARGO Cluster is a ~2000 core Linux cluster also providing a scheduled job environment using Slurm. There are a number of different node configurations on ARGO ranging from 16 cores to 32 cores per node. Each node provides a minimum of 4 GB/core, however it has some nodes with 512 GB of RAM and one node with 1.5 TB of RAM. Additionally there are also several kinds of GPU nodes.

We will use HOPPER in this class.

Setting up Software for your computer

There are several ways to connect to HOPPER. By far the easiest is via the Dashboard in your browser. This requires no special software installation or setup on your laptop. There are a number of commonly-requested apps provided from the dashboard, and a link to a more complete list of apps. In this course, we will be using two of the options listed:

1. The HOPPER Desktop - a Linux GUI OS interface to HOPPER, and
2. Jupyter Lab - an interface to create and run Python notebooks on HOPPER

As with all things GMU, you will have to log in with your Mason ID and password. If you have your browser setup properly, you will only have to do this every 14 days (all secure GMU pages including the ORC Dashboard should recognize the same authentication).

Connecting to HOPPER via the Dashboard

If you do not already have an ORC account, please see the new user information. The pre-course survey should have informed us whether you already have an account.

1. Click on the “HOPPER Desktop option”.
2. You will be moved to a page called “Interactive Sessions” and given options to launch an interactive desktop session.
   • Number of hours: Your session will expire automatically after the selected number of hours. Since this class lasts 1:15, select 2 hours (you may want to set a longer time when you are working outside of class).
   • Number of cores: Select 1. We will not do any massively parallel computing that requires more than one core. We will run some Python packages that take advantage of multiple CPUs simultaneously within the same core.
   • There is an option to be notified by email when your session becomes available. For such a small request, your session is typically ready after only a few seconds - it is not necessary to request email notification unless you are making a very large request (many nodes) for which resources may not be immediately available.
3. A new browser tab will open and present you with a virtual Linux desktop. Click on the “terminal emulator” icon at the bottom to launch a session in a terminal window. Your session will start in your home directory on HOPPER.

Connecting to HOPPER via ssh

As an alternative to the HOPPER desktop, you may access HOPPER via a terminal window on your laptop running secure shell (ssh) software. The software differs based on what type of computer you have.

Mac OS

For a Mac computer, use software called Xquartz, which takes advantage of the fact that Mac OS is built on top of Unix.

Windows

For a Windows computer, use software called MobusXterm.

Linux

The default Shell for Linux operating systems is usually bash. On most versions of Linux, it is accessible by running the (Gnome) Terminal or (KDE) Konsole or xterm, which can be found via the applications menu or the search bar. If your machine is set up to use something other than bash, you can run it by opening a terminal session and typing bash.

Download and install the correct ssh software for your computer

If you have not already done so, it would be a good idea to download the correct ssh software for your computer

Follow the instructions on your computer to install the software

On macOS

Launch the XQuartz software you downloaded and select Shell-> New Window from the menu in the upper left. A window will appear with a command line prompt, waiting for you to type.

To connect to HOPPER, type the following and replace username with your Mason ID:

$ ssh -Y username@hopper.orc.gmu.edu

Enter your password when prompted.

On Windows

1. Launch the MobusXterm software you downloaded.
2. Click Session->SSH
3. In the Remote host box, enter hopper.orc.gmu.edu
4. Check the Specify username box and enter your username
5. Click OK
6. Enter your password when prompted
7. Select No when asked to save your password.

If this is the first time you have logged in, you will be required to change your password.

Password Policies

• Keep your password secure and in a safe place - remember, this is your password for everything at GMU, not just this computer account!
• Do not share your password.
• No one should ever ask you for your password. If you ever get an email or other electronic message asking for your password DO NOT respond to it, it is a phishing attempt.
• Do not store your password on your laptop in plain text.
• Passwords must be changed every 6-months. You will get an alert from the university when the time is approaching.
• Follow the Mason Responsible Use of Technology Policies

Key Points

• You need to use GMU computers for this class

• The GMU computer that you will log into is a cluster called HOPPER

• Secure (ssh) software is used to login to GMU computers

Shell Commands

Overview

Teaching: 10 min
Exercises: 5 min

Questions

• How do I use the bash shell?

Objectives

• Introduce Unix Commands, switches, and how to get more info

Background

Now that you have logged into HOPPER, the computer is waiting for you to tell it what to do.

Humans and computers commonly interact in many different ways, such as through a keyboard and mouse, touch screen interfaces, or using speech recognition systems. The most widely used way to interact with personal computers is called a graphical user interface (GUI). With a GUI, we give instructions by clicking a mouse and using menu-driven interactions.

While the visual aid of a GUI makes it intuitive to learn, this way of delivering instructions to a computer scales very poorly. Imagine the following task: for a literature search, you have to copy the third line of one thousand text files in one thousand different directories and paste it into a single file. Using a GUI, you would not only be clicking at your desk for several hours, but you could potentially also commit an error in the process of completing this repetitive task. This is where we take advantage of the power of a computing shells.

The shell is used to receive commands telling the computer what to do. The shell is both a command-line interface (CLI) and a basic scripting language, allowing repetitive tasks to be done automatically and fast. With the proper commands, the shell can repeat tasks with or without some modification as many times as we want. Using the shell, the task in the literature example above could be accomplished in a second.

The Shell

The shell is actually just (another) program where users can type commands to interface with the computer. With the shell, it’s possible to invoke complicated programs like climate modeling software or simple commands that create an empty directory, often with only one line of code. There have been many shells created over the years. The very first, as part of the original Unix operating system in the early 1970s, is sh which stands for… “shell”! One of the most popular shells is bash (the Bourne Again SHell — so-called because it’s derived from a shell, that was also named sh, written by Stephen Bourne in the late 1970s). bash was written in the late 1980s, and has stood the test of time. It is the default shell on most modern implementations of Unix/Linux and in most packages that provide Unix-like tools for Windows.

When the shell is first opened, you are presented with a prompt, indicating that the shell is waiting for input:

$

The shell typically uses $ as the prompt, but may use a different symbol. In the examples for this lesson, we’ll show the prompt as $ . Most importantly: when typing commands, either from these lessons or from other sources, do not type the prompt, only the commands that follow it.

Many of you indicated in the Pre-Course Survey that you use the Unix/Linux shell daily or weekly. Often students know a small set of basic commands, but very few useful options or switches and are not familiar with commands that can greatly help your work and the work we will need to do in this course. Therefore, we will review some Unix basics here and you will review some additional helpful commands on your own prior to next class (note that the possibilities of what you can do with shell commands is practically limitless - feel free to read up, search the Internet and explore to learn more).

ls means list the contents of the current directory.

$ ls

$ ls -F /

ls is the command, with an option -F (also called switches or flags) and an argument /.

ls prints the names of the files and directories in the current directory. We make its output more comprehensible by using -F which tells ls to classify the output by adding a marker to file and directory names to indicate what they are: - a trailing / indicates that this is a directory - @ indicates a link - * indicates an executable

Depending on your default options, the shell might also use colors to indicate whether each entry is a file or directory.

Unix commands have many options that are very useful. Let’s look at some more for ls

Exploring More ls Flags

You can also use two options at the same time. What does the command ls do when used with the -l option? What about if you use both the -l and the -h option?

Some of its output is about properties that we do not cover in this lesson (such as file permissions and ownership), but the rest should be useful nevertheless.

Solution

The -l option makes ls use a long listing format, showing not only the file/directory names but also additional information such as the file size and the time of its last modification. If you use both the -h option and the -l option, this makes the file size ‘human readable’, i.e. displaying something like 5.3K instead of 5369.

Listing in Reverse Chronological Order

By default ls lists the contents of a directory in alphabetical order by name. The command ls -t lists items by time of last change instead of alphabetically. The command ls -r lists the contents of a directory in reverse order. Which file is displayed last when you combine the -t and -r flags? Hint: You may need to use the -l flag to see the last changed dates.

Solution

The most recently changed file is listed last when using -rt. This can be very useful for finding your most recent edits or checking to see if a new output file was written.

Getting help

ls has lots of other options. There are two common ways to find out how to use a command and what options it accepts:

1. We can pass a --help option to the command, such as:

$ ls --help

1. Many systems (but not HOPPER) allow viewing of a command’s manual page with man, such as:

$ man ls

1. More modern than man, Linux systems and many Unix systems have a differently-formatted manual invoked with the command info, such as:

$ info ls

More about options in Unix commands

You may have noticed that there may be single-dash options, e.g., ls -F and double-dash options, e.g., ls --help. You will usually see both kinds listed when you get help for a command.

Single-dash options are always single characters, and they can be chained together, e.g.: ls -asr is the same as ls -a -s -r.

Double-dash options are longer strings, usually in “plain English”, and must be listed separately, e.g.: ls --all --size --reverse.

Usually the order doesn’t matter - but check the documentation if things do not behave the way you expect.

Some options have both forms, some have one or the other. Single-dash options are the original approach from the early days of Unix, but restricts the number of options that can be defined, so double-dash was developed to allow for limitless options.

Finally, some options take arguments, e.g.: ls --color=always, which prints the names of files and directories in different colors depending on their types. Again, see documentation for valid choices.

Key Points

• Most commands consist of the command, options or switches, and input to the command

• Additional switches can be found using info or --help

Navigating Files and Directories

Overview

Teaching: 15 min
Exercises: 5 min

Questions

• How can I move around the file system on the computer?

• How can I see what files and directories I have?

• How can I specify the location of a file or directory on the computer?

Objectives

• Explain the similarities and differences between a file and a directory.

• Translate an absolute path into a relative path and vice versa.

• Construct absolute and relative paths that identify specific files and directories.

• Use options and arguments to change the behaviour of a shell command.

• Demonstrate the use of tab completion, and explain its advantages.

The part of the operating system responsible for managing files and directories (where all your data are stored) is called the file system. It organizes our data into files, which hold information, and directories (also called ‘folders’), which hold files or other directories. You are probably familiar with this idea from your own laptop. It works very similarly on HOPPER and other Linux/Unix systems.

Several commands are frequently used to create, inspect, rename, and delete files and directories. To start exploring them, we’ll go to our open shell window.

First let’s find out where we are by running a command called pwd (which stands for ‘print working directory’). Directories are like places - at any time while we are using the shell we are in exactly one place, called our current working directory. Commands, by default, read, write and execute files in the current working directory, i.e. ‘here’, so knowing where you are before running a command is important. pwd shows you where you are:

$ pwd

Home Directory Variation

The home directory path will look different on different operating systems. For ficticious student Nelle Nematode, on Linux it may look like /home/nelle (or if she’s at GMU: /home/nnematod), and on Windows it will be similar to C:\Documents and Settings\nelle or C:\Users\nelle (Note that it may look slightly different for different versions of Windows.) Meanwhile, on MacOS: /Users/nelle.

To understand what a ‘home directory’ is, let’s have a look at how the file system as a whole is organized. For the sake of this example, we’ll be illustrating the filesystem on our scientist Nelle’s computer. After this illustration, you’ll be learning commands to explore your own filesystem, which will be constructed in a similar way, but not be exactly identical.

On Nelle’s computer, the filesystem looks like this:

At the top is the root directory that holds everything else. We refer to it using a slash character, /, on its own; this is the leading slash in /Users/nelle.

Inside that directory are several other directories: bin (which is where some built-in programs are stored), data (for miscellaneous data files), Users (where users’ personal directories are located), tmp (for temporary files that don’t need to be stored long-term), and so on.

We know that our current working directory /Users/nelle is stored inside /Users because /Users is the first part of its name. Similarly, we know that /Users is stored inside the root directory / because its name begins with /.

Slashes

Notice that there are two meanings for the / character. When it appears at the front of a file or directory name, it refers to the root directory. When it appears inside a path, it’s just a separator.

Underneath /Users, we find one directory for each user with an account on Nelle’s machine, her colleagues imhotep and larry.

The user imhotep’s files are stored in /Users/imhotep, user larry’s in /Users/larry, and Nelle’s in /Users/nelle. Because Nelle is the user in our examples here we get /Users/nelle as our home directory. Typically, when you open a new command prompt you will be in your home directory to start.

Moving around files and directories

$ cd 

$ cd ..

.. is a special directory name meaning “the directory containing this one”, or more succinctly, the parent of the current directory.

The basic commands for navigating the filesystem on your computer: pwd, ls and cd. Let’s explore some variations on those commands.

What happens if you type cd on its own, without giving a directory?

$ cd

How can you check what happened? pwd gives us the answer! It turns out that cd without an argument will return you to your home directory, which is great if you’ve gotten lost in your own filesystem.

$ pwd

/home/pdirmeye

Let’s get some data and examples to work with for this lesson:

First, copy the file /home/pdirmeye/classes/clim680_2022/data-shell.zip to your home directory

$ cd
$ cp /home/pdirmeye/classes/clim680_2022/data-shell.zip .

This is a lot to type, but we can let the shell do most of the work through what is called tab completion. If we type:

$ cp /home/pdi

and then presses Tab (the tab key on your keyboard), the shell automatically completes the directory name:

$ cp /home/pdirmeye/

Pressing Tab again does nothing, because there are several possibilities; pressing Tab twice brings up a list of all the options, and so on. This is called tab completion, and we will see it in many other tools as we go on.

Once you have copied the file to your home directory, let’s unzip it:

$ unzip data-shell.zip

Let’s go explore the directories we copied:

$ cd data-shell/data

Check that we’ve moved to the right place by running pwd and ls -F

Here we used the relative path to specify the directory. When you use a relative path with a command like ls or cd, it tries to find that location from where we are, rather than from the root of the file system.

However, it is possible to specify the absolute path to a directory by including its entire path from the root directory, which is indicated by a leading slash. The leading / tells the computer to follow the path from the root of the file system, so it always refers to exactly one directory, no matter where we are when we run the command.

This allows us to move to our data-shell directory from anywhere on the filesystem (including from inside data). To find the absolute path we’re looking for, we can use pwd and then extract the piece we need to move to data-shell.

$ pwd

/home/pdirmeye/data-shell/data

$ cd /home/pdirmeye/data-shell

Run pwd and ls -F to ensure that we’re in the directory we expect.

Two More Shortcuts

The shell interprets the character ~ (tilde) at the start of a path to mean “the current user’s home directory”. For example, if Nelle’s home directory is /home/nelle, then ~/data is equivalent to /home/nelle/data. This only works if it is the first character in the path: here/there/~/elsewhere is not here/there/home/nelle/elsewhere.

Another shortcut is the - (dash) character. cd will translate - into the previous directory I was in, which is faster than having to remember, then type, the full path. This is a very efficient way of moving back and forth between directories. The difference between cd .. and cd - is that the former brings you up, while the latter brings you back. You can think of it as the Last Channel button on a TV remote.

Absolute vs Relative Paths

Starting from /home/amanda/data, which of the following commands could Amanda use to navigate to her home directory, which is /home/amanda?

1. cd .
2. cd /
3. cd /home/amanda
4. cd ../..
5. cd ~
6. cd home
7. cd ~/data/..
8. cd
9. cd ..

Solution

1. No: . stands for the current directory.
2. No: / stands for the root directory.
3. Yes: Amanda’s home directory is /home/amanda.
4. No: this goes up two levels, i.e. ends in /home.
5. Yes: ~ stands for the user’s home directory, in this case /home/amanda.
6. No: this would navigate into a directory home in the current directory if it exists.
7. Yes: unnecessarily complicated, but correct.
8. Yes: shortcut to go back to the user’s home directory.
9. Yes: goes up one level.

Key Points

• The file system is responsible for managing information on the disk.

• Information is stored in files, which are stored in directories (folders).

• Directories can also store other directories, which forms a directory tree.

• cd path changes the current working directory.

• ls path prints a listing of a specific file or directory; ls on its own lists the current working directory.

• pwd prints the user’s current working directory.

• / on its own is the root directory of the whole file system.

• A relative path specifies a location starting from the current location.

• An absolute path specifies a location from the root of the file system.

• Directory names in a path are separated with / on Unix/Linux and Mac OS, but \ on Windows.

• .. means ‘the directory above the current one’; . on its own means ‘the current directory’.

Working With Files and Directories

Overview

Teaching: 15 min
Exercises: 5 min

Questions

• How can I create, copy, and delete files and directories?

Objectives

• Create a directory hierarchy that matches a given diagram.

• Create files in that hierarchy using an editor or by copying and renaming existing files.

• Delete, copy and move specified files and/or directories.

Creating directories

We now know how to explore files and directories, but how do we create them in the first place?

Step one: see where we are and what we already have

Let’s go back to our data-shell directory on the Desktop and use ls -F to see what it contains:

$ pwd

/home/pdirmeye/data-shell

$ ls -F

creatures/  data/  molecules/  north-pacific-gyre/  notes.txt  pizza.cfg  solar.pdf  writing/

Create a directory

Let’s create a new directory called thesis using the command mkdir thesis (which has no output):

$ mkdir thesis

As you might guess from its name, mkdir means ‘make directory’. Since thesis is a relative path (i.e., does not have a leading slash, like /what/ever/thesis), the new directory is created in the current working directory:

$ ls -F

creatures/  data/  molecules/  north-pacific-gyre/  notes.txt  pizza.cfg  solar.pdf  thesis/  writing/

Good names for files and directories

Complicated names of files and directories can make your life painful when working on the command line. But extremely short and cryptic filenames will make your life painful when you return after days (weeks, months…) and try to remember what you were doing. Here we provide a few useful tips for the names of your files.

1. Don’t use spaces.

   Spaces can make a name more meaningful, but since spaces are used to separate arguments on the command line it is better to avoid them in names of files and directories. You can use - or _ instead (e.g. north-pacific-gyre/ rather than north pacific gyre/).

   You may encounter files with spaces in their names if they have been uploaded from a laptop. To reference such files from the command line, enclose the entire name in quotes ("" or '').

2. Don’t begin the name with - (dash).

   Commands treat names starting with - as options.

3. Stick with letters, numbers, . (period or ‘full stop’), - (dash) and _ (underscore).

   Many other characters have special meanings on the command line. We will learn about some of these during this lesson. There are special characters that can cause your command to not work as expected and can even result in data loss.

If you need to refer to names of files or directories that have spaces or other special characters, you should surround the name in quotes ("").

Since we’ve just created the thesis directory, there’s nothing in it yet:

$ ls -F thesis

Creating A File

$ touch my_file.txt

1. What did the touch command do?
2. Use ls -l to inspect the files. How large is my_file.txt?

Solution

1. The touch command generates a new file called my_file.txt in your current directory.

2. When you inspect the file with ls -l, note that the size of my_file.txt is 0 bytes. In other words, it contains no data. If you open my_file.txt using your text editor it is blank.

3. touching an existing file does not change it, but will update the date/time of its last edit to the current date/time.

Moving files and directories

Returning to the data-shell directory,

cd ~/data-shell/

Let’s create a file called draft.txt in our thesis directory using a text editor:

$ vi thesis/draft.txt

Let’s type a few lines. First, type the i character (this stands for insert)

It's not "publish or perish" any more, it's "share and thrive"

To save the file using this editor, use the esc key and then type ZZ

In our thesis directory we now have a file draft.txt that contains a quote. This is not a particularly informative name, so let’s change the file’s name using mv, which is short for ‘move’:

$ mv thesis/draft.txt thesis/quotes.txt

The first argument tells mv what we’re ‘moving’, while the second is where it’s to go. In this case, we’re moving thesis/draft.txt to thesis/quotes.txt, which has the same effect as renaming the file. Sure enough, ls shows us that thesis now contains one file called quotes.txt:

$ ls thesis

quotes.txt

One has to be careful when specifying the target file name, since mv will silently overwrite any existing file with the same name, which could lead to data loss. An additional option, mv -i (or mv --interactive), can be used to make mv ask you for confirmation before overwriting.

Note that mv also works on directories.

Let’s move quotes.txt into the current working directory. We use mv once again, but this time we’ll use just the name of a directory as the second argument to tell mv that we want to keep the filename, but put the file somewhere new. (This is why the command is called ‘move’.) In this case, the directory name we use is the special directory name . that we mentioned earlier.

$ mv thesis/quotes.txt .

The effect is to move the file from the directory it was in to the current working directory. ls now shows us that thesis is empty:

$ ls thesis

Further, ls with a filename or directory name as an argument only lists that file or directory. We can use this to see that quotes.txt is still in our current directory:

$ ls quotes.txt

quotes.txt

Moving Files to a new folder

After running the following commands, Jamie realizes that she put the files sucrose.dat and maltose.dat into the wrong folder. The files should have been placed in the raw folder.

$ ls -F
 analyzed/ raw/
$ ls -F analyzed
fructose.dat glucose.dat maltose.dat sucrose.dat
$ cd analyzed

Fill in the blanks to move these files to the raw/ folder (i.e. the one she forgot to put them in)

$ mv sucrose.dat maltose.dat ____/____

Solution

$ mv sucrose.dat maltose.dat ../raw

Recall that .. refers to the parent directory (i.e. one above the current directory) and that . refers to the current directory.

Copying files and directories

The cp command works very much like mv, except it copies a file instead of moving it. We can check that it did the right thing using ls with two paths as arguments — like most Unix/Linux commands, ls can be given multiple paths at once:

$ cp quotes.txt thesis/quotations.txt
$ ls quotes.txt thesis/quotations.txt

quotes.txt   thesis/quotations.txt

We can also copy a directory and all its contents by using the recursive option -r, e.g. to back up a directory:

$ cp -r thesis thesis_backup

We can check the result by listing the contents of both the thesis and thesis_backup directory:

$ ls thesis thesis_backup

thesis:
quotations.txt

thesis_backup:
quotations.txt

Renaming Files

Suppose that you created a plain-text file in your current directory to contain a list of the statistical tests you will need to do to analyze your data, and named it: statstics.txt

After creating and saving this file you realize you misspelled the filename! You want to correct the mistake, which of the following commands could you use to do so?

1. cp statstics.txt statistics.txt
2. mv statstics.txt statistics.txt
3. mv statstics.txt .
4. cp statstics.txt .

Solution

1. No. While this would create a file with the correct name, the incorrectly named file still exists in the directory and would need to be deleted.
2. Yes, this would work to rename the file.
3. No, the period(.) indicates where to move the file, but does not provide a new file name; identical file names cannot be created.
4. No, the period(.) indicates where to copy the file, but does not provide a new file name; identical file names cannot be created.

Moving and Copying

What is the output of the closing ls command in the sequence shown below?

$ pwd

/home/jamie/data

$ ls

proteins.dat

$ mkdir recombine
$ mv proteins.dat recombine/
$ cp recombine/proteins.dat ../proteins-saved.dat
$ ls

1. proteins-saved.dat recombine
2. recombine
3. proteins.dat recombine
4. proteins-saved.dat

Solution

We start in the /home/jamie/data directory, and create a new folder called recombine. The second line moves (mv) the file proteins.dat to the new folder (recombine). The third line makes a copy of the file we just moved. The tricky part here is where the file was copied to. Recall that .. means ‘go up a level’, so the copied file is now in /home/jamie. Notice that .. is interpreted with respect to the current working directory, not with respect to the location of the file being copied. So, the only thing that will show using ls (in /home/jamie/data) is the recombine folder.

1. No, see explanation above. proteins-saved.dat is located at /home/jamie
2. Yes
3. No, see explanation above. proteins.dat is located at /home/jamie/data/recombine
4. No, see explanation above. proteins-saved.dat is located at /home/jamie

Removing files and directories

Returning to the data-shell directory, let’s tidy up this directory by removing the quotes.txt file we created. The command we’ll use for this is rm (short for ‘remove’):

$ rm quotes.txt

We can confirm the file has gone using ls:

$ ls quotes.txt

ls: cannot access 'quotes.txt': No such file or directory

Deleting Is Forever

Shells don’t have a trash bin that we can recover deleted files from (though most graphical interfaces to Unix do - but interestingly, the one on HOPPER does not. But if you try to delete a file in HOPPER’s GUI, it will ask you if you are sure). Instead, when we delete files, they are unlinked from the file system so that their storage space on disk can be recycled. Tools for finding and recovering deleted files do exist, but there’s no guarantee they’ll work in any particular situation, since the computer may reuse the file’s disk space right away.

Using rm Safely

What happens when we execute rm -i thesis_backup/quotations.txt? Why would we want this protection when using rm?

Solution

$ rm: remove regular file 'thesis_backup/quotations.txt'? y

The -i option will prompt before (every) removal (use Y to confirm deletion or N to keep the file). The shell doesn’t have a trash bin, so all the files removed will disappear forever. By using the -i option, we have the chance to check that we are deleting only the files that we want to remove. Some systems have the -i option enabled by default. HOPPER accessed via a ssh terminal does not!

If we try to remove the thesis directory using rm thesis, we get an error message:

$ rm thesis

rm: cannot remove `thesis': Is a directory

This happens because rm by default only works on files, not directories.

rm can remove a directory and all its contents if we use the recursive option -r, and it will do so without any confirmation prompts:

$ rm -r thesis

Given that there is no way to retrieve files deleted using the shell, rm -r should be used with extreme caution (you might consider adding the interactive option rm -r -i).

If a directory is empty, rmdir (the complement of mkdir) will also remove it.

Operations with multiple files and directories

Oftent one needs to copy or move several files at once. This can be done by providing a list of individual filenames, or specifying a naming pattern using wildcards.

Copy with Multiple Filenames

For this exercise, you can test the commands in the data-shell/data directory.

In the example below, what does cp do when given several filenames and a directory name?

$ mkdir backup
$ cp amino-acids.txt animals.txt backup/

In the example below, what does cp do when given three or more file names?

$ ls -F

amino-acids.txt  animals.txt  backup/  elements/  morse.txt  pdb/  planets.txt  salmon.txt  sunspot.txt

$ cp amino-acids.txt animals.txt morse.txt

Solution

If given more than one file name followed by a directory name (i.e. the destination directory must be the last argument), cp copies the files to the named directory.

If given three file names, cp throws an error such as the one below, because it is expecting a directory name as the last argument.

cp: target ‘morse.txt’ is not a directory

Using wildcards for accessing multiple files at once

Wildcards

* is a wildcard, which matches zero or more characters. Let’s consider the data-shell/molecules directory: *.pdb matches ethane.pdb, propane.pdb, and every file that ends with ‘.pdb’. On the other hand, p*.pdb only matches pentane.pdb and propane.pdb, because the ‘p’ at the front only matches filenames that begin with the letter ‘p’.

? is also a wildcard, but it matches exactly one character. So ?ethane.pdb would match methane.pdb whereas *ethane.pdb matches both ethane.pdb, and methane.pdb.

Wildcards can be used in combination with each other e.g. ???ane.pdb matches three characters followed by ane.pdb, giving cubane.pdb ethane.pdb octane.pdb.

When the shell sees a wildcard, it expands the wildcard to create a list of matching filenames before running the command that was asked for. As an exception, if a wildcard expression does not match any file, Bash will pass the expression as an argument to the command as it is. For example typing ls *.pdf in the molecules directory (which contains only files with names ending with .pdb) results in an error message that there is no file called *.pdf. However, generally commands like wc and ls see the lists of file names matching these expressions, but not the wildcards themselves. It is the shell, not the other programs, that deals with expanding wildcards.

There are many other, fancier wildcards as well. For example:

[0-9] will match only numbers

[a-Z] will match any letters of either case. Unix/Linux alphabetizes as aAbBcCdD..zZ, so the mixed cases a and Z are necessary to include all the letters in the range.

[[:lower:]] will match only lower-case letters

[[:upper:]] will match only upper case letters

List filenames matching a pattern

When run in the molecules directory, which ls command(s) will produce this output?

ethane.pdb methane.pdb

1. ls *t*ane.pdb
2. ls *t?ne.*
3. ls *t??ne.pdb
4. ls ethane.*

Solution

The solution is 3.

1. shows all files whose names contain zero or more characters (*) followed by the letter t, then zero or more characters (*) followed by ane.pdb. This gives ethane.pdb methane.pdb octane.pdb pentane.pdb.

2. shows all files whose names start with zero or more characters (*) followed by the letter t, then a single character (?), then ne. followed by zero or more characters (*). This will give us octane.pdb and pentane.pdb but doesn’t match anything which ends in thane.pdb.

3. fixes the problems of option 2 by matching two characters (??) between t and ne. This is the solution.

4. only shows files starting with ethane..

More on Wildcards

Sam has a directory containing calibration data, datasets, and descriptions of the datasets:

.   
├── 2015-10-23-calibration.txt
├── 2015-10-23-dataset1.txt
├── 2015-10-23-dataset2.txt
├── 2015-10-23-dataset_overview.txt
├── 2015-10-26-calibration.txt
├── 2015-10-26-dataset1.txt
├── 2015-10-26-dataset2.txt
├── 2015-10-26-dataset_overview.txt
├── 2015-11-23-calibration.txt
├── 2015-11-23-dataset1.txt
├── 2015-11-23-dataset2.txt
├── 2015-11-23-dataset_overview.txt
├── backup
│   ├── calibration
│   └── datasets
└── send_to_bob
    ├── all_datasets_created_on_a_23rd
    └── all_november_files

Before heading off to another field trip, she wants to back up her data and send some datasets to her colleague Bob. Sam uses the following commands to get the job done:

$ cp *dataset* backup/datasets
$ cp ____calibration____ backup/calibration
$ cp 2015-____-____ send_to_bob/all_november_files/
$ cp ____ send_to_bob/all_datasets_created_on_a_23rd/

Help Sam by filling in the blanks.

The resulting directory structure should look like this

.
├── 2015-10-23-calibration.txt
├── 2015-10-23-dataset1.txt
├── 2015-10-23-dataset2.txt
├── 2015-10-23-dataset_overview.txt
├── 2015-10-26-calibration.txt
├── 2015-10-26-dataset1.txt
├── 2015-10-26-dataset2.txt
├── 2015-10-26-dataset_overview.txt
├── 2015-11-23-calibration.txt
├── 2015-11-23-dataset1.txt
├── 2015-11-23-dataset2.txt
├── 2015-11-23-dataset_overview.txt
├── backup
│   ├── calibration
│   │   ├── 2015-10-23-calibration.txt
│   │   ├── 2015-10-26-calibration.txt
│   │   └── 2015-11-23-calibration.txt
│   └── datasets
│       ├── 2015-10-23-dataset1.txt
│       ├── 2015-10-23-dataset2.txt
│       ├── 2015-10-23-dataset_overview.txt
│       ├── 2015-10-26-dataset1.txt
│       ├── 2015-10-26-dataset2.txt
│       ├── 2015-10-26-dataset_overview.txt
│       ├── 2015-11-23-dataset1.txt
│       ├── 2015-11-23-dataset2.txt
│       └── 2015-11-23-dataset_overview.txt
└── send_to_bob
    ├── all_datasets_created_on_a_23rd
    │   ├── 2015-10-23-dataset1.txt
    │   ├── 2015-10-23-dataset2.txt
    │   ├── 2015-10-23-dataset_overview.txt
    │   ├── 2015-11-23-dataset1.txt
    │   ├── 2015-11-23-dataset2.txt
    │   └── 2015-11-23-dataset_overview.txt
    └── all_november_files
        ├── 2015-11-23-calibration.txt
        ├── 2015-11-23-dataset1.txt
        ├── 2015-11-23-dataset2.txt
        └── 2015-11-23-dataset_overview.txt

Solution

$ cp *calibration.txt backup/calibration
$ cp 2015-11-* send_to_bob/all_november_files/
$ cp *-23-dataset* send_to_bob/all_datasets_created_on_a_23rd/

Organizing Directories and Files

Jamie is working on a project and she sees that her files aren’t very well organized:

$ ls -F

analyzed/  fructose.dat    raw/   sucrose.dat

The fructose.dat and sucrose.dat files contain output from her data analysis. What command(s) covered in this lesson does she need to run so that the commands below will produce the output shown?

$ ls -F

analyzed/   raw/

$ ls analyzed

fructose.dat    sucrose.dat

Solution

mv *.dat analyzed

Jamie needs to move her files fructose.dat and sucrose.dat to the analyzed directory. The shell will expand *.dat to match all .dat files in the current directory. The mv command then moves the list of .dat files to the ‘analyzed’ directory.

Reproduce a folder structure

You’re starting a new experiment, and would like to duplicate the directory structure from your previous experiment so you can add new data.

Assume that the previous experiment is in a folder called ‘2016-05-18’, which contains a data folder that in turn contains folders named raw and processed that contain data files. The goal is to copy the folder structure of the 2016-05-18-data folder into a folder called 2016-05-20 so that your final directory structure looks like this:

2016-05-20/
└── data
    ├── processed
    └── raw

Which of the following set of commands would achieve this objective? What would the other commands do?

$ mkdir 2016-05-20
$ mkdir 2016-05-20/data
$ mkdir 2016-05-20/data/processed
$ mkdir 2016-05-20/data/raw

$ mkdir 2016-05-20
$ cd 2016-05-20
$ mkdir data
$ cd data
$ mkdir raw processed

$ mkdir 2016-05-20/data/raw
$ mkdir 2016-05-20/data/processed

$ mkdir 2016-05-20
$ cd 2016-05-20
$ mkdir data
$ mkdir raw processed

Solution

The first two sets of commands achieve this objective. The first set uses relative paths to create the top level directory before the subdirectories.

The third set of commands will give an error because mkdir won’t create a subdirectory of a non-existant directory: the intermediate level folders must be created first.

The final set of commands generates the ‘raw’ and ‘processed’ directories at the same level as the ‘data’ directory.

Key Points

• cp old new copies a file.

• mkdir path creates a new directory.

• mv old new moves (renames) a file or directory.

• rm path removes (deletes) a file.

• * matches zero or more characters in a filename, so *.txt matches all files ending in .txt.

• ? matches any single character in a filename, so ?.txt matches a.txt but not any.txt.

• Use of the Control key may be notated in many ways, including Ctrl-X, Control-X, and ^X.

• The shell does not have a trash bin: once something is deleted, it’s really gone. REALLY!

• Most files’ names are something.extension. The extension isn’t required, and doesn’t guarantee anything, but is normally used to indicate the type of data in the file.