Contents

Introduction II - the (unix) command line: bash

Contents

Introduction II - the (unix) command line: bash

Peer Herholz (he/him)
Habilitation candidate - Fiebach Lab, Neurocognitive Psychology at Goethe-University Frankfurt
Research affiliate - NeuroDataScience lab at MNI/McGill
Member - BIDS, ReproNim, Brainhack, Neuromod, OHBM SEA-SIG, UNIQUE

logo logo   @peerherholz

Before we get started 1…


  • most of what you’ll see within this lecture was prepared by Ross Markello and further adapted by Peer Herholz

  • based on the Software Carpentries “Introduction to the Shell” under CC-BY 4.0

Before we get started 2…

We’re going to be working with a dataset from https://swcarpentry.github.io/shell-novice/data/shell-lesson-data.zip.

Download that file and unzip it on your Desktop!

Goals

  • learn basic and efficient usage of the shell for various tasks

    • navigating directories

    • file handling: copy, paste, create, delete

What is the “shell”?

  • The shell is a command-line interface (CLI) to your computer

    • This is in contrast to the graphical user interfaces (GUIs) that you normally use!

  • The shell is also a scripting language that can be used to automate repetitive tasks

But what’s this “bash shell”?

It’s one of many available shells!

  • sh - Bourne SHell

  • ksh - Korn SHell

  • dash - Debian Almquist SHell

  • csh - C SHell

  • tcsh - TENEX C SHell

  • zsh - Z SHell

  • bash - Bourne Again SHell <– We’ll focus on this one!

WHY so many?

  • They all have different strengths / weaknesses

  • You will see many of them throughout much of neuroimaging software, too!

    • sh is most frequently used in FSL

    • csh/tcsh is very common in FreeSurfer and AFNI

So we’re going to focus on the bash shell?

Yes! It’s perhaps the most common shell, available on almost every OS:

  • It’s the default shell on most Linux systems

  • It’s the default shell in the Windows Subsytem for Linux (WSL)

  • It’s the default shell on Mac <=10.14

    • zsh is the new default on Mac Catalina (for licensing reasons 🙄)

    • But bash is still available!!

Alright, but why use the shell at all?

Isn’t the GUI good enough?

  • Yes, but the shell is very powerful

  • Sequences of shell commands can be strung together to quickly and reproducibly make powerful pipelines

  • Also, you need to use the shell to accesss remote machine / high-performance computing environments (like Compute Canada)

NOTE: We will not be able to cover all (or even most) aspects of the shell today.

But, we’ll get through some basics that you can build on going forward.

The (bash) shell

Now, let’s open up your terminal!

  • Windows: Open the Ubuntu application

  • Mac/Linux: Open the Terminal

When the shell is first opened, you are presented with a prompt, indicating that the shell is waiting for input:

$

The shell typically uses $ as the prompt, but may use a different symbol.

IMPORTANT: When typing commands, either in this lesson or from other sources, do not type the prompt, only the commands that follow it!

Am I using bash?

Let’s check! You can use the following command to determine what shell you’re using:

echo $SHELL

/bin/bash

If that doesn’t say something like /bin/bash, then simply type bash, press Enter, and try running the command again.

  • then simply type bash, press Enter, and try running the command again

  • there might be other ways depending on your OS/installation, please let us know

Note: The echo command does exactly what its name implies: it simply echoes whatever we provide it to the screen!

(It’s like print in Python / R or disp in MATLAB or printf in C or …)

What’s with the $SHELL?

  • Things prefixed with $ in bash are (mostly) environmental variables

    • All programming languages have variables!

  • We can assign variables in bash but when we want to reference them we need to add the $ prefix

  • We’ll dig into this a bit more later, but by default our shell comes with some preset variables

    • $SHELL is one of them!

Soooo, let’s try our ~first~ second command in bash!

This command lists the contents of our current directory:

ls

fancy_analyzes.py             intro_to_shell.ipynb
gui_cli_example_bash.sh       introduction.md
gui_cli_example_python.py     introduction_1.md
intro_jupyter.ipynb           introduction_2.md
intro_to_git_and_github.ipynb

What happens if we make a typo? Or if the program we want isn’t installed on our computer?

Will the computer magically understand what we were trying to do?

ks

bash: ks: command not found

Nope! But you will get a (moderately) helpful error message 😁

The cons of the CLI

  • You need to know the names of the commands you want to run!

  • Sometimes, commands are not immediately obvious

    • E.g., why ls over list_contents?

Key Points

  • A shell is a program whose primary purpose is to accept commands and run programs

  • The shell’s main advantages are its high action-to-keystroke ratio, its support for automating repetitive tasks, and its capacity to access remote machines

  • The shell’s main disadvantages are its primarily textual nature and how cryptic its commands and operation can be

Working with Files and Directories

How do we actually make new files and directories from the command line?

First, let’s remind ourselves of where we are:

cd ~/Desktop/shell-lesson-data
pwd

/Users/peerherholz/Desktop/shell-lesson-data

ls -F

creatures/          molecules/          numbers.txt         writing/
data/               north-pacific-gyre/ pizza.cfg
history.txt         notes.txt           solar.pdf

Creating a directory

We can create new directories with the mkdir (make directory) command:

mkdir thesis

Since we provided a relative path, we can expect that to have been created in our current working directory:

ls -F

creatures/          molecules/          numbers.txt         thesis/
data/               north-pacific-gyre/ pizza.cfg           writing/
history.txt         notes.txt           solar.pdf

(You could have also opened up the file explorer and made a new folder that way, too!)

Good naming conventions

  1. Don’t use spaces

  2. Don’t begin the name with -

  3. Stick with letters, numbers, ., -, and _

    • That is, avoid other special characters like ~!@#$%^&*()

Creating a text file

Let’s navigate into our (empty) thesis directory and create a new file:

cd thesis

We can make a file via the following command:

touch draft.txt

touch creates an empty file. We can see that with ls -l:

ls -l

total 0
-rw-r--r--  1 peerherholz  staff  0 Nov  4 18:52 draft.txt

Moving files and directories

Let’s start by going back to the data-shell directory:

cd ~/Desktop/shell-lesson-data

We now have a thesis/draft.txt file, which isn’t very informatively named. Let’s move it:

mv thesis/draft.txt thesis/quotes.txt

The first argument of mv is the file we’re moving, and the last argument is where we want it to go!

Let’s make sure that worked:

ls thesis

quotes.txt

Note: we can provide more than two arguments to mv, as long as the final argument is a directory! That would mean “move all these things into this directory”.

Also note: mv is quite dangerous, because it will silently overwrite files if the destination already exists! Refer to the -i flag for “interactive” moving (with warnings!).

More on mv

Note that we use mv to change files to a different directory (rather than just re-naming):

mv thesis/quotes.txt .

The . means “the current directory”, so we should have moved quotes.txt out of the thesis directory into our current directory.

Let’s check that worked as expected:

ls thesis

ls quotes.txt

quotes.txt

(Note: providing a filename to ls instead of a directory will list only that filename if it exists. Otherwise, it will throw an error.)

Exercise: Moving files to a new folder

After running the following commands, Jamie realizes that she put the files sucrose.dat and maltose.dat into the wrong folder. The files should have been placed in the raw folder.

$ ls -F
 analyzed/ raw/
$ ls -F analyzed
fructose.dat glucose.dat maltose.dat sucrose.dat
$ cd analyzed

Fill in the blanks to move these files to the raw/ folder (i.e. the one she forgot to put them in):

$ mv sucrose.dat maltose.dat ____/____

mv sucrose.dat maltose.dat ../raw

Remember, the .. refers to the parent directory (i.e., one above the current directory)

Copying files and directories

The cp (copy) command is like mv, but copies instead of moving!

cp quotes.txt thesis/quotations.txt

ls quotes.txt thesis/quotations.txt

quotes.txt            thesis/quotations.txt

We can use the -r (recursive) flag to copy a directory and all its contents:

cp -r thesis thesis_backup

ls thesis thesis_backup

thesis:
quotations.txt

thesis_backup:
quotations.txt

Exercise: Renaming files

Suppose that you created a plain-text file in your current directory to contain a list of the statistical tests you will need to do to analyze your data, and named it: statstics.txt

After creating and saving this file you realize you misspelled the filename! You want to correct the mistake and remove the incorrectly named file. Which of the following commands could you use to do so?

  1. cp statstics.txt statistics.txt

  2. mv statstics.txt statistics.txt

  3. mv statstics.txt .

  4. cp statstics.txt .

  1. No: this would create a file with the correct name but would not remove the incorrectly named file

  1. Yes: this would rename the file!

  1. No, the . indicates where to move the file but does not provide a new name.

  1. No, the . indicates where to copy the file but does not provide a new name.

Moving and Copying

What is the output of the closing ls command in the aequence shown below:

$ pwd
/Users/jamie/data
$ ls
proteins.dat
$ mkdir recombine
$ mv proteins.dat recombine
$ cp recombine/proteins.dat ../proteins-saved.dat
$ ls

  1. proteins-saved.dat recombine

  2. recombine

  3. proteins.dat recombine

  4. proteins-saved.dat

  1. No: proteins-saved.dat is located at /Users/jamie

  1. Yes!

  1. No: proteins.dat is located at /Users/jamie/data/recombine

  1. No, proteins-saved.dat is located at /Users/jamie

Removing files

Let’s go back to data-shell and remove the quotes.txt file we created:

cd ~/Desktop/shell-lesson-data

rm quotes.txt

The rm command deletes files. Let’s check that the file is gone:

ls quotes.txt

ls: quotes.txt: No such file or directory

Deleting is FOREVER 💀💀

  • The shell DOES NOT HAVE A TRASH BIN.

  • You CANNOT recover files that have been deleted with rm

  • But, you can use the -i flag to do things a bit more safely!

    • This will prompt you to type Y or N before every file that is going to be deleted.

Removing directories

Let’s try and remove the thesis directory:

rm thesis

rm: thesis: is a directory

rm only works on files, by default, but we can tell it to recursively delete a directory and all its contents with the -r flag:

rm -r thesis

Because deleting is forever 💀💀, the rm -r command should be used with GREAT CAUTION.

Operations with multiple files and directories

Oftentimes you need to copy or move several files at once. You can do this by specifiying a list of filenames

Exercise: Copy with Multiple Filenames

(Work through these in the data-shell/data directory.)

In the example below, what does cp do when given several filenames and a directory name?

$ mkdir backup
$ cp amino-acids.txt animals.txt backup/

What does cp do when given three or more filenames?

$ ls
amino-acids.txt  animals.txt  backup/  elements/  morse.txt  pdb/  planets.txt  salmon.txt  sunspot.txt
$ cp amino-acids.txt animals.txt morse.txt

  1. When given multiple filenames followed by a directory all the files are copied into the directory.

  1. When give multiple filenames with no directory, cp throws an error:

cp: target morse.txt is not a directory

Using wildcards for accessing multiple files at once

* is a wildcard which matches zero or more characters.

Consider the data-shell/molecules directory:

ls molecules/*

molecules/cubane.pdb  molecules/methane.pdb molecules/pentane.pdb
molecules/ethane.pdb  molecules/octane.pdb  molecules/propane.pdb

This matches every file in the molecules directory.

ls molecules/*pdb

molecules/cubane.pdb  molecules/methane.pdb molecules/pentane.pdb
molecules/ethane.pdb  molecules/octane.pdb  molecules/propane.pdb

This matches every file in the molecules directory ending in .pdb.

ls molecules/p*.pdb

molecules/pentane.pdb molecules/propane.pdb

This matches all files in the molecules directory starting with p and ending with .pdb

Using wildcards for accessing multiple files at once (cont’d)

? is a wildcard matching exactly one character.

ls molecules/?ethane.pdb

molecules/methane.pdb

This matches any file in molecules that has one character followed by ethane.pdb. Compare to:

ls molecules/*ethane.pdb

molecules/ethane.pdb  molecules/methane.pdb

Which matches any file in molecules that ends in ethane.pdb.

Using wildcards for accessing multiple files at once (cont’d)

You can string wildcards together, too!

ls molecules/???ane.pdb

molecules/cubane.pdb molecules/ethane.pdb molecules/octane.pdb

This matches and file in molecules that has any three characters and ends in ane.pdb

Wildcards are said to be “expanded” to create a list of matching files. This happens before running the relevant command. For example, the following command will fail:

ls molecules/*pdf

ls: molecules/*pdf: No such file or directory

Exercise: List filenames matching a pattern

When run in the molecules directory, which ls command(s) will produce this output?

ethane.pdb methane.pdb

  1. ls *t*ane.pdb

  2. ls *t?ne.*

  3. ls *t??ne.pdb

  4. ls ethane.*

  1. No: This will give ethane.pdb methane.pdb octane.pdb pentane.pdb

  1. No: this will give octane.pdb pentane.pdb

  1. Yes!

  1. No: This only shows file starting with ethane

Key points

  • cp old new copies a file

  • mkdir path creates a new directory

  • mv old new moves (renames) a file or directory

  • rm path removes (deletes) a file

  • * matches zero or more characters in a filename, so *.txt matches all files ending in .txt

  • ? matches any single character in a filename, so ?.txt matches a.txt but not any.txt

  • The shell does not have a trash bin: once something is deleted, it’s really gone

Summary

  • The bash shell is very powerful!

  • It offers a command-line interface to your computer and file system

  • It makes it easy to operate on files quickly and efficiently (copying, renaming, etc.)

  • Sequences of shell commands can be strung together to quickly and reproducibly make powerful pipelines

Soapbox

  • Bash is fantastic and you will (likely) find yourself using it a lot!

  • However, for complex pipelines and programs we would strongly encourage you to use a “newer” programming lanuage

    • Like Python, which we will also be discussed in this workshop!

  • There are a number of reasons for this (e.g., better control flow, error handling, and debugging)

References

There are lots of excellent resources online for learning more about bash:

Finding Things

Oftentimes, our file system can be quite complex, with sub-directories inside sub-directories inside sub-directories.

What happens in we want to find one (or several) files, without having to type ls hundreds or thousands of times?

First, let’s navigate to the data-shell/writing directory:

cd ~/Desktop/shell-lesson-data/writing

The directory structure of data-shell/writing looks like:

Let’s get our bearings with ls:

ls

data      haiku.txt thesis    tools

Unfortunately, this doesn’t list any of the files in the sub-directories. Enter find:

find .

.
./tools
./tools/old
./tools/old/oldtool
./tools/format
./tools/stats
./haiku.txt
./thesis
./thesis/empty-draft.md
./data
./data/two.txt
./data/LittleWomen.txt
./data/one.txt

Remember, . means “the current working directory”. Here, find provides us a full list of the entire directory structure!

Filtering find

We can add some helpful options to find to filter things a bit:

find . -type d

.
./tools
./tools/old
./thesis
./data

This will list only the directories underneath our current directory (incluing sub-directories).

Alternatively, we can list only the files with:

find . -type f

./tools/old/oldtool
./tools/format
./tools/stats
./haiku.txt
./thesis/empty-draft.md
./data/two.txt
./data/LittleWomen.txt
./data/one.txt

We can also match things by name:

find . -name *.txt

./haiku.txt

Why didn’t this also get the other files??

Remember: wildcards are expanded BEFORE being passed to the command. So, we really want:

find . -name "*.txt"

./haiku.txt
./data/two.txt
./data/LittleWomen.txt
./data/one.txt

Executing with find

What if we want to perform some operation on the output of our find command? Say, list the file sizes for each file (as in ls -lh)?

We can do that with a bit of extra work:

find . -name "*.txt" -exec ls -lh {} \;

-rw-rw-r--@ 1 peerherholz  staff   218B Jul 30 10:41 ./haiku.txt
-rw-rw-r--@ 1 peerherholz  staff    18K Jul 30 10:41 ./data/two.txt
-rw-rw-r--@ 1 peerherholz  staff   1.0M Jul 30 10:41 ./data/LittleWomen.txt
-rw-rw-r--@ 1 peerherholz  staff   3.9K Jul 30 10:41 ./data/one.txt

Note the very funky syntax:

  • The -exec option means execute the following command,

  • ls -lh is the command we want to execute,

  • {} signifies where the output of find should go so as to be provided to the command we’re executing, and

  • \; means “this is the end of command we want to execute”

We can also “pipe” the output of find to the ls -lh command as follows:

ls -lh $( find . -name "*.txt" )

-rw-rw-r--@ 1 peerherholz  staff   1.0M Jul 30 10:41 ./data/LittleWomen.txt
-rw-rw-r--@ 1 peerherholz  staff   3.9K Jul 30 10:41 ./data/one.txt
-rw-rw-r--@ 1 peerherholz  staff    18K Jul 30 10:41 ./data/two.txt
-rw-rw-r--@ 1 peerherholz  staff   218B Jul 30 10:41 ./haiku.txt

Here, the $( ) syntax means “run this command first and insert it’s output here”, so ls -lh is provided the output of the find . -name "*.txt" command as arguments.

previous

Introduction II

next

Introduction III