May the 'command line' be with you

Published: 26 May 2019
Last updated: 12 September 2021

Motivation

Sounds of keystrokes:

tick tick - tock

I was listening that repetitive sounds again and again.

I became curious:

Hi! How are you? What are you doing?

Hi! I’m removing lines with one space. I’m just about to finish…

Introduction

We are used to hearing about automation for repetitive and regular tasks.

However, time can be saved with tasks although they are executed only once.

Furthermore, the personal satisfaction instead of doing a manual task is priceless.

This post is based on some useful examples of commands with that in mind.

About the examples included

We need a command-line interpreter for reading lines from either the terminal or a file: the Shell.

There are different implementations of the shell:

Korn shell (ksh)
Bash (bash)
Dash (dash)
Z shell (zsh)
…

This is the current shell in my Linux distribution (Ubuntu):

$ echo $0
bash

Bash has stayed with me from the late nineties. However, Z shell is widely chosen by most of my friends who are using Linux or macOS and it’s cool when customizing it with Oh My Zsh!.

Its basic usage is similar to Bash and all the examples included in this post work for both of them.

These are my steps to move to Z shell in the last days:

Installing Z shell:
```
$ sudo apt install zsh
```

Installing Oh My Zsh!:

$ sh -c "$(curl -fsSL \ 
https://raw.github.com/robbyrussell/oh-my-zsh/master/tools/install.sh)"

Recovering Bash as the default shell (it was changed in the previous step):
```
$ chsh -s /bin/bash
```

When I feel confident and comfortable with Z shell, I’ll change my default shell definitively. Meanwhile, I’ll continue experimenting by opening a new terminal with that shell specifically:

$ gnome-terminal -e zsh &

Even though I’m a Linux user, I’ve been working with Windows operating systems as well and it’s possible to pursue the same goal with Command Prompt, PowerShell or Cygwin (a Unix emulation layer that runs on top of Windows).

Some examples

Remember that this post is focused on saving time when avoiding manual tasks.

There are more commands and more ways to do the same.

Files

Removing lines

# Format for removing lines with a pattern:
#  sed '/pattern/d' FILE

# Remove blank lines 
#  ^ = start of line
#  $ = end of line
$ sed -i '/^$/d' file.txt

# Remove lines with one space
$ sed -i '/^ $/d' file.txt

# Remove lines with whitespaces 
#  \s = whitespace
#  * = zero or more times the preceding character
$ sed -i '/^\s*$/d' file.txt

Replacing strings

# Format for replacing a pattern for another one:
# sed -i 's/pattern1/pattern2/g' FILE

# Replace 'foo' by 'bar' in file.txt
$ sed -i 's/foo/bar/g' file.txt

# Replace 'foo' by 'bar' in file.txt from line 3 to line 7
$ sed -i '3,7 s/foo/bar/g' file.txt

# Replace 'foo' by 'bar' in all the TXT files in a second level
$ sed -i 's/foo/bar/g' **/*.txt

Without the final g, it would just replace the first occurrence for every line.

For doing other types of massive replacements, take a look at the examples with find ... -exec ... or Renaming files with the same pattern section.

Getting part of the content

Imagine that you have a CSV file, colon as a separator, with users data.

You only need the emails which appear in the second column:

# d = delimiter (colon in this case)
# f = field (second field in this case)
$ cut -d: -f2 my_file.csv

# Skip lines without the delimiter
$ cut -d: -f2 -s my_file.csv

# Save the output into a file
$ cut -d: -f2 -s my_file.csv > second_column.txt

Looking at the end of a file

Instead of opening a file and moving to the end, the last lines can be obtained as follows:

# Get the last 10 lines
$ tail file.txt

# Get the last 20 lines
$ tail -n 20 file.txt

# Get all the lines from the line number 20
$ tail -n +20 file.txt

Adding line numbers

# Include line numbers for every line, using 2 columns and adding a whitespace afterwards
$ nl -ba -w2 -s' ' file.txt

File system

Creating nested directories

Instead of:

$ mkdir dir1
$ mkdir dir1/dir2
$ mkdir dir1/dir2/dir3
$ mkdir dir1/dir2/dir3/dir4
$ mkdir dir1/dir2/dir3/dir4/dir5
$ mkdir dir1/dir2/dir3/dir4/dir5/dir6

get the same in one line:

$ mkdir -p dir1/dir2/dir3/dir4/dir5/dir6

Renaming files with the same pattern

Renaming all the files in the current directory when appending “previous_”:

$ for file in *
do
    mv "$file" "previous_$file"
done

Or in one line:

$ for file in *; do mv "$file" "previous_$file"; done

If we want to make the change for only regular files, not directories:

# File exists and is a regular file:
#  test -f file.txt
# Equivalent to: [ -f file.txt ]

$ for file in *
do
    if [ -f "$file" ]
    then
        mv "$file" "previous_$file"
    fi
done

Or in one line:

$ for file in *; do if [ -f "$file" ]; then \
mv "$file" "previous_$file"; fi; done

Note: These examples wouldn't need quotes when executing on Z shell because it doesn't do word splitting by default, so they would work properly even with whitespaces in the filenames.

Searching

Searching for text

# Search for all the files that contain the text 'date=' 
# from the current directory
$ grep -r 'date=' .

# or only the names of the files
$ grep -rl 'date=' .

Searching for files

# Search for all the regular files from the current directory
$ find . -type f

# Search for all the JPG files from the current directory
$ find . -name '*.jpg'

# Search for all the JPG and PNG files from the current directory
$ find . \( -name '*.png' -or -name '*.jpg' \)

Remember the use of quotes around the pattern. Otherwise, it will behave strangely when finding a matching file in the current directory.

The find command can be expanded to execute a command for each file found:

# Remove all the JPG files from the current directory
#  {} is replaced by the name of the found file
#  ';' indicates the end of the value for the 'exec' option
$ find . -name '*.jpg' -exec rm {} ';'

Aren’t you sure about executing it? Ask for confirmation when replacing exec by ok:

$ find . -name '*.jpg' -ok rm {} ';'

Moving faster

Changing to another directory

To come back to the previous directory, it’s not necessary to type the full path. This command will drive you directly:

$ cd -

Z Shell allows you to do this kind of changes:

# pwd: print name of current/working directory

$ pwd
/home/rachel/lab/application/3.0/src
$ cd 3.0 5.0
$ pwd
/home/rachel/lab/application/5.0/src

Executing a previous command

The exclamation point !, also called bang, is useful to recall a previous command line:

# For example, we execute these commands
$ cat my_file.txt
$ vi my_file.txt

# bang bang 
# (it's more funny than thinking on 'exclamation point exclamation point')
$ !!
# It executes the last command: 'vi my_file.txt'

# Another useful example
$ vi /etc/hosts
# Oh! I forgot to execute it as a superuser
$ sudo !!

# bang cat
$ !cat
# It executes the last command that starts with 'cat': 'cat my_file.txt'

# Or extending the last command
$ echo Hello
Hello
$ !! world!
echo Hello world!
Hello world!

On the other hand, instead of using the upper arrow key to move backwards in the history to look for other executed commands, press Control + R:

$
(reverse-i-search)`':

Then, write a fragment of the command you’re looking for.

If you don’t find the command, keep pressing Control + R for getting the next match.

Once you find the command, press Return. Or Control + G to exit.

Managing a long command line

Control + A: it moves the cursor to the beginning of the line (A = first letter of the alphabet)
Control + E: it moves the cursor to the end of the line (E = end)
Control + U: it deletes all the command line (U = undo)
Control + K: it cuts the text from the current cursor position to the end
Control + Y: it pastes the previously cut text

Why K and Y?

Because of killing and yanking. This idea comes from Emacs and its kill ring.

Actually, when killing the text with Control + K, the text is saved in a “kill ring”.

The “kill ring” can store up to 10 elements.

After yanking text with Control + Y, this option is available:

Esc + Y: it replaces the last yanked text by the previously killed text available in the ring. Keep pressing Esc + Y until you get the desired text. It’s called a “ring”, because it works as a cycle.

Renaming a file

Instead of:

$ mv this/is/a/long/path/for/filename.txt \
     this/is/a/long/path/for/filename.txt.bk

execute:

$ mv this/is/a/long/path/for/filename.txt{,.bk}

Avoiding long paths

Imagine that you usually work in a directory and there is a long path to reach it:

$ cd directory1/directory2/directory3/directory4/directory5

A link with a simple name can be created:

# Create a link
$ ln -s directory1/directory2/directory3/directory4/directory5 workspace

# Access the directory
$ cd workspace

However, the last command will only work if it’s executed under the directory that contains the created link.

An alternative is to set the variable CDPATH:

export CDPATH='directory1/directory2/directory3/directory4'

and the following command will work from any directory:

cd directory5

Why?

When executing it, the shell will try to make the change under the current directory. If the directory doesn’t exists, it will try to find it under the directory saved in CDPATH variable.

CDPATH variable can contain several directories in the same way as PATH, so every directory will be checked in order.

Avoiding long commands

For example, this long command for getting your public IP address:

$ dig +short myip.opendns.com @resolver1.opendns.com

An alias can be created and saved in the ~/.bash_aliases file:

alias public_ip='dig +short myip.opendns.com @resolver1.opendns.com'

Then you will get your public IP address when typing:

$ public_ip

NOTES:

Remember to run source ~/.bash_aliases or . ~/.bash_aliases if you want to see the changes applied on the current open terminal.
Check that this piece of code exists into your ~/.bashrc file in order to have that alias available forever:
```
if [ -f ~/.bash_aliases ]; then
  . ~/.bash_aliases
fi
```

Git provides its own way to create and to save aliases. For example, if you want to rebase with every pull operation:

$ git config --global alias.pr 'pull --rebase'
$ git pr
# It executes: git pull --rebase
# ---
# Get all the existing aliases in the global scope
$ git config --global --get-regexp alias
# or
$ git config --global -l | grep alias
# or looking at [alias] section in ~/.gitconfig file

You can keep all these customizations in a repository as a backup. For example, GitHub is full of repositories named as dotfiles.

Friendly reminders

Anyone who does not have the command line at their beck-and-call is really missing something. - Tim O’Reilly

The power of tools

IDEs are getting better and better. However, I notice that commands and scripting supplement my work. And I even still continue to use VIM for some tasks.

The more tools you know, the easier it will be for you to do your tasks.

Command line tools vs GUI clients

It’s easy to find GUI clients for command-line tools. However, those GUI tools are usually limited to a list of operations.

If you like freedom, choose the command line ;)

Moreover, in the case of Git, sometimes people only push buttons on the GUI client until everything is green without knowing what they are doing. Yodra López had a similar experience and she includes some stories about it in her great talk (in Spanish): Mira lo que ha hecho… Git.

Freedom brings with it responsibilities, so maybe it forces people to know what they are doing.

Acknowledgments

Thanks to Marc Cornellà for letting me know the alternative to enabling word-splitting after my small pull request. It encouraged me to know more about it and to discard my decision of setting sh_word_split in Z shell global settings.