Command Line Quick Reference

superior_hosting_service

Command

quick reference on command line tools and techniques for the people with limited time

The command line quick reference is a github repository by Utsav Barman

1. Introduction


1.1. Scope

  • This document will mainly focus on things that are usually done in the command line
  • The focus is on tools and techniques for one shot adhoc tasks.
  • Most of the things in the document can be applied to shell scripting. However, shell scripting is not the focus of this document
  • This document is for Linux/Unix

1.2. Background

  • I am documenting things I learnt and am still learning
  • I had multiple files with random snippets.
  • I decided to organize the mess into a single piece, so that I can efficiently look up things
  • I cannot claim to be an expert in command line.

1.3. Purpose

  • The document will not help anyone become an expert.
  • However, it may help people become effective.
  • I have kept the document concise. I skipped over details that I never used much and can be looked up later in the internet.
  • I have focused on concrete examples of common or interesting use cases.

1.4. Next steps

  • rephrase unclear material
  • correct incorrect material
  • add missing material
  • add further reading material
  • Contributions are welcome

2. Basics


2.1. Common commands

commanddescription
manget help on commands. e.g.
man date
dategives the date and time
calshows the calender
lstells which files are present in the current working directory
when a -l option is used, the command returns owner, size, date of file, persissions, etc.
catshows contents of a file
cpcopy a file
mvmove a file
difflists differences between 2 files
rmremoves file
grepfind occurences of strings in one or more files
pwdprint the present working directory’s name
cdchanges the current directory

2.2. Shortcuts

  • 2.2.1. Navigation
keyboard shortcutdescription
Ctrl+Amove the cursor to the start of the line
Ctrl+Emove the cursor to the end of the line
Alt+Fmove the cursor forward by one word
Ctrl+Fmove the cursor forward by one character
Alt+Bmove the cursor backward by one word
Ctrl+Bmove the cursor backward by one charater
Ctrl+XXtoggle the cursor’s position between the current position and the previous position
  • 2.2.2. Editing
keyboard shortcutdescription
Ctrl+Ucut all the characters to the left of the cursor
Ctrl+Kcut all the characters to the right of the cursor
Ctrl+Wcut one word to the left of the cursor
Ctrl+Hcut one character to the left of the cursor
Alt+Dcut one word to the right of the cursor
Ctrl+Dcut one character to the right of the cursor
Ctrl+Ypaste the cut characters
Ctrl+_undo the last deletion
Tabcomplete arguments or list all available commands
  • 2.2.3. Recall from history
keyboard shortcutdescription
Crtl+Rsearch the command history
Ctrl+Gabort
Ctrl+P/UPthe previous command in the history
Ctrl+N/DOWNthe next command in the history

3. Streams, Pipes and Redirects


3.1. Streams

A linux shell’s inputs and outputs are sequence of characters called streams. There are three standard I/O streams:

namedescriptionfile descriptor
stdoutdisplays output from commands1
stderrdisplays error from commands2
stdinprovides inputs to commands0

3.2. Redirections

  • 3.2.1. Types

Input and output redirections are done using angular brackets (<>)

Bracket typedescription
>send stream to a file. E.g.
ls a > o.txt
>>append stream to a file. E.g.
ls b >> o.txt
>&write into stream. E.g.
ls c > o2.txt 2>&1
<receive stream from a file. E.g.
wc < o.txt
<<embed the text that will be fed to a command within the script
Example
cat << EOF > output.txt
line 1
line 2
line 3
EOF
echo done
also known as heredoc
<<<expand variable and feed into the stdin of the command
Example
wc <<< $word
also known as herestring
  • 3.2.2. Additional Examples

3.2.2.1. Send standard output to sout.txt and standard error to serr.txt

command1 > sout.txt 2> serr.txt

3.2.2.2. Send standard output and standard error streams to the same file sone.txt

command1 > sone.txt 2>&1

OR

command1 &> sone.txt

3.2.2.3. Check if a string is present in a shell variable

grep "ABC" <<< $var

3.2.2.4. Ignore both standard input and output

command1 &> /dev/null

/dev/null is a null device file. This will discard anything written to it, and will return EOF on reading.

3.3. Pipe

Piping can redirect the standard output of one command to the input of another command.

command1 | command2 paramater1 | command3 parameter1 parameter2 | command4

Examples:

  • Print file contents only once. remove duplicate records:
    sort file1 | uniq
  • print 5 most frequently used commands
    history | awk '{a[$2]++}END{for(i in a){print a[i] " " i}}' | sort -rn | head -5
  • print the file types and their frequencies
    ls | rev | cut -f1 -d'.' | rev | sort | uniq -c | sort -n

3.4. xargs

  • short for extended arguments
  • some commands can take arguments from both standard input and as command-line arguments.
  • however, there are some commands that cannot take input from standard input. They accept inputs only from arguments. For these commands we need to use args.
  • xargs converts input from standard input into arguments to a command.
  • divides the arguments to a permitted number and runs the command repeatedly over each greoup of arguments.
  • using -n option, the number of arguments per command can be specified. find . | xargs -n1 basename
  • in order to assign the std input to a placeholder , use -I{}. the placeholder is usually used when we want to place the std input in the middle of a command
    ls | xargs -I{} echo {} file is found
  • xargs, by default, divides the input into arguments using whitespace. When -0 option is used, xargs uses null termination to identify arguments.

Examples:

  • print the number lines/words/characters in the files in a directory
    ls | wc
  • print the file types and their frequencies
    find . -type f | xargs basename -a | grep "\." | rev | cut -f1 -d'.' | rev | sort | uniq -c | sort -n
  • rename all files in a directory
    ls | xargs -I{} mv {} {}.bkp
  • copy all .c files to a specific directory
    find . -name "*.c" -print0 | xargs -0 -n1 -I{} cp {} some/folder/
    The print0 option of the find command prints the file names with the null character as termination character. By using the -0 option with xargs, it starts looking for the null character for reading filenames from the stdin.

3.5. tee

  • the command reads from standard input and writes it to a standard output and to one or more files
  • this is useful when we want to see the output of a command both on the screen and also want to save the output in a file for later analysis.

Examples:

  • ls | tee fileList.txt

4. Filename expansion


  • a wildcard character is a character that is used to represent one or more characters in a filename or foldername.
  • file globbing is the operation that recognizes these wildcard characters and does the expansion.
wildcarddescriptionexamplematchesdoes not match
*matches 0 or more charactersls to*to, tom, ton, tow, tommy, tommietata, tea
?matches 1 characterls to?tom, tow, tonto, tommy, tommie, tata, tea
[abc]matches any one of the characters in the bracketsls [bc]atbat, catBat, Cat, rat
[a-z]matches any one of the characters within the range in the bracketsls day[1-9]day1, day2 upto day9day11, day
[!abc]matches any one character that is not in the bracketsls [!r]atbat, cat, Bat, Cat, Ratrat
[!a-z]matches any one character that is not in the range in the bracketsls day[!1-9]day0, daysday1 upto day9

5. Brace expansion


  • it is generation of a range of strings to be used on the command line
  • useful for generating a list of sequential data or running any command on sequence of data.
  • a range or comma-separated list of data can be used for expansion. They are enclosed within curly braces.
  • example:
    • {0..4} generates a sequence from 0 to 4
    • {alice, bob, charlie} generates alice, bob and charlie
  • the list can be preceded by an optional string (called preamble or prefix). this is prefixed to each string in the braces
  • the list can be followed by an optional string (called postscript or postfix). this is appended to each string in the braces
  • syntax –
    • string list
      {String1, String2}
    • range list
      {<start>..<end>}
    • range list with custom increment
      {<start>..<end>..<increment>}
    • <prefix>{ string or range }<postfix>

5.1. Examples

  • create files for each day of month January
    touch {1..31}
  • print all even numbers from 1 to 100
    echo {2..100..2}
  • create input, output, backup and error folders
    mkdir /some/path/{input, output, backup, error}
  • print the alphabets in reverse order
    echo {z..a}
  • create a backup of a file
    cp file.txt{,.bkp}
  • restore from .bkp file
    mv file.txt{.bkp,}
  • create folder for each day of January, February and March
    mkdir -p ~/logs/{January/{1..31},February/{1..28},March/{1..31}}

6. Job control


  • command line provides the ability to stop/suspend the execution of a process and resume a suspended process at a later point in time
  • each running program is called job
  • unique id is assigned to every job
commanddescription
jobslist all the jobs that the current shell is running or has suspended
fgbring job to foreground
bgsend job to background
killterminate job
stopsuspend job
Ctrl+cterminate job
Ctrl+zsuspend job
  • to start a job in the background, we need to add an ampersand(&) at the end of the command

Examples:

  • start a job in the background
    sleep 1000 &
  • bring the job to the foreground
    fg %2 # where 2 is the job number
  • suspend a job
    stop %2 # where 2 is the job number
  • resume a job in the background
    bg %2 # where 2 is the job number
  • terminate a job
    kill %2 # where 2 is the job number

7. Process handling


  • Get a snapshot of processes running with ps command
  • Get the process id of a command by ps -ef | grep command
  • wait for a process to finish by using bash wait <process id>
  • kill a process by using kill <process id>
  • wait for completion of all child processes
    wait
  • wait for completion of specific process
    wait 1234 # where 1234 is the process id

7.1. difference between ps and jobs

  • jobs tells the list of jobs the current shell is managing
  • ps tells the list of all the processes running in the system

8. Quoting


  • Quoting is used to remove special meanings from characters or words
  • single quotes – when the single quotes are used, every character within the quotes is preserved and is not evaluated
  • double quotes – when the double qoutes are used, the dollar sign, back quotes and blackslashes are evaluated and interpreted.
  • escape character – \ is used to preserve the literal value of the following character.
  • ANSI C quoting – backslash escaped characters are treated according to ANSI C standard. Format:- $'string

Examples:

examplescommandresult
no quoteecho $HOME/home/user1/
escape characterecho \$HOME$HOME
single quoteecho '$HOME'$HOME
double quoteecho "$HOME"/home/user1/
ANSI C quotingecho $'There\'s a quote in my string!'There’s a quote in my string

9. Basic file management


9.1. list directory ls

commanddescription
lslist the contents of current directory
ls *list contents of the directory along with the subdirectory
ls -llist the contents of a directory along with the owner, permission, date, size
ls -alist hidden file
ls -tlist files in descending order of last modified date
ls -rtlist files in ascending order of last modified date
ls -Rlist files of the current directory and the subdirectory recursively to the last subdirectory
ls /path/to/dirextorylist files in the directory mentioned

9.2. show file contents

commanddescription
cat demo.txtshow the contents of file. use for relatively small files
head demo.txtshow the first part of the file
tail demo.txtshow the last part of the file
tail -f demo.txtshow the text appended to the file as the file grows
less demo.txtshow contents of file one screen at a time
less -p "regular" demo.txtshow contents of the file from the first line with which the pattern matches
strings -a binaryfileprint the sequence of all the printable characters in the file
diff file1 file2shows difference between 2 files
comm file1 file2compare 2 sorted file

9.3. file handling

  • 9.3.1. copy
commanddescription
cp file1 file2copy file1 to file2
cp file1 file2 directory1copy file1 and file2 to directory1
cp -R directory1 directory2copy contents of directory1 to directory2
cp *.txt directory1copy all files ending with .txt to directory1
  • 9.3.2. move or rename
commanddescription
mv file1 file2rename file1 to file2
mv file1 directory1/move file1 to directory1
mv *.jpg directory1/move all files ending with .jpg to directory1
  • 9.3.3. delete
commanddescription
rm file1remove file1
rm file1 file2remove file1 and file2
rm *.pngremove all files ending with .png
rm -d emptyDirectoryremove an empty directory
rm -r directory1recursively remove all files, subdirectories and directory of directory1

9.3.4. file linking

  • a symbolic link is a file that points to another file or directory
  • there are 2 types of links
    • Hard link -> it is an additional name for the existing file. Each file is associated with an unique number. This unique number is called inode. Hardlink associate 2 or more filenames to the same inode and in turn the same file. If the original file is removed, the contents are still available via the hardlink
    • Soft link -> it is an indirect pointer to a file or directory. It has only the path of the original file and not its contents
commanddescription
ln file1 link1create a hardlink link1 to file file1
ln -s file1 link1create a softlink link1 to file file1

9.3.5. change directory

commanddescription
cdChange to home directory
cd ~Change to home directory
cd -Change to the previous directory
cd ..Change to the parent directory
cd /Change to the root directory
cd ~/dir1/dir2Change to directory relative to home directory

9.3.6. display disk usage

commanddescription
duestimates and displays the disk space used by the files
du -h *prints size in human readable’

9.3.7. file ownership and permissions

commanddescription
chmodchange permissions
chownchange ownership

10. Special shell variables


  • there are special variables that are set internally
  • these variables are available to the user
variabledescription
$0name of the shell
$$the process id of the current shell
$?the exit status of the last executed command
$!the process id of the last background process
$_the last argument of the previous command

11. Exit codes


  • also known as return code
  • it is the code that is returned by command or process after it finishes
  • following are few exit codes and their meaning
exit codedescription
0Success
1general error
126command invoked does not have execute permissions
127command not found
130command terminated using Ctrl+C

12. grep


  • searches for pattern in files and prints each line that matches the input pattern grep -<options> <pattern> <filenames>
  • grep can be used to search in a single file or in multiple files

12.1. Useful grep options

optiondescription
-iignore case
-ndisplay line numbers along with lines
-vdisplay lines that do not match the pattern
-ccount the number of matching lines
-rsearch in all files under each directory
-ldiplay the filename of the file which has the matching pattern
-oprint only the matched string. The whole line with the matched string is not printed
-Iignore binary files
-A<n>include n lines after match
-B<n>include n lines before match
-C<n>include n lines before and after the match

12.1.1. Examples

  • lets say a demo file(demo.txt) has following content
  • THIS IS UPPER CASE LINE
  • this is lower case line
  • This is regular line
  • This line is also regular
  • Line number four
  • Line #5
  • Search for a string in a file
    grep "this" demo.txt
  • Search for a string in a file, without matching case
    grep -i "this" demo.txt
  • Search for a string in a file and get both the line number and output
    grep -n "this" demo.txt
  • Get the number of lines matching the searched string
    grep -c "this" demo.txt
  • Get the filename in which the searched string is found
    grep -l "this" demo.txt
  • Get 2 lines after the matching line
    grep -A2 "This" demo.txt
  • Get 2 line before the matching line
    grep -B2 "This" demo.txt
  • Get 2 lines before and after the matching line
    grep -C2 "This" demo.txt
  • Search recursively in all files in subdirectories
    grep -inrI 'some text' /path/to/dir

12.2. Regular expression in grep

  • a regular expression is a sequence of characters that specifies the search pattern in text.
  • different characters have special meaning in regular expressions
characterdescription
[abc]matches any one of the characters in the square brackets
[a-d]matches any one of the characters in the range specified in the square brackets
^startmatches the pattern only if the pattern is at the start of the line
end$matches the pattern only if the pattern is at the end of the line
[^abc]matches any one character that is NOT present in the square brackets
[^a-d]matches any one character that is NOT present in the range
.matches any one character
*mathces 0 or more occurences of the preceding character
.*matches zero or more of any character

12.3. Examples

  • Match any one character
    grep "[Tt]his" demo.txt
  • Search for line starting with the search pattern
    grep "^last" demo.txt
  • Search for line ending with the search patter
    grep "regular$" demo.txt
  • Search for line with a character in specified range
    grep "[0-9]" demo.txt
  • Search for line without a character in specified range
    grep "[^0-9]" demo.txt
  • Search for a line where the middle characters are not known
    grep "line.*regular" demo.txt

Note: grep can have regular expressions in the search pattern part, and can have wildcards in the files to search section.

  • last line

13. find


  • the find command is used to search and locate the list of files and directories
  • the list returned will satisfy the conditions used in find command.
  • syntax is find [starting point] [expression]
  • -exec [command] can be used to run command on the files located by find
  • by default all the results are newline terminated. -print0 option can be used to make the results null terminated

13.1. Examples

  • find files with specific name
    find . -name demo.txt
  • find files with a specific pattern
    find ./Codes -name *.cpp
  • find directories with specific name
    find . -name Codes -type d
  • find files with permission as 777 and change permissions to 644
    find . -type f -perm 0777 -print -exec chmod 644 {} \;
  • find and remove files
    find . -type f -name "*.bkp" -exec rm -f {};

14. sed filter and transform text


14.1. Overview

  • looks for pattern and edits them
  • works on both files and stdin
  • original files is not updated
  • results are put into standard output
  • syntax is sed 'instructions' file
  • instruction is of format '[address]command/regex/replace/modifier'
    • example – ‘5,15s/abc/ABC/g’ -> this will substitute abc with ABC in lines 5 to 15.
      • modifier g specifies that all occurences of abc in line will be replaced with ABC
  • when address is used, the lines belonging to the address are examined/modified. -> sed '1,100 s/A/a/' file.txt. this restricts substition to first 100 lines.
  • By default all the lines are printed. When ‘-n’ flag is used, this behavior is suppressed. Nothing will be printed unless an explicit request to print is found.
  • A pattern can be used as a address -> sed -n '/start/,/stop/ p' file.txt
  • When ‘!’ is used, the command is run outside of the address -> sed -n '/match/ !p' file.txt
commanddescription
ddelete
pprint
ssubstitute
qquit
ainsert a line after pattern
iinsert a line before pattern
cchange a line
ytransform

14.2. Examples

  • remove blank lines
    sed '/^$/d' file.txt
  • remove all lines with search string
    sed '/Search/d' file.txt
  • remove all instances of search string
    sed 's/Search//g' file.txt
  • print lines with the search string
    sed '/Search/ p' file.txt
  • substitue a string with another
    sed 's/oldString/newString/g' file.txt
  • remove trailing spaces
    sed 's/ *$//' file.txt
  • remove leading spaces
    sed `s/^ *//' file.txt
  • add spaces to start of everyline
    sed 's/^/ /' file.txt
  • print the first 10 line
    sed '10 q' file.txt
    sed -n '1,10 p' file.txt
  • append line after a pattern
    sed '/pattern/ a add line here' file.txt
  • insert line before a pattern
    sed '/pattern/ i add line here' file.txt
  • change a line with a pattern
    sed '/pattern/ c line changed here' file.txt
  • change a->p, b->q, c->r
    sed 'y/abc/pqr/' file.txt

14.3. Grouping

  • sed allows capturing specific parts of text into groups.
  • these groups can be manipulated
  • group is enclosed within parentheses expression “(” and “)” in the search string
  • each group is assigned a number. The first group is assigned \1 and so on.
  • \1 can be both in pattern string and replacement string

14.3.1. Grouping Examples:

  • switch first and second columns
    sed 's/\([a-z]*\) \([a-z]*\)/\2 \1/' file.txt
  • print lines which have consecutive duplicate words
    sed -n '/\([a-z][a-z]*\) \1/p' file.txt
  • remove consecutive duplicate words in a line
    sed 's/\([a-z][a-z]*\) \1/\1/' file.txt

14.4. Hold Buffer

  • When sed read text, each line is placed into a temporary space.
  • When a new line is read, the old text is replaced by the new line in the temporary space.
  • This temporary space is called pattern space.
  • Hold buffer is like a long term storage. text can be copied to and from pattern space.
commanddescription
xexchange hold space and pattern space
hcopy pattern buffer into hold space
Happend pattern buffer into hold space
gcopy hold space to pattern space
Gappend hold buffer into pattern buffer

14.4.1. Example

  • print one line after and before the pattern match
    sed -n '/999/ !{x;d};/999/ {x;p;x;p;n;p}' file.txt
  • add space after every line
    sed 'G' file.txt
  • insert blank line above every line which matches pattern
    sed '/start/ {x;p;x}' file.txt
  • Insert blank line after every line which matches pattern
    sed '/start/ {G}' file.txt
  • Insert blank line before and after every line which matches pattern
    sed '/start/ {x;p;x;G}' file.txt

15. awk


  • command line utility to find, process and transform text files
  • the basic syntax is pattern { action }
    • the pattern is compared with every input line. pattern can be any regular expression
    • when the pattern matches, the action is performed
    • when no pattern is provided, the action is applied to every line
    • example -> /^HTTP/ {print}
      • every line starting with HTTP is printed
    • there are 2 other important patterns
      • BEGIN – specifies actions to be performed, before any line is read
      • END – specifies actions to be performed, after all lines are read
      • Example -> awk 'BEGIN{print "start"} {print} END{print "end"}' file.txt
        • first start is printed, then all lines of file.txt are printed and then end is printed
  • awk interprets each line as a record of fields
  • one or more consecutive spaces or tabs are considered as a single delimiter between fields
  • $1, $2, etc. represents the first field, second field, and so on.
  • $0 represents the entire input line
  • awk has 2 data types – strings and integers
  • awk internally converts variable according to context
  • awk supports associative arrays. example var[key] = value
  • on integers, basic arithmatic operations (+-*/%) are supported. autoincrement(++) and decrement(–) is also supported.

15.1. Actions

  • following are few actions that can be performed
actiondescription
{ print $0; }print records
{ exit; }ends program
{ next; }skips current line
{a=$1; b=”X”}variable assignment
{ c[$1] = $2 }array varaible assignment
{if (condition) { action } else if (condition) { action } else { action }}if else conditions
{ for (i=1; i < x; i++) { action } }for loop
{ for (item in c) { action } }for loop iterating over a list

15.2. Special variables

variabledesc
FSInput field separator. can be modified
RSInput record separator. default value is newline. can be modified by user
OFSOutput field separator. can be modified
ORSOutput record separator. default value is newline. can be modified by user
NFNumber of fields in the current line (record). cannot be updated by user
NRNumber of lines processed so far. cannot be updated by user

Note: -F option can be used to update the input field separator -> awk -F":"'{ print $1 }' file.txt

15.3. Examples

  • split up “,” (comma) separated fields and print the third field ($3)
    awk -F"," '{print $2}' file.txt
  • print the 3rd field of a csv if the second field ($2) exists and is not empty
    awk -F"," '{if ($2)print $3}' file.txt
  • print the last field in each line
    awk -F"," '{ print $NF }' file.txt
  • print the line after the line matching search pattern
    awk '/pattern/ { i=1; next; } {if(i) {i--; print;}}' file.txt
  • print the line and the 2 lines after the line matching search pattern
    awk '/regexp/ {i=3;} { if(i) {i--; print;}}' file.txt
  • print the lines from a file starting at the line matching start until the line matching stop
    awk '/start/,/stop/' file.txt
  • count lines (wc -l)
    awk 'END{print NR}' file.txt
  • print matching lines (grep)
    awk '/pattern/'
  • print non matching lines (grep -v)
    awk '!/pattern/'
  • remove duplicate consecutive lines (uniq)
    awk 'a !~ $0 {print}; {a=$0}' file.txt
  • print first 10 lines of file (head)
    awk 'NR < 11' file.txt
  • print last 10 lines of file (tail)
    awk '{vect[NR]=$0;} END{for(i=NR-9;i<=NR;i++) {print vect[i];}}' file.txt
  • print the total number of bytes used by files
    ls -l | awk '{ x += $5 } END { print "Total bytes: " x }'
  • read a csv(comma separated) file and print the first and third field in semicolon separated format
    awk 'BEGIN{FS=",";OFS=";"}{print $1, $3}' file.txt

16. Arithmetic expansion


  • arithmetic expression can be evaluated and the result can then be used.
  • syntax – $(( expression ))

16.1. Example

  • print sum of 2 numbers
    echo $((1+3))

17. Command substitution


  • In command substitution, the output of a command replaces the command
  • the output of a command can be used an argument to another command
  • syntax is `command` and $(command)
  • using backticks (`) is discouraged and has been deprecated
  • $(commmand) supports nesting i.e. $(command1 $(command2))

17.1. Examples

  • assign the output of a command to a variable
    thedate=`date`
  • use the output of command as a parameter of another command
    vi $(grep -l 123 *)
  • assign contents of file to a variable
    filedata=$(cat file.txt)
    filedata=$(<file.txt)
    second option is faster

18. Process substitution


  • the input or output of a command can appear as a file. This is known as process substitution
  • this technique is useful when we want to use the output of multiple commands as the input to a command
  • process substitution can also be used to capture output and redirect it to the input of a process
  • template – <(command) and >(command)

18.1. Examples:

  • sort and compare two files
    diff <(sort file1) <(sort file2)
  • compare 2 folders
    diff <(ls $first_directory) <(ls $second_directory)

19. Subshell


  • a subshell is a child process launched by a shell
  • whenever a shell script is run, a subshell is created and the script is run in the subshell
  • variables defined in parent shell can be accessed if export is used while defining the variable
  • subshells can also be created using parentheses
    (command1; command2; command3)
  • subshells are a convenient way to group commands.
  • can be used to temporarily move to a different directory
#do something in current directory
(cd some/other/directory; other-command)
#back in the original directory
  • to run a command or script in the current shell, without creating a subshell, use ‘.’ as in . script.sh

20. Command grouping


  • sometimes we need to run multiple commands and redirect all the output to a single file. Command grouping helps in this.
  • without command grouping we need to redirect each command output individually to a file

20.1. using parentheses

  • Grouping list of commands can be done using parentheses ()
  • a subshell is created
  • example – (date; uptime) > file.txt

20.2. using curly braces

  • Grouping commands can be done using curly braces {}
  • causes the list to be executed in the current shell context
  • no subshell is created
  • example – {date; uptime;} > file.txt

21. Text editing with cut, paste and join


21.1. cut

  • cut command cuts out sections from each line and writes result to standard output
  • syntax is cut OPTION [FILE]
optiondescription
-ccharacter range which will be selected
-ddelimiter which will separate each fields in line in file
-ffields which will be printed

when -c option is used, ranges can be specified. Each range can be one of:

range typedescription
NNth character
N-fron Nth character to end of line
N-Mfrom Nth character to Mth character
-Mfrom first to Mth character

21.1.1. Examples

  • print the first and third columns of a csv file
    cut -f1,3 -d"," file.txt
  • print the first 3 characters of each line
    cut -c -3 file.txt

21.2. paste

  • merges lines of files
  • by default, the lines from each files are delimited by tab
  • when ‘-‘ is used instead of filename, the command reads from standard input
  • syntax paste [OPTION] [FILE] [FILE]
optiondescription
-dused to specify the delimiter
-spaste one file at a time

21.2.1. Examples

lets take 2 files – number.txt and name.txt

  • cat number.txt 1
  • 2
  • 3
  • 4
  • cat name.txt Alice
  • Bob
  • Charlie
  • David
  • merge 2 files, first file will give the first column and second file will give the second column
    paste number.txt name.txt
  • merge 2 files, delimited by ‘,’
    paste -d"," number.txt name.txt
  • merge 2 files, sequentially, i.e first only first file is printed and then only the second file
    paste -s number.txt name.txt

21.3. join

  • join lines of two files on a common field
  • syntax join [OPTIONS] FILE1 FILE2

21.3.1. Examples

lets take 2 files – number.txt and name.txt

  • cat number.txt 1 100
  • 2 101
  • 3 102
  • 4 103
  • 5 104
  • cat name.txt 1 Alice
  • 2 Bob
  • 3 Charlie
  • 4 David

join 2 files based on the first column
join number.txt name.txt

22. Aliases


  • aliases are short names for long commands
  • when we need to execute long commands multiple times, it is advisable to create aliases
  • syntax – alias [-p] [name[=value]]
  • we can create permanent alias by storing them in configuration files
  • to temporarily bypass an alias, use \ -> \ll
  • creating a alias
    alias name='values'
  • removing alias
    unalias name
  • print all defined alias
    alias -p

22.1. useful aliases

alias gh='history|grep'
alias c=clear
alias cx='chmod +x'
alias ..='cd ..'
alias sl=ls
alias left='ls -t -1'
alias count='find . -type f | wc -l'
alias f='find . |grep '

23. Functions


  • set of commands that accomplish a specific task
  • can be used numerous times
  • helps avoid writing the same code repeatedly
  • can use loops and conditions within them
  • arguments can be passed to functions
  • by using export -f functionname, we can make the functions available to shell scripts
    • export can be added to configuration files like .bashrc
    • To keep things modular, create a new file called ~/.bash_functions and then have your .bashrc load it
  • syntax
function_name () {
  commands
}
function_name () { commands; }

23.1. useful functions

mcd() { mkdir -p "$1"; cd "$1";}
cdl() { cd "$1"; ls;}

24. sort


  • sort lines text files
optiondescription
-rsort in reverse order
-nsort in numberical order
-ksort based on nth column
-usort and remove duplicates

25. uniq


  • report or omit repeated lines
  • the input file must be sorted
optiondescription
-cshow how many times a line is repeated
-dprints only the repeated lines only once
-uprints only the unique lines
-icase insensitive comparison

26. Conditions


26.1. If else

  • if-then-else is supported in command line
  • Syntax
if [ condition1 ]; then command1;
elif [ condition2 ]; then command2;
else command3; fi
if [ condition1 ]; then command1; elif [ condition2 ]; then command2; else command3; fi
  • there are different types of conditions
    1. file based condiitons
conditionsdescription
-echeck if file exists
-rcheck if file exists and is readable
-wcheck if file exists and is writable
-dcheck if file exists and is a directory
  1. string based conditions
conditionsdescription
==check if both the strings are equal
!=check if both the strings are not equal
>check if the first string is lexicographically greater than the second
<check if the first string is lexicographically smaller than the second
-ncheck if the string has a length more than 0
-zcheck if the string is an empty string
  1. number based conditions
conditionsdescription
-eqcheck if the numbers are equal
-necheck if the numbers are not equal
-gtcheck if the first number is greater than the second
-gecheck if the first number is greater than or equal to the second
-ltcheck if the first number is less than the second
-lecheck if the first number is less than or equal to the second
  • 0 is considered true and numbers greater than 0 are considered false.
    • this is because in Unix/Linix, when a process ends successfuly, it returns 0

26.2. Short circuiting

  • an alternative way of using conditions is by using logical AND (&&) and logical OR(||)
  • evaluation of a logical expression is stopped, as soon as the outcome has been determined. This is known as short-circuiting.
  • in case of Logical AND, as soon as sub-expression becomes false, the whole expression evaluates to false
    • in case of expr1 && expr2, if expr1 evaluates to false, then the whole expression will evaluate to false. So, expr2 is not evaluated at all.
    • && can be used to ensure that command2 is run only if command1 ends successfully. example ->command1 && command2
  • in case of Logical OR, as soon as sub-expression becomes true, the whole expression evaluates to true
    • in case of expr1 || expr2, if expr1 evaluates to true, then the whole expression will evaluate to true. So, expr2 is not evaluated at all.
    • || can be used to ensure that command2 is run only if command1 fails. example -> command1 || command2

26.2.1. example

  • create folder if it does not exist
  [ -d ./some/path/folder ] || mkdir /some/path/folder
  • create create file only if folder exists
  cd /some/path/folder && touch file.txt

27. Loops


27.1. while loops

  • the loop runs as long as the given condition is true
  • syntax while [ condition ]; do commands; done

27.1.1. example

  • print all the folders with .c files
find . -name *.c | {while read -r filename; do dirname $filename; done;} | sort | uniq # dirname returns the directory name

27.2. for loops

  • the loop iterates over a list of values or preset number of times
  • syntax for <variable name> in <a list of items>;do <some command> $<variable name>;done;

27.2.1. example

  • copy files from one folder to another
for file in ./code/*.txt; do cp $file /home/code/backup; done

28. ssh


  • ssh (SSH client) is a program for logging into a remote machine and for executing commands on a remote machine
  • syntax ssh user@host
  • running a single command on remote server
    ssh user@host command_to_run
  • logging into server with different port
    ssh -p portnum user@host
  • ssh connection using host in the middlessh -t reachable_host ssh unreachable_host

29. curl


  • used to transfer data from one server to another
  • syntax – curl [options] [URL]
optionsdescription
-osave the output in a file with name as specified after the option
-Osave the output in a file with the same name as in the url
-C –resume a download
-Ifetch headers of a url
-Lfollow redirects

29.1. Examples

  • retrieve a webpage
    curl example.com
  • save a webpage
    curl example.com -o example.html
  • resume a download
    curl -C - -O https://releases.ubuntu.com/21.10/ubuntu-21.10-desktop-amd64.iso
  • fetch headers only
    curl -I example.com
  • fetch weather
    curl wttr.in/london

29.2. handling REST calls

29.2.1. GET

  • GET is used to fetch resource
  • the GET method is the default method
  • example curl https://reqres.in/api/users/2

29.2.2. POST

  • POST is used to create resource in a server
  • To send a curl POST request we use the option -X POST
  • example curl -X POST -H "Content-Type: application/json" -d '{"email": "eve.holt@reqres.in","password": "pistol"}' https://reqres.in/api/register

29.2.3. PUT

  • PUT is used to update resource
  • To send a curl PUT request we use the option -X PUT
  • example curl -X PUT -H "Content-Type: application/json" -d '{"name": "morpheus","job": "zion resident"}' https://reqres.in/api/users/2

29.2.4. DELETE

  • DELETE is used to remove resource
  • To send a curl DELETE request we use the option -X DELETE
  • example curl -X DELETE https://reqres.in/api/users/2

30. wget


  • command line utility to download file
optionsdescription
-Odownload file under different name
-cresume a download

30.1. Examples

  • download a file
    wget https://releases.ubuntu.com/21.10/ubuntu-21.10-desktop-amd64.iso
  • resume a download
    wget -c https://releases.ubuntu.com/21.10/ubuntu-21.10-desktop-amd64.iso

31. One liners


  • print the files and directories in tree structure
    find . | sed -e "s/[^-][^\/]*\// |/g" -e "s/|\([^ ]\)/|-\1/"
  • print the directories in tree structure
    find . -type d | sed -e "s/[^-][^\/]*\// |/g" -e "s/|\([^ ]\)/| - \1/"
  • quick access to ascii table
    man ascii
  • find common lines in 2 files
    cat file1 file2 | uniq -d
  • find lines in file1 that is not present in file2
    cat file1 file2 file2 | uniq -u
  • find most frequently used commands
    history | cut -c8- | sort | uniq -c | sort -rn | head
  • recursively remove only directories with no files
    find . -depth -type d -exec rmdir {} \;
  • count the number of total active network interfaces
    wc -w <<<$(netstat -i | cut -d" " -f1 | egrep -v "^Kernel|Iface|lo")
  • display all the files/directories sorted by size
    du -ah . 2>/dev/null | sort -hr | less
  • make multiple copies of file
    tee filecopy{1..10}.txt < file.txt

32. Further reading


33. Change History


  • presented in reversed chronological order i.e. the latest change is at the top
NameDateChange Description
Utsav Barman26 Jan 2022added: [arithmetic expansion, grep recursively search, command substitution example]
Utsav Barman26 Jan 2022added: [ANSI C quoting]
Utsav Barman26 Jan 2022added: [brace expansion, find piped to xargs, herestrings, awk example]
Utsav Barman26 Jan 2022added: [command grouping]
Utsav Barman25 Jan 2022added: [chown, chmod and du]
Utsav Barman25 Jan 2022added: [& to start a background job, added curl and wget]
Utsav Barman25 Jan 2022read should be used with option -r ; used appropriate variable names in for loop example;
Utsav Barman25 Jan 2022Updated variable DATE to thedate. Uppercase variables are conventionally used for environment variables. Added $() for command substitution. backticks are discouraged; replaced -a with -e
Utsav Barman25 Jan 2022fixed typo for cd to root directory; added section for special shell variables and exit codes;
Utsav Barman25 Jan 2022Added wait, globbing, grep -o; fixed typos
Utsav Barman24 Jan 2022Initial draft