5 text processing tools `grep, sed, awk, cut, and tr` to score some marks in System Commands OPPE

This tutorial shall help anyone to score some marks in the IIT Madras System Commands OPPE exam.

Introduction

This tutorial covers five essential Linux text processing tools: grep, sed, awk, cut, and tr.

1. grep (Global Regular Expression Print)

grep is used for searching text using patterns.

Example 1: Basic text search

$ echo -e "This is an error\nThis is a warning\nThis is info" | grep "error"
This is an error

Example 2: Case-insensitive search

$ echo -e "Warning: System overload\nwarning: Low memory" | grep -i "warning"
Warning: System overload
warning: Low memory

Example 3: Display line numbers

$ echo -e "Line 1\nLine 2\nCritical error\nLine 4" | grep -n "Critical"
3:Critical error

Example 4: Recursive search in directories

Assume we have a directory structure:

project/
├── file1.txt (contains "TODO: Fix this")
└── subdir/
 └── file2.txt (contains "TODO: Implement feature")

$ grep -r "TODO" project/
project/file1.txt:TODO: Fix this
project/subdir/file2.txt:TODO: Implement feature

Example 5: Invert match

$ echo -e "Success: Task 1\nError: Task 2\nSuccess: Task 3" | grep -v "Success"
Error: Task 2

Example 6: Use regular expressions

$ echo -e "Phone: 123-456-7890\nInvalid: 123-45-6789" | grep -E "[0-9]-[0-9]-[0-9]"
Phone: 123-456-7890

Example 7: Print only matched parts (-o option)

$ echo "The IP address is 192.168.1.1 and 10.0.0.1" | grep -oE "\b([0-9]\.)[0-9]\b"
192.168.1.1
10.0.0.1

2. sed (Stream Editor)

sed is used for text transformation.

Example 1: Basic substitution

$ echo "The color is red" | sed 's/red/blue/'
The color is blue

Example 2: Global substitution

$ echo "one two one two" | sed 's/one/three/g'
three two three two

Example 3: In-place editing

Assume we have a file named "colors.txt" with content "The color is red"

$ sed -i 's/red/green/' colors.txt
$ cat colors.txt
The color is green

Example 4: Delete lines

$ echo -e "Keep this\nDelete me\nKeep this too" | sed '/Delete/d'
Keep this
Keep this too

Example 5: Print specific lines

$ seq 10 | sed -n '3,7p'
3
4
5
6
7

Example 6: Multiple operations

$ echo -e "foo bar\nbaz foo" | sed -e 's/foo/qux/g' -e '/baz/d'
qux bar

3. awk (Aho, Weinberger, and Kernighan)

awk is a powerful text-processing tool for working with structured data.

Example 1: Print specific fields

$ echo -e "John Doe 25\nJane Smith 30" | awk ''
John 25
Jane 30

Example 2: Use custom field separator

$ echo "name:John:age:30" | awk -F':' ''
John

Example 3: Sum values in a column

$ echo -e "10\n20\n30" | awk ' END '
60

Example 4: Conditional printing

$ echo -e "John 25\nJane 35\nMike 40" | awk '$2 > 30 '
Jane 35
Mike 40

Example 5: Calculate average

$ echo -e "10\n20\n30\n40\n50" | awk ' END '
30

Example 6: Use regular expressions

$ echo -e "apple\nbanana\ncherry" | awk '/^b/ '
banana

4. cut

cut is used for extracting sections from each line of files.

Example 1: Extract specific characters

$ echo "Hello, World!" | cut -c 1-5
Hello

Example 2: Extract specific fields with delimiter

$ echo "John,Doe,30,New York" | cut -d',' -f 2,4
Doe,New York

Example 3: Extract range of fields

$ echo "field1:field2:field3:field4:field5" | cut -d':' -f 1-3
field1:field2:field3

Example 4: Extract to end of line

$ echo "Name: John Doe, Age: 30, City: New York" | cut -d':' -f 2-
 John Doe, Age: 30, City: New York

Example 5: Exclude specific fields

$ echo "1,2,3,4,5" | cut -d',' --complement -f 2,4
1,3,5

5. tr (Translate)

tr is used for translating or deleting characters.

Example 1: Convert lowercase to uppercase

$ echo "hello world" | tr 'a-z' 'A-Z'
HELLO WORLD

Example 2: Delete specific characters

$ echo "hello 123 world" | tr -d '0-9'
hello world

Example 3: Squeeze repeating characters

$ echo "hello world" | tr -s ' '
hello world

Example 4: Translate characters

$ echo "hello world" | tr 'hw' 'HW'
Hello World

Example 5: Delete all except specified characters

$ echo "hello 123 world" | tr -cd '0-9\n'
123

Conclusion

These five tools - grep, sed, awk, cut, and tr - are powerful utilities for text processing in Linux. The examples provided demonstrate their basic usage and some advanced features. Practice with these examples and explore their man pages for more advanced usage using man command.