Editing Text with sed & awk

sed and awk are two of the oldest tools in Unix, and they are still everywhere: in install scripts, in server automation and in one-line fixes typed during an outage. sed is a stream editor: it edits text as it flows past, line by line, without opening an editor. awk is a small language for columns: it splits every line into fields and lets you filter, calculate and print.

In this lesson you will learn the parts of each that cover most real work. The examples use orders.csv from the previous lesson and a small settings file, app.conf:

# Sharma Sweets app settings
port=8080
debug=true

# database
db_host=localhost
db_name=sweets

sed: find and replace

The most used sed command is s/old/new/, substitute. sed prints every line, changed or not, to stdout; the file itself stays the same unless you ask otherwise.

sed 's/true/false/' app.conf | grep debug
sed 's/Pune/Pune (MH)/g' orders.csv | head -n 2

debug=false
order_id,customer,city,item,amount
1001,Aarav,Pune (MH),Notebook,120

Without the g flag, s replaces only the first match on each line; with g it replaces all of them. Add i (for example s/pune/Pune/gi) to ignore case. If your text contains slashes, such as paths, use another separator character: s|/var/www/old|/var/www/shop|g works exactly the same.

sed: print and delete lines

sed commands can have an address in front, which limits them to certain lines: a line number, a range such as 2,4, or a regex between slashes. p prints and d deletes. With -n, sed prints nothing except what you ask for.

sed -n '2,4p' orders.csv
sed -n '/Kochi/p' orders.csv
sed '/^#/d; /^$/d' app.conf

1001,Aarav,Pune,Notebook,120
1002,Priya,Jaipur,Pen Set,250
1003,Kabir,Pune,Backpack,1299
1004,Meera,Kochi,Notebook,120
1008,Isha,Kochi,Notebook,120
port=8080
debug=true
db_host=localhost
db_name=sweets

The last command deletes comment lines (starting with #) and empty lines (^$), showing only the real settings. Administrators run this on long config files such as sshd_config to see what is actually set. A semicolon separates several sed commands.

sed: edit files in place

-i makes sed change the file itself. Give it a suffix (-i.bak) and sed first saves the original as a backup:

sed -i.bak 's/^port=.*/port=9090/' app.conf
grep port app.conf app.conf.bak

app.conf:port=9090
app.conf.bak:port=8080

The pattern ^port=.* means "a line starting with port= followed by anything", so the whole line is replaced, whatever the old port was. This is how many setup scripts change one setting without opening an editor.

With -E sed uses extended regular expressions, and \1, \2 in the replacement insert what the bracketed groups matched:

sed -E 's/^([0-9]+),([A-Za-z]+),.*/\2 placed order \1/' orders.csv | sed -n '2,3p'

Aarav placed order 1001
Priya placed order 1002

awk: fields and patterns

awk reads each line, splits it into fields called $1, $2 and so on ($0 is the whole line), and runs your program on it. By default fields are separated by spaces and tabs; -F, changes the separator to a comma. An awk program is a list of pattern { action } pairs: the action runs only on lines where the pattern is true.

awk -F, '{print $2, $5}' orders.csv | head -n 3
awk -F, 'NR > 1 && $5 > 300 {print $2 " spent Rs " $5}' orders.csv
awk '{print $1}' access.log | sort | uniq -c | sort -rn

customer amount
Aarav 120
Priya 250
Kabir spent Rs 1299
Rohan spent Rs 349
Ananya spent Rs 1299
      4 203.0.113.7
      3 192.0.2.55
      2 198.51.100.23
      1 198.51.100.89

A comma in print puts a space between values; strings placed side by side are joined. NR is the current line number (number of records), so NR > 1 skips the header, and $5 > 300 compares the amount as a number. The third command prints the first field of a web log, the visitor's IP address, and counts the visits from each one. Other useful built-in variables are NF (number of fields on the line, so $NF is the last field) and FS (the field separator).

awk: totals and reports

awk has variables, arithmetic and arrays, and two special patterns: BEGIN runs before the first line and END runs after the last. Variables start at zero, so you can add to them straight away.

awk -F, 'NR > 1 {total += $5} END {print "Total sales:", total}' orders.csv
awk -F, 'NR > 1 {sum[$3] += $5} END {for (c in sum) print c, sum[c]}' orders.csv | sort -k2,2nr
awk -F, 'NR > 1 {printf "%-8s %-12s %6d\n", $2, $4, $5}' orders.csv | head -n 3

Total sales: 3807
Pune 1669
Jaipur 1549
Lucknow 349
Kochi 240
Aarav    Notebook        120
Priya    Pen Set         250
Kabir    Backpack       1299

sum[$3] is an associative array: the city name is the key, and awk adds each amount to the right city. The order of for (c in sum) is not guaranteed, so the result is piped to sort. printf formats output like in C: %-8s is a left-aligned text column 8 characters wide and %6d a right-aligned whole number.

sed or awk?

Use sed for text changes: replace a word, delete lines, edit a config file in place. Use awk for columns and numbers: pick fields from space-separated output, filter rows by value, add up totals. When a task needs real logic or a proper CSV parser, write a small Python or Bash script instead of a very long one-liner.

Common mistakes

• Using double quotes around awk programs. The shell then replaces $1 and $5 with its own (empty) variables. Always use single quotes.
• Forgetting -F, for CSV. awk splits on spaces by default, so $2 of a CSV line is whatever follows the first space.
• Running sed -i untested. Check the output first, and keep a .bak copy.
• Forgetting g. s/a/b/ changes only the first match on each line.

Key points

• sed 's/old/new/g' file replaces text; -i.bak edits the file in place and keeps a backup.
• sed -n '2,4p' prints lines 2 to 4; sed '/^#/d' deletes comment lines.
• awk -F, '{print $2}' prints the second field; NR is the line number and NF the field count.
• pattern {action} runs the action only on matching lines; END {} runs after the last line.
• awk arrays such as sum[$3] += $5 build group totals; pass shell values with -v.

sed -n '/,Jaipur,/p' orders.csv
sed 's/Notebook/Notebook A5/g' orders.csv > orders-new.csv
awk -F, 'NR > 1 {sum += $5; n++} END {print "Average:", sum / n}' orders.csv
awk -F, 'NR > 1 {count[$4]++} END {for (i in count) print count[i], i}' orders.csv | sort -rn

1002,Priya,Jaipur,Pen Set,250
1006,Ananya,Jaipur,Backpack,1299
Average: 475.875
3 Notebook
2 Pen Set
2 Backpack
1 Water Bottle