cut, sort, uniq, wc & tr

grep finds lines. The tools in this lesson work on what is inside the lines: wc counts, cut picks out columns, sort orders lines, uniq removes or counts repeats, and tr changes characters. Each one is small, but joined with pipes they turn a CSV file or a log into a quick report, often faster than opening a spreadsheet.

All examples use Priya's shop data, orders.csv:

order_id,customer,city,item,amount
1001,Aarav,Pune,Notebook,120
1002,Priya,Jaipur,Pen Set,250
1003,Kabir,Pune,Backpack,1299
1004,Meera,Kochi,Notebook,120
1005,Rohan,Lucknow,Water Bottle,349
1006,Ananya,Jaipur,Backpack,1299
1007,Farhan,Pune,Pen Set,250
1008,Isha,Kochi,Notebook,120

wc: count lines, words and bytes

wc (word count) prints three numbers: lines, words and bytes. Options pick just one: -l lines, -w words, -c bytes, -m characters.

wc orders.csv
wc -l orders.csv
ls /etc | wc -l

  9  12 281 orders.csv
9 orders.csv
208

The file has 9 lines (a header and 8 orders), 12 "words" (wc splits on spaces, so Pen Set counts as two) and 281 bytes. wc -l at the end of a pipe is the standard way to count results of any command.

cut: pick columns

cut extracts parts of each line. With -d you choose the delimiter (the character that separates fields) and with -f the field numbers. With -c you cut by character position instead.

cut -d, -f2,3 orders.csv | head -n 4
cut -d: -f1,7 /etc/passwd | tail -n 2
cut -c1-4 orders.csv | head -n 3

customer,city
Aarav,Pune
Priya,Jaipur
Kabir,Pune
priya:/bin/bash
kabir:/bin/bash
orde
1001
1002

Field lists can be single numbers (-f3), lists (-f1,3) or ranges (-f2-4, or -f3- for "3 to the end"). /etc/passwd uses : as its delimiter, so -d: works there.

sort: put lines in order

sort sorts lines alphabetically by default. Its options control how:

printf '10\n9\n100\n' | sort
printf '10\n9\n100\n' | sort -n
tail -n +2 orders.csv | sort -t, -k5,5nr -k2,2 | head -n 4

10
100
9
9
10
100
1006,Ananya,Jaipur,Backpack,1299
1003,Kabir,Pune,Backpack,1299
1005,Rohan,Lucknow,Water Bottle,349
1007,Farhan,Pune,Pen Set,250

Without -n, numbers sort as text, character by character, so 100 comes before 9. The last command skips the header, sorts by the amount (field 5) as a number in reverse, and breaks ties by customer name (field 2), giving the four biggest orders. -k5,5 means "start and end at field 5"; writing only -k5 would mean "from field 5 to the end of the line".

uniq: remove or count repeats

uniq collapses adjacent identical lines into one. Because it only looks at neighbours, you almost always sort first:

cut -d, -f4 orders.csv | tail -n +2 | sort | uniq
cut -d, -f4 orders.csv | tail -n +2 | sort | uniq -c
cut -d, -f4 orders.csv | tail -n +2 | sort | uniq -d

Backpack
Notebook
Pen Set
Water Bottle
      2 Backpack
      3 Notebook
      2 Pen Set
      1 Water Bottle
Backpack
Notebook
Pen Set

uniq -c puts a count in front of each line: the classic "how many of each" report. uniq -d prints only lines that appear more than once, and uniq -u only those that appear exactly once. Add | sort -rn after uniq -c to rank the results, which you saw in the previous lesson.

tr: translate or delete characters

tr reads stdin (it does not take file names) and replaces, deletes or squeezes characters:

echo "Sharma Sweets, Rohtak" | tr 'a-z' 'A-Z'
echo "9876-543-210" | tr -d '-'
echo "too    many   spaces" | tr -s ' '
tr ',' '\t' < orders.csv > orders.tsv

SHARMA SWEETS, ROHTAK
9876543210
too many spaces

The first line changes lower case to upper case. -d deletes every listed character, here the dashes in a phone number. -s squeezes runs of a repeated character into one, which is very useful before cut on space-separated output. The last line converts the CSV to tab-separated values.

Putting it together

These tools shine when combined. Here is a small report: how many distinct cities ordered, and a neat table of city and amount.

tail -n +2 orders.csv | cut -d, -f3 | sort -u | wc -l
tail -n +2 orders.csv | cut -d, -f3,5 | tr ',' ' ' | column -t | head -n 3

4
Pune     120
Jaipur   250
Pune     1299

sort -u is a shortcut for sort | uniq. column -t lines text up in neat columns, which makes output much easier to read.

Common mistakes

• Using uniq without sort. uniq only merges neighbouring lines, so b a b a stays as four lines.
• Sorting numbers as text. Add -n (or -h for sizes like 4.0K).
• Counting the header row. Skip it with tail -n +2 before counting.
• Forgetting -d with cut. The default delimiter is a Tab, so cut -f2 orders.csv prints whole lines.
• Passing a file name to tr. It reads only stdin: use tr ... < file.

Key points

• wc -l counts lines, and is the usual last step of a counting pipeline.
• cut -d, -f2 picks fields by delimiter; cut -c1-4 picks character positions.
• sort -n sorts numbers, -r reverses, -t, -k5,5 sorts by one field, -u removes duplicates.
• uniq merges adjacent repeats, so sort first; uniq -c counts them.
• tr changes (a-z A-Z), deletes (-d) or squeezes (-s) characters from stdin.

cut -d, -f4 orders.csv | grep -c '^Notebook$'
tail -n +2 orders.csv | sort -t, -k5,5nr | head -n 1 | cut -d, -f2
cut -d, -f4 orders.csv | tail -n +2 | sort | uniq -c | sort -rn

3
Kabir
      3 Notebook
      2 Pen Set
      2 Backpack
      1 Water Bottle