Pipes, Redirection, and the Text-Filter Toolkit

Three streams

Every command has three data streams: stdin (input), stdout (normal output), stderr (errors) - kept separate so you can handle them differently:

command > file          # stdout to a file, overwriting it
command >> file           # stdout to a file, appended to what's there
command 2> file             # stderr only, to a file
command 2>&1                 # send stderr wherever stdout is already going

> on its own only redirects stdout - the normal output channel. Run a command that fails and you'll see why 2> (stderr specifically) is a separate flag:

ls /no-such-directory > out.txt
cat out.txt

$

out.txt is empty, even though ls clearly printed something to your screen - "No such file or directory" went to stderr, not stdout, so > never captured it. To catch that error message too, redirect stderr explicitly:

ls /no-such-directory > out.txt 2> err.txt
cat err.txt

ls: cannot access '/no-such-directory': No such file or directory

This is exactly why the two streams are kept separate: it lets you save a command's real output while still seeing (or logging, separately) anything that went wrong.

Pipes

| sends one command's stdout directly into the next command's stdin, no file in between:

command-1 | command-2

That's the whole idea. The power comes from chaining small, single-purpose tools together.

The filter toolkit

cut -d' ' -f1        # print field 1 of each line
sort                    # sort lines alphabetically
sort -rn                 # sort as numbers, highest first
uniq                       # remove duplicate lines that sit next to each other
uniq -c                      # ...and also print how many times each one appeared
wc -l                          # count how many lines there are
tr 'a-z' 'A-Z'                   # replace characters, e.g. every lowercase letter with its uppercase version

cut splits each line into pieces called fields, using some character as the divider. -d' ' tells it the divider is a space. -f1 tells it to print only the first field. Change -f1 to -f2 and you'd get the second field instead.

sort -rn is two flags combined: -n sorts by numeric value instead of by plain text (so 9 correctly comes before 10 - plain text sorting would put 10 first), and -r reverses the order, so the biggest number comes first instead of last.

uniq -c only removes duplicates that are sitting right next to each other in the file - it does not scan the whole file for repeats. That's exactly why it's almost always used right after sort: sorting moves every copy of the same line next to each other first, so uniq -c actually has something to collapse.

awk, in the one shape you need here

awk reads a file one line at a time and can act on it. awk '{print $1}' means: for every line, print field 1 - where a "field" is a chunk of text separated by whitespace (spaces or tabs), counted from 1, not from 0. It does almost the same job as cut -d' ' -f1, with one difference that matters on real logs: cut needs exactly one space between fields, while awk still works correctly even if there are two or three spaces in a row.

Answer a question with one line

~/access.log has ten lines, one request per line, IP address first:

10.0.0.5 - - "GET /orders HTTP/1.1" 200
10.0.0.9 - - "GET /health HTTP/1.1" 200
10.0.0.5 - - "GET /orders HTTP/1.1" 200
10.0.0.2 - - "GET /cart HTTP/1.1" 200
10.0.0.5 - - "POST /orders HTTP/1.1" 201
10.0.0.9 - - "GET /health HTTP/1.1" 200
10.0.0.7 - - "GET /cart HTTP/1.1" 200
10.0.0.5 - - "GET /orders HTTP/1.1" 200
10.0.0.2 - - "GET /cart HTTP/1.1" 200
10.0.0.9 - - "GET /health HTTP/1.1" 200

Find which IP shows up the most, without opening the file and counting by eye. Build the pipeline one stage at a time so you can see what each step actually does. First, just the IPs:

awk '{print $1}' ~/access.log

10.0.0.5
10.0.0.9
10.0.0.5
10.0.0.2
10.0.0.5
10.0.0.9
10.0.0.7
10.0.0.5
10.0.0.2
10.0.0.9

Now sort them, so identical lines sit next to each other, then collapse and count with uniq -c:

awk '{print $1}' ~/access.log | sort | uniq -c

      2 10.0.0.2
      4 10.0.0.5
      1 10.0.0.7
      3 10.0.0.9

That's the real shape of the data: 10.0.0.5 really did show up 4 times, more than any other IP. Sort this small table by count, highest first, and keep only the top line:

awk '{print $1}' ~/access.log | sort | uniq -c | sort -rn | head -1

      4 10.0.0.5

Read the whole pipeline left to right, now that you've seen every stage: print the first field of every line (the IP), sort those IPs so duplicates are adjacent, collapse and count them, sort that count highest-first, keep only the top line. Pull out just the IP address, without the count next to it, and save it to a file:

awk '{print $1}' ~/access.log | sort | uniq -c | sort -rn | head -1 | awk '{print $2}' > ~/top-ip.txt

That's the same pipeline, with one more step tacked on the end: the line it produces looks like 4 10.0.0.5 (count, then IP), so the final awk '{print $2}' keeps only field 2 - the IP itself, without the count next to it.