Manipulating, Summarizing, and Extracting Text

🎯 Learning Objective

Master the wc, sort, and uniq commands to count, order, and de-duplicate text data, forming a powerful toolkit for data analysis and summarization on the command line.

📚 Concept Introduction

Raw data is rarely useful until it's summarized. Imagine you have a log file and need answers to basic questions: How many lines does it contain? What are the most common entries? How can I view numerical data in order?

Linux provides a suite of small, powerful tools that work together to answer these questions. In this unit, we'll focus on three of them: wc (word count), sort, and uniq (unique).

📁 Pre-created for this unit:

• A file named fruit_list.txt - A simple list with duplicate entries for sorting and counting.
• A file named numbers.txt - A list of numbers to demonstrate numeric sorting.

📏 wc: Counting Lines, Words, and Bytes

When you first encounter a file, you might ask: how big is it? The wc command gives you a quick summary. Think of it as your file's vital statistics - telling you at a glance whether you're dealing with a small configuration file or a massive log file.

◆ Basic wc Usage

Running it on our fruit list:

wc fruit_list.txt

The output:

9  9 50 fruit_list.txt

means 9 lines, 9 words, and 50 bytes. Most often, you only need one of these, which you can get with a specific flag.

◆ Common wc Options

For system administration, line counting is particularly valuable - it tells you how many entries are in a log file, how many users are in a system file, or how many configuration directives you have.

🔠 sort: Ordering Data

The sort command arranges the lines of a file in a specific order. By default, it sorts alphabetically.

◆ Basic Alphabetical Sorting

sort fruit_list.txt

This default alphabetical sorting works perfectly for text, but creates unexpected results with numbers. If you have a file with numbers like "1, 10, 2, 20", alphabetical sorting gives you "1, 10, 2, 20" instead of the numerical order "1, 2, 10, 20".

◆ Numeric Sorting

Compare the output of sort numbers.txt with sort -n numbers.txt.

The difference is crucial: -n tells sort to understand that "100" is larger than "2", not that "1" comes before "2" in the alphabet.

🔁 uniq: Finding and Counting Unique Lines

The uniq command is simple but powerful: it filters out adjacent duplicate lines. The key word here is "adjacent" - uniq only compares each line with the line immediately before it.

◆ Basic Unique Filtering

Because it only works on adjacent lines, you must almost always sort a file first before piping it to uniq. This is one of the most important command patterns in Linux - sort first, then uniq.

To get a simple list of unique fruits:

sort fruit_list.txt | uniq

Without the sort first, uniq might miss duplicates that aren't next to each other. The pipe (|) feeds the sorted output directly into uniq, creating a clean workflow.

◆ Counting Occurrences

The most powerful feature of uniq is counting occurrences with the -c flag. This is a classic and extremely common command-line pattern.

To get a count of each unique fruit:

sort fruit_list.txt | uniq -c

This combination answers the question "what appears in this file and how often?" - incredibly useful for analyzing log files, survey responses, or any repetitive data.

📋 Essential Command Reference

💡 Key Takeaways

• wc, sort, and uniq are a fundamental toolkit for text-file analysis.
• wc -l is perfect for getting a quick line count of a file.
• Always use sort -n when sorting numbers to avoid alphabetical sorting errors.
• uniq only works on sorted data. The sort file | uniq pattern is extremely common.
• The sort file | uniq -c pipeline is the standard way to get a frequency count of every unique line in a file.

By combining these three simple utilities, you can perform powerful data summarization and analysis directly on the command line, forming the basis of many shell scripts and diagnostic procedures.

🎯 Learning Objective

Master cut and awk to extract, reorder, and format columns from structured text data, a critical skill for processing log files, CSVs, and other delimited data.

📚 Concept Introduction

System logs, CSV files, and many command outputs are structured in columns. Rarely do you need all the columns. More often, you need to pull out just the first and third columns, or maybe print the last field of every line.

This is where field processing tools come in. cut is a simple, fast tool for extracting columns. awk is a more powerful programming language that excels at manipulating and formatting those columns.

📁 Pre-created for this unit:

• A file named employee_data.csv - A comma-separated list of employee data for our extraction tasks.

✂️ cut: The Simple Column Extractor

cut is the perfect tool when you just need to "cut out" a few columns from a file based on a delimiter. Think of it as a digital pair of scissors that can slice through structured data with precision.

When you're dealing with CSV files, log entries, or any delimited data, you rarely need every column. Maybe you only care about timestamps and error codes from a log file, or just names and salaries from an employee database. cut excels at these simple extraction tasks.

Syntax:

cut -d 'delimiter' -f field(s) filename

• -d: Specifies the delimiter. For a comma, you'd use -d ','.
• -f: Specifies which fields to select. The fields are numbered starting from 1.

To get just the employee names (the 1st field) from our CSV:

cut -d ',' -f 1 employee_data.csv

This command tells cut to use commas as separators and extract only the first field from each line. The result is a clean list of names with no other data cluttering the output.

🧠 awk: The Advanced Field Processor

While cut is great for simple extraction, awk is a powerful programming language designed for text processing. Think of awk as having a smart assistant that can not only extract columns but also rearrange them, format them nicely, and even make decisions based on the data content.

The beauty of awk lies in its ability to treat every line of text as a record with fields, making it perfect for processing structured data like CSV files, log entries, or command output.

◆ awk Basics

Syntax:

awk -F 'delimiter' '{ action }' filename

• -F: Sets the Field separator (delimiter). For CSV, this is -F ','.
• '{ action }': The code to run for each line. Inside the action:
  • Fields are referenced with a $ prefix: $1 is the first field, $2 is the second, and so on.
  • $0 refers to the entire line.

Understanding the $ notation is key: think of each field as a numbered variable that awk automatically creates for you from each line of input.

To print just the names (the 1st field) from our CSV:

awk -F ',' '{print $1}' employee_data.csv

This does the same job as the cut command above, but awk's real power shows when you want to do more than simple extraction. You can reorder fields and add descriptive text:

awk -F ',' '{print "Name:", $1, "ID:", $3}' employee_data.csv

Notice how awk lets you mix field values with custom text, creating formatted output that's much more readable than raw data columns.

◆ Controlling Output with Built-in Variables

awk provides special variables that give you powerful control over your data. These variables automatically track useful information about the input, letting you make intelligent decisions about what to process.

◆ Example: Skipping the Header Row

This is one of the most common real-world uses of NR. CSV files typically have a header row that describes the columns, but you don't want that header in your processed output. awk makes this easy:

awk -F ',' 'NR > 1 {print $1, $4}' employee_data.csv

This command processes only lines where the record number (NR) is greater than 1, effectively skipping the header. This pattern is incredibly useful when processing exported data from databases or spreadsheets.

The condition NR > 1 acts as a filter - awk only executes the print statement for lines that meet this criteria. This conditional processing is what makes awk so much more powerful than simple extraction tools.

◆ Example: Working with the Last Field

Sometimes you don't know how many columns a file has, but you always want the last one. $NF is perfect for this. This command will print the salary column from our CSV, as it is the last field.

awk -F ',' '{print $NF}' employee_data.csv

📋 Essential Command Reference

💡 Key Takeaways

• cut is a simple and fast tool for extracting columns when the delimiter is a single character.
• awk is a more powerful and versatile tool for field processing. It can reorder, format, and conditionally process data.
• Use -d with cut and -F with awk to specify your delimiter.
• awk fields are referenced by $1, $2, etc. $0 represents the entire line.
• awk's NR and NF variables give you powerful control over which lines to process and how to handle fields.

Choosing the right tool is key: for quick and simple column extraction, use cut. For anything more complex involving reordering, formatting, or conditional logic, awk is the superior choice.