Terminal · intermediate
Field processing with awk
awk processes input as records split into fields and runs pattern-action rules that can filter, calculate, aggregate, and format output.
Why it matters
Its data model handles column-oriented text and stateful summaries more clearly than long chains of single-purpose filters.
Mental model
How to reason about field processing with awk
For each input record, derive fields using FS, test each rule's pattern, and execute matching actions while variables can retain state across records.
Analogy
It is a programmable spreadsheet reader that visits each row, names its columns by position, and can keep running totals between rows.
Examples
See the boundary, not just the happy path
Worked example · Filter and format fields
awk -F, '$3 == "failed" { print $1, $2 }' jobs.csv-F, selects comma as the simple field separator, the pattern tests field 3, and the action emits fields 1 and 2 separated by OFS.
Worked example · Aggregate a column
awk '{ total += $2 } END { print total }' usage.txtThe action updates total for every record, and the END rule runs once after input is exhausted to emit the aggregate.
Avoid · Treating quoted CSV as simple fields
awk -F, '{ print $1 }' customers.csvStandard field splitting does not implement CSV quoting, so embedded commas require a CSV-aware parser rather than ordinary -F,.
Common mistakes
Misconceptions to remove early
Letting the shell expand awk variables
Single-quote the awk program so $1 and other awk field expressions reach awk instead of becoming shell positional parameters.
Assuming default fields mean fixed columns
The default FS treats runs of whitespace specially; explicitly choose parsing rules that match the actual input format.
Quick check
Can you predict the result?
1. In awk, when does an END rule run?
- • After all input records have been processed
- • After every field
- • Before the first record
2. Why is an awk program commonly enclosed in single quotes in a shell command?
Keep building