Terminal · foundation
Sorting and deduplicating text
sort orders input lines according to keys, locale, and options, while uniq compares adjacent lines and reports or removes adjacent repetitions.
Why it matters
The common sort | uniq combination can count distinct records correctly, but locale, numeric ordering, and adjacency affect the result.
Mental model
How to reason about sorting and deduplicating text
sort groups equal keys into neighboring positions; uniq then performs a one-pass comparison with only the immediately preceding line.
Analogy
First alphabetize a stack of cards so identical labels touch, then walk the stack and keep or count one run at a time.
Examples
See the boundary, not just the happy path
Worked example · Count repeated values
sort status-codes.txt | uniq -c | sort -nrThe first sort makes equal codes adjacent, uniq -c counts each run, and the numeric reverse sort ranks the resulting counts.
Worked example · Numeric rather than lexical order
printf '%s\n' 2 10 3 | sort -n-n compares numeric values and produces 2, 3, 10; ordinary lexical comparison may place 10 before 2.
Useful contrast · uniq sees runs, not global duplicates
printf '%s\n' a b a | uniqBoth a lines remain because they are separated by b; uniq does not remember every value seen earlier.
Common mistakes
Misconceptions to remove early
Using uniq on unsorted records for global deduplication
uniq only compares adjacent lines, so sort first unless the producer already groups equal records.
Ignoring locale-dependent collation
Locale can change ordering and equality behavior; use an explicit locale such as LC_ALL=C when bytewise reproducibility is required.
Quick check
Can you predict the result?
1. Why is sort commonly placed before uniq?
- • It brings equal lines together because uniq only compares adjacent lines.
- • uniq accepts only sorted syntax.
- • sort converts lines to numbers.
2. Which sort option is needed to order 2, 10, and 3 by numeric value?
Keep building