Terminal · foundation
Pipelines
A pipeline connects the standard output of each command to the standard input of the next, allowing multiple processes to transform a byte stream concurrently.
Why it matters
Pipelines replace temporary-file choreography with small composable stages whose intermediate data can be inspected and tested.
Mental model
How to reason about pipelines
The shell connects each stage's output stream to the next stage's input. On typical Unix-like systems this is a bounded kernel pipe, so stages overlap and a full buffer applies backpressure upstream.
Analogy
A pipeline is an assembly line whose stations work at the same time, each consuming the previous station's output and passing on a transformed result.
Examples
See the boundary, not just the happy path
Worked example · Count matching requests
grep ' 500 ' access.log | wc -lgrep emits only matching lines to stdout, and wc consumes that stream to count them without an intermediate file.
Worked example · Inspect an intermediate stage
generate | tee raw.txt | validatetee copies the same stream to raw.txt and its stdout, preserving the pipeline while making generated data available for debugging.
Useful contrast · stderr is not piped by default
compile | lessOnly compile's stdout enters less; compile's stderr remains connected to its prior destination unless separately redirected.
Common mistakes
Misconceptions to remove early
Assuming stages run one after another
Pipeline processes normally overlap; designs that require an upstream command to finish before downstream starts need a different boundary.
Checking only the last stage accidentally
Many shells report the last command's status for a pipeline by default, so an earlier failure may require explicit PIPESTATUS handling or a carefully chosen pipefail mode.
Quick check
Can you predict the result?
1. In a | b, which streams does | connect by default?
- • a's standard output to b's standard input
- • a's standard error to b's standard input
- • Both output streams from a to b
2. What happens when a downstream stage reads more slowly than an upstream stage writes?
Keep building