Pipes (no tobacco or Vaping allowed)

[dos2unix]

The Linux Pipe Command: Connecting Tools Together
The pipe (|) is one of the most powerful features in Linux, allowing you to chain commands together. It takes the output of one command and feeds it as input to another.
Finding the pipe key:
The pipe symbol is located above the backslash key (Shift + ), usually between Backspace and Enter, or to the right of the square brackets.

Basic Concept
The pipe connects commands in a chain:

command1 | command2 | command3

Output flows left to right: command1 → command2 → command3

Common Examples
1. Count files in a directory:

ls | wc -l

2. Search for running processes:

ps aux | grep firefox

3. View large log files page-by-page:

cat /var/log/syslog | less
dmesg | less

4. Sort and remove duplicates:

cat names.txt | sort | uniq

5. Count failed SSH login attempts:

cat /var/log/auth.log | grep "Failed password" | wc -l

6. Find largest files in current directory:

ls -lh | sort -k5 -h | tail -10

7. Search for specific text in multiple files:

cat *.log | grep "error"

Advanced Multi-Pipe Chains
1. Find top 10 memory-consuming processes:

ps aux | sort -k4 -r | head -10

2. Count unique IP addresses in access log:

cat /var/log/apache2/access.log | awk '{print $1}' | sort | uniq | wc -l

3. Find files modified today, sorted by size:

find /var/log -type f -mtime 0 | xargs ls -lh | sort -k5 -h

4. Monitor live log file for specific errors:

tail -f /var/log/syslog | grep "error"

5. List installed packages containing "python":

dpkg -l | grep python
rpm -qa | grep python

6. Count total disk space used by user:

find /home/username -type f | xargs du -ch | tail -1

7. Network connections sorted by state:

netstat -an | grep ESTABLISHED | wc -l
ss -tan | grep ESTAB | wc -l

8. Extract and count unique email addresses from file:

cat file.txt | grep -oE '[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+.[a-zA-Z]{2,}' | sort | uniq | wc -l

Practical System Administration Examples
1. Find which service is using port 80:

netstat -tulpn | grep :80
ss -tulpn | grep :80

2. Monitor CPU usage in real-time:

top -b -n 1 | head -20

3. Find files larger than 100MB:

find / -type f -size +100M | xargs ls -lh | sort -k5 -h

4. Check disk usage by directory:

du -h /var | sort -h | tail -20

5. Find recently modified config files:

find /etc -type f -mtime -7 | xargs ls -lt

6. Count lines of code in project:

find . -name "*.py" | xargs cat | wc -l

7. Search all logs for specific error:

find /var/log -type f -name "*.log" | xargs grep "connection refused"

Combining Pipes with Other Features
Redirect output to file while viewing:

ps aux | tee processes.txt | grep firefox

Count specific pattern occurrences:

journalctl | grep -c "error"

Extract specific columns:

cat data.csv | cut -d',' -f1,3 | sort

Replace text in pipeline:

cat file.txt | sed 's/old/new/g' | grep "new"

Tips and Best Practices

Each command in the pipe runs simultaneously, not sequentially
Data flows through the pipe in real-time
Use tee to save intermediate results while continuing the pipe
Pipes work with any command that reads from stdin and writes to stdout
You can chain as many commands as needed
Press Ctrl+C to stop a running pipe chain


Common Pipe Command Combinations

grep          Filter lines matching pattern
sort          Sort lines alphabetically or numerically
uniq          Remove duplicate adjacent lines
wc            Count lines, words, or characters
head          Show first N lines
tail          Show last N lines
less/more     Page through output
awk           Process and format text
sed           Stream editor for text manipulation
cut           Extract specific columns
tr            Translate or delete characters

The pipe command embodies the Unix philosophy: "Do one thing and do it well." By chaining simple tools together, you can accomplish complex tasks without writing scripts.

 

[dos2unix]

I see example 8 above has a lot of regex, maybe I should do an article on that sometime?