Filtering text processing

Raw command output is noise. Your job is to extract the signal. Master these tools and you’ll spend less time staring at walls of text and more time finding what matters.

The Mindset

Linux tools follow the Unix philosophy: each command does one thing well, and you chain them together with pipes. The real power isn’t in any single command — it’s in combining them.

File Descriptors & Redirection

Before filtering, understand how data flows in Linux. Every process has three standard streams:

FD	Name	Description
`0`	`STDIN`	Input fed into a program
`1`	`STDOUT`	Normal output
`2`	`STDERR`	Error output

Think of file descriptors like a ticket system — the OS uses the number to know which stream to interact with.

Redirect Output

# Redirect STDOUT to a file (overwrites)
find /etc/ -name shadow > results.txt

# Append STDOUT to an existing file
find /etc/ -name passwd >> results.txt

# Redirect STDERR to /dev/null (silence errors)
find / -name "*.conf" 2>/dev/null

# Redirect STDOUT and STDERR to separate files
find /etc/ -name shadow 1>results.txt 2>errors.txt

# Redirect a file into a command as STDIN
cat < results.txt

Pipes

Pipes (|) take the STDOUT of one command and feed it as STDIN to the next. This is how you build processing chains.

# Find systemd config files, suppress errors, count results
find /etc/ -name "*.conf" 2>/dev/null | grep systemd | wc -l

2>/dev/null is your best friend. Add it to the end of almost every find or enumeration command to keep your output clean.

Filtering & Searching

`grep` — Pattern Matching

The most-used filtering tool. Searches for lines matching a pattern.

# Find users with bash shell
cat /etc/passwd | grep "/bin/bash"

# Exclude lines matching a pattern (-v = invert)
cat /etc/passwd | grep -v "nologin\|false"

# Case-insensitive search
grep -i "password" config.php

# Recursive search through a directory
grep -r "password" /var/www/ 2>/dev/null

# Show line numbers
grep -n "root" /etc/passwd

# Show only the matching part (not the whole line)
grep -o "password=[^ ]*" config.txt

# Extended regex (use | for OR, + for one-or-more)
grep -E "(password|passwd|secret|key)" config.php

Pentest one-liners with grep:

# Hunt for credentials in web configs
grep -r "password" /var/www/ 2>/dev/null
grep -r "DB_PASS\|db_password\|mysqli" /var/www/ 2>/dev/null

# Find cleartext passwords in logs
grep -i "password" /var/log/auth.log 2>/dev/null

# Search bash history for sensitive commands
grep -i "password\|sudo\|ssh\|curl\|wget" ~/.bash_history

Transforming Output

`cut` — Extract Specific Fields

Splits lines by a delimiter and picks the columns you want.

# Extract just usernames from /etc/passwd (field 1, delimiter :)
cat /etc/passwd | grep -v "nologin\|false" | cut -d":" -f1

# root
# sync
# cry0l1t3
# htb-student

# Extract username AND shell (fields 1 and 7)
cut -d":" -f1,7 /etc/passwd

`tr` — Translate / Replace Characters

# Replace colons with spaces
cat /etc/passwd | tr ":" " "
# Before: root:x:0:0:root:/root:/bin/bash
# After:  root x 0 0 root /root /bin/bash

# Convert lowercase to uppercase
echo "hello world" | tr "[:lower:]" "[:upper:]"
# HELLO WORLD

# Delete characters
echo "hello 123" | tr -d "[:digit:]"
# hello

# Replace newlines with spaces (useful for compressing multi-line output)
cat /etc/passwd | cut -d: -f1 | tr "\n" " "
# root daemon bin sys sync cry0l1t3 htb-student

`column` — Align Output Into a Table

Makes messy output readable. Great after using tr to replace delimiters.

# Without column -t
cat /etc/passwd | grep -v "nologin\|false" | tr ":" " "
# root x 0 0 root /root /bin/bash
# cry0l1t3 x 1001 1001 /home/cry0l1t3 /bin/bash
# htb-student x 1002 1002 /home/htb-student /bin/bash

# With column -t
cat /etc/passwd | grep -v "nologin\|false" | tr ":" " " | column -t
# root         x  0     0     root  /root              /bin/bash
# cry0l1t3     x  1001  1001        /home/cry0l1t3     /bin/bash
# htb-student  x  1002  1002        /home/htb-student  /bin/bash

Power Tools

`awk` — Field-Based Processing

More powerful than cut — can print specific fields, do math, and filter simultaneously.

# Print first and last field of each line
cat /etc/passwd | grep -v "nologin\|false" | tr ":" " " | awk '{print $1, $NF}'

# root /bin/bash
# cry0l1t3 /bin/bash
# htb-student /bin/bash

# Print lines where field 3 (UID) equals 0 (root-level accounts)
awk -F: '$3 == 0 {print $1}' /etc/passwd

`sed` — Stream Editor (Find & Replace)

# Replace all occurrences of "bin" with "HTB"
cat /etc/passwd | sed 's/bin/HTB/g'

# Delete lines matching a pattern
sed '/nologin/d' /etc/passwd

# Print only lines matching a pattern
sed -n '/bash/p' /etc/passwd

Regular Expressions (RegEx)

RegEx lets you define flexible search patterns instead of literal strings.

Key Operators

Operator	Description	Example
`.`	Any single character	`p.ss` matches `pass`, `p4ss`
`*`	Zero or more of the previous	`ba*` matches `b`, `ba`, `baa`
`+`	One or more of the previous	`ba+` matches `ba`, `baa`
`?`	Zero or one of the previous	`colou?r` matches `color`, `colour`
`^`	Start of line	`^root` — lines starting with root
`$`	End of line	`bash$` — lines ending with bash
`[abc]`	Any character in the set	`[aeiou]` — any vowel
`[a-z]`	Character range	`[0-9]` — any digit
`(a\|b)`	OR — match a or b	`(bash\|sh)`
`{2,4}`	Repeat 2 to 4 times	`[0-9]{1,3}`

Examples with `grep -E`

# Lines containing "my" OR "false"
grep -E "(my|false)" /etc/passwd

# Lines containing BOTH "my" AND "false" (AND pattern)
grep -E "(my.*false)" /etc/passwd

# Find IP addresses in a log file
grep -E "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}" /var/log/auth.log

# Lines that do NOT start with #
grep -v "^#" /etc/ssh/sshd_config

# Lines ending with "yes"
grep -E "yes$" /etc/ssh/sshd_config

Quick Reference

Tool	Primary Use	Key Flag
`grep`	Filter lines by pattern	`-v` invert, `-r` recursive, `-E` extended regex
`cut`	Extract fields by delimiter	`-d` delimiter, `-f` field number
`tr`	Replace or delete characters	`-d` delete
`awk`	Field processing and logic	`'{print $1, $NF}'`
`sed`	Find and replace in streams	`'s/old/new/g'`
`sort`	Sort output	`-n` numeric, `-r` reverse, `-u` unique
`wc`	Count results	`-l` lines
`column`	Format output as table	`-t`
`head` / `tail`	First or last N lines	`-n` number, `-f` follow
`less`	Page through output	`q` quit, `/` search

Next: Permission Analysis — reading permission strings, finding SUID misconfigs, and understanding what sticky bits mean for exploitation.

​The Mindset

​File Descriptors & Redirection

​Redirect Output

​Pipes

​Filtering & Searching

​grep — Pattern Matching

​Transforming Output

​cut — Extract Specific Fields

​tr — Translate / Replace Characters

​column — Align Output Into a Table

​Power Tools

​awk — Field-Based Processing

​sed — Stream Editor (Find & Replace)

​Regular Expressions (RegEx)

​Key Operators

​Examples with grep -E

​Quick Reference

The Mindset

File Descriptors & Redirection

Redirect Output

Pipes

Filtering & Searching

`grep` — Pattern Matching

Transforming Output

`cut` — Extract Specific Fields

`tr` — Translate / Replace Characters

`column` — Align Output Into a Table

Power Tools

`awk` — Field-Based Processing

`sed` — Stream Editor (Find & Replace)

Regular Expressions (RegEx)

Key Operators

Examples with `grep -E`

Quick Reference