Standard Streams and File Descriptors

9.1. Standard Streams and File Descriptors#

9.1.1. Common Pitfalls#

1. Confusing descriptor numbers (0=stdin, 1=stdout, 2=stderr)

# Wrong: trying to redirect stdin
command > 0 /dev/null  # Invalid

# Right: redirect stdout to null
command > /dev/null

# Right: redirect stderr to null
command 2> /dev/null

2. Forgetting that > overwrites files

# Dangerous: this overwrites the log!
echo "Line 1" > log.txt
echo "Line 2" > log.txt  # Line 1 is gone!

# Better: use append
echo "Line 1" > log.txt
echo "Line 2" >> log.txt

3. Losing stderr by redirecting only stdout

# command_with_error runs, error goes to console
command_with_error > /tmp/output.txt

# Better: also redirect stderr
command_with_error > /tmp/output.txt 2> /tmp/errors.txt

# Or combine them
command_with_error > /tmp/output.txt 2>&1

4. Order matters in redirections

# Wrong: 2>&1 must come after >
echo "test" 2>&1 > file.txt  # stderr goes to console!

# Right: specify in correct order
echo "test" > file.txt 2>&1  # Both go to file

9.1.2. Real-World Example: Logging with Multiple Streams#

#!/bin/bash

LOG_FILE="/var/log/app.log"
ERROR_LOG="/var/log/app.err"

# Function to log messages
log_message() {
  local msg="$1"
  local timestamp=$(date '+%Y-%m-%d %H:%M:%S')
  
  # Log to file and console
  echo "[$timestamp] $msg" | tee -a "$LOG_FILE"
}

# Function to log errors
log_error() {
  local msg="$1"
  local timestamp=$(date '+%Y-%m-%d %H:%M:%S')
  
  # Log to stderr and error file
  echo "[$timestamp] ERROR: $msg" | tee -a "$ERROR_LOG" >&2
}

# Example usage
log_message "Application started"
log_error "Failed to connect to database"

# Process file while logging progress
while IFS= read -r line; do
  log_message "Processing: $line"
done < input_file.txt

This pattern separates stdout logging (via tee) from error reporting.

9.1.3. Here Documents and Here Strings#

9.1.3.1. Here Documents (<<)#

A here document sends multiple lines to a command’s stdin:

#!/bin/bash

# Basic here document
cat << EOF
This is a here document.
It can span multiple lines.
Variables are expanded: $USER
EOF

# Prevent variable expansion (quote the delimiter)
cat << 'EOF'
Variables are NOT expanded: $USER
EOF

# Use - to indent the closing delimiter
cat <<- EOF
    This content is indented
    The - allows indentation of EOF
EOF

9.1.3.2. Here Strings (<<<)#

Send a single string to stdin:

#!/bin/bash

# Here string
read var <<< "Hello from here string"
echo $var  # Output: Hello from here string

# Process a string with a filter
grep "bash" <<< "This is a bash script"

# Calculate with bc
result=$(bc <<< "2 + 2")
echo $result  # Output: 4

9.1.4. File Descriptors and Advanced Redirection#

File descriptors are numeric references to open files/streams. You can work with any descriptor 0-9 (or higher in some shells).

9.1.4.1. Using Custom File Descriptors#

#!/bin/bash

# Open a file for reading on descriptor 3
exec 3< /etc/passwd

# Read from descriptor 3
read -u 3 first_line
echo "First line of /etc/passwd: $first_line"

# Close descriptor 3
exec 3<-

# Open a file for writing on descriptor 4
exec 4> /tmp/output.txt
echo "Line written to descriptor 4" >&4
exec 4>-

9.1.4.2. Duplicating File Descriptors#

#!/bin/bash

# Save stdout to descriptor 3
exec 3>&1

# Redirect stdout to a file
exec 1> /tmp/output.txt

echo "This goes to /tmp/output.txt"

# Restore stdout from descriptor 3
exec 1>&3

echo "This goes to the terminal"

# Close descriptor 3
exec 3>&-

9.1.5. Redirecting Output#

9.1.5.1. Redirect to File#

# Overwrite file (>)
echo "First line" > /tmp/log.txt
echo "Second line" > /tmp/log.txt  # This overwrites!

# Append to file (>>)
echo "First line" > /tmp/log.txt
echo "Second line" >> /tmp/log.txt  # This appends

# Result:
# First line
# Second line

9.1.5.2. Redirect stderr Separately#

# Send success to file, errors to console
ls /tmp /nonexistent > /tmp/success.txt

# Send errors to file, success to console
ls /tmp /nonexistent 2> /tmp/errors.txt

# Send both to different files
ls /tmp /nonexistent > /tmp/success.txt 2> /tmp/errors.txt

# Send both to same file
ls /tmp /nonexistent > /tmp/all.txt 2>&1
# Or in newer bash:
ls /tmp /nonexistent &> /tmp/all.txt

9.1.5.3. Suppress Output#

# Discard stdout
ls /tmp > /dev/null

# Discard stderr
ls /nonexistent 2> /dev/null

# Discard both
ls /tmp /nonexistent > /dev/null 2>&1

9.1.6. The Three Standard Streams#

Every Unix process has three standard I/O streams:

Stream

Descriptor

Default

Purpose

stdin

0

keyboard

Input to the program

stdout

1

terminal

Normal output from the program

stderr

2

terminal

Error messages from the program

9.1.6.1. Using stdin (file descriptor 0)#

# Read from stdin (waits for user input)
read -p "Enter your name: " name
echo "Hello, $name"

# Read from a file instead of keyboard
read variable < /path/to/file

# Read from a pipe
echo "some data" | read data

9.1.6.2. Using stdout (file descriptor 1)#

# Normal output (default)
echo "This goes to stdout"
ls /tmp

# Explicit redirect to stdout
echo "Message" 1> /tmp/output.txt

# Append to file
echo "Line 2" 1>> /tmp/output.txt

9.1.6.3. Using stderr (file descriptor 2)#

# Error output goes to stderr
ls /nonexistent 2> /tmp/errors.txt

# Both stdout and stderr to same file
ls /tmp /nonexistent &> /tmp/all.txt

# Or use 2>&1 syntax
ls /tmp /nonexistent > /tmp/all.txt 2>&1
Contents