Secure File Transfer: SCP and Rsync

13.2. Secure File Transfer: SCP and Rsync#

13.2.1. Common Pitfalls#

1. Trailing slashes matter in rsync

# Different behavior!
# With slash: copies contents of /src into /dst
rsync -avz /src/ /dst/

# Without slash: copies /src directory into /dst (creates /dst/src)
rsync -avz /src /dst/

# Be explicit:
# Source with /, destination with / = merge contents
rsync -avz /src/ /dst/

2. Not using –delete when expecting mirror behavior

# Bad: Old files remain on remote
rsync -avz /src/ user@remote:/dst/

# Good: Remote becomes exact mirror of source
rsync -avz --delete /src/ user@remote:/dst/

3. Copying sensitive files unencrypted

# Bad: Files exposed during transfer
scp /etc/shadow user@remote:/backup/

# Better: Use rsync with SSH (default)
rsync -avz /etc/shadow user@remote:/backup/

# Or use specific SSH key:
rsync -avz -e "ssh -i ~/.ssh/backup_key" /src user@remote:/dst

4. Not verifying transferred data

# Bad: Assumes transfer is complete and correct
scp large-file.iso user@remote:/backup/

# Good: Verify with checksums
rsync -avz --checksum /src user@remote:/dst
rsync -avz --checksum --dry-run /src user@remote:/dst | grep transferred && echo "Verification failed"

5. Killing transfers without cleanup

# Bad: Incomplete files left on destination
rsync -avz /src user@remote:/dst &
kill $!  # Leaves partial files

# Better: Use --partial-dir for safe resumption
rsync -avz --partial-dir=.rsync-partial /src user@remote:/dst
# Can resume later: rsync -avz /src user@remote:/dst

13.2.2. Real-World Backup Strategy#

#!/bin/bash
set -euo pipefail

# COMPREHENSIVE BACKUP WITH RSYNC
# Daily incremental + weekly full with verification

readonly BACKUP_HOST="backup.example.com"
readonly BACKUP_USER="backup"
readonly BACKUP_ROOT="/backups"
readonly SOURCE="/data"
readonly LOG_FILE="/var/log/backup.log"

backup_daily() {
  local backup_date=$(date +%Y-%m-%d-%H%M%S)
  local backup_path="$BACKUP_ROOT/daily/$backup_date"
  
  echo "[$(date)] Starting daily backup..." >> "$LOG_FILE"
  
  # Perform incremental sync
  if rsync -avz \
    --exclude='.cache' \
    --exclude='.tmp' \
    "$SOURCE/" \
    "$BACKUP_USER@$BACKUP_HOST:$backup_path/"; then
    
    echo "[$(date)] Daily backup completed: $backup_path" >> "$LOG_FILE"
    return 0
  else
    echo "[$(date)] Daily backup failed" >&2 | tee -a "$LOG_FILE"
    return 1
  fi
}

backup_weekly_full() {
  local backup_date=$(date +%Y-W%V)
  local backup_path="$BACKUP_ROOT/weekly/$backup_date"
  
  echo "[$(date)] Starting weekly full backup..." >> "$LOG_FILE"
  
  # Full backup with verification
  rsync -avz --checksum \
    "$SOURCE/" \
    "$BACKUP_USER@$BACKUP_HOST:$backup_path/" || {
    echo "[$(date)] Weekly backup failed" >&2 | tee -a "$LOG_FILE"
    return 1
  }
  
  # Verify backup
  if rsync -avz --checksum --dry-run \
    "$SOURCE/" \
    "$BACKUP_USER@$BACKUP_HOST:$backup_path/" | \
    grep -q 'would be transferred'; then
    
    echo "[$(date)] Verification failed" >&2 | tee -a "$LOG_FILE"
    return 1
  fi
  
  echo "[$(date)] Weekly backup verified: $backup_path" >> "$LOG_FILE"
}

cleanup_old_backups() {
  local retention_days=30
  
  echo "[$(date)] Cleaning backups older than $retention_days days..." >> "$LOG_FILE"
  
  ssh "$BACKUP_USER@$BACKUP_HOST" \
    "find $BACKUP_ROOT/daily -type d -mtime +$retention_days -exec rm -rf {} \;"
}

# Main execution
case "${1:-daily}" in
  daily) backup_daily;;
  weekly) backup_weekly_full; cleanup_old_backups;;
  *) echo "Usage: $0 {daily|weekly}"; exit 1;;
esac

13.2.3. Efficient Synchronization with Rsync#

Rsync is superior to SCP for synchronization: it only transfers changed files, supports incremental backups, and handles deletions.

13.2.3.1. Basic Rsync Usage#

#!/bin/bash

# Sync local to remote
rsync -avz /path/to/local/ user@remote.com:/path/to/remote/
# -a: archive mode (preserves permissions, timestamps, etc.)
# -v: verbose
# -z: compress during transfer

# Sync remote to local
rsync -avz user@remote.com:/path/to/remote/ /path/to/local/

# Sync with delete (remove files not in source)
rsync -avz --delete /path/to/local/ user@remote.com:/path/to/remote/

# Exclude patterns
rsync -avz --exclude='*.log' --exclude='.git' /src/ user@remote.com:/dst/

# Include/exclude with file
rsync -avz --filter=':- .rsyncignore' /src/ user@remote.com:/dst/

# Dry run (show what would be transferred)
rsync -avz --dry-run /src/ user@remote.com:/dst/

# Custom SSH port and key
rsync -avz -e "ssh -p 2222 -i ~/.ssh/key" /src/ user@remote.com:/dst/

13.2.3.2. Advanced Rsync Patterns#

#!/bin/bash

# Incremental backups with snapshots
rsync_backup() {
  local source=$1
  local destination=$2
  local backup_dir="${destination}/backup-$(date +%Y%m%d-%H%M%S)"
  
  # Create new backup directory with hardlinks to previous
  if [[ -d "$destination/latest" ]]; then
    cp -al "$destination/latest" "$backup_dir"
  else
    mkdir -p "$backup_dir"
  fi
  
  # Sync with delete
  rsync -avz --delete "$source" "$backup_dir/"
  
  # Update latest link
  ln -sfn "$backup_dir" "$destination/latest"
}

# Bandwidth-limited sync
rsync_limited() {
  local source=$1
  local destination=$2
  
  rsync -avz --bwlimit=10000 "$source" "$destination"
  # Limit to 10 MB/s
}

# Parallel rsync for multiple directories
rsync_parallel() {
  local source_dir=$1
  local dest=$2
  
  # Rsync each subdirectory in parallel
  find "$source_dir" -maxdepth 1 -type d ! -name '.' | \
    xargs -P 4 -I {} rsync -avz {} "$dest"
}

# Verify transfer with checksums
rsync_verify() {
  local source=$1
  local destination=$2
  
  # Transfer with checksums
  rsync -avz --checksum "$source" "$destination"
  
  # Verify by comparing
  rsync -avz --checksum --dry-run "$source" "$destination" | \
    grep 'would be transferred' && {
      echo "Verification failed: files differ"
      return 1
    }
  echo "✓ Verification passed"
}

13.2.4. Secure Copy with SCP#

SCP (Secure Copy) transfers files over SSH, providing secure file transmission.

13.2.4.1. Basic SCP Operations#

#!/bin/bash

# Copy file from local to remote
scp /path/to/local/file user@remote.com:/path/to/remote/

# Copy file from remote to local
scp user@remote.com:/path/to/remote/file /path/to/local/

# Copy entire directory (recursive)
scp -r /path/to/local/dir/ user@remote.com:/path/to/remote/
scp -r user@remote.com:/path/to/remote/dir/ /path/to/local/

# Use specific SSH port
scp -P 2222 /path/to/file user@remote.com:/path/to/

# Copy between two remote hosts
scp -3 user1@remote1.com:/file user2@remote2.com:/path/

# Preserve file attributes (timestamps, permissions)
scp -p /path/to/file user@remote.com:/path/to/

13.2.4.2. SCP with Options#

#!/bin/bash

# Limit bandwidth (in kbps)
scp -l 1024 /path/to/file user@remote.com:/path/to/

# Use specific SSH key
scp -i ~/.ssh/custom_key /path/to/file user@remote.com:/path/to/

# Verbose output
scp -v /path/to/file user@remote.com:/path/to/

# Batch copy from list
while IFS= read -r file; do
  scp "$file" user@remote.com:/backup/
done < files.txt

# Transfer with progress indicator and error handling
secure_copy() {
  local source=$1
  local destination=$2
  
  if scp -p "$source" "$destination"; then
    echo "✓ Transfer successful: $source"
    return 0
  else
    echo "✗ Transfer failed: $source" >&2
    return 1
  fi
}

secure_copy "/etc/config" "user@remote.com:/backup/config"
Contents