Bash Power: Automating Log Rotation and Batch Processing
In the early months of 2001, managing a Linux server (likely running Kernel 2.2 or the new 2.4) means dealing with a lot of text. Apache logs, mail logs, and system messages are constantly filling up our meager 10GB or 20GB hard drives. If /var hits 100%, the whole system grinds to a halt.
While logrotate exists, sometimes you need a custom solution to process logs before archiving them. Let's build a script that finds the "Top 10 IP Addresses" in an Apache access_log, mails the report to the admin, and then compresses the log.
The Analysis Script
We'll use the classic Unix philosophy: small tools connected by pipes. awk, sort, uniq, and head are our primary weapons.
#!/bin/bash
# log_processor.sh - 2001 style
LOG_FILE="/usr/local/apache/logs/access_log"
REPORT_FILE="/tmp/daily_report.txt"
ADMIN_EMAIL="admin@example.com"
echo "Daily Access Report for $(date)" > $REPORT_FILE
echo "--------------------------------" >> $REPORT_FILE
echo "Top 10 IPs by request count:" >> $REPORT_FILE
# The Magic Pipe:
# 1. Get the first column (IP)
# 2. Sort them so uniq can count them
# 3. Count unique occurrences
# 4. Sort numerically in reverse
# 5. Take the top 10
awk '{print $1}' $LOG_FILE | sort | uniq -c | sort -nr | head -n 10 >> $REPORT_FILE
# Send the mail using the 'mail' command
mail -s "Daily Log Report" $ADMIN_EMAIL < $REPORT_FILE
# Log Rotation: Move and Compress
mv $LOG_FILE $LOG_FILE.1
kill -HUP $(cat /usr/local/apache/logs/httpd.pid)
gzip $LOG_FILE.1
rm $REPORT_FILE
Understanding the -HUP Signal
In 2001, we don't have systemctl. To tell Apache to start writing to a new log file after we've moved the old one, we send a SIGHUP (Signal Hang Up). This tells the process to reload its configuration and reopen its log files without dropping current connections.
Batch Processing with Loops
If you have a directory full of legacy logs to process, a for loop is your best friend.
#!/bin/bash
# Batch process all .log files in a directory
for f in /var/logs/legacy/*.log; do
echo "Processing $f..."
# Search for 404 errors and count them
COUNT=$(grep " 404 " $f | wc -l)
echo "$f had $COUNT errors." >> /var/log/error_summary.txt
done
Bash scripting is the glue that holds the internet together in 2001. Mastering these text-processing pipelines is what separates the junior operators from the senior sysadmins.
Aunimeda provides DevOps engineering and infrastructure services - CI/CD pipelines, containerization, cloud deployments, and monitoring setups.
Contact us to discuss your infrastructure needs. See also: DevOps Services, Custom Software Development