Surviving Traffic Spikes Before Auto-Scaling Existed: PHP in 2010

In November 2010 a post about one of our client's apps landed on the front page of Hacker News. Within 12 minutes, the server was returning 502s. By minute 18, MySQL had stopped responding. By minute 25, the VPS itself was unreachable.

We brought it back up in 40 minutes. The traffic window lasted 3 hours. We captured maybe 15% of it.

What happened in the aftermath shaped how we architected PHP applications for the next four years.

---

What the stack looked like

Standard LAMP stack, circa 2010: Ubuntu 10.04, Apache 2.2 with mod_php, MySQL 5.1, no caching layer. Every page request hit PHP, which queried MySQL. No object caching, no opcode cache, no reverse proxy.

The server: a single VPS with 2 CPU cores and 2GB RAM. Hosting cost: $40/month.

Under normal load — 500 daily visitors — this was fine. Under a sudden 15,000 concurrent visitors, it was catastrophically undersized. But the problem wasn't just hardware.

---

What actually killed it

MySQL connections. Each Apache worker held a persistent MySQL connection. Apache's default MaxClients was 150. At 150 concurrent workers, each with an open connection, MySQL hit its default max_connections of 100 (old configs had lower defaults), and began refusing new connections. PHP responded with fatal errors. Apache logged them. Logs filled disk. Disk full. MySQL crashed. Server unresponsive.

The cascade:

Traffic spike → Apache max workers hit → MySQL connections exhausted 
→ PHP errors → error log fills disk → MySQL crashes → 502 everywhere

The hardware wasn't the bottleneck first. The architecture was.

---

The immediate fixes (deployed under fire)

In order of implementation, while the server was live:

1. opcode cache. Install APC (Alternative PHP Cache, the pre-opcache standard in 2010). PHP's interpreter compiled every .php file on every request. APC cached the compiled bytecode. Single command, immediate effect:

pecl install apc
echo "extension=apc.so" >> /etc/php5/apache2/php.ini
echo "apc.shm_size=64M" >> /etc/php5/apache2/php.ini
service apache2 restart

Result: request execution time dropped ~40%. Fewer Apache workers blocked, fewer MySQL connections opened per second.

2. Memcached in front of MySQL. Every read query that could tolerate 60 seconds of staleness got wrapped:

function getCachedQuery($key, $ttl, callable $query) {
    $memcache = new Memcache();
    $memcache->connect('127.0.0.1', 11211);
    
    $result = $memcache->get($key);
    if ($result !== false) {
        return $result;
    }
    
    $result = $query();
    $memcache->set($key, $result, 0, $ttl);
    return $result;
}

// Before
$posts = $db->query("SELECT * FROM posts ORDER BY created_at DESC LIMIT 10");

// After
$posts = getCachedQuery('homepage_posts', 60, function() use ($db) {
    return $db->query("SELECT * FROM posts ORDER BY created_at DESC LIMIT 10");
});

Homepage load: from 47 MySQL queries per request to 2.

3. Static file offload. Every image, CSS, and JS file was being served through Apache + PHP (due to a misconfigured .htaccess rewrite rule catching everything). Fix: serve static files directly from nginx, added as a reverse proxy in front of Apache.

server {
    listen 80;
    
    location ~* \.(jpg|jpeg|png|gif|css|js|ico)$ {
        root /var/www/myapp/public;
        expires 7d;
        access_log off;
    }
    
    location / {
        proxy_pass http://127.0.0.1:8080; # Apache behind nginx
        proxy_set_header Host $host;
    }
}

Apache's worker count halved immediately. It was no longer spending threads serving 40KB images.

---

What we built into every project after this

MySQL connection pooling. We switched to PDO with persistent connections and added wait_timeout monitoring. More importantly: we set Apache's MaxClients based on the database's max_connections, not hardware capacity.

# Before: 150 workers, MySQL on default 100 connections → guaranteed crash
MaxClients 150

# After: calculate from DB limit with headroom
# max_connections = 150, reserve 20 for admin/cron
MaxClients 100

Output buffering with etag validation. Pages that didn't change between requests returned 304 Not Modified instead of re-rendering. One line in the PHP bootstrap:

// If content hash matches, send 304 and exit
$etag = md5($content);
header("ETag: $etag");
if (isset($_SERVER['HTTP_IF_NONE_MATCH']) && $_SERVER['HTTP_IF_NONE_MATCH'] === $etag) {
    http_response_code(304);
    exit;
}

Separate read replicas for expensive queries. MySQL replication to a read-only slave for reporting queries, search, and archive pages. Kept the master exclusively for writes and cache-miss reads.

Log rotation before it fills the disk. After filling a disk with error logs during a spike, we treated log rotation as a first-class concern, not a sysadmin afterthought.

---

The cloud in 2010

AWS existed. EC2 had launched in 2006. But in 2010, it was not "the default" — it was an advanced option. Most production PHP apps ran on shared hosting or single VPS instances. Managed databases, auto-scaling groups, CDN edge networks — these tools existed but were expensive, complex, and designed for teams with dedicated ops engineers.

The real lesson from 2010 wasn't "use the cloud." It was: your architecture must degrade gracefully under load, not catastrophically. APC, Memcached, nginx as a reverse proxy, and connection management weren't scaling strategies — they were correctness. The app was broken under moderate load without them.

The traffic spike didn't expose a capacity problem. It exposed missing architectural fundamentals that would have caused issues at any meaningful traffic level. We just didn't know until Hacker News showed up.

Everything we reached for in 2010 under pressure — caching layers, static file separation, connection pooling — became standard defaults in every project afterward. That spike cost us 85% of a significant traffic event and cost a client real conversions. It bought us an architectural education we never had to relearn.