A practical baseline for publisher servers

Servers for content websites rarely fail in exotic ways. When a site goes down, the cause is usually ordinary: a disk filled with logs, a certificate that expired, a configuration change made in a hurry, or a backup that turned out not to restore. A short baseline, applied to every server, prevents most of it.

Access

Log in with SSH keys, not passwords, and give each person their own account so changes can be traced. Disable direct root login, restrict SSH to the addresses that need it where you can, and remove access when someone leaves or a contractor finishes. Keep a written list of who has access to what.

Updates

Turn on automatic security updates for the operating system (unattended-upgrades on Debian and Ubuntu) and set a monthly window for everything else, including the web server, PHP and database versions. Know when your distribution and PHP version reach end of life, and plan the upgrade before that date rather than after it.

Caching in front of the application

Most visitors to a content site are not logged in and see the same pages. Serving those pages from a cache, whether a CDN, Nginx’s built-in cache or Varnish, means the application and database only do work when something changes. Even a cache lifetime of one or two minutes removes most of the load during a traffic spike. A minimal Nginx setup for a PHP site looks like this:

# http block
fastcgi_cache_path /var/cache/nginx/pages levels=1:2 keys_zone=pages:100m
                   max_size=10g inactive=60m use_temp_path=off;
fastcgi_cache_key "$scheme$request_method$host$request_uri";

# inside the PHP location block
fastcgi_cache pages;
fastcgi_cache_valid 200 301 2m;
fastcgi_cache_use_stale error timeout updating http_500 http_503;
fastcgi_cache_background_update on;
fastcgi_cache_lock on;
fastcgi_cache_bypass $cookie_session_id;   # use your login cookie name
fastcgi_no_cache $cookie_session_id;
add_header X-Cache $upstream_cache_status;

Static files such as images, CSS and JavaScript should have long cache lifetimes and versioned file names, so that a change is picked up immediately without waiting for caches to expire.

Backups you have restored

Keep backups off the server, ideally with a second provider, and keep several generations so a problem noticed late can still be undone. Back up databases with a consistent dump or snapshot rather than by copying live files. Most importantly, restore a backup to a separate machine on a schedule and check that the site works. A backup that has never been restored is an assumption, not a backup.

Monitoring that catches problems early

An uptime check tells you when a site is already down. Add alerts for the things that cause downtime: disk usage above 80 per cent, memory pressure and swapping, a rising error rate in the web server logs, database replication lag, and TLS certificates within 14 days of expiry. Send alerts somewhere a person will see them, and keep the list short enough that nobody learns to ignore it.

Changes with a way back

Before changing a production server, write down what will change, how you will know it worked and how to undo it. Take a dated copy of every configuration file before editing it, or keep configuration in version control. Make one change at a time, check the site after each, and avoid changes late in the day before a weekend. Most incidents that last hours rather than minutes are ones where nobody knew exactly what had changed.