Server Management Tips from the Trenches

Getting Monitoring Right from Day One

Skip the fancy dashboards at first. Set up Prometheus with Node Exporter on every box and point it at a simple Grafana instance. Last quarter we spotted a slow disk on a Hetzner dedicated server three days before it failed, because basic metrics were already in place. CPU steal time, disk I/O wait, and memory pressure tell you far more than uptime alone. We also forward syslog to a central Loki instance so queries across ten servers take seconds instead of manual SSH hops. Alerts go to a single PagerDuty rotation, not scattered email lists. Without these basics, you spend outages guessing instead of knowing exactly where the bottleneck sits.

Backup Strategies That Actually Work

Do not rely on provider snapshots alone. We run daily Borg backups to an offsite rsync.net target plus weekly full dumps of PostgreSQL with pg_dump. A few weeks back a RAID controller died on one of our Linode boxes; the Borg restore had us back online in under ninety minutes. Test restores quarterly on a spare machine. Keep at least three copies across two geographic regions and verify checksums on every transfer. Object storage from Backblaze B2 works well for cold archives when you encrypt with age before upload. Never assume the hosting provider will recover your data faster than you can.

Hardening Without Slowing Everything Down

Lock down SSH with key-only access and fail2ban, then run unattended-upgrades on Debian systems. We also pin UFW rules to the minimum ports needed. After a routine audit last month we found an old Redis port left open on a staging host; closing it took five minutes and removed the only real exposure. Use AppArmor profiles for Nginx and PostgreSQL. Rotate secrets with Vault instead of hard-coding them in scripts. Kernel parameters like net.ipv4.tcp_syncookies and vm.swappiness get tuned once and committed to /etc/sysctl.conf. These steps add negligible latency yet close the gaps that matter most in production.

Automation Beats Manual Fixes

Anything you touch more than twice gets an Ansible playbook. We keep playbooks for Nginx config, Certbot renewals, and kernel updates. When we migrated eight servers from Ubuntu 20.04 to 22.04 two months ago, the entire fleet was done in one evening with zero downtime. GitOps via Terraform handles new VPS provisioning on DigitalOcean and Hetzner alike. Idempotent playbooks mean you can rerun them safely after a partial failure. Manual steps belong only in the runbook for true emergencies; everything else must be scripted and version-controlled.

Scaling on Real Hardware Limits

Vertical scaling hits a wall fast. We moved our heaviest PostgreSQL workload off a single 32 GB DigitalOcean droplet onto two smaller Hetzner AX41 boxes with Patroni for failover. Throughput went up and monthly cost dropped by 18 percent. Watch for NUMA effects on larger AMD EPYC hosts and pin PostgreSQL processes accordingly. Load testing with realistic traffic via Locust reveals when you actually need another node. Never guess at headroom; measure connection counts, query latency, and disk queue depth under peak load.

Handling Real Outages

Keep a printed runbook. When a core switch failed at our colocation last year, the printed steps for failing over to the backup site got us live again before most customers noticed. Cloud dashboards are useless when the network is down. Post-incident reviews happen within 24 hours and focus on what changed, not who was on call. Document every manual intervention so it becomes the next Ansible task. Outages teach more than any monitoring graph ever will.

Capacity Planning Before It Bites You

Review disk growth weekly and project six months ahead. We once let a logging partition fill on a production Nginx box because weekly checks had slipped. The fix required an emergency resize during business hours. Track both allocated and actual usage across CPU, RAM, and storage. Tools like Prometheus plus simple shell scripts catch trends early. Plan hardware refreshes on a two-year cycle for dedicated servers rather than reacting to failures. This approach keeps surprises to a minimum and budgets predictable.

This is Allan Ali for Sylt.ing.