Hosting Infrastructure Insights This Week

Production Lessons from Recent Outages

Last week a major AWS availability zone suffered partial network loss. Teams running single-region EC2 clusters took hours to recover. Those with pre-provisioned standby capacity in a second zone restored traffic in under ten minutes using simple Terraform apply commands. The key difference came down to having runbooks that accounted for partial failures rather than full region loss. In practice this meant keeping a small set of warm instances idle but ready instead of relying solely on auto-scaling groups that needed time to spin up new nodes from scratch.

One team I know avoided the worst by maintaining a secondary Postgres replica on Hetzner dedicated hardware. When the primary zone went dark they promoted the replica and updated DNS records within fifteen minutes. The lesson here is simple: test your failover paths quarterly and keep at least one copy of critical data outside the primary cloud provider’s ecosystem.

Scaling with Kubernetes on Bare Metal

DigitalOcean’s new managed Kubernetes offering still trails self-managed setups for latency-sensitive workloads. One production cluster moved to bare-metal nodes with Calico networking cut tail latency by 40 percent. The trade-off is manual node patching, but the performance gain justified the extra ops work. We used Ansible playbooks to handle kernel updates across the fleet without downtime by draining nodes one at a time.

Bare metal also gave us direct control over network interfaces. Instead of fighting overlay network overhead we bonded two 10 Gbps links per node and passed traffic straight through. Container runtime remained Docker with a custom CNI plugin tuned for our specific traffic patterns. Results showed consistent sub-millisecond pod-to-pod communication that managed services could not match at the same price point.

Database Hosting Choices

PostgreSQL on managed services continues to surprise on cost. A mid-size workload migrated from AWS RDS to self-hosted Postgres on Hetzner dedicated servers dropped monthly bills by roughly half. Connection pooling via PgBouncer proved essential once query volume climbed past 5,000 per second. We also added read replicas across two physical locations using streaming replication with a simple WAL archiving script written in Bash.

Backups moved to daily full dumps plus continuous archiving to an S3-compatible bucket on Linode. Recovery testing showed we could restore a 200 GB database to a new server in under forty minutes. The migration required careful schema review and index optimization but paid for itself within the first billing cycle.

CDN and Edge Delivery Tuning

Cloudflare Workers handling origin shielding reduced cache-miss costs noticeably this week. One site serving 2 million daily requests cut origin bandwidth by 65 percent after adding a simple Worker that rewrites cache-control headers based on request path. We layered this on top of existing Nginx configuration that already handled compression and Brotli encoding at the origin.

Edge rules also allowed us to block bad bots early using IP reputation lists updated hourly. The combination kept our origin servers focused on dynamic content while static assets stayed at the edge. Latency for users in Europe dropped by an average of 120 milliseconds after the changes went live.

Cost Controls on Compute

Spot instance interruptions hit harder than expected on mixed workloads. Switching to reserved capacity for the baseline and keeping only stateless jobs on spot saved money without the 3 a.m. pager alerts. Monitoring via Prometheus and simple alert rules caught the pattern early. We set up node affinity rules in Kubernetes so critical pods never landed on spot capacity.

Additional savings came from right-sizing instances every quarter. Tools like kubectl top and custom scripts that exported resource usage to Grafana dashboards made it easy to spot over-provisioned nodes. One cluster reduced its footprint from twenty c5.2xlarge instances to twelve c5.xlarge instances after three weeks of observation.

Effective Monitoring Strategies

Prometheus with Thanos sidecar storage handled our growing metrics volume this month. Instead of paying for long-term storage in managed services we shipped data to an object store on Vultr and queried it on demand. Alertmanager rules stayed simple: focus on symptoms rather than every possible cause. This kept the on-call rotation manageable even as the number of services grew.

Log aggregation moved to a self-hosted Loki stack running on dedicated nodes. Retention stayed at thirty days for most workloads and ninety days for audit logs. The setup integrated cleanly with existing Grafana dashboards so the team did not need to learn new query languages.

This is Allan Ali for Sylt.ing.