Docker Production Pitfalls We Learned the Hard Way

Lessons from Switching to Multi-Stage Dockerfiles

Eighteen months ago we took over a client infrastructure on DigitalOcean that relied on single-stage Dockerfiles for every service. Build pipelines dragged on for forty-five minutes while final images exceeded 1.2 GB because every build tool and dependency stayed in the runtime layer. We rewrote the files using multi-stage builds with explicit FROM statements for compilation and final COPY —from= stages pulling only the compiled binaries onto Alpine bases. Image sizes dropped to 180 MB and builds finished in under eight minutes. The change also eliminated a class of security issues since build-time secrets never reached production hosts. Since then every new service at the same client starts with a two-stage template that separates build and runtime concerns.

We repeated the pattern across an Nginx reverse proxy and a Redis cache layer. The smaller images meant faster pulls during rolling updates and lower storage costs on the registry. Teams that skip this step continue to ship bloated containers that take minutes to start and waste bandwidth on every deploy.

Networking Breaks When You Ignore Host Modes

Last quarter an application spread across three AWS availability zones revealed the fragility of default bridge networking. Service discovery through Consul worked until a routine host reboot triggered IP address churn that left the Traefik load balancer with stale backends. Connections reset randomly and latency spiked. We moved the reverse proxy tier to host networking mode and pinned critical ports with static mappings. Average latency fell thirty milliseconds and the ghost resets disappeared. Host mode is not a blanket solution but it remains the practical choice for edge proxies that must survive node failures without extra service mesh overhead.

We also learned to avoid overlay networks for anything latency sensitive. When a payment processing service needed sub-ten-millisecond responses we placed it on the same host network segment as its database instead of relying on Docker overlay routing. The configuration has stayed stable through multiple kernel upgrades.

Volumes and Data Persistence Still Trip Teams Up

A production Postgres container lost twelve hours of transaction data after an unplanned host reboot because the volume driver defaulted to local instead of an EBS-backed driver. The container restarted with an empty data directory. We now declare every stateful workload with named volumes and explicit driver options. Automated restic jobs run every four hours and ship snapshots to an offsite S3 bucket. Recovery tests happen monthly so the team knows exactly how long a restore takes.

The same rule applies to message queues. A client using RabbitMQ on Docker volumes without backups discovered the hard way that container recreation wipes the queue state. We switched those workloads to managed persistence outside the container and treat the Docker layer as ephemeral. No production volume should ever hold data you cannot rebuild from source in under ten minutes.

Resource Limits Prevent the 3 a.m. Pager

Without memory limits an errant Java process inside a container once consumed the entire 32 GB of a bare-metal host. The machine became unresponsive and required a physical power cycle. We introduced --memory and --cpus flags on every docker run command plus adjusted OOM scores so containers die before the host does. After these limits went live we recorded zero host-level out-of-memory events across six months. Grafana dashboards now track container memory usage against the declared limits so teams receive alerts before any single process can starve the node.

CPU limits also matter for noisy neighbors. A batch job that previously stole cycles from a web tier now runs inside a constrained container with a 2.0 CPU cap. The web tier response times stabilized and we stopped seeing correlated slowdowns during nightly reports.

Logging to Stdout Is Not Enough at Scale

Directing every container to stdout worked for the first year. Then a misconfigured service emitted forty gigabytes of logs in a single night and filled the root disk on three production hosts. We adopted a sidecar pattern that ships logs through Fluentd to a central Loki instance. Disk usage on application hosts dropped to a few hundred megabytes and searchability improved because logs carry structured labels instead of raw text. The sidecar containers run with their own resource limits so a logging spike cannot affect the main workload.

We also rotate logs at the Docker daemon level and forward only error-level events to PagerDuty. The combination keeps noise low while preserving the ability to debug production incidents quickly. Teams that treat stdout as the final destination eventually face the same disk-full outage we experienced.

This is Allan Ali for Sylt.ing.