Docker in Production: Hard Lessons from the Trenches

Container Image Management

Two years ago I inherited a fleet where every image topped 1.2 GB. Builds took twelve minutes and deploys dragged on EC2 instances running out of disk. Switching to multi-stage builds with Alpine or Distroless cut sizes to under 80 MB. We also started pinning exact digests instead of tags. That single change stopped a surprise outage when an upstream maintainer force-pushed a tag. At a mid-size fintech client we saw similar bloat from base images pulled from Docker Hub without verification. Introducing Harbor as an internal registry with vulnerability scanning added friction at first but paid off when a critical OpenSSL CVE hit three months later. Teams now rebuild weekly on a schedule rather than on every commit. The result is smaller attack surface and faster pull times across our bare-metal hosts in two data centers.

Networking Realities on Bare Metal

Docker’s default bridge network works fine on a laptop. In production it falls apart once you need service discovery across hosts. We moved to an overlay with Calico and explicit subnet planning. One customer’s setup kept dropping packets between containers on different racks until we disabled Docker’s iptables mangling and let the host firewall handle it. Latency dropped from 4 ms to sub-millisecond. We also learned the hard way that MTU mismatches between physical switches and VXLAN overlays cause silent fragmentation. Setting MTU to 1450 across the board and testing with iperf between nodes became part of every new cluster rollout. Traefik sits in front for ingress and we keep host ports reserved only for health checks. This approach has kept network-related incidents under two per quarter for the past year.

Logging Without the Noise

JSON logging to stdout sounds clean until a single misbehaving container floods your central collector. We switched to a sidecar pattern using Fluent Bit writing directly to Loki. That gave us per-container rate limits and automatic rotation. A few months back this caught a memory leak in an Nginx container before it took down the whole node. We also forward critical logs to an on-prem Graylog instance for longer retention. The key lesson was separating application logs from Docker daemon logs early. Daemon logs now go to journald with a 500 MB cap per host. This separation saved us during a recent kernel update that spammed dmesg entries across every node. Without the split our Loki cluster would have filled overnight.

Orchestration Choices That Actually Matter

Kubernetes is the default answer yet many teams still reach for Docker Swarm or even plain systemd units. We evaluated Nomad last quarter and kept it for stateless services because the operational surface is smaller. Persistent volumes remain the pain point; we now use Longhorn for block storage instead of relying on cloud provider CSI drivers that add unpredictable latency. For a recent e-commerce migration we ran Nomad on thirty dedicated servers with Ceph-backed volumes. Failover times stayed under thirty seconds. We still use Kubernetes for teams that need complex CRDs and operators. The decision comes down to team size and tolerance for YAML. Both platforms run on the same physical hardware so we can shift workloads when needed without changing the underlying network fabric.

Security and Runtime Hardening

Running containers as root is still the fastest way to get pwned. We enforce non-root users, read-only root filesystems, and seccomp profiles generated from actual workloads. AppArmor profiles are applied at the host level and updated through Ansible. Six months ago an audit found three services still running privileged mode for GPU access. We moved those to dedicated nodes with strict device plugins and removed the flag everywhere else. Image signing with cosign is now required in our pipeline. Any unsigned image is rejected at the admission stage. These steps reduced our CIS Docker benchmark score gaps from forty-two to nine in the last assessment. The remaining items are mostly kernel tuning that we handle outside the container layer.

Resource Limits and Monitoring in Practice

Unlimited container resources will eventually exhaust the host. We set CPU and memory limits on every service and use cAdvisor plus Prometheus for collection. Alerts fire when a container approaches eighty percent of its limit for more than five minutes. Last year a Java service with no heap limit grew to 14 GB and triggered OOM kills across two nodes. After adding proper JVM flags and limits the same workload stabilized at 2.8 GB. We also reserve ten percent headroom on every host for the Docker daemon and system processes. Grafana dashboards show both container and host metrics side by side so on-call engineers can spot contention quickly. This setup has kept our average node utilization around sixty-five percent even during peak traffic events.

This is Allan Ali for Sylt.ing.