Securing Your Production Servers in Practice

Assessing Your Current Setup

Last month I walked into a client environment running on DigitalOcean droplets where SSH allowed root login with password auth. That single misstep had already drawn repeated brute force attempts logged in auth.log. The fix started with a full inventory of open ports and running services before touching anything else. I used netstat and nmap to map everything from the host level out to the AWS EC2 security groups. Too many teams skip this step and end up patching the wrong holes while leaving others wide open. In another case two weeks ago on an Ubuntu 22.04 box the nginx process was listening on an unexpected high port because of a leftover development container. Documenting every listening socket and every cron job saved hours of later troubleshooting.

Locking Down Remote Access

Disable root login entirely in sshd_config and switch to key-based authentication only. I also bind SSH to a non-standard port and restrict it by IP where possible. On one Ubuntu 22.04 fleet I added the AllowUsers directive so only specific service accounts could connect. Fail2Ban then watches the logs and blocks repeat offenders after three failed attempts. I keep the ban time at 24 hours and whitelist internal monitoring IPs so our Nagios checks never get locked out. In production I have seen password attempts drop from thousands per day to zero once key auth and Fail2Ban were combined. Never rely on obscurity alone; combine it with strict key management and rotate keys every ninety days using a simple shell script that pulls from a central vault.

Configuring Host Firewalls

ufw remains the simplest tool for most Linux hosts. Enable it, then explicitly allow only the ports your application stack needs. For an Nginx plus PostgreSQL setup I allow 80, 443, and 5432 from the application tier while denying everything else. Cloud provider security groups add another layer, but the host firewall still catches misrouted traffic inside the VPC. I also drop invalid packets and limit new connections to 20 per minute on the SSH port to blunt any remaining scans. On a recent AWS migration the combination of ufw rules and security group tightening stopped an internal port scan that would have otherwise exposed the Redis instance. Test the rules with ufw status verbose after every change and keep a rollback script ready.

Keeping Systems Patched

Unattended-upgrades handles the base OS on Debian and Ubuntu derivatives, but I still schedule a monthly maintenance window to review kernel updates and restart services. A few weeks ago an unpatched libssl vulnerability on a production Nginx box was caught only because I forced the review instead of relying solely on automatic updates. I also pin critical packages like nginx and postgresql to known-good versions in apt preferences so surprise upgrades do not break the stack. Reboot testing happens in a staging environment that mirrors production exactly, including the same kernel version. For larger fleets I use Ansible to push updates across all hosts in a rolling fashion so no more than one server is down at a time. Skipping patches because of fear of breakage is the fastest way to invite real incidents.

Adding Monitoring and Logging

Centralize logs with rsyslog forwarding to a dedicated logging server or ELK stack. I also run OSSEC for file integrity checking and active response. When a binary changed unexpectedly on a web server last quarter, the alert arrived within minutes rather than days later during the next manual audit. Set up alerts for failed logins, unusual process spawns, and disk space below twenty percent. Integrate the ELK dashboards with simple Slack webhooks so the on-call engineer sees the important events without digging through raw logs. Keep log retention at ninety days minimum for compliance and incident review. Without centralized logging you are flying blind the moment a single host is compromised.

This is Allan Ali for Sylt.ing.