Security Hardening

Date: April 2026  |  Environment: Ubuntu 24.04 LTS on VPS (Exabytes)  |  Stack: Strapi, MySQL, Metabase, Frontend (nginx), Cloudflare Tunnel

---

Background

In April 2026 the production server was compromised via a malicious npm package introduced during a large dependency upgrade (strapi 5.11 → 5.41, vuetify 3 → 4). The malicious package exploited the fact that the ubuntu user was a member of the docker group, which gave it full root access to the host. This page documents the security measures taken after the incident and the reasoning behind each decision.

Root cause

Running yarn install directly on the production server with 25,000+ npm packages is an enormous attack surface. Any one of those packages can execute a postinstall script automatically — and if the process user owns the Docker socket, it has root. Production servers must never run package managers.

This incident directly motivated the new deployment approach where images are built on staging and only the finished artefact reaches production.

---

What Went Wrong

1. Docker Ports Exposed to the Internet

When Docker publishes a port like this:

ports:
  - "3306:3306"

Docker binds to all network interfaces (0.0.0.0), including the public internet-facing interface. Critically, this bypasses ufw because Docker injects its own iptables rules that take effect before the firewall.

Services that were publicly accessible before hardening:

Service	Port	Risk
MySQL	3306	Direct database access
Adminer	8080	Web-based database management
Strapi	1337	API and admin panel
Metabase	3003	Analytics dashboard

2. The ubuntu User Was in the Docker Group

Membership in the docker group is effectively equivalent to root access. Any process running as ubuntu can do:

docker run --rm -v /:/host alpine chroot /host bash

This mounts the entire host filesystem inside a container — no password required. When yarn install ran the malicious postinstall script as ubuntu, it used exactly this technique to install a rootkit.

3. Running yarn install on the Production Server

With 25,000+ packages being installed, any single one of them can run arbitrary code via postinstall. Running this on a server that holds live database credentials and customer data is an unacceptable risk regardless of other controls.

---

Security Measures Implemented

Measure 1 — Docker Network Architecture

The most significant architectural change separates all application containers into an internal-only Docker network that has no internet access. Only the Cloudflare tunnel container sits on both networks.

---
title: Docker Network Architecture
config:
    look: handDrawn
---
flowchart TD

classDef internal fill:#A5D6A7,stroke:#43A047,stroke-width:2px
classDef external fill:#BBDEFB,stroke:#1E88E5,stroke-width:2px
classDef internet fill:#FFCDD2,stroke:#E53935,stroke-width:2px

INTERNET["🌐 Internet"]:::internet
CF["Cloudflare Edge<br/>(global CDN)"]:::external
TUNNEL["cloudflare_tunnel container<br/>(tunnel + frontend networks)"]:::external

subgraph FRONTEND_NET ["frontend network — internal, no internet"]
  direction LR
  STRAPI["strapi<br/>expose: 1337"]:::internal
  NGINX["frontend/nginx<br/>expose: 80"]:::internal
  MYSQL["mysql-db<br/>expose: 3306"]:::internal
  ADMINER["adminer<br/>127.0.0.1:8080"]:::internal
  METABASE["metabase<br/>expose: 3000"]:::internal
end

subgraph TUNNEL_NET ["tunnel network — internet-facing"]
  TUNNEL
end

INTERNET -->|HTTPS| CF
CF -->|Cloudflare tunnel| TUNNEL
TUNNEL -->|frontend network| STRAPI & NGINX & METABASE

style FRONTEND_NET fill:#E8F5E9,stroke:#43A047,stroke-width:2px
style TUNNEL_NET fill:#E3F2FD,stroke:#1E88E5,stroke-width:2px

frontend network — created with --internal:

• No outbound internet access for any container
• All app services (Strapi, MySQL, Metabase, nginx) live here
• Prevents compromised containers from phoning home or downloading payloads

tunnel network — internet-facing:

• Only the cloudflare_tunnel container is attached
• Has a manual iptables MASQUERADE rule for its subnet
• All other containers are explicitly excluded

How to verify:

docker run --rm --network frontend alpine \
  sh -c "wget -q -O- http://cloudflare.com && echo 'BAD' || echo 'GOOD — internet blocked'"

Measure 2 — Ports: expose Instead of ports

All application services use expose: instead of ports:. This makes the port reachable only within Docker networks — not from the host, and certainly not from the internet.

Service	Before	After
MySQL	"3306:3306" (public)	expose: ["3306"] (Docker-internal only)
Adminer	"8080:8080" (public)	"127.0.0.1:8080:8080" (localhost → Tailscale only)
Strapi	"1337:1337" (public)	expose: ["1337"]
Frontend	"8090:80" (public)	expose: ["80"]
Metabase	"3003:3000" (public)	expose: ["3000"]
Piwigo	"80:80" (public)	expose: ["80"]

Adminer remains reachable via Tailscale because tailscale serve --bg 8080 proxies from the Tailscale IP to localhost:8080.

How to verify:

sudo ss -tnlp | grep docker-proxy
# All entries should show 127.0.0.1:PORT — none should show 0.0.0.0:PORT

Measure 3 — Docker Daemon daemon.json

A server-wide default prevents any port binding from accidentally reaching the public internet:

{
  "ip": "127.0.0.1",
  "iptables": true,
  "ip-masq": false,
  "bip": "10.20.29.1/24",
  "default-address-pools": [
    {"base": "10.20.0.0/16", "size": 24}
  ]
}

Setting	Effect
"ip": "127.0.0.1"	Default bind address for all port bindings — safety net if a compose file is missing the explicit 127.0.0.1: prefix
"ip-masq": false	Disables automatic NAT masquerading — containers cannot reach the internet unless a manual iptables rule is added
"bip"	Custom IP range for the default bridge — less predictable than the default 172.17.x.x
"default-address-pools"	Custom IP ranges for all Docker networks

The tunnel network gets a manual masquerade rule (added by Ansible) so only the Cloudflare container can reach the internet:

iptables -t nat -A POSTROUTING -s 10.20.31.0/24 -j MASQUERADE
netfilter-persistent save

Measure 4 — ubuntu Removed from Docker Group

The ubuntu user has been removed from the docker group. Docker commands now require sudo:

sudo docker ps
sudo docker compose up -d

In Ansible all community.docker.* tasks and any shell task invoking docker now carry become: true to run as root.

How to verify:

groups ubuntu | grep docker
# Should return nothing

Operational impact

sudo is now required for all docker commands when logged in to the server. Ansible handles this automatically via become: true. Scripts and cron jobs that call docker directly must also be updated to use sudo.

Measure 5 — Passwords Rotated

All database credentials that were publicly accessible while MySQL was exposed on port 3306 have been rotated. New credentials are stored exclusively in Ansible Vault (inventory/group_vars/klhhh/vault.yml) and never appear in any file committed to git.

Changing MySQL passwords

MYSQL_ROOT_PASSWORD in docker-compose.yml is only used when the container is first created. Changing it in the compose file does not change the password in an existing database. Always change the password inside MySQL first:

ALTER USER 'root'@'%' IDENTIFIED BY 'NEW_ROOT_PASSWORD';
ALTER USER 'your_app_user'@'%' IDENTIFIED BY 'NEW_APP_PASSWORD';
FLUSH PRIVILEGES;

Measure 6 — No More Builds on Production

The root cause of the incident — running yarn install on production — is structurally eliminated. Production servers now only receive finished Docker images. All compilation, package installation, and building happens on the isolated staging server.

See Deployment Process for the full pipeline.

---

Security Verification Checklist

Run these checks periodically to verify the server remains properly hardened:

# 1. No ports publicly exposed (all should show 127.0.0.1:PORT)
sudo ss -tnlp | grep docker-proxy

# 2. Internal network cannot reach internet
docker run --rm --network frontend alpine \
  sh -c "wget -q -O- http://cloudflare.com && echo 'BAD — internet reachable' || echo 'GOOD — internet blocked'"

# 3. ubuntu is not in docker group (should return nothing)
groups ubuntu | grep docker

# 4. Check for rootkit indicators
sudo stat /etc/ld.so.preload 2>/dev/null && echo "WARNING: file exists" || echo "Clean"
sudo crontab -l 2>/dev/null | grep -v "^#" | grep -v "^$" \
  && echo "WARNING: root has cron jobs" || echo "Clean"

# 5. Docker iptables rules are in place
sudo iptables -t nat -L POSTROUTING -n | grep MASQUERADE

---

Pending Improvements

The following measures are recommended for a future release:

• SSH restricted to Tailscale interface only — edit /etc/ssh/sshd_config so sshd only listens on the Tailscale IP, not the public interface
• Automatic security updates — unattended-upgrades for automatic security patches
• Cloudflare WAF rules — rate limiting on public API endpoints
• Remove Adminer from production — use docker exec for emergency database access instead of an always-running web UI