Automated Security Audit | Joshua Nowland

BACKGROUND

Why I built it

As my homelab grew, I ended up running more services, more networking, more credentials and more configuration than I wanted to manually inspect every day. I wanted a way to quickly answer a simple question: is the server still in the state I expect it to be?

Manual checking does not scale

I was already checking things such as Docker containers, firewall rules, updates, WireGuard peers and backups manually. That works at first, but it becomes easy to miss something as the environment grows.

Detection before remediation

I deliberately designed the first version to be read-only. The audit detects and reports unexpected changes, but does not automatically modify firewall rules, delete keys, restart services or remove containers.

DISCOVERY

Establishing a known-good state

Before automating anything, I manually inspected the server and recorded what normal looked like. This gave the audit something meaningful to compare against rather than relying on generic assumptions.

Firewall

UFW was verified as active with a default deny policy for inbound traffic. SSH was restricted to the home LAN and WireGuard network, while the WireGuard UDP port remained intentionally reachable.

SSH

Root SSH login was disabled, password authentication was disabled and public-key authentication was enabled. I also checked for unexpected privileged users and UID 0 accounts.

Docker exposure

I compared container port mappings against actual listening sockets to understand what was internal to Docker and what was genuinely published onto the host network.

Backups

Vaultwarden backups were checked beyond just the systemd exit status. The audit verifies that a recent database backup actually exists and is non-empty.

AUDIT CONTROLS

What the audit checks

The script classifies each result as PASS, WARN or CRITICAL and returns a matching exit code so systemd can distinguish a healthy audit from a serious security condition.

Firewall & network exposure

Checks UFW status, the default inbound policy, globally published Docker ports and listening network sockets.

Authentication

Checks SSH configuration and recent SSH authentication failures while avoiding false positives from normal local sudo password mistakes.

Privileged access

Checks sudo group membership, UID 0 accounts and unexpected NOPASSWD rules.

System health

Checks failed systemd services, disk usage, pending package updates and unhealthy Docker containers.

WireGuard

Confirms the expected peer count is still present and helps highlight unexpected changes to the VPN configuration.

Backup freshness

Checks that the latest Vaultwarden database backup is recent and has real content rather than trusting the backup job status alone.

CONFIGURATION DRIFT

Known-good baselines

The audit does not just ask whether a service is running. It also checks whether the server still looks like the version of Odysseus that I deliberately approved.

Expected containers

The currently running Docker container list is compared against a stored baseline. A new or missing container produces a CRITICAL result.

Docker port mappings

Container port configuration is compared against the known-good state so accidental exposure of a service can be detected.

Listening sockets

Active TCP and UDP listeners are compared against the approved list. Unexpected listeners are treated as critical.

Security-sensitive files

SHA-256 hashes are stored for SSH, sudo, UFW, WireGuard and authorized_keys configuration files. Any change causes the integrity check to fail.

AUTOMATION

systemd service and timer

The audit is run automatically by systemd rather than depending on me remembering to execute it manually.

Daily scheduled audit

A oneshot systemd service runs the security audit each morning. A persistent timer means that if the server is offline at the scheduled time, the missed audit runs after the machine returns.

Immediate path monitoring

A systemd path unit watches high-risk configuration locations such as SSH, sudo, UFW, WireGuard and authorized_keys. A change can trigger the audit immediately instead of waiting until the next daily run.

ALERTING

Telegram reporting

I reused the Telegram bot already running on Odysseus rather than creating another notification system. The audit sends a compact summary directly to my phone.

Daily summary

The message includes PASS, WARN and CRITICAL totals, the overall status and the checks responsible for warnings or failures.

Secret handling

The Telegram token is not stored inside the audit script or Git repository. The audit references the existing protected token file instead.

TROUBLESHOOTING

Problems I had to solve

Building the audit exposed several cases where a security control could create misleading results if I did not understand what was actually happening underneath.

systemd sandbox caused false UFW failures

I initially hardened the audit service with ProtectSystem=strict. UFW still needed to create /run/ufw.lock even when only checking status, so the audit incorrectly reported the firewall as CRITICAL.

I identified the problem through journald and changed the service protection to ProtectSystem=full, keeping useful hardening while allowing UFW to operate correctly.

Backup job success was not enough

The Vaultwarden backup service was reporting success, but I realised that checking the service result alone did not prove a usable backup existed. The audit now verifies the actual artifact, freshness and file size.

Authentication false positives

An early version counted a mistyped local sudo password as a security warning. I refined the check so it focuses on SSH authentication events instead of treating every PAM authentication failure equally.

Docker exposure vs internal ports

I learned that a port shown by Docker is not automatically reachable from the host or internet. The audit distinguishes internal container ports from host-published mappings and compares both against the known-good design.

VALIDATION

Testing a real CRITICAL condition

I did not want to trust the alerting system without proving it could detect a genuine deviation.

Controlled test

I temporarily started a local HTTP listener on 127.0.0.1:55555. Because that socket was not part of the approved baseline, the audit detected the change and returned CRITICAL.

End-to-end alert

The service exited with the critical status, the Telegram notification reached my phone, and after the temporary listener was removed the next audit returned HEALTHY.

DESIGN DECISION

Why it does not automatically fix everything

Automatic remediation sounds attractive, but actions such as deleting SSH keys, modifying firewall rules or stopping containers can create more damage if the detection is wrong.

Detect first

The current system automatically detects, records and reports security drift while leaving higher-risk remediation under manual control.

Future low-risk automation

My next improvement is to classify safe, low-risk actions that can be automated separately, while keeping firewall, credential, key and other high-impact changes approval-gated.

OUTCOME

What I learned

The project gave me practical exposure to security monitoring, Linux service management and the difference between checking that something is alive and checking that it is still configured the way I intended.

Configuration drift

I learned how a known-good baseline can detect unexpected changes to containers, ports, network listeners and security configuration.

systemd

I used services, timers, path units, exit codes and sandboxing to turn a Bash script into an unattended security process.

Security monitoring

I learned why useful monitoring needs to avoid false positives, preserve evidence and report context rather than simply saying that something failed.

Operational thinking

The biggest lesson was that a security control needs to be maintainable. The project was designed around one person being able to understand, verify and operate it without creating unnecessary complexity.