1. BLUF (Bottom Line Up Front)

secure_homelab.py is a cross-platform Python automation script (macOS Apple Silicon + Linux) that installs and configures Fail2ban, a host firewall (UFW), kernel hardening, SSH hardening, file integrity monitoring (AIDE), rootkit detection (rkhunter), audit logging (auditd), and a lightweight SIEM-like log monitor with email alerting. Run it once as root and it sets up a layered defensive posture for a self-hosted home lab environment. On a red team assessment laptop, disable AIDE, kernel ICMP blocking, and PasswordAuthentication no until you have confirmed your SSH public key is already in place.

2. Architecture

graph TD
    Script["secure_homelab.py (root)"]

    subgraph Setup["One-time Setup Mode"]
        F2B["Fail2ban\nSSH / HTTP / HTTPS jails\npf (macOS) / iptables / nftables (Linux)"]
        FW["UFW Firewall\ndefault-deny incoming\nrate-limit SSH"]
        KERN["Kernel Hardening\nsysctl - SYN cookies, no redirects,\nrp_filter, dmesg_restrict"]
        SSH["SSH Hardening\nno root login, no password auth,\nMaxAuthTries 3"]
        AIDE["AIDE\nFile integrity baseline\ndaily cron check"]
        RKH["rkhunter\nRootkit detection\ndaily cron scan"]
        AUD["auditd\nSystem call auditing\nexecve, passwd, shadow, sudoers"]
        LYN["Lynis\nSecurity posture scanner\n(manual run)"]
    end

    subgraph Daemon["--daemon / Continuous Mode"]
        LOG["LogMonitor\ntail -f on auth/nginx/apache/fail2ban logs\nregex pattern matching"]
        HEALTH["Health Check\ndisk, CPU load, fail2ban status"]
        UPDATES["SystemUpdater\napt/dnf/yum / brew / softwareupdate"]
        ALERT["Email Alerting\nSMTP / STARTTLS"]
    end

    Script --> F2B
    Script --> FW
    Script --> KERN
    Script --> SSH
    Script --> AIDE
    Script --> RKH
    Script --> AUD
    Script --> LYN
    Script --> LOG
    Script --> HEALTH
    Script --> UPDATES
    LOG --> ALERT
    HEALTH --> ALERT

3. Tools, Techniques, Procedures (TTPs)

4. Walkthrough Summary

The script is invoked as root with optional flags (--launchd, --config, --daemon, --update-only). On first run without flags it performs a full one-time hardening pass: installs and configures Fail2ban with conditional service jails (only adds Apache/Nginx jails if the log paths exist), sets up a UFW default-deny firewall with SSH rate-limiting, writes kernel hardening sysctl parameters, appends hardening directives to /etc/ssh/sshd_config, initialises an AIDE file integrity baseline, installs and schedules rkhunter, configures auditd with custom rules, and installs Lynis. It then starts the background log monitor thread and performs an initial health check. If any step fails, RollbackManager executes registered undo actions in reverse order. With --daemon, the script runs as a persistent process doing periodic health checks and auto-updates.

5. Detailed Step-by-Step

Step 1 — Configuration & Logging

# SecurityConfig is a dataclass with all tunable parameters.
# Loaded from JSON (--config) or defaulted.
# Key defaults: ban_time=3600s, maxretry=5 for SSH, 10 for HTTP,
# find_time=600s (sliding window), health_check=300s.
config = load_config(config_file)

# Logs to /var/log/secure-homelab/security.log AND stdout.
# NOTE: log dir is created as root — on macOS /var/log is writable by root only.
logger = setup_logging()

Consequence: If you set suspicious_threshold too low (e.g., 1–2), expect a flood of email alerts from normal sudo usage triggering the privilege_escalation pattern.

Step 2 — RollbackManager

# Every destructive action (file write, service start, cron creation)
# registers a paired undo function. On exception or Ctrl+C, all undo
# functions run in reverse order (LIFO) to restore the system.
rollback_manager.add_action("Restore /etc/pf.conf from backup", restore_func)

Consequence: Rollback does NOT guarantee a clean state if the OS state changed mid-run (e.g., if fail2ban partially started and banned an IP before crashing). Always verify manually after a failed run.

Step 3 — Fail2ban Setup (macOS)

# pf is macOS's packet filter. The script appends an anchor rule to
# /etc/pf.conf so Fail2ban can dynamically add/remove IP tables.
anchor_line = 'anchor "f2b/*"'

# ⚠️  CONSEQUENCE: Modifying /etc/pf.conf and reloading pf
# (pfctl -f /etc/pf.conf) will instantly apply ALL rules in the file.
# If an existing rule was misconfigured, you can cut off your own SSH
# session or block your management IP before you can intervene.
# Always test first: pfctl -nf /etc/pf.conf (the script does this).
run(["pfctl", "-nf", "/etc/pf.conf"])  # dry-run test — good practice
run(["pfctl", "-f", "/etc/pf.conf"])   # live apply
run(["pfctl", "-e"], check=False)      # enable pf if not already on

# jail.local is built conditionally — Apache/Nginx jails only added
# if the corresponding log files already exist on disk.
# This prevents fail2ban startup failures from missing logpath entries.
if apache_error_log.exists():
    jail_config_parts.append(f"[apache-auth]...")

Consequence: If you later install Apache/Nginx after running this script, you must manually add the jail entries or re-run the script to pick them up.

Step 4 — Fail2ban Setup (Linux)

# Auto-detects nftables vs iptables for banaction.
# nftables preferred on modern distros (Debian 11+, RHEL 9+).
banaction = detect_banaction_linux()

# ⚠️  CONSEQUENCE: Using iptables-multiport on a system running
# firewalld will cause conflicts. firewalld manages its own iptables
# chains and will overwrite fail2ban entries on reload.
# Fix: set banaction = firewallcmd-rich-rules in jail.local instead.

Step 5 — UFW Firewall

# Resets UFW to clean state before applying rules.
# ⚠️  CONSEQUENCE: --force reset wipes ALL existing rules silently.
# If you had custom UFW rules (e.g., allowing VPN, custom ports),
# they are gone. There is no backup of previous UFW state.
run(["ufw", "--force", "reset"])

# Default-deny incoming: blocks all inbound connections by default.
# ⚠️  CONSEQUENCE: If you rely on any inbound service (NFS, Samba,
# Docker exposed ports, VNC, etc.) that isn't explicitly allowed,
# it will be silently blocked after enable. Test from another host.
run(["ufw", "default", "deny", "incoming"])

# SSH is rate-limited (6 connections per 30s) rather than fully blocked.
# ⚠️  CONSEQUENCE: ufw limit is relatively permissive — combine with
# Fail2ban for proper brute force protection.
run(["ufw", "limit", f"{ssh_port}/tcp"])

Step 6 — Kernel Hardening (sysctl)

# Written to /etc/sysctl.d/99-security-hardening.conf and applied live.

net.ipv4.ip_forward = 0
# ⚠️  CONSEQUENCE: If this host runs Docker, LXC, or any VM bridging,
# ip_forward=0 will BREAK container networking. Docker will fail to
# route traffic between containers and the host. Fix: set to 1, or
# add Docker-specific sysctl overrides in /etc/sysctl.d/99-docker.conf.

net.ipv4.icmp_echo_ignore_all = 1
# ⚠️  CONSEQUENCE: Host will not respond to ping. Useful for stealth,
# but breaks network diagnostics and monitoring tools (Zabbix, Nagios,
# Prometheus node_exporter ping checks). On a red team workstation
# this will also break your own recon tooling that relies on ICMP alive checks.

kernel.dmesg_restrict = 1
kernel.kptr_restrict = 2
# ⚠️  CONSEQUENCE: Non-root users cannot read kernel logs or kernel
# symbols. This breaks some debugging tools (perf, bpftrace) for
# non-root users. On a pentesting workstation this can impact BPF-based
# tools unless run as root.

Step 7 — SSH Hardening

PasswordAuthentication no
# ⚠️  CONSEQUENCE — HIGH RISK: This is the single most dangerous line
# in the script for a home lab. If you run this before confirming your
# SSH public key is in ~/.ssh/authorized_keys AND sshd has loaded it,
# you will be locked out of the machine remotely. The script does
# NOT verify authorized_keys exist before flipping this setting.
# MITIGATION: Always run from a local console session, not over SSH,
# the first time. Or add a pre-flight authorized_keys check.

MaxAuthTries 3
# ⚠️  CONSEQUENCE: Typos count. 3 failed attempts within a session
# drops the connection. With a password manager auto-filling wrong
# credentials this fires quickly. Consider MaxAuthTries 5 for home use.

MaxSessions 2
# ⚠️  CONSEQUENCE: Limits SSH multiplexing. If you use ControlMaster
# or tmux with multiple panes SSHing into the same host, you may hit
# this limit. Increase to 4-6 for interactive use.

Step 8 — AIDE File Integrity

# aideinit scans the entire filesystem and builds a baseline database.
# ⚠️  CONSEQUENCE: On a live system, takes 5-20 minutes and generates
# significant disk I/O. Don't run during production hours.
run(["aideinit"], check=False)

# Daily cron at 3 AM compares current state against baseline.
# ⚠️  CONSEQUENCE: Every legitimate software update (apt upgrade,
# pip install, tool update) will generate AIDE alerts the following
# morning. You must re-run --propupd / aideinit after intentional
# changes or tune AIDE's monitored paths in /etc/aide/aide.conf.
# On a red team workstation where tools change daily, AIDE noise
# will be extremely high — consider disabling or scoping to /etc only.

Step 9 — Log Monitor (SIEM-like)

# Spawns one `tail -f` process per log file.
# Reads with readline() in a single thread with time.sleep(1).

# ⚠️  BUG: readline() on a Popen stdout is BLOCKING. If one log file
# produces no new lines, the loop stalls on that file's readline()
# and misses events from all other files. On a quiet system with
# multiple active logs this causes event loss.
# FIX: Use select.select() or per-file threads to avoid blocking.

line = proc.stdout.readline()  # BLOCKING — stalls entire monitor loop

# Pattern for privilege escalation:
'privilege_escalation': re.compile(r'sudo:.*COMMAND=')
# ⚠️  CONSEQUENCE: EVERY legitimate sudo command by any user fires this
# pattern. With suspicious_threshold=10, you'll get an alert after 10
# sudo uses — which happens within minutes on an active system.
# This pattern should be scoped to unexpected users or commands,
# not all sudo invocations.

# event_counts is a simple dict that grows unbounded in daemon mode.
# ⚠️  BUG: No TTL or reset on event_counts. After a brute force attack
# the counter for an attacker IP never resets. This means the threshold
# re-fires on every subsequent event from that IP, generating email spam.
# FIX: Use a time-windowed counter (e.g., collections.deque with maxlen).
self.event_counts[key] = self.event_counts.get(key, 0) + 1

Step 10 — Log Cleanup

# Deletes files matching *.log* from /var/log and /opt/homebrew/var/log
# based purely on mtime older than log_retention_days.
for log_file in log_path.glob("*.log*"):
    if mtime < cutoff_date:
        log_file.unlink()

# ⚠️  CONSEQUENCE — HIGH RISK: This will delete logrotated archives
# (auth.log.1, auth.log.2.gz, syslog.1, etc.) that logrotate is still
# managing. It can also delete application logs you want to keep
# (e.g., nginx/access.log.14, fail2ban.log.7).
# FIX: Scope to /var/log/secure-homelab/ only, or use
# find with -name pattern rather than glob("*.log*") on the entire /var/log.

Step 11 — Email Alerting

# Stores SMTP password in plaintext in the JSON config file.
# ⚠️  CONSEQUENCE: Anyone who can read secure_homelab_config.json
# has your email credentials. The file has no mandatory restrictive
# permissions set by the script. Use an app-specific password (not your
# main account password) and chmod 600 the config file manually.
email_pass: str = ""

# ⚠️  BUG: send_alert() references `logger` (line 193) which is a
# module-level variable that only exists after setup_logging() is called.
# If send_alert() is called before setup_logging() (e.g., from load_config
# error handler), it raises NameError: name 'logger' is not defined.
# FIX: Replace `logger.info(...)` with `logging.getLogger(__name__).info(...)`.
logger.info(f"Alert sent: {subject}")  # NameError risk

6. Commands Used

# Run full setup (Linux/macOS root)
sudo python3 secure_homelab.py

# Run with launchd persistence (macOS only)
sudo python3 secure_homelab.py --launchd

# Generate config file (no root needed)
python3 secure_homelab.py --create-config

# Run with custom config
sudo python3 secure_homelab.py --config secure_homelab_config.json

# Run updates only (for cron/daemon use)
sudo python3 secure_homelab.py --update-only

# Run as persistent daemon
sudo python3 secure_homelab.py --daemon

# Post-setup verification
sudo fail2ban-client status
sudo ufw status verbose
sudo ausearch -k passwd_changes
sudo lynis audit system
tail -f /var/log/secure-homelab/security.log
sudo rkhunter --check --skip-keypress

7. Issues, Fixes, and Troubleshooting

8. Final Notes

Use this on a home server or NAS — not as-is on a red team assessment laptop.

For a red team / pentesting workstation, apply the following modifications before running:

# In SecurityConfig or your JSON config:
auto_updates   = False      # Never auto-update during an engagement
log_monitoring = False      # Disable — you want clean logs, not a monitor eating CPU

# In harden_kernel(): remove or comment out:
# net.ipv4.icmp_echo_ignore_all = 1   <-- breaks your own recon
# net.ipv4.ip_forward = 0             <-- breaks Docker/VM pivoting

# In harden_ssh(): add pre-flight key check before flipping PasswordAuth
# In setup_file_integrity(): skip entirely — AIDE noise is constant

Recommended run order for a new home server:

1. python3 secure_homelab.py --create-config — generate config, fill in email
2. chmod 600 secure_homelab_config.json
3. From a local console (not SSH): sudo python3 secure_homelab.py --config secure_homelab_config.json --launchd
4. Verify: sudo fail2ban-client status, sudo ufw status, sudo lynis audit system
5. Re-run aideinit after any intentional system changes to reset the baseline