Linux Log System

Overview

On a typical modern distro the key Linux logs live under /var/log, each covering a different slice of the system. From a security perspective you care most about auth, sudo, SSH, audit, kernel, cron, and general syslog/journal logs — plus how they're rotated, centralized, and protected at rest and in transit.

Systemd-based distros also keep a binary journal via journald (/var/log/journal/ or in-memory only); journalctl is your interface and it can mirror or replace classic text logs.

────────────────────────

Core /var/log Files and Purposes

Exact names vary by distro (Debian/Ubuntu vs RHEL/CentOS), but the purpose is the same.

────────────────────────

Quick Recon Commands

# --- Text logs: live tail ---
tail -f /var/log/auth.log                        # SSH/sudo events (Debian/Ubuntu)
tail -f /var/log/secure                          # same on RHEL/CentOS

# --- journald: systemd journal ---
journalctl -f                                    # follow everything
journalctl -u sshd --since "1 hour ago"         # one unit, recent
journalctl -p err -b                             # errors this boot
journalctl _UID=0 --since today                  # all root activity today

# --- Binary logs ---
last   -F                                        # successful logins (wtmp)
lastb  -F                                        # failed logins (btmp) [needs root]
lastlog                                          # last login per user

# --- Currently logged-in ---
who; w

# --- Audit log ---
ausearch -m USER_LOGIN -ts today                 # logins via auditd
aureport --auth --summary                        # auth summary report

────────────────────────

Auth Log: What to Look For

# --- Brute force / failed SSH ---
grep "Failed password" /var/log/auth.log | awk '{print $11}' | sort | uniq -c | sort -rn | head -20

# --- Successful logins after failures (compromise indicator) ---
grep "Accepted password" /var/log/auth.log

# --- Sudo usage ---
grep "sudo" /var/log/auth.log | grep -v "PAM"

# --- New user accounts created ---
grep "useradd\|adduser" /var/log/auth.log

# --- SSHD: accepted public key logins ---
grep "Accepted publickey" /var/log/auth.log

# --- Logins from IPs (extract unique sources) ---
grep "sshd" /var/log/auth.log | grep -oP 'from \K[\d.]+' | sort | uniq -c | sort -rn

────────────────────────

auditd: High-Value Rules

# /etc/audit/rules.d/hardening.rules

# Track privilege escalation
-a always,exit -F arch=b64 -S execve -F euid=0 -k root_cmd

# Watch sensitive files
-w /etc/passwd        -p wa -k passwd_change
-w /etc/shadow        -p wa -k shadow_change
-w /etc/sudoers       -p wa -k sudoers_change
-w /etc/ssh/sshd_config -p wa -k sshd_config_change

# Watch log files themselves (detect tampering)
-w /var/log/auth.log  -p wa -k log_tamper
-w /var/log/audit/    -p wa -k audit_tamper

# Cron modifications
-w /etc/crontab       -p wa -k cron_change
-w /var/spool/cron/   -p wa -k cron_change

# Module loads (rootkit indicator)
-a always,exit -F arch=b64 -S init_module -S finit_module -k module_load

# Apply without reboot
auditctl -R /etc/audit/rules.d/hardening.rules

────────────────────────

Log Rotation: logrotate Basics

# Main config
/etc/logrotate.conf
/etc/logrotate.d/*          # per-service overrides

# Typical stanza
/var/log/auth.log {
    daily                   # rotate daily
    rotate 14               # keep 14 rotated copies
    compress                # gzip old files
    delaycompress           # keep last rotation uncompressed
    missingok               # no error if file missing
    notifempty              # skip if empty
    postrotate
        /usr/lib/rsyslog/rsyslog-rotate  # signal rsyslog
    endscript
}

# Force rotation now (debug/test)
logrotate -f /etc/logrotate.d/rsyslog

# Show what would happen (dry-run)
logrotate -d /etc/logrotate.conf

────────────────────────

Attack Vectors Against Logs (Red Team)

Common techniques attackers use once they have elevated access:

Key insight: On a hardened system most of the above requires root. Attackers either aim for low and slow with legit creds to avoid generating logs, or gain root first and then surgically clean up.

────────────────────────

Protecting Logs at Rest

# --- Immutable / append-only flags ---
chattr +a /var/log/auth.log    # append-only: even root can't overwrite, only append
chattr +i /var/log/auth.log    # fully immutable: nothing can modify/delete
lsattr /var/log/auth.log       # verify flags

# --- Protect the log directory ---
chmod 640  /var/log/auth.log   # root:adm only
chmod 750  /var/log/audit/
chown root:root /var/log/audit/audit.log

# --- LUKS encrypted /var partition (full disk) ---
# At boot the key unlocks the partition; once mounted, logs are plaintext to root.
# Key protection comes from limiting root access itself.
cryptsetup luksOpen /dev/sda3 var_crypt
mount /dev/mapper/var_crypt /var

# --- Minimal local retention: forward and shred ---
# After forwarding off-box, truncate local copy to reduce exposure window
truncate -s 0 /var/log/auth.log

────────────────────────

Forwarding Logs Securely (rsyslog + TLS)

# /etc/rsyslog.d/99-remote.conf  (CLIENT side)

# Load TLS module
module(load="imtcp")
global(
  DefaultNetstreamDriver="gtls"
  DefaultNetstreamDriverCAFile="/etc/ssl/certs/logserver-ca.pem"
  DefaultNetstreamDriverCertFile="/etc/ssl/certs/client-cert.pem"
  DefaultNetstreamDriverKeyFile="/etc/ssl/private/client-key.pem"
)

# Forward all auth events to central syslog over TLS
auth,authpriv.*  action(
  type="omfwd"
  target="syslog.example.com"
  port="6514"
  protocol="tcp"
  StreamDriver="gtls"
  StreamDriverMode="1"
  StreamDriverAuthMode="x509/name"   # NOT anon; validate cert CN
  StreamDriverPermittedPeers="syslog.example.com"
)

# Restart and verify
systemctl restart rsyslog
journalctl -u rsyslog -n 30   # check for TLS handshake errors

# Alternative: run syslog inside a WireGuard / IPsec VPN tunnel
# plain UDP/TCP syslog but encrypted at the network layer
wg-quick up wg0 && systemctl restart rsyslog

────────────────────────

journald Remote Forwarding

# /etc/systemd/journald.conf
[Journal]
Storage=persistent          # ensure journal survives reboot
Compress=yes
ForwardToSyslog=yes         # bridge to rsyslog for TLS forwarding above

# OR use systemd-journal-remote (native journal protocol over HTTPS)
# Server: install systemd-journal-remote, enable systemd-journal-remote.socket
# Client:
systemd-journal-upload \
  --url=https://logserver.example.com:19532 \
  --server-key=/etc/ssl/private/client-key.pem \
  --server-certificate=/etc/ssl/certs/client-cert.pem \
  --trust=/etc/ssl/certs/logserver-ca.pem

────────────────────────

Integrity: Detecting Tampering

# --- AIDE / Tripwire: hash-based FIM ---
aide --init                          # build initial database
aide --check                         # compare against database
# Protect the database itself off-box; on-box DB = useless against root

# --- auditd watches on log files (see rules above) ---
ausearch -k log_tamper               # see who touched auth.log

# --- Verify binary log files haven't been zeroed ---
wc -c /var/log/wtmp  # size should be non-zero if any logins ever happened
file /var/log/btmp   # should be "data", non-empty

# --- Off-box append-only storage ---
# S3 / object-store bucket with Object Lock (WORM) -- once written, cannot be deleted by root
# Central SIEM with immutable indexing (Splunk, Elastic frozen tier, Loki + S3)

────────────────────────

Distro Cheat-Sheet: File Name Equivalents

────────────────────────

One-Liner Triage (First 60 Seconds on a Compromised Box)

# Who is/was here?
echo "=== NOW ===";      w; who
echo "=== RECENT ===";   last -F | head -20
echo "=== FAILED ===";   lastb -F 2>/dev/null | head -20

# Recent auth events
echo "=== AUTH ===";     tail -100 /var/log/auth.log 2>/dev/null || journalctl -u sshd -n 50

# Sudo / root escalations
echo "=== SUDO ===";     grep sudo /var/log/auth.log | tail -20

# Cron / persistence
echo "=== CRON ===";     tail -50 /var/log/cron 2>/dev/null; crontab -l 2>/dev/null; ls /etc/cron*

# Package installs (attacker tooling)
echo "=== PKGS ===";     tail -50 /var/log/dpkg.log 2>/dev/null || tail -50 /var/log/yum.log 2>/dev/null

# Signs of log wipe
echo "=== WTMP SIZE ==="; wc -c /var/log/wtmp /var/log/btmp
echo "=== AUTH SIZE ==="; wc -c /var/log/auth.log 2>/dev/null || echo "missing"

────────────────────────

Resources

#BlueTeam #RedTeam #Linux #Logs #DFIR #Forensics #CyberSecurity #SOC #auditd #rsyslog