rick-infra/docs/security-hardening.md

Security Hardening Guide

Comprehensive security hardening for rick-infra, implementing defense-in-depth with network, container, database, and application security layers.

Security Architecture Overview

┌─────────────────────────────────────────────────────────────────┐
│ rick-infra Security Architecture (Defense in Depth)            │
│                                                                 │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ Layer 1: Network Security                                  │ │
│ │ • SSH Hardening  • Firewall  • Fail2ban  • TLS/HTTPS      │ │
│ └─────────────────────────────────────────────────────────────┘ │
│                                ▼                               │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ Layer 2: Authentication & Authorization                    │ │
│ │ • Authentik SSO  • MFA  • Forward Auth  • Policy Engine   │ │
│ └─────────────────────────────────────────────────────────────┘ │
│                                ▼                               │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ Layer 3: Container Security                                │ │
│ │ • Rootless Podman  • User Namespaces  • Capabilities      │ │
│ │ • SELinux  • Resource Limits  • Security Labels           │ │
│ └─────────────────────────────────────────────────────────────┘ │
│                                ▼                               │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ Layer 4: Database & Cache Security                         │ │
│ │ • Unix Sockets Only  • No Network Exposure  • Group ACLs  │ │
│ │ • Encrypted Storage  • Backup Security  • Access Logging  │ │
│ └─────────────────────────────────────────────────────────────┘ │
│                                ▼                               │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ Layer 5: Application Security                              │ │
│ │ • Secret Management  • Secure Headers  • Audit Logging    │ │
│ │ • Input Validation  • Session Security  • CSRF Protection │ │
│ └─────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘

Layer 1: Network Security

SSH Hardening

Configuration Applied by Security Role

# SSH hardening settings
sshd_config:
  # Authentication Security
  PasswordAuthentication: no
  ChallengeResponseAuthentication: no
  PermitRootLogin: no
  PubkeyAuthentication: yes
  AuthenticationMethods: "publickey"
  
  # Protocol Security
  Protocol: 2
  PermitEmptyPasswords: no
  MaxAuthTries: 3
  LoginGraceTime: 60
  
  # Connection Security
  X11Forwarding: no
  AllowAgentForwarding: no
  AllowTcpForwarding: no
  PermitTunnel: no
  
  # Session Security
  ClientAliveInterval: 300
  ClientAliveCountMax: 2
  MaxSessions: 2
  MaxStartups: "2:30:10"

Verification Commands

# Verify SSH hardening
ssh root@your-vps "sshd -T | grep -E '(passwordauth|rootlogin|pubkey)'"

# Test SSH security
ssh-audit your-vps  # External tool for SSH security assessment

# Monitor SSH attempts
ssh root@your-vps "journalctl -u sshd | grep -i 'failed\|invalid'"

Firewall Configuration

nftables Firewall Rules

# Deployed firewall configuration
table inet filter {
    chain input {
        type filter hook input priority 0; policy drop;
        
        # Allow loopback traffic
        iifname "lo" accept
        
        # Allow established connections
        ct state established,related accept
        
        # Allow SSH (rate limited)
        tcp dport 22 ct state new limit rate 5/minute accept
        
        # Allow HTTP/HTTPS
        tcp dport {80, 443} accept
        
        # Allow ICMP (rate limited)
        icmp type echo-request limit rate 1/second accept
        
        # Log dropped packets
        log prefix "DROPPED: " drop
    }
    
    chain forward {
        type filter hook forward priority 0; policy drop;
    }
    
    chain output {
        type filter hook output priority 0; policy accept;
    }
}

Firewall Management

# Check firewall status
ssh root@your-vps "nft list ruleset"

# Monitor dropped connections
ssh root@your-vps "journalctl -k | grep DROPPED"

# Temporary rule addition (emergency access)
ssh root@your-vps "nft add rule inet filter input tcp dport 8080 accept"

Intrusion Detection (fail2ban)

fail2ban Configuration

# SSH protection jail
[sshd]
enabled = true
port = ssh
filter = sshd
logpath = /var/log/auth.log
maxretry = 3
findtime = 600
bantime = 3600
backend = systemd

# HTTP brute force protection  
[caddy-auth]
enabled = true
port = http,https
filter = caddy-auth
logpath = /var/log/caddy/access.log
maxretry = 10
findtime = 600
bantime = 1800

# Authentik protection
[authentik-auth]
enabled = true
port = http,https
filter = authentik-auth
logpath = /opt/authentik/logs/server.log
maxretry = 5
findtime = 300
bantime = 3600

Intrusion Monitoring

# Check fail2ban status
ssh root@your-vps "fail2ban-client status"
ssh root@your-vps "fail2ban-client status sshd"

# View banned IPs
ssh root@your-vps "fail2ban-client get sshd banned"

# Unban IP (if needed)
ssh root@your-vps "fail2ban-client set sshd unbanip 203.0.113.100"

TLS/HTTPS Security

Caddy TLS Configuration

# Automatic HTTPS with security headers
(security_headers) {
    header {
        # HSTS (HTTP Strict Transport Security)
        Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
        
        # Content Security Policy
        Content-Security-Policy "default-src 'self'; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline'"
        
        # Additional security headers
        X-Content-Type-Options "nosniff"
        X-Frame-Options "DENY"  
        X-XSS-Protection "1; mode=block"
        Referrer-Policy "strict-origin-when-cross-origin"
        
        # Remove server information
        -Server
    }
}

# Apply to all sites
*.jnss.me {
    import security_headers
    tls {
        protocols tls1.2 tls1.3
        curves x25519 secp384r1
    }
}

TLS Monitoring

# Check certificate status
curl -vI https://auth.jnss.me/ 2>&1 | grep -E "(certificate|expire|protocol)"

# Test TLS configuration
nmap --script ssl-enum-ciphers -p 443 your-domain.com

# Monitor certificate renewal
ssh root@your-vps "journalctl -u caddy | grep -i cert"

Layer 2: Authentication & Authorization Security

Authentik Security Configuration

Password Policy

# Authentik password policy
password_policy:
  length_min: 12
  amount_lowercase: 1
  amount_uppercase: 1
  amount_digits: 1
  amount_symbols: 1
  password_field: "password"
  check_static_rules: true
  check_have_i_been_pwned: true
  check_zxcvbn: true
  zxcvbn_score_threshold: 3

Multi-Factor Authentication

# MFA enforcement policies
mfa_policies:
  - name: "Admin MFA Required"
    bound_to: "group:admins"
    authenticators: ["totp", "webauthn"]
    enforce: true
    
  - name: "Sensitive Services MFA"
    bound_to: "application:admin_*"
    authenticators: ["totp"]
    enforce: true
    
  - name: "External Access MFA"
    bound_to: "source_ip:!10.0.0.0/8"
    authenticators: ["totp", "sms"]
    enforce: true

Session Security

# Session security settings
session_security:
  cookie_secure: true
  cookie_httponly: true
  cookie_samesite: "Strict"
  session_timeout: 3600  # 1 hour
  remember_me_timeout: 86400  # 24 hours
  concurrent_sessions_limit: 3
  session_fixation_protection: true

Access Control Policies

Time-Based Access Control

# Business hours access policy
policies:
  - name: "Business Hours Only"
    type: "time"
    parameters:
      start_time: "08:00"
      end_time: "18:00"
      days: ["monday", "tuesday", "wednesday", "thursday", "friday"]
      timezone: "UTC"
    applications: ["admin_dashboard", "financial_systems"]

IP-Based Access Control

# Geographic/Network access restrictions
policies:
  - name: "Office Network Only"
    type: "source_ip"
    parameters:
      cidr: ["10.0.0.0/8", "192.168.0.0/16"]
    applications: ["admin_*", "backup_*"]
    
  - name: "Block High-Risk Countries"
    type: "geoip"
    parameters:
      countries: ["CN", "RU", "KP"]
      action: "deny"
    applications: ["*"]

Layer 3: Container Security

Rootless Container Architecture

Security Benefits of Rootless Podman

# Rootless container verification
ssh root@your-vps "sudo -u authentik podman info | grep -A10 'security'"

# User namespace mapping
ssh root@your-vps "sudo -u authentik cat /proc/self/uid_map"
ssh root@your-vps "sudo -u authentik cat /proc/self/gid_map"

# Process tree verification (containers run as unprivileged user)
ssh root@your-vps "ps aux | grep -E '(authentik|gitea)'"

Container User Security

# Container configuration with security
[Container]
# Run as specific UID/GID (not root)
User=963:963

# Preserve supplementary groups for socket access
Annotation=run.oci.keep_original_groups=1

# Minimal capabilities
AddCapability=IPC_OWNER
DropCapability=ALL

# Security labels
SecurityLabelDisable=true
SecurityLabelType=container_t

Container Isolation

Filesystem Security

# Read-only root filesystem where possible
security_settings:
  read_only_root_fs: true
  temp_fs_mounts:
    - "/tmp:rw,noexec,nosuid,size=100M"
    - "/var/tmp:rw,noexec,nosuid,size=100M"
  
  # Minimal volume mounts
  volume_mounts:
    - "/opt/authentik/data:/data:Z"  # SELinux label
    - "/opt/authentik/media:/media:Z"
    - "/var/run/postgresql:/var/run/postgresql:Z"
    - "/var/run/valkey:/var/run/valkey:Z"

Resource Limits

# systemd resource controls
[Service]
# Memory limits
MemoryMax=1G
MemorySwapMax=0

# CPU limits  
CPUQuota=200%  # 2 CPU cores max

# Process limits
TasksMax=100
LimitNOFILE=1024

# I/O limits
IODeviceWriteBandwidthMax=/dev/sda 50M

SELinux Security

Container Labeling

# Verify SELinux labels on container volumes
ssh root@your-vps "ls -Z /var/run/postgresql/"
ssh root@your-vps "ls -Z /opt/authentik/"

# Check container SELinux context
ssh root@your-vps "sudo -u authentik podman exec authentik-server cat /proc/self/attr/current"

# SELinux audit logs
ssh root@your-vps "ausearch -m AVC -ts recent"

Custom SELinux Policies

# Generate custom SELinux policy for containers
ssh root@your-vps "audit2allow -a -m rick_containers"

# Apply custom policy
ssh root@your-vps "semodule -i rick_containers.pp"

Layer 4: Database & Cache Security

Unix Socket Security Model

Socket File Permissions

# Verify socket permissions
ssh root@your-vps "ls -la /var/run/postgresql/"
# Should show: srwxrwx--- postgres postgres .s.PGSQL.5432

ssh root@your-vps "ls -la /var/run/valkey/"  
# Should show: srwxrwx--- valkey valkey valkey.sock

# Verify group access
ssh root@your-vps "groups authentik"
# Should include: postgres valkey

Network Isolation

# Database configuration (no network exposure)
postgresql_config:
  listen_addresses: ''  # Unix sockets only
  port: 0              # Disable TCP port
  unix_socket_directories: '/var/run/postgresql'
  unix_socket_permissions: 0770

valkey_config:
  bind: ''             # No network binding
  port: 0              # Disable TCP port  
  unixsocket: '/var/run/valkey/valkey.sock'
  unixsocketperm: 770

Database Security Hardening

PostgreSQL Security

-- Database security configuration
-- Remove public schema permissions
REVOKE ALL ON SCHEMA public FROM PUBLIC;

-- Create application-specific schemas
CREATE SCHEMA authentik AUTHORIZATION authentik;
CREATE SCHEMA gitea AUTHORIZATION gitea;

-- Row-level security (if needed)
ALTER TABLE authentik.sensitive_data ENABLE ROW LEVEL SECURITY;
CREATE POLICY user_data_policy ON authentik.sensitive_data 
  FOR ALL TO authentik USING (user_id = current_setting('app.user_id'));

-- Audit logging
CREATE EXTENSION IF NOT EXISTS pgaudit;
ALTER SYSTEM SET pgaudit.log = 'all';

Database Access Monitoring

# Monitor database connections
ssh root@your-vps "sudo -u postgres psql -h /var/run/postgresql -c 'SELECT datname, usename, application_name, client_addr FROM pg_stat_activity;'"

# Check for suspicious queries
ssh root@your-vps "sudo -u postgres psql -h /var/run/postgresql -c 'SELECT query, query_start, state FROM pg_stat_activity WHERE state != '\''idle'\'';'"

# Review authentication logs
ssh root@your-vps "grep -i 'authentication\|connection' /var/log/postgresql/postgresql.log"

Cache Security

Valkey/Redis Security Configuration

# Valkey security settings
CONFIG SET requirepass "strong_password"
CONFIG SET rename-command FLUSHDB ""
CONFIG SET rename-command FLUSHALL ""
CONFIG SET rename-command DEBUG ""
CONFIG SET rename-command CONFIG "CONFIG_b3a0f6c8"

# Disable dangerous commands
CONFIG SET protected-mode yes
CONFIG SET save ""  # Disable automatic snapshots

Cache Access Control

# Verify cache authentication
ssh root@your-vps "redis-cli -s /var/run/valkey/valkey.sock -a \$VALKEY_PASSWORD ping"

# Monitor cache operations
ssh root@your-vps "redis-cli -s /var/run/valkey/valkey.sock -a \$VALKEY_PASSWORD monitor"

# Check cache configuration
ssh root@your-vps "redis-cli -s /var/run/valkey/valkey.sock -a \$VALKEY_PASSWORD CONFIG GET '*'"

Layer 5: Application Security

Secret Management

Ansible Vault Security

# Vault encryption verification
vault_security:
  encryption_method: "AES256"
  key_derivation: "PBKDF2"
  vault_password_source: "prompt"  # Never store in files
  
# Vault best practices
vault_practices:
  - rotate_vault_password: "quarterly"
  - separate_vault_files: "by_environment"  
  - backup_encrypted_vaults: "securely"
  - audit_vault_access: "regularly"

Environment Variable Security

# Verify secrets are not in process environment
ssh root@your-vps "sudo -u authentik cat /proc/\$(pgrep -f authentik-server)/environ | tr '\0' '\n' | grep -v SECRET"

# Check for secrets in logs
ssh root@your-vps "journalctl --user -M authentik@ -u authentik-server | grep -i 'password\|secret\|token'"

Secure Headers and Middleware

Application-Level Security Headers

# Example application security middleware
security_middleware = {
    'SECURE_BROWSER_XSS_FILTER': True,
    'SECURE_CONTENT_TYPE_NOSNIFF': True,
    'SECURE_HSTS_SECONDS': 31536000,
    'SECURE_HSTS_INCLUDE_SUBDOMAINS': True,
    'SECURE_HSTS_PRELOAD': True,
    'SECURE_REDIRECT_EXEMPT': [],
    'SECURE_SSL_REDIRECT': True,
    'SESSION_COOKIE_SECURE': True,
    'SESSION_COOKIE_HTTPONLY': True,
    'SESSION_COOKIE_SAMESITE': 'Strict',
    'CSRF_COOKIE_SECURE': True,
    'X_FRAME_OPTIONS': 'DENY'
}

Content Security Policy

# Strict CSP for web applications
Content-Security-Policy: default-src 'self'; 
                        script-src 'self' 'unsafe-inline' https://cdn.example.com; 
                        style-src 'self' 'unsafe-inline'; 
                        img-src 'self' data: https:; 
                        font-src 'self' https://fonts.gstatic.com; 
                        connect-src 'self' https://api.example.com; 
                        frame-ancestors 'none'; 
                        base-uri 'self'; 
                        form-action 'self'

Audit Logging and Monitoring

System-Wide Audit Configuration

# auditd configuration for security events
auditctl -w /etc/passwd -p wa -k user_modification
auditctl -w /etc/shadow -p wa -k password_modification
auditctl -w /opt/authentik -p wa -k authentik_access
auditctl -w /var/run/postgresql -p wa -k database_access
auditctl -w /var/run/valkey -p wa -k cache_access

# Monitor systemd service changes
auditctl -w /etc/systemd/system -p wa -k service_modification
auditctl -w /home/*/.*config/containers/systemd -p wa -k container_modification

Application Audit Logging

# Application-level audit events
audit_events:
  authentication:
    - user_login_success
    - user_login_failure  
    - password_change
    - mfa_enrollment
    - session_timeout
    
  authorization:
    - access_granted
    - access_denied
    - privilege_escalation
    - policy_violation
    
  data_access:
    - sensitive_data_access
    - data_export
    - configuration_change
    - backup_access

Security Monitoring and Alerting

Real-Time Security Monitoring

# Security monitoring scripts
#!/bin/bash
# security-monitor.sh

# Monitor failed authentication attempts
FAILED_LOGINS=$(grep "Failed password" /var/log/auth.log | wc -l)
if [ $FAILED_LOGINS -gt 10 ]; then
    echo "ALERT: High number of failed login attempts: $FAILED_LOGINS"
fi

# Monitor unusual database connections
DB_CONNECTIONS=$(sudo -u postgres psql -h /var/run/postgresql -t -c "SELECT count(*) FROM pg_stat_activity WHERE application_name NOT IN ('authentik', 'gitea');")
if [ $DB_CONNECTIONS -gt 5 ]; then
    echo "ALERT: Unusual database connections: $DB_CONNECTIONS"
fi

# Monitor container security violations
SELINUX_VIOLATIONS=$(ausearch -m AVC -ts recent 2>/dev/null | wc -l)
if [ $SELINUX_VIOLATIONS -gt 0 ]; then
    echo "ALERT: SELinux violations detected: $SELINUX_VIOLATIONS"
fi

Log Aggregation and Analysis

# Log analysis configuration
log_analysis:
  sources:
    - /var/log/auth.log        # SSH and system authentication
    - /var/log/audit/audit.log # System audit events
    - /opt/authentik/logs/     # Application logs
    - /var/log/caddy/          # Web server logs
    - /var/log/postgresql/     # Database logs
    
  alerts:
    - pattern: "Failed password.*from.*"
      threshold: 5
      timeframe: "5m"
      action: "block_ip"
      
    - pattern: "AUTHENTICATION_FAILED.*authentik"
      threshold: 10
      timeframe: "10m"  
      action: "alert_admin"

Security Best Practices

Regular Security Maintenance

Security Updates

# Automated security updates (weekly)
#!/bin/bash
# security-updates.sh

# System package updates
pacman -Syu --noconfirm

# Container image updates
for user in authentik gitea; do
    sudo -u $user podman auto-update
done

# Certificate renewal check
systemctl reload caddy

# Restart services if security updates require it
if [ -f /var/run/reboot-required ]; then
    systemctl reboot
fi

Security Auditing

# Monthly security audit checklist
#!/bin/bash
# security-audit.sh

echo "=== Security Audit Report $(date) ==="

# Check for unauthorized users
echo "Checking system users:"
awk -F: '$3 >= 1000 {print $1, $3}' /etc/passwd

# Check sudo access
echo "Checking sudo access:"
grep -v '^#' /etc/sudoers

# Check listening services
echo "Checking listening services:"
netstat -tulpn | grep LISTEN

# Check file permissions on critical files
echo "Checking critical file permissions:"
ls -la /etc/shadow /etc/passwd /etc/ssh/sshd_config

# Check for SUID/SGID files
echo "Checking SUID/SGID files:"
find /usr -perm /6000 -type f 2>/dev/null

# Check container security
echo "Checking container security:"
for user in authentik gitea; do
    echo "User: $user"
    sudo -u $user podman ps --format "table {{.Names}}\t{{.RunningFor}}\t{{.Status}}"
done

# Check database security
echo "Checking database connections:"
sudo -u postgres psql -h /var/run/postgresql -c "SELECT datname, usename, client_addr FROM pg_stat_activity;"

Incident Response Procedures

Security Incident Response Plan

incident_response:
  detection:
    - automated_alerts: "fail2ban, audit logs, monitoring"
    - manual_review: "daily log review, weekly security audit"
    
  assessment:
    - severity_levels: ["low", "medium", "high", "critical"]
    - response_time: 
        critical: "15 minutes"
        high: "1 hour"
        medium: "4 hours"
        low: "24 hours"
    
  containment:
    - isolate_affected_systems: "fail2ban, iptables blocks"
    - preserve_evidence: "log snapshots, memory dumps"
    - maintain_operations: "failover procedures"
    
  eradication:
    - remove_threats: "malware removal, account cleanup"
    - patch_vulnerabilities: "system updates, configuration fixes"
    - strengthen_defenses: "additional monitoring, access controls"
    
  recovery:
    - restore_services: "service restart, data restoration"  
    - monitor_operations: "enhanced monitoring period"
    - validate_security: "penetration testing, vulnerability scans"

Emergency Response Commands

# Emergency security response toolkit
#!/bin/bash
# emergency-response.sh

case "$1" in
    "isolate")
        # Emergency network isolation
        iptables -I INPUT -j DROP
        iptables -I OUTPUT -j DROP
        echo "System isolated from network"
        ;;
        
    "lockdown")
        # Lock all user accounts except emergency admin
        for user in $(cat /etc/passwd | cut -d: -f1); do
            if [ "$user" != "root" ] && [ "$user" != "emergency" ]; then
                usermod -L $user
            fi
        done
        echo "All user accounts locked"
        ;;
        
    "audit")
        # Emergency audit collection
        mkdir -p /tmp/emergency-audit
        cp -r /var/log/* /tmp/emergency-audit/
        journalctl --no-pager > /tmp/emergency-audit/journalctl.log
        netstat -tulpn > /tmp/emergency-audit/network.log
        ps auxf > /tmp/emergency-audit/processes.log
        echo "Audit data collected in /tmp/emergency-audit/"
        ;;
        
    "restore")
        # Restore from isolation
        iptables -F
        iptables -P INPUT ACCEPT
        iptables -P OUTPUT ACCEPT
        echo "Network access restored - apply proper firewall rules"
        ;;
esac

Compliance and Standards

Security Standards Alignment

CIS (Center for Internet Security) Benchmarks

cis_compliance:
  level_1_controls:
    - filesystem_configuration: "implemented"
    - software_updates: "automated" 
    - secure_boot_settings: "implemented"
    - additional_process_hardening: "implemented"
    - mandatory_access_controls: "selinux_enforcing"
    
  level_2_controls:
    - address_space_layout_randomization: "enabled"
    - prelink_disabled: "implemented"
    - activate_gpg_check: "implemented" 
    - disable_automounting: "implemented"

NIST Cybersecurity Framework

nist_framework:
  identify:
    - asset_management: "complete inventory of systems and services"
    - risk_assessment: "quarterly risk assessments"
    
  protect:
    - access_control: "authentik SSO with MFA"
    - awareness_training: "security documentation and procedures"
    - data_security: "encryption at rest and in transit"
    
  detect:
    - anomalies_events: "automated monitoring and alerting"
    - security_monitoring: "continuous monitoring of security events"
    
  respond:
    - response_planning: "documented incident response procedures"
    - communications: "stakeholder notification procedures"
    
  recover:
    - recovery_planning: "documented recovery procedures"
    - improvements: "lessons learned and security enhancements"

Documentation and Training

Security Documentation

required_documentation:
  policies:
    - information_security_policy
    - access_control_policy
    - incident_response_policy
    - backup_recovery_policy
    
  procedures:
    - user_onboarding_offboarding
    - security_incident_response
    - vulnerability_management
    - change_management
    
  technical_guides:
    - system_hardening_guide
    - application_security_guide
    - network_security_guide
    - monitoring_alerting_guide

---

This comprehensive security hardening guide implements defense-in-depth security across all layers of the rick-infra stack, emphasizing the security benefits of native database services, Unix socket IPC, rootless containers, and centralized authentication.

References

• Authentication Architecture - Detailed authentication and authorization
• Architecture Decisions - Security-focused architectural choices
• Authentik Deployment Guide - Secure authentication deployment
• Service Integration Guide - Secure service integration patterns