# Security Hardening Guide Comprehensive security hardening for rick-infra, implementing defense-in-depth with network, container, database, and application security layers. ## Security Architecture Overview ``` ┌─────────────────────────────────────────────────────────────────┐ │ rick-infra Security Architecture (Defense in Depth) │ │ │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ Layer 1: Network Security │ │ │ │ • SSH Hardening • Firewall • Fail2ban • TLS/HTTPS │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ Layer 2: Authentication & Authorization │ │ │ │ • Authentik SSO • MFA • Forward Auth • Policy Engine │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ Layer 3: Container Security │ │ │ │ • Rootless Podman • User Namespaces • Capabilities │ │ │ │ • SELinux • Resource Limits • Security Labels │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ Layer 4: Database & Cache Security │ │ │ │ • Unix Sockets Only • No Network Exposure • Group ACLs │ │ │ │ • Encrypted Storage • Backup Security • Access Logging │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ Layer 5: Application Security │ │ │ │ • Secret Management • Secure Headers • Audit Logging │ │ │ │ • Input Validation • Session Security • CSRF Protection │ │ │ └─────────────────────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────────┘ ``` ## Layer 1: Network Security ### SSH Hardening #### Configuration Applied by Security Role ```yaml # SSH hardening settings sshd_config: # Authentication Security PasswordAuthentication: no ChallengeResponseAuthentication: no PermitRootLogin: no PubkeyAuthentication: yes AuthenticationMethods: "publickey" # Protocol Security Protocol: 2 PermitEmptyPasswords: no MaxAuthTries: 3 LoginGraceTime: 60 # Connection Security X11Forwarding: no AllowAgentForwarding: no AllowTcpForwarding: no PermitTunnel: no # Session Security ClientAliveInterval: 300 ClientAliveCountMax: 2 MaxSessions: 2 MaxStartups: "2:30:10" ``` #### Verification Commands ```bash # Verify SSH hardening ssh root@your-vps "sshd -T | grep -E '(passwordauth|rootlogin|pubkey)'" # Test SSH security ssh-audit your-vps # External tool for SSH security assessment # Monitor SSH attempts ssh root@your-vps "journalctl -u sshd | grep -i 'failed\|invalid'" ``` ### Firewall Configuration #### nftables Firewall Rules ```bash # Deployed firewall configuration table inet filter { chain input { type filter hook input priority 0; policy drop; # Allow loopback traffic iifname "lo" accept # Allow established connections ct state established,related accept # Allow SSH (rate limited) tcp dport 22 ct state new limit rate 5/minute accept # Allow HTTP/HTTPS tcp dport {80, 443} accept # Allow ICMP (rate limited) icmp type echo-request limit rate 1/second accept # Log dropped packets log prefix "DROPPED: " drop } chain forward { type filter hook forward priority 0; policy drop; } chain output { type filter hook output priority 0; policy accept; } } ``` #### Firewall Management ```bash # Check firewall status ssh root@your-vps "nft list ruleset" # Monitor dropped connections ssh root@your-vps "journalctl -k | grep DROPPED" # Temporary rule addition (emergency access) ssh root@your-vps "nft add rule inet filter input tcp dport 8080 accept" ``` ### Intrusion Detection (fail2ban) #### fail2ban Configuration ```ini # SSH protection jail [sshd] enabled = true port = ssh filter = sshd logpath = /var/log/auth.log maxretry = 3 findtime = 600 bantime = 3600 backend = systemd # HTTP brute force protection [caddy-auth] enabled = true port = http,https filter = caddy-auth logpath = /var/log/caddy/access.log maxretry = 10 findtime = 600 bantime = 1800 # Authentik protection [authentik-auth] enabled = true port = http,https filter = authentik-auth logpath = /opt/authentik/logs/server.log maxretry = 5 findtime = 300 bantime = 3600 ``` #### Intrusion Monitoring ```bash # Check fail2ban status ssh root@your-vps "fail2ban-client status" ssh root@your-vps "fail2ban-client status sshd" # View banned IPs ssh root@your-vps "fail2ban-client get sshd banned" # Unban IP (if needed) ssh root@your-vps "fail2ban-client set sshd unbanip 203.0.113.100" ``` ### TLS/HTTPS Security #### Caddy TLS Configuration ```caddyfile # Automatic HTTPS with security headers (security_headers) { header { # HSTS (HTTP Strict Transport Security) Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" # Content Security Policy Content-Security-Policy "default-src 'self'; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline'" # Additional security headers X-Content-Type-Options "nosniff" X-Frame-Options "DENY" X-XSS-Protection "1; mode=block" Referrer-Policy "strict-origin-when-cross-origin" # Remove server information -Server } } # Apply to all sites *.jnss.me { import security_headers tls { protocols tls1.2 tls1.3 curves x25519 secp384r1 } } ``` #### TLS Monitoring ```bash # Check certificate status curl -vI https://auth.jnss.me/ 2>&1 | grep -E "(certificate|expire|protocol)" # Test TLS configuration nmap --script ssl-enum-ciphers -p 443 your-domain.com # Monitor certificate renewal ssh root@your-vps "journalctl -u caddy | grep -i cert" ``` ## Layer 2: Authentication & Authorization Security ### Authentik Security Configuration #### Password Policy ```yaml # Authentik password policy password_policy: length_min: 12 amount_lowercase: 1 amount_uppercase: 1 amount_digits: 1 amount_symbols: 1 password_field: "password" check_static_rules: true check_have_i_been_pwned: true check_zxcvbn: true zxcvbn_score_threshold: 3 ``` #### Multi-Factor Authentication ```yaml # MFA enforcement policies mfa_policies: - name: "Admin MFA Required" bound_to: "group:admins" authenticators: ["totp", "webauthn"] enforce: true - name: "Sensitive Services MFA" bound_to: "application:admin_*" authenticators: ["totp"] enforce: true - name: "External Access MFA" bound_to: "source_ip:!10.0.0.0/8" authenticators: ["totp", "sms"] enforce: true ``` #### Session Security ```yaml # Session security settings session_security: cookie_secure: true cookie_httponly: true cookie_samesite: "Strict" session_timeout: 3600 # 1 hour remember_me_timeout: 86400 # 24 hours concurrent_sessions_limit: 3 session_fixation_protection: true ``` ### Access Control Policies #### Time-Based Access Control ```yaml # Business hours access policy policies: - name: "Business Hours Only" type: "time" parameters: start_time: "08:00" end_time: "18:00" days: ["monday", "tuesday", "wednesday", "thursday", "friday"] timezone: "UTC" applications: ["admin_dashboard", "financial_systems"] ``` #### IP-Based Access Control ```yaml # Geographic/Network access restrictions policies: - name: "Office Network Only" type: "source_ip" parameters: cidr: ["10.0.0.0/8", "192.168.0.0/16"] applications: ["admin_*", "backup_*"] - name: "Block High-Risk Countries" type: "geoip" parameters: countries: ["CN", "RU", "KP"] action: "deny" applications: ["*"] ``` ## Layer 3: Container Security ### Rootless Container Architecture #### Security Benefits of Rootless Podman ```bash # Rootless container verification ssh root@your-vps "sudo -u authentik podman info | grep -A10 'security'" # User namespace mapping ssh root@your-vps "sudo -u authentik cat /proc/self/uid_map" ssh root@your-vps "sudo -u authentik cat /proc/self/gid_map" # Process tree verification (containers run as unprivileged user) ssh root@your-vps "ps aux | grep -E '(authentik|gitea)'" ``` #### Container User Security ```ini # Container configuration with security [Container] # Run as specific UID/GID (not root) User=963:963 # Preserve supplementary groups for socket access Annotation=run.oci.keep_original_groups=1 # Minimal capabilities AddCapability=IPC_OWNER DropCapability=ALL # Security labels SecurityLabelDisable=true SecurityLabelType=container_t ``` ### Container Isolation #### Filesystem Security ```yaml # Read-only root filesystem where possible security_settings: read_only_root_fs: true temp_fs_mounts: - "/tmp:rw,noexec,nosuid,size=100M" - "/var/tmp:rw,noexec,nosuid,size=100M" # Minimal volume mounts volume_mounts: - "/opt/authentik/data:/data:Z" # SELinux label - "/opt/authentik/media:/media:Z" - "/var/run/postgresql:/var/run/postgresql:Z" - "/var/run/valkey:/var/run/valkey:Z" ``` #### Resource Limits ```ini # systemd resource controls [Service] # Memory limits MemoryMax=1G MemorySwapMax=0 # CPU limits CPUQuota=200% # 2 CPU cores max # Process limits TasksMax=100 LimitNOFILE=1024 # I/O limits IODeviceWriteBandwidthMax=/dev/sda 50M ``` ### SELinux Security #### Container Labeling ```bash # Verify SELinux labels on container volumes ssh root@your-vps "ls -Z /var/run/postgresql/" ssh root@your-vps "ls -Z /opt/authentik/" # Check container SELinux context ssh root@your-vps "sudo -u authentik podman exec authentik-server cat /proc/self/attr/current" # SELinux audit logs ssh root@your-vps "ausearch -m AVC -ts recent" ``` #### Custom SELinux Policies ```bash # Generate custom SELinux policy for containers ssh root@your-vps "audit2allow -a -m rick_containers" # Apply custom policy ssh root@your-vps "semodule -i rick_containers.pp" ``` ## Layer 4: Database & Cache Security ### Unix Socket Security Model #### Socket File Permissions ```bash # Verify socket permissions ssh root@your-vps "ls -la /var/run/postgresql/" # Should show: srwxrwx--- postgres postgres .s.PGSQL.5432 ssh root@your-vps "ls -la /var/run/valkey/" # Should show: srwxrwx--- valkey valkey valkey.sock # Verify group access ssh root@your-vps "groups authentik" # Should include: postgres valkey ``` #### Network Isolation ```yaml # Database configuration (no network exposure) postgresql_config: listen_addresses: '' # Unix sockets only port: 0 # Disable TCP port unix_socket_directories: '/var/run/postgresql' unix_socket_permissions: 0770 valkey_config: bind: '' # No network binding port: 0 # Disable TCP port unixsocket: '/var/run/valkey/valkey.sock' unixsocketperm: 770 ``` ### Database Security Hardening #### PostgreSQL Security ```sql -- Database security configuration -- Remove public schema permissions REVOKE ALL ON SCHEMA public FROM PUBLIC; -- Create application-specific schemas CREATE SCHEMA authentik AUTHORIZATION authentik; CREATE SCHEMA gitea AUTHORIZATION gitea; -- Row-level security (if needed) ALTER TABLE authentik.sensitive_data ENABLE ROW LEVEL SECURITY; CREATE POLICY user_data_policy ON authentik.sensitive_data FOR ALL TO authentik USING (user_id = current_setting('app.user_id')); -- Audit logging CREATE EXTENSION IF NOT EXISTS pgaudit; ALTER SYSTEM SET pgaudit.log = 'all'; ``` #### Database Access Monitoring ```bash # Monitor database connections ssh root@your-vps "sudo -u postgres psql -h /var/run/postgresql -c 'SELECT datname, usename, application_name, client_addr FROM pg_stat_activity;'" # Check for suspicious queries ssh root@your-vps "sudo -u postgres psql -h /var/run/postgresql -c 'SELECT query, query_start, state FROM pg_stat_activity WHERE state != '\''idle'\'';'" # Review authentication logs ssh root@your-vps "grep -i 'authentication\|connection' /var/log/postgresql/postgresql.log" ``` ### Cache Security #### Valkey/Redis Security Configuration ```bash # Valkey security settings CONFIG SET requirepass "strong_password" CONFIG SET rename-command FLUSHDB "" CONFIG SET rename-command FLUSHALL "" CONFIG SET rename-command DEBUG "" CONFIG SET rename-command CONFIG "CONFIG_b3a0f6c8" # Disable dangerous commands CONFIG SET protected-mode yes CONFIG SET save "" # Disable automatic snapshots ``` #### Cache Access Control ```bash # Verify cache authentication ssh root@your-vps "redis-cli -s /var/run/valkey/valkey.sock -a \$VALKEY_PASSWORD ping" # Monitor cache operations ssh root@your-vps "redis-cli -s /var/run/valkey/valkey.sock -a \$VALKEY_PASSWORD monitor" # Check cache configuration ssh root@your-vps "redis-cli -s /var/run/valkey/valkey.sock -a \$VALKEY_PASSWORD CONFIG GET '*'" ``` ## Layer 5: Application Security ### Secret Management #### Ansible Vault Security ```yaml # Vault encryption verification vault_security: encryption_method: "AES256" key_derivation: "PBKDF2" vault_password_source: "prompt" # Never store in files # Vault best practices vault_practices: - rotate_vault_password: "quarterly" - separate_vault_files: "by_environment" - backup_encrypted_vaults: "securely" - audit_vault_access: "regularly" ``` #### Environment Variable Security ```bash # Verify secrets are not in process environment ssh root@your-vps "sudo -u authentik cat /proc/\$(pgrep -f authentik-server)/environ | tr '\0' '\n' | grep -v SECRET" # Check for secrets in logs ssh root@your-vps "journalctl --user -M authentik@ -u authentik-server | grep -i 'password\|secret\|token'" ``` ### Secure Headers and Middleware #### Application-Level Security Headers ```python # Example application security middleware security_middleware = { 'SECURE_BROWSER_XSS_FILTER': True, 'SECURE_CONTENT_TYPE_NOSNIFF': True, 'SECURE_HSTS_SECONDS': 31536000, 'SECURE_HSTS_INCLUDE_SUBDOMAINS': True, 'SECURE_HSTS_PRELOAD': True, 'SECURE_REDIRECT_EXEMPT': [], 'SECURE_SSL_REDIRECT': True, 'SESSION_COOKIE_SECURE': True, 'SESSION_COOKIE_HTTPONLY': True, 'SESSION_COOKIE_SAMESITE': 'Strict', 'CSRF_COOKIE_SECURE': True, 'X_FRAME_OPTIONS': 'DENY' } ``` #### Content Security Policy ```http # Strict CSP for web applications Content-Security-Policy: default-src 'self'; script-src 'self' 'unsafe-inline' https://cdn.example.com; style-src 'self' 'unsafe-inline'; img-src 'self' data: https:; font-src 'self' https://fonts.gstatic.com; connect-src 'self' https://api.example.com; frame-ancestors 'none'; base-uri 'self'; form-action 'self' ``` ### Audit Logging and Monitoring #### System-Wide Audit Configuration ```bash # auditd configuration for security events auditctl -w /etc/passwd -p wa -k user_modification auditctl -w /etc/shadow -p wa -k password_modification auditctl -w /opt/authentik -p wa -k authentik_access auditctl -w /var/run/postgresql -p wa -k database_access auditctl -w /var/run/valkey -p wa -k cache_access # Monitor systemd service changes auditctl -w /etc/systemd/system -p wa -k service_modification auditctl -w /home/*/.*config/containers/systemd -p wa -k container_modification ``` #### Application Audit Logging ```yaml # Application-level audit events audit_events: authentication: - user_login_success - user_login_failure - password_change - mfa_enrollment - session_timeout authorization: - access_granted - access_denied - privilege_escalation - policy_violation data_access: - sensitive_data_access - data_export - configuration_change - backup_access ``` ### Security Monitoring and Alerting #### Real-Time Security Monitoring ```bash # Security monitoring scripts #!/bin/bash # security-monitor.sh # Monitor failed authentication attempts FAILED_LOGINS=$(grep "Failed password" /var/log/auth.log | wc -l) if [ $FAILED_LOGINS -gt 10 ]; then echo "ALERT: High number of failed login attempts: $FAILED_LOGINS" fi # Monitor unusual database connections DB_CONNECTIONS=$(sudo -u postgres psql -h /var/run/postgresql -t -c "SELECT count(*) FROM pg_stat_activity WHERE application_name NOT IN ('authentik', 'gitea');") if [ $DB_CONNECTIONS -gt 5 ]; then echo "ALERT: Unusual database connections: $DB_CONNECTIONS" fi # Monitor container security violations SELINUX_VIOLATIONS=$(ausearch -m AVC -ts recent 2>/dev/null | wc -l) if [ $SELINUX_VIOLATIONS -gt 0 ]; then echo "ALERT: SELinux violations detected: $SELINUX_VIOLATIONS" fi ``` #### Log Aggregation and Analysis ```yaml # Log analysis configuration log_analysis: sources: - /var/log/auth.log # SSH and system authentication - /var/log/audit/audit.log # System audit events - /opt/authentik/logs/ # Application logs - /var/log/caddy/ # Web server logs - /var/log/postgresql/ # Database logs alerts: - pattern: "Failed password.*from.*" threshold: 5 timeframe: "5m" action: "block_ip" - pattern: "AUTHENTICATION_FAILED.*authentik" threshold: 10 timeframe: "10m" action: "alert_admin" ``` ## Security Best Practices ### Regular Security Maintenance #### Security Updates ```bash # Automated security updates (weekly) #!/bin/bash # security-updates.sh # System package updates pacman -Syu --noconfirm # Container image updates for user in authentik gitea; do sudo -u $user podman auto-update done # Certificate renewal check systemctl reload caddy # Restart services if security updates require it if [ -f /var/run/reboot-required ]; then systemctl reboot fi ``` #### Security Auditing ```bash # Monthly security audit checklist #!/bin/bash # security-audit.sh echo "=== Security Audit Report $(date) ===" # Check for unauthorized users echo "Checking system users:" awk -F: '$3 >= 1000 {print $1, $3}' /etc/passwd # Check sudo access echo "Checking sudo access:" grep -v '^#' /etc/sudoers # Check listening services echo "Checking listening services:" netstat -tulpn | grep LISTEN # Check file permissions on critical files echo "Checking critical file permissions:" ls -la /etc/shadow /etc/passwd /etc/ssh/sshd_config # Check for SUID/SGID files echo "Checking SUID/SGID files:" find /usr -perm /6000 -type f 2>/dev/null # Check container security echo "Checking container security:" for user in authentik gitea; do echo "User: $user" sudo -u $user podman ps --format "table {{.Names}}\t{{.RunningFor}}\t{{.Status}}" done # Check database security echo "Checking database connections:" sudo -u postgres psql -h /var/run/postgresql -c "SELECT datname, usename, client_addr FROM pg_stat_activity;" ``` ### Incident Response Procedures #### Security Incident Response Plan ```yaml incident_response: detection: - automated_alerts: "fail2ban, audit logs, monitoring" - manual_review: "daily log review, weekly security audit" assessment: - severity_levels: ["low", "medium", "high", "critical"] - response_time: critical: "15 minutes" high: "1 hour" medium: "4 hours" low: "24 hours" containment: - isolate_affected_systems: "fail2ban, iptables blocks" - preserve_evidence: "log snapshots, memory dumps" - maintain_operations: "failover procedures" eradication: - remove_threats: "malware removal, account cleanup" - patch_vulnerabilities: "system updates, configuration fixes" - strengthen_defenses: "additional monitoring, access controls" recovery: - restore_services: "service restart, data restoration" - monitor_operations: "enhanced monitoring period" - validate_security: "penetration testing, vulnerability scans" ``` #### Emergency Response Commands ```bash # Emergency security response toolkit #!/bin/bash # emergency-response.sh case "$1" in "isolate") # Emergency network isolation iptables -I INPUT -j DROP iptables -I OUTPUT -j DROP echo "System isolated from network" ;; "lockdown") # Lock all user accounts except emergency admin for user in $(cat /etc/passwd | cut -d: -f1); do if [ "$user" != "root" ] && [ "$user" != "emergency" ]; then usermod -L $user fi done echo "All user accounts locked" ;; "audit") # Emergency audit collection mkdir -p /tmp/emergency-audit cp -r /var/log/* /tmp/emergency-audit/ journalctl --no-pager > /tmp/emergency-audit/journalctl.log netstat -tulpn > /tmp/emergency-audit/network.log ps auxf > /tmp/emergency-audit/processes.log echo "Audit data collected in /tmp/emergency-audit/" ;; "restore") # Restore from isolation iptables -F iptables -P INPUT ACCEPT iptables -P OUTPUT ACCEPT echo "Network access restored - apply proper firewall rules" ;; esac ``` ## Compliance and Standards ### Security Standards Alignment #### CIS (Center for Internet Security) Benchmarks ```yaml cis_compliance: level_1_controls: - filesystem_configuration: "implemented" - software_updates: "automated" - secure_boot_settings: "implemented" - additional_process_hardening: "implemented" - mandatory_access_controls: "selinux_enforcing" level_2_controls: - address_space_layout_randomization: "enabled" - prelink_disabled: "implemented" - activate_gpg_check: "implemented" - disable_automounting: "implemented" ``` #### NIST Cybersecurity Framework ```yaml nist_framework: identify: - asset_management: "complete inventory of systems and services" - risk_assessment: "quarterly risk assessments" protect: - access_control: "authentik SSO with MFA" - awareness_training: "security documentation and procedures" - data_security: "encryption at rest and in transit" detect: - anomalies_events: "automated monitoring and alerting" - security_monitoring: "continuous monitoring of security events" respond: - response_planning: "documented incident response procedures" - communications: "stakeholder notification procedures" recover: - recovery_planning: "documented recovery procedures" - improvements: "lessons learned and security enhancements" ``` ### Documentation and Training #### Security Documentation ```yaml required_documentation: policies: - information_security_policy - access_control_policy - incident_response_policy - backup_recovery_policy procedures: - user_onboarding_offboarding - security_incident_response - vulnerability_management - change_management technical_guides: - system_hardening_guide - application_security_guide - network_security_guide - monitoring_alerting_guide ``` --- This comprehensive security hardening guide implements defense-in-depth security across all layers of the rick-infra stack, emphasizing the security benefits of native database services, Unix socket IPC, rootless containers, and centralized authentication. ## References - **[Authentication Architecture](authentication-architecture.md)** - Detailed authentication and authorization - **[Architecture Decisions](architecture-decisions.md)** - Security-focused architectural choices - **[Authentik Deployment Guide](authentik-deployment-guide.md)** - Secure authentication deployment - **[Service Integration Guide](service-integration-guide.md)** - Secure service integration patterns