rick-infra/docs/authentik-deployment-guide.md

Authentik Deployment Guide

A comprehensive guide for deploying Authentik authentication server with native database services and Unix socket IPC in the rick-infra environment.

Overview

This guide covers the complete deployment process for Authentik, a modern authentication and authorization server, integrated with:

• Native PostgreSQL - High-performance database with Unix socket IPC
• Native Valkey - Redis-compatible cache with Unix socket IPC
• Podman Containers - System-level container orchestration via systemd/Quadlet
• Caddy Reverse Proxy - TLS termination and forward authentication

Architecture Summary

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────────────┐
│   Internet      │    │   Caddy Proxy   │    │  Authentik (systemd)    │
│                 │───▶│ auth.jnss.me    │───▶│  /system.slice/         │
│ HTTPS/443       │    │ TLS + Forward   │    │                         │
└─────────────────┘    │ Auth            │    │ ┌─────────────────────┐ │
                       └─────────────────┘    │ │ Pod + Server/Worker │ │
                                              │ │ User: 966:966       │ │
                                              │ │ Groups: 961,962     │ │
                                              │ └─────────────────────┘ │
                                              └─────────────────────────┘
                                                        │
                       ┌─────────────────────────────────┼─────────────────┐
                       │         Host Infrastructure     │                 │
                       │                                 ▼                 │
                       │  ┌─────────────┐    ┌─────────────────────────┐   │
                       │  │ PostgreSQL  │    │       Valkey            │   │
                       │  │ (Native)    │    │    (Redis-compatible)   │   │
                       │  │ Unix Socket │    │     Unix Socket         │   │
                       │  │ Group: 962  │    │     Group: 961          │   │
                       │  └─────────────┘    └─────────────────────────┘   │
                       └─────────────────────────────────────────────────────┘

Prerequisites

Infrastructure Requirements

Before deploying Authentik, ensure the following infrastructure services are running:

# Verify required services are active
ssh root@your-vps "systemctl is-active postgresql valkey caddy podman"

Expected output: All services should show active

Required Infrastructure Components

1. PostgreSQL Database

   • Native systemd service with Unix socket enabled
   • Socket location: /var/run/postgresql/.s.PGSQL.5432

2. Valkey Cache Service

   • Native systemd service with Unix socket enabled
   • Socket location: /var/run/valkey/valkey.sock

3. Podman Container Runtime

   • Container runtime installed
   • systemd integration (Quadlet) configured

4. Caddy Web Server

   • TLS/SSL termination configured
   • API management enabled for dynamic configuration

DNS Configuration

Ensure DNS records are configured for your authentik domain:

# Verify DNS resolution
dig +short auth.jnss.me
# Should return your VPS IP address

Network Requirements

• Port 80/443: Open for HTTP/HTTPS traffic
• Internal communications: Unix sockets (no additional ports required)

Vault Variables Setup

Required Vault Variables

Create or update host_vars/arch-vps/vault.yml with the following encrypted variables:

---
# Authentik Database Password
vault_authentik_db_password: "secure_random_password_32_chars"

# Authentik Secret Key (generate with: openssl rand -base64 32)
vault_authentik_secret_key: "your_generated_secret_key_here"

# Authentik Admin Password  
vault_authentik_admin_password: "secure_admin_password"

# Infrastructure Dependencies (should already exist)
vault_valkey_password: "valkey_password"

Generate Required Secrets

# Generate Authentik secret key
openssl rand -base64 32

# Generate secure passwords
openssl rand -base64 20

Encrypt Vault File

# Encrypt the vault file
ansible-vault encrypt host_vars/arch-vps/vault.yml

# Verify vault variables
ansible-vault view host_vars/arch-vps/vault.yml

Pre-deployment Validation

Infrastructure Health Check

Run the following commands to verify infrastructure readiness:

# 1. Check PostgreSQL Unix socket
ssh root@your-vps "ls -la /var/run/postgresql/"
# Should show .s.PGSQL.5432 socket file

# 2. Check Valkey Unix socket  
ssh root@your-vps "ls -la /var/run/valkey/"
# Should show valkey.sock file

# 3. Test PostgreSQL connectivity
ssh root@your-vps "sudo -u postgres psql -h /var/run/postgresql -c 'SELECT version();'"
# Should return PostgreSQL version

# 4. Test Valkey connectivity
ssh root@your-vps "redis-cli -s /var/run/valkey/valkey.sock ping"
# Should return "PONG"

# 5. Verify Caddy is responsive
curl -I https://jnss.me/
# Should return HTTP/2 200

Ansible Configuration Validation

# Test Ansible connectivity
ansible arch-vps -m ping

# Verify vault variables can be decrypted
ansible arch-vps -m debug -a "var=vault_authentik_secret_key" --ask-vault-pass

Step-by-Step Deployment

Step 1: Enable Authentik Role

Update site.yml to include the authentik role:

- name: Deploy Core Infrastructure
  hosts: arch-vps
  become: true
  gather_facts: true
  
  roles:
    # Infrastructure dependencies handled automatically via meta/main.yml
    - role: authentik
      tags: ['authentik', 'auth', 'sso']

Step 2: Execute Deployment

# Full deployment with verbose output
ansible-playbook -i inventory/hosts.yml site.yml --tags authentik --ask-vault-pass -v

# Alternative: Deploy with specific components
ansible-playbook -i inventory/hosts.yml site.yml --tags authentik,database,containers --ask-vault-pass

Step 3: Monitor Deployment Progress

During deployment, monitor the following:

# Monitor deployment logs in real-time (separate terminal)
ssh root@your-vps "journalctl -f"

# Watch for authentik-specific services
ssh root@your-vps "systemctl --user -M authentik@ status"

Step 4: Verify Container Deployment

After deployment completion:

# Check systemd services (system scope)
ssh root@your-vps "systemctl list-units 'authentik*'"

# Verify service location
ssh root@your-vps "systemctl status authentik-server | grep CGroup"
# Expected: /system.slice/authentik-server.service

# Verify containers are running
ssh root@your-vps "podman ps"

# Check pod status
ssh root@your-vps "podman pod ps"

Step 5: Health Check Verification

# Test internal HTTP endpoint
ssh root@your-vps "curl -I http://127.0.0.1:9000/"
# Expected: HTTP/1.1 302 Found (redirect to login)

# Test external HTTPS endpoint  
curl -I https://auth.jnss.me/
# Expected: HTTP/2 200 or 302

# Test authentik API endpoint
curl -s https://auth.jnss.me/api/v3/admin/version/
# Expected: JSON response with authentication error (proves API is responsive)

Post-deployment Configuration

Initial Admin Access

1. Access Web Interface:

   # Open in browser
   https://auth.jnss.me/
   

2. Admin Login:

   • Username: admin@auth.jnss.me (or configured admin email)
   • Password: Value from vault_authentik_admin_password

3. Verify Admin Access:

   • Navigate to /if/admin/
   • Confirm admin interface loads successfully
   • Check system status in admin dashboard

Essential Configuration Tasks

1. Configure OAuth2 Provider

# Navigate to Applications → Providers → Create
# Provider Type: OAuth2/OpenID Provider
# Name: "Default OAuth2 Provider"  
# Client Type: Confidential
# Authorization Grant Type: Authorization Code
# Redirect URIs: Add your application callback URLs

2. Create Application

# Navigate to Applications → Applications → Create
# Name: "Your Application Name"
# Slug: "your-app"
# Provider: Select OAuth2 provider created above
# Launch URL: Your application URL

3. Configure Forward Auth (for Caddy integration)

# Navigate to Applications → Providers → Create
# Provider Type: Proxy Provider
# Name: "Forward Auth Provider"
# External Host: https://your-service.jnss.me
# Internal Host: http://localhost:8080 (your service backend)

Service Integration Examples

Example 1: Protect Existing HTTP Service with Forward Auth

Add to your service's Caddy configuration:

# In /etc/caddy/sites-enabled/myservice.caddy
myservice.jnss.me {
    # Forward authentication to authentik
    forward_auth https://auth.jnss.me {
        uri /outpost.goauthentik.io/auth/caddy
        copy_headers Remote-User Remote-Name Remote-Email Remote-Groups
    }
    
    # Your service backend
    reverse_proxy localhost:8080
}

Example 2: OAuth2 Integration for Custom Applications

For applications that can handle OAuth2 directly:

# Application configuration
OAUTH2_PROVIDER_URL: "https://auth.jnss.me/application/o/authorize/"
OAUTH2_TOKEN_URL: "https://auth.jnss.me/application/o/token/"
OAUTH2_USER_INFO_URL: "https://auth.jnss.me/application/o/userinfo/"
OAUTH2_CLIENT_ID: "your_client_id"
OAUTH2_CLIENT_SECRET: "your_client_secret"
OAUTH2_REDIRECT_URI: "https://yourapp.jnss.me/oauth/callback"

Troubleshooting Guide

Common Issues and Solutions

Issue: Containers fail to start with socket permission errors

Symptoms:

Error: failed to connect to database: permission denied

Solution:

# Check authentik user group membership
ssh root@your-vps "groups authentik"
# Should show: authentik postgres-clients valkey-clients

# Verify container process groups
ssh root@your-vps "ps aux | grep authentik-server | head -1 | awk '{print \$2}' | xargs -I {} cat /proc/{}/status | grep Groups"
# Should show: Groups: 961 962 966 (valkey-clients postgres-clients authentik)

# Verify socket permissions
ssh root@your-vps "ls -la /var/run/postgresql/ /var/run/valkey/"

# Fix group membership if missing
ansible-playbook site.yml --tags authentik,user,setup --ask-vault-pass

Issue: HTTP binding errors (address already in use)

Symptoms:

Error: bind: address already in use (port 9000)

Solution:

# Check what's using port 9000
ssh root@your-vps "netstat -tulpn | grep 9000"

# Stop conflicting services
ssh root@your-vps "systemctl stop authentik-pod"

# Restart with correct configuration
ansible-playbook site.yml --tags authentik,containers --ask-vault-pass

Issue: Database connection failures

Symptoms:

FATAL: database "authentik" does not exist

Solution:

# Recreate database and user
ansible-playbook site.yml --tags authentik,database --ask-vault-pass

# Verify database creation
ssh root@your-vps "sudo -u postgres psql -h /var/run/postgresql -c '\l'" 

Issue: Cache connection failures

Symptoms:

Error connecting to Redis: Connection refused

Solution:

# Check Valkey service status
ssh root@your-vps "systemctl status valkey"

# Test socket connectivity
ssh root@your-vps "redis-cli -s /var/run/valkey/valkey.sock ping"

# Redeploy cache configuration if needed
ansible-playbook site.yml --tags authentik,cache --ask-vault-pass

Diagnostic Commands

Container Debugging

# Check container logs
ssh root@your-vps "podman logs authentik-server"
ssh root@your-vps "podman logs authentik-worker"

# Inspect container configuration
ssh root@your-vps "podman inspect authentik-server"

# Check container user/group mapping
ssh root@your-vps "podman exec authentik-server id"
# Expected: uid=966(authentik) gid=966(authentik) groups=966(authentik),961(valkey-clients),962(postgres-clients)

Service Status Verification

# Check all authentik systemd services
ssh root@your-vps "systemctl status authentik-pod authentik-server authentik-worker"

# View service dependencies
ssh root@your-vps "systemctl list-dependencies authentik-pod"

# Verify services are in system.slice
ssh root@your-vps "systemctl status authentik-server | grep CGroup"
# Expected: /system.slice/authentik-server.service

Network Connectivity Testing

# Test internal HTTP binding
ssh root@your-vps "curl -v http://127.0.0.1:9000/"

# Test Caddy reverse proxy
ssh root@your-vps "curl -v http://127.0.0.1:80/ -H 'Host: auth.jnss.me'"

# Test external HTTPS
curl -v https://auth.jnss.me/

Log Analysis

Key Log Locations

Authentik uses journald for centralized logging. Both journalctl and podman logs provide access to the same log stream:

# View Authentik logs via journalctl (system-wide logging)
ssh root@your-vps "journalctl -u authentik-server -f"
ssh root@your-vps "journalctl -u authentik-worker -f"

# View Authentik logs via podman (container-specific)
ssh root@your-vps "podman logs -f authentik-server"
ssh root@your-vps "podman logs -f authentik-worker"

# View recent logs with timestamp (last 50 lines)
ssh root@your-vps "journalctl -u authentik-server --lines 50 --no-pager"

# Filter logs by time
ssh root@your-vps "journalctl -u authentik-server --since '10 minutes ago'"
ssh root@your-vps "journalctl -u authentik-server --since '2025-12-14 16:00:00'"

# Search logs for specific patterns
ssh root@your-vps "journalctl -u authentik-server | grep ERROR"
ssh root@your-vps "journalctl -u authentik-worker | grep 'database connection'"

# Caddy logs for reverse proxy issues
ssh root@your-vps "journalctl -u caddy -f"

Note: Logs are in JSON format with structured fields (timestamp, level, logger, event, etc.).

Common Log Patterns

Authentik logs are in JSON format for structured analysis. Here are common patterns:

Successful API request:

{"auth_via": "secret_key", "domain_url": "0.0.0.0", "event": "/api/v3/outposts/proxy/", 
 "level": "info", "logger": "authentik.asgi", "method": "GET", "status": 200, 
 "timestamp": "2025-12-14T16:13:17.269312"}

Startup and initialization:

{"event": "updating brand certificates", "level": "info", 
 "logger": "authentik.router.brand_tls", "timestamp": "2025-12-14T16:13:17Z"}

Warning patterns:

{"event": "No providers assigned to this outpost, check outpost configuration", 
 "level": "warning", "logger": "authentik.outpost.proxyv2"}

Filtering JSON logs by level:

# Filter by error level
ssh root@your-vps "journalctl -u authentik-server --since today | grep '\"level\":\"error\"'"

# Filter by specific event
ssh root@your-vps "journalctl -u authentik-server | grep '\"event\":\"database connection\"'"

Performance Monitoring

Resource Usage Monitoring

# Monitor container resource usage
ssh root@your-vps "podman stats"

# Monitor service memory usage
ssh root@your-vps "systemctl status authentik-server | grep Memory"

# Monitor database connections
ssh root@your-vps "sudo -u postgres psql -h /var/run/postgresql -c 'SELECT * FROM pg_stat_activity;'"

Performance Optimization Tips

1. Database Performance:

   • Monitor PostgreSQL slow query log
   • Consider database connection pooling for high traffic
   • Regular database maintenance (VACUUM, ANALYZE)

2. Cache Performance:

   • Monitor Valkey memory usage and hit rate
   • Adjust cache TTL settings based on usage patterns
   • Consider cache warming for frequently accessed data

3. Container Performance:

   • Monitor container memory limits and usage
   • Optimize shared memory configuration if needed
   • Review worker process configuration

Maintenance Tasks

Regular Maintenance

Update Authentik Version

# Update version in defaults or inventory
authentik_version: "2025.12.1"  # New version

# Deploy update
ansible-playbook site.yml --tags authentik,containers --ask-vault-pass

Backup Procedures

# Database backup
ssh root@your-vps "sudo -u postgres pg_dump -h /var/run/postgresql authentik > /backup/authentik-$(date +%Y%m%d).sql"

# Media files backup
ssh root@your-vps "tar -czf /backup/authentik-media-$(date +%Y%m%d).tar.gz -C /opt/authentik media"

# Configuration backup (run from ansible control machine)
ansible-vault view host_vars/arch-vps/vault.yml > backup/authentik-vault-$(date +%Y%m%d).yml

Health Monitoring

Set up regular health checks:

#!/bin/bash
# Health check script
HEALTH_URL="https://auth.jnss.me/if/health/live/"
if ! curl -f -s "$HEALTH_URL" > /dev/null; then
    echo "Authentik health check failed"
    # Add alerting logic
fi

Security Maintenance

Certificate Monitoring

# Check certificate expiration
ssh root@your-vps "curl -vI https://auth.jnss.me/ 2>&1 | grep expire"

# Caddy handles renewal automatically, but monitor logs
ssh root@your-vps "journalctl -u caddy | grep -i cert"

Security Updates

# Update container images regularly
ansible-playbook site.yml --tags authentik,image-pull --ask-vault-pass

# Monitor for Authentik security advisories
# https://github.com/goauthentik/authentik/security/advisories

Support and Resources

Documentation References

• Authentik Official Documentation: https://docs.goauthentik.io/
• rick-infra Architecture Decisions: docs/architecture-decisions.md
• Service Integration Guide: docs/service-integration-guide.md
• Security Model: docs/security-hardening.md

Community Resources

• Authentik Community Forum: https://community.goauthentik.io/
• GitHub Issues: https://github.com/goauthentik/authentik/issues
• Discord Community: https://discord.gg/jg33eMhnj6

Emergency Procedures

Service Recovery

# Emergency service restart
ssh root@your-vps "systemctl restart authentik-pod"

# Fallback: Direct container management
ssh root@your-vps "podman pod restart authentik"

# Last resort: Full service rebuild
ansible-playbook site.yml --tags authentik --ask-vault-pass --limit arch-vps

Rollback Procedures

# Rollback to previous container version
authentik_version: "previous_working_version"
ansible-playbook site.yml --tags authentik,containers --ask-vault-pass

# Database rollback (if needed)
ssh root@your-vps "sudo -u postgres psql -h /var/run/postgresql authentik < /backup/authentik-backup.sql"

---

Deployment Checklist

Use this checklist to ensure complete deployment:

Pre-deployment

• Infrastructure services (PostgreSQL, Valkey, Caddy, Podman) running
• DNS records configured for auth.jnss.me
• Vault variables configured and encrypted
• Ansible connectivity verified

Deployment

• Authentik role enabled in site.yml
• Deployment executed successfully
• Health checks passing
• Containers running and responsive

Post-deployment

• Admin web interface accessible
• Initial admin login successful
• OAuth2 provider configured
• Test application integration
• Forward auth configuration tested

Production Readiness

• Backup procedures implemented
• Monitoring and alerting configured
• Security review completed
• Documentation updated
• Team training completed

---

This comprehensive deployment guide provides everything needed to successfully deploy and maintain Authentik in the rick-infra environment, emphasizing the security and performance benefits of our native database + Unix socket architecture.