rick-infra/docs/authentik-deployment-guide.md

Authentik Deployment Guide

A comprehensive guide for deploying Authentik authentication server with native database services and Unix socket IPC in the rick-infra environment.

Overview

This guide covers the complete deployment process for Authentik, a modern authentication and authorization server, integrated with:

• Native PostgreSQL - High-performance database with Unix socket IPC
• Native Valkey - Redis-compatible cache with Unix socket IPC
• Podman Containers - System-level container orchestration via systemd/Quadlet
• Caddy Reverse Proxy - TLS termination, OAuth2/OIDC provider, and forward authentication fallback

Architecture Summary

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────────────┐
│   Internet      │    │   Caddy Proxy   │    │  Authentik (systemd)    │
│                 │───▶│ auth.jnss.me    │───▶│  /system.slice/         │
│ HTTPS/443       │    │ TLS + Forward   │    │                         │
└─────────────────┘    │ Auth            │    │ ┌─────────────────────┐ │
                       └─────────────────┘    │ │ Pod + Server/Worker │ │
                                              │ │ User: 966:966       │ │
                                              │ │ Groups: 961,962     │ │
                                              │ └─────────────────────┘ │
                                              └─────────────────────────┘
                                                        │
                       ┌─────────────────────────────────┼─────────────────┐
                       │         Host Infrastructure     │                 │
                       │                                 ▼                 │
                       │  ┌─────────────┐    ┌─────────────────────────┐   │
                       │  │ PostgreSQL  │    │       Valkey            │   │
                       │  │ (Native)    │    │    (Redis-compatible)   │   │
                       │  │ Unix Socket │    │     Unix Socket         │   │
                       │  │ Group: 962  │    │     Group: 961          │   │
                       │  └─────────────┘    └─────────────────────────┘   │
                       └─────────────────────────────────────────────────────┘

Prerequisites

Infrastructure Requirements

Before deploying Authentik, ensure the following infrastructure services are running:

# Verify required services are active
ssh root@your-vps "systemctl is-active postgresql valkey caddy podman"

Expected output: All services should show active

Required Infrastructure Components

1. PostgreSQL Database

   • Native systemd service with Unix socket enabled
   • Socket location: /var/run/postgresql/.s.PGSQL.5432

2. Valkey Cache Service

   • Native systemd service with Unix socket enabled
   • Socket location: /var/run/valkey/valkey.sock

3. Podman Container Runtime

   • Container runtime installed
   • systemd integration (Quadlet) configured

4. Caddy Web Server

   • TLS/SSL termination configured
   • API management enabled for dynamic configuration

DNS Configuration

Ensure DNS records are configured for your authentik domain:

# Verify DNS resolution
dig +short auth.jnss.me
# Should return your VPS IP address

Network Requirements

• Port 80/443: Open for HTTP/HTTPS traffic
• Internal communications: Unix sockets (no additional ports required)

Vault Variables Setup

Required Vault Variables

Create or update host_vars/arch-vps/vault.yml with the following encrypted variables:

---
# Authentik Database Password
vault_authentik_db_password: "secure_random_password_32_chars"

# Authentik Secret Key (generate with: openssl rand -base64 32)
vault_authentik_secret_key: "your_generated_secret_key_here"

# Authentik Admin Password  
vault_authentik_admin_password: "secure_admin_password"

# Infrastructure Dependencies (should already exist)
vault_valkey_password: "valkey_password"

Generate Required Secrets

# Generate Authentik secret key
openssl rand -base64 32

# Generate secure passwords
openssl rand -base64 20

Encrypt Vault File

# Encrypt the vault file
ansible-vault encrypt host_vars/arch-vps/vault.yml

# Verify vault variables
ansible-vault view host_vars/arch-vps/vault.yml

Pre-deployment Validation

Infrastructure Health Check

Run the following commands to verify infrastructure readiness:

# 1. Check PostgreSQL Unix socket
ssh root@your-vps "ls -la /var/run/postgresql/"
# Should show .s.PGSQL.5432 socket file

# 2. Check Valkey Unix socket  
ssh root@your-vps "ls -la /var/run/valkey/"
# Should show valkey.sock file

# 3. Test PostgreSQL connectivity
ssh root@your-vps "sudo -u postgres psql -h /var/run/postgresql -c 'SELECT version();'"
# Should return PostgreSQL version

# 4. Test Valkey connectivity
ssh root@your-vps "redis-cli -s /var/run/valkey/valkey.sock ping"
# Should return "PONG"

# 5. Verify Caddy is responsive
curl -I https://jnss.me/
# Should return HTTP/2 200

Ansible Configuration Validation

# Test Ansible connectivity
ansible arch-vps -m ping

# Verify vault variables can be decrypted
ansible arch-vps -m debug -a "var=vault_authentik_secret_key" --ask-vault-pass

Step-by-Step Deployment

Step 1: Enable Authentik Role

Update site.yml to include the authentik role:

- name: Deploy Core Infrastructure
  hosts: arch-vps
  become: true
  gather_facts: true
  
  roles:
    # Infrastructure dependencies handled automatically via meta/main.yml
    - role: authentik
      tags: ['authentik', 'auth', 'sso']

Step 2: Execute Deployment

# Full deployment with verbose output
ansible-playbook -i inventory/hosts.yml site.yml --tags authentik --ask-vault-pass -v

# Alternative: Deploy with specific components
ansible-playbook -i inventory/hosts.yml site.yml --tags authentik,database,containers --ask-vault-pass

Step 3: Monitor Deployment Progress

During deployment, monitor the following:

# Monitor deployment logs in real-time (separate terminal)
ssh root@your-vps "journalctl -f"

# Watch for authentik-specific services
ssh root@your-vps "systemctl --user -M authentik@ status"

Step 4: Verify Container Deployment

After deployment completion:

# Check systemd services (system scope)
ssh root@your-vps "systemctl list-units 'authentik*'"

# Verify service location
ssh root@your-vps "systemctl status authentik-server | grep CGroup"
# Expected: /system.slice/authentik-server.service

# Verify containers are running
ssh root@your-vps "podman ps"

# Check pod status
ssh root@your-vps "podman pod ps"

Step 5: Health Check Verification

# Test internal HTTP endpoint
ssh root@your-vps "curl -I http://127.0.0.1:9000/"
# Expected: HTTP/1.1 302 Found (redirect to login)

# Test external HTTPS endpoint  
curl -I https://auth.jnss.me/
# Expected: HTTP/2 200 or 302

# Test authentik API endpoint
curl -s https://auth.jnss.me/api/v3/admin/version/
# Expected: JSON response with authentication error (proves API is responsive)

Post-deployment Configuration

Initial Admin Access

1. Access Web Interface:

   # Open in browser
   https://auth.jnss.me/
   

2. Admin Login:

   • Username: admin@auth.jnss.me (or configured admin email)
   • Password: Value from vault_authentik_admin_password

3. Verify Admin Access:

   • Navigate to /if/admin/
   • Confirm admin interface loads successfully
   • Check system status in admin dashboard

Essential Configuration Tasks

1. Configure OAuth2 Provider

# Navigate to Applications → Providers → Create
# Provider Type: OAuth2/OpenID Provider
# Name: "Default OAuth2 Provider"  
# Client Type: Confidential
# Authorization Grant Type: Authorization Code
# Redirect URIs: Add your application callback URLs

2. Create Application

# Navigate to Applications → Applications → Create
# Name: "Your Application Name"
# Slug: "your-app"
# Provider: Select OAuth2 provider created above
# Launch URL: Your application URL

3. Configure Forward Auth (for services without OAuth support)

Note: Only use Forward Auth for services that don't support OAuth2/OIDC integration. For services like Nextcloud, Gitea, or Grafana, use OAuth2 providers (see step 1 above) instead.

# Navigate to Applications → Providers → Create
# Provider Type: Proxy Provider
# Name: "Forward Auth Provider"
# External Host: https://your-service.jnss.me
# Internal Host: http://localhost:8080 (your service backend)
# Use this only for: static sites, legacy apps, simple tools without OAuth support

Service Integration Examples

Choosing the Right Integration Method

Primary Method - OAuth2/OIDC (Use when service supports it):

• ✅ Nextcloud: Native OIDC support
• ✅ Gitea: Native OAuth2 support
• ✅ Grafana: Native OAuth2 support
• ✅ Custom applications: Applications with OAuth2 client libraries

Fallback Method - Forward Auth (Use when service doesn't support OAuth):

• ⚠️ Static sites: No authentication capabilities
• ⚠️ Legacy applications: Cannot be modified
• ⚠️ Simple tools: whoami, monitoring dashboards without auth

---

Example 1: OAuth2/OIDC Integration (Nextcloud)

Recommended for services with native OAuth support

For applications that can handle OAuth2/OIDC directly (like Nextcloud, Gitea):

# Nextcloud OIDC configuration (config.php)
'oidc_login_provider_url' => 'https://auth.jnss.me/application/o/nextcloud/',
'oidc_login_client_id' => 'nextcloud-client-id',
'oidc_login_client_secret' => 'your_client_secret',
'oidc_login_auto_redirect' => true,
'oidc_login_end_session_redirect' => true,
'oidc_login_button_text' => 'Login with SSO',
'oidc_login_hide_password_form' => true,
'oidc_login_use_id_token' => true,
'oidc_login_attributes' => [
    'id' => 'preferred_username',
    'name' => 'name',
    'mail' => 'email',
    'groups' => 'groups',
],
'oidc_login_default_group' => 'users',
'oidc_login_scope' => 'openid profile email groups',
'oidc_login_tls_verify' => true,

Caddy configuration (no forward auth needed):

# In /etc/caddy/sites-enabled/nextcloud.caddy
cloud.jnss.me {
    reverse_proxy localhost:8080
}

---

Example 2: Forward Auth for Services Without OAuth Support

Use only when OAuth2/OIDC is not available

Add to your service's Caddy configuration:

# In /etc/caddy/sites-enabled/whoami.caddy
whoami.jnss.me {
    # Forward authentication to authentik
    forward_auth https://auth.jnss.me {
        uri /outpost.goauthentik.io/auth/caddy
        copy_headers Remote-User Remote-Name Remote-Email Remote-Groups
    }
    
    # Your service backend
    reverse_proxy localhost:8080
}

Log Analysis

Key Log Locations

Authentik uses journald for centralized logging. Both journalctl and podman logs provide access to the same log stream:

# View Authentik logs via journalctl (system-wide logging)
ssh root@your-vps "journalctl -u authentik-server -f"
ssh root@your-vps "journalctl -u authentik-worker -f"

# View Authentik logs via podman (container-specific)
ssh root@your-vps "podman logs -f authentik-server"
ssh root@your-vps "podman logs -f authentik-worker"

# View recent logs with timestamp (last 50 lines)
ssh root@your-vps "journalctl -u authentik-server --lines 50 --no-pager"

# Filter logs by time
ssh root@your-vps "journalctl -u authentik-server --since '10 minutes ago'"
ssh root@your-vps "journalctl -u authentik-server --since '2025-12-14 16:00:00'"

# Search logs for specific patterns
ssh root@your-vps "journalctl -u authentik-server | grep ERROR"
ssh root@your-vps "journalctl -u authentik-worker | grep 'database connection'"

# Caddy logs for reverse proxy issues
ssh root@your-vps "journalctl -u caddy -f"

Note: Logs are in JSON format with structured fields (timestamp, level, logger, event, etc.).

Common Log Patterns

Authentik logs are in JSON format for structured analysis. Here are common patterns:

Successful API request:

{"auth_via": "secret_key", "domain_url": "0.0.0.0", "event": "/api/v3/outposts/proxy/", 
 "level": "info", "logger": "authentik.asgi", "method": "GET", "status": 200, 
 "timestamp": "2025-12-14T16:13:17.269312"}

Startup and initialization:

{"event": "updating brand certificates", "level": "info", 
 "logger": "authentik.router.brand_tls", "timestamp": "2025-12-14T16:13:17Z"}

Warning patterns:

{"event": "No providers assigned to this outpost, check outpost configuration", 
 "level": "warning", "logger": "authentik.outpost.proxyv2"}

Filtering JSON logs by level:

# Filter by error level
ssh root@your-vps "journalctl -u authentik-server --since today | grep '\"level\":\"error\"'"

# Filter by specific event
ssh root@your-vps "journalctl -u authentik-server | grep '\"event\":\"database connection\"'"