rick-infra/docs/authentik-deployment-guide.md

# Authentik Deployment Guide

A comprehensive guide for deploying Authentik authentication server with native database services and Unix socket IPC in the rick-infra environment.

## Overview

This guide covers the complete deployment process for Authentik, a modern authentication and authorization server, integrated with:

- **Native PostgreSQL** - High-performance database with Unix socket IPC
- **Native Valkey** - Redis-compatible cache with Unix socket IPC  
- **Podman Containers** - System-level container orchestration via systemd/Quadlet
- **Caddy Reverse Proxy** - TLS termination, OAuth2/OIDC provider, and forward authentication fallback

## Architecture Summary

```
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────────────┐
│   Internet      │    │   Caddy Proxy   │    │  Authentik (systemd)    │
│                 │───▶│ auth.jnss.me    │───▶│  /system.slice/         │
│ HTTPS/443       │    │ TLS + Forward   │    │                         │
└─────────────────┘    │ Auth            │    │ ┌─────────────────────┐ │
                       └─────────────────┘    │ │ Pod + Server/Worker │ │
                                              │ │ User: 966:966       │ │
                                              │ │ Groups: 961,962     │ │
                                              │ └─────────────────────┘ │
                                              └─────────────────────────┘
                                                        │
                       ┌─────────────────────────────────┼─────────────────┐
                       │         Host Infrastructure     │                 │
                       │                                 ▼                 │
                       │  ┌─────────────┐    ┌─────────────────────────┐   │
                       │  │ PostgreSQL  │    │       Valkey            │   │
                       │  │ (Native)    │    │    (Redis-compatible)   │   │
                       │  │ Unix Socket │    │     Unix Socket         │   │
                       │  │ Group: 962  │    │     Group: 961          │   │
                       │  └─────────────┘    └─────────────────────────┘   │
                       └─────────────────────────────────────────────────────┘
```

## Prerequisites

### Infrastructure Requirements

Before deploying Authentik, ensure the following infrastructure services are running:

```bash
# Verify required services are active
ssh root@your-vps "systemctl is-active postgresql valkey caddy podman"
```

Expected output: All services should show `active`

### Required Infrastructure Components

1. **PostgreSQL Database** 
   - Native systemd service with Unix socket enabled
   - Socket location: `/var/run/postgresql/.s.PGSQL.5432`

2. **Valkey Cache Service**
   - Native systemd service with Unix socket enabled  
   - Socket location: `/var/run/valkey/valkey.sock`

3. **Podman Container Runtime**
   - Container runtime installed
   - systemd integration (Quadlet) configured

4. **Caddy Web Server**
   - TLS/SSL termination configured
   - API management enabled for dynamic configuration

### DNS Configuration

Ensure DNS records are configured for your authentik domain:

```bash
# Verify DNS resolution
dig +short auth.jnss.me
# Should return your VPS IP address
```

### Network Requirements

- **Port 80/443**: Open for HTTP/HTTPS traffic
- **Internal communications**: Unix sockets (no additional ports required)

## Vault Variables Setup

### Required Vault Variables

Create or update `host_vars/arch-vps/vault.yml` with the following encrypted variables:

```yaml
---
# Authentik Database Password
vault_authentik_db_password: "secure_random_password_32_chars"

# Authentik Secret Key (generate with: openssl rand -base64 32)
vault_authentik_secret_key: "your_generated_secret_key_here"

# Authentik Admin Password  
vault_authentik_admin_password: "secure_admin_password"

# Infrastructure Dependencies (should already exist)
vault_valkey_password: "valkey_password"
```

### Generate Required Secrets

```bash
# Generate Authentik secret key
openssl rand -base64 32

# Generate secure passwords
openssl rand -base64 20
```

### Encrypt Vault File

```bash
# Encrypt the vault file
ansible-vault encrypt host_vars/arch-vps/vault.yml

# Verify vault variables
ansible-vault view host_vars/arch-vps/vault.yml
```

## Pre-deployment Validation

### Infrastructure Health Check

Run the following commands to verify infrastructure readiness:

```bash
# 1. Check PostgreSQL Unix socket
ssh root@your-vps "ls -la /var/run/postgresql/"
# Should show .s.PGSQL.5432 socket file

# 2. Check Valkey Unix socket  
ssh root@your-vps "ls -la /var/run/valkey/"
# Should show valkey.sock file

# 3. Test PostgreSQL connectivity
ssh root@your-vps "sudo -u postgres psql -h /var/run/postgresql -c 'SELECT version();'"
# Should return PostgreSQL version

# 4. Test Valkey connectivity
ssh root@your-vps "redis-cli -s /var/run/valkey/valkey.sock ping"
# Should return "PONG"

# 5. Verify Caddy is responsive
curl -I https://jnss.me/
# Should return HTTP/2 200
```

### Ansible Configuration Validation

```bash
# Test Ansible connectivity
ansible arch-vps -m ping

# Verify vault variables can be decrypted
ansible arch-vps -m debug -a "var=vault_authentik_secret_key" --ask-vault-pass
```

## Step-by-Step Deployment

### Step 1: Enable Authentik Role

Update `site.yml` to include the authentik role:

```yaml
- name: Deploy Core Infrastructure
  hosts: arch-vps
  become: true
  gather_facts: true
  
  roles:
    # Infrastructure dependencies handled automatically via meta/main.yml
    - role: authentik
      tags: ['authentik', 'auth', 'sso']
```

### Step 2: Execute Deployment

```bash
# Full deployment with verbose output
ansible-playbook -i inventory/hosts.yml site.yml --tags authentik --ask-vault-pass -v

# Alternative: Deploy with specific components
ansible-playbook -i inventory/hosts.yml site.yml --tags authentik,database,containers --ask-vault-pass
```

### Step 3: Monitor Deployment Progress

During deployment, monitor the following:

```bash
# Monitor deployment logs in real-time (separate terminal)
ssh root@your-vps "journalctl -f"

# Watch for authentik-specific services
ssh root@your-vps "systemctl --user -M authentik@ status"
```

### Step 4: Verify Container Deployment

After deployment completion:

```bash
# Check systemd services (system scope)
ssh root@your-vps "systemctl list-units 'authentik*'"

# Verify service location
ssh root@your-vps "systemctl status authentik-server | grep CGroup"
# Expected: /system.slice/authentik-server.service

# Verify containers are running
ssh root@your-vps "podman ps"

# Check pod status
ssh root@your-vps "podman pod ps"
```

### Step 5: Health Check Verification

```bash
# Test internal HTTP endpoint
ssh root@your-vps "curl -I http://127.0.0.1:9000/"
# Expected: HTTP/1.1 302 Found (redirect to login)

# Test external HTTPS endpoint  
curl -I https://auth.jnss.me/
# Expected: HTTP/2 200 or 302

# Test authentik API endpoint
curl -s https://auth.jnss.me/api/v3/admin/version/
# Expected: JSON response with authentication error (proves API is responsive)
```

## Post-deployment Configuration

### Initial Admin Access

1. **Access Web Interface**:
   ```bash
   # Open in browser
   https://auth.jnss.me/
   ```

2. **Admin Login**:
   - **Username**: `admin@auth.jnss.me` (or configured admin email)
   - **Password**: Value from `vault_authentik_admin_password`

3. **Verify Admin Access**:
   - Navigate to `/if/admin/` 
   - Confirm admin interface loads successfully
   - Check system status in admin dashboard

### Essential Configuration Tasks

#### 1. Configure OAuth2 Provider

```bash
# Navigate to Applications → Providers → Create
# Provider Type: OAuth2/OpenID Provider
# Name: "Default OAuth2 Provider"  
# Client Type: Confidential
# Authorization Grant Type: Authorization Code
# Redirect URIs: Add your application callback URLs
```

#### 2. Create Application  

```bash
# Navigate to Applications → Applications → Create
# Name: "Your Application Name"
# Slug: "your-app"
# Provider: Select OAuth2 provider created above
# Launch URL: Your application URL
```

#### 3. Configure Forward Auth (for services without OAuth support)

**Note**: Only use Forward Auth for services that don't support OAuth2/OIDC integration. For services like Nextcloud, Gitea, or Grafana, use OAuth2 providers (see step 1 above) instead.

```bash
# Navigate to Applications → Providers → Create
# Provider Type: Proxy Provider
# Name: "Forward Auth Provider"
# External Host: https://your-service.jnss.me
# Internal Host: http://localhost:8080 (your service backend)
# Use this only for: static sites, legacy apps, simple tools without OAuth support
```

## Service Integration Examples

### Choosing the Right Integration Method

**Primary Method - OAuth2/OIDC** (Use when service supports it):
- ✅ **Nextcloud**: Native OIDC support
- ✅ **Gitea**: Native OAuth2 support
- ✅ **Grafana**: Native OAuth2 support
- ✅ **Custom applications**: Applications with OAuth2 client libraries

**Fallback Method - Forward Auth** (Use when service doesn't support OAuth):
- ⚠️ **Static sites**: No authentication capabilities
- ⚠️ **Legacy applications**: Cannot be modified
- ⚠️ **Simple tools**: whoami, monitoring dashboards without auth

---

### Example 1: OAuth2/OIDC Integration (Nextcloud)

**Recommended for services with native OAuth support**

For applications that can handle OAuth2/OIDC directly (like Nextcloud, Gitea):

```yaml
# Nextcloud OIDC configuration (config.php)
'oidc_login_provider_url' => 'https://auth.jnss.me/application/o/nextcloud/',
'oidc_login_client_id' => 'nextcloud-client-id',
'oidc_login_client_secret' => 'your_client_secret',
'oidc_login_auto_redirect' => true,
'oidc_login_end_session_redirect' => true,
'oidc_login_button_text' => 'Login with SSO',
'oidc_login_hide_password_form' => true,
'oidc_login_use_id_token' => true,
'oidc_login_attributes' => [
    'id' => 'preferred_username',
    'name' => 'name',
    'mail' => 'email',
    'groups' => 'groups',
],
'oidc_login_default_group' => 'users',
'oidc_login_scope' => 'openid profile email groups',
'oidc_login_tls_verify' => true,
```

**Caddy configuration** (no forward auth needed):
```caddyfile
# In /etc/caddy/sites-enabled/nextcloud.caddy
cloud.jnss.me {
    reverse_proxy localhost:8080
}
```

---

### Example 2: Forward Auth for Services Without OAuth Support

**Use only when OAuth2/OIDC is not available**

Add to your service's Caddy configuration:

```caddyfile
# In /etc/caddy/sites-enabled/whoami.caddy
whoami.jnss.me {
    # Forward authentication to authentik
    forward_auth https://auth.jnss.me {
        uri /outpost.goauthentik.io/auth/caddy
        copy_headers Remote-User Remote-Name Remote-Email Remote-Groups
    }
    
    # Your service backend
    reverse_proxy localhost:8080
}
```

### Log Analysis

#### Key Log Locations

Authentik uses **journald** for centralized logging. Both `journalctl` and `podman logs` provide access to the same log stream:

```bash
# View Authentik logs via journalctl (system-wide logging)
ssh root@your-vps "journalctl -u authentik-server -f"
ssh root@your-vps "journalctl -u authentik-worker -f"

# View Authentik logs via podman (container-specific)
ssh root@your-vps "podman logs -f authentik-server"
ssh root@your-vps "podman logs -f authentik-worker"

# View recent logs with timestamp (last 50 lines)
ssh root@your-vps "journalctl -u authentik-server --lines 50 --no-pager"

# Filter logs by time
ssh root@your-vps "journalctl -u authentik-server --since '10 minutes ago'"
ssh root@your-vps "journalctl -u authentik-server --since '2025-12-14 16:00:00'"

# Search logs for specific patterns
ssh root@your-vps "journalctl -u authentik-server | grep ERROR"
ssh root@your-vps "journalctl -u authentik-worker | grep 'database connection'"

# Caddy logs for reverse proxy issues
ssh root@your-vps "journalctl -u caddy -f"
```

**Note**: Logs are in JSON format with structured fields (timestamp, level, logger, event, etc.).

#### Common Log Patterns

Authentik logs are in **JSON format** for structured analysis. Here are common patterns:

**Successful API request**:
```json
{"auth_via": "secret_key", "domain_url": "0.0.0.0", "event": "/api/v3/outposts/proxy/", 
 "level": "info", "logger": "authentik.asgi", "method": "GET", "status": 200, 
 "timestamp": "2025-12-14T16:13:17.269312"}
```

**Startup and initialization**:
```json
{"event": "updating brand certificates", "level": "info", 
 "logger": "authentik.router.brand_tls", "timestamp": "2025-12-14T16:13:17Z"}
```

**Warning patterns**:
```json
{"event": "No providers assigned to this outpost, check outpost configuration", 
 "level": "warning", "logger": "authentik.outpost.proxyv2"}
```

**Filtering JSON logs by level**:
```bash
# Filter by error level
ssh root@your-vps "journalctl -u authentik-server --since today | grep '\"level\":\"error\"'"

# Filter by specific event
ssh root@your-vps "journalctl -u authentik-server | grep '\"event\":\"database connection\"'"
```