rick-infra/docs/deployment-guide.md

Deployment Guide

This guide explains how to deploy your infrastructure including core services, authentication, and applications.

Overview

The rick-infra deployment system provides:

• Native Infrastructure: PostgreSQL, Valkey, Podman, Caddy managed by systemd
• Authentication Services: Authentik SSO with forward auth integration
• Application Services: Containerized services with Unix socket IPC
• Security First: All services deployed with security hardening by default

Architecture Overview

┌─────────────────────────────────────────────────────────────┐
│ rick-infra Infrastructure Stack                            │
│                                                             │
│ ┌─────────────────┐  ┌─────────────────┐  ┌───────────────┐ │
│ │   Applications  │  │  Authentication │  │ Reverse Proxy │ │
│ │                 │  │                 │  │               │ │
│ │ Gitea (Git)     │  │ Authentik (SSO) │  │ Caddy (HTTPS) │ │
│ │ Gallery (Media) │  │ Forward Auth    │  │ Auto TLS      │ │
│ │ Custom Services │  │ OAuth2/OIDC     │  │ Load Balance  │ │
│ └─────────────────┘  └─────────────────┘  └───────────────┘ │
│           │                    │                    │       │
│           └────────────────────┼────────────────────┘       │
│                                │                            │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ Infrastructure Services (Native systemd)               │ │
│ │                                                         │ │
│ │ ┌─────────────┐  ┌─────────────┐  ┌─────────────────┐   │ │
│ │ │ PostgreSQL  │  │   Valkey    │  │     Podman      │   │ │
│ │ │ (Database)  │  │ (Cache)     │  │ (Containers)    │   │ │
│ │ │Unix Sockets │  │Unix Sockets │  │ Rootless        │   │ │
│ │ └─────────────┘  └─────────────┘  └─────────────────┘   │ │
│ └─────────────────────────────────────────────────────────┘ │
│                                                             │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ Security Foundation                                     │ │
│ │ SSH Hardening • Firewall • Fail2ban • Updates          │ │
│ └─────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘

Infrastructure Overview

Rick-infra now manages two separate environments:

Homelab (arch-vps)

Personal services and experimentation platform at jnss.me:

• PostgreSQL, Valkey, Podman infrastructure
• Caddy reverse proxy with auto-HTTPS
• Nextcloud (cloud.jnss.me)
• Authentik SSO (auth.jnss.me)
• Gitea (git.jnss.me)

Production (mini-vps)

Client projects requiring high uptime:

• Caddy reverse proxy with auto-HTTPS
• Sigvild Gallery (sigvild.no, api.sigvild.no)

Available Deployments

1. site.yml - Deploy All Environments

Deploys both homelab and production infrastructure.

ansible-playbook site.yml --ask-vault-pass

2. Environment-Specific Deployments

# Deploy only homelab services
ansible-playbook playbooks/homelab.yml --ask-vault-pass

# Deploy only production services
ansible-playbook playbooks/production.yml --ask-vault-pass

# Or use site.yml with limits
ansible-playbook site.yml -l homelab --ask-vault-pass
ansible-playbook site.yml -l production --ask-vault-pass

2. Service-Specific Deployments

Deploy individual components using tags:

# Deploy only infrastructure services
ansible-playbook site.yml --tags postgresql,valkey,podman,caddy --ask-vault-pass

# Deploy only authentication
ansible-playbook site.yml --tags authentik --ask-vault-pass

# Deploy only applications  
ansible-playbook site.yml --tags gitea,sigvild-gallery --ask-vault-pass

3. Security-Only Deployment

# Deploy only security hardening
ansible-playbook playbooks/security.yml --ask-vault-pass

Deployment Patterns

First-Time Complete Deployment

⚠️ Important: First-time deployments include security hardening that may require a system reboot.

Prerequisites

1. VPS Setup: Fresh Arch Linux VPS with root access
2. DNS Configuration: Domain pointed to VPS IP address
3. Vault Variables: Required secrets configured (see Vault Setup)
4. SSH Access: Key-based authentication configured

Step-by-Step First Deployment

# 1. Test connectivity
ansible arch-vps -m ping

# 2. Deploy complete infrastructure stack
ansible-playbook -i inventory/hosts.yml site.yml --ask-vault-pass

# 3. Verify deployment
curl -I https://auth.jnss.me/
curl -I https://git.jnss.me/

What happens during first deployment:

1. Security Hardening (1-2 minutes)

   • System package updates
   • SSH configuration hardening
   • nftables firewall setup
   • fail2ban intrusion detection
   • Kernel security parameters
   • May require reboot if kernel updated

2. Infrastructure Services (2-3 minutes)

   • PostgreSQL database with Unix sockets
   • Valkey cache with Unix sockets
   • Podman rootless container setup
   • Caddy reverse proxy with auto-HTTPS

3. Authentication Service (3-5 minutes)

   • Authentik container deployment
   • Database and cache integration
   • Forward auth configuration
   • Admin user initialization

4. Application Services (2-4 minutes)

   • Gitea Git service
   • Gallery media service
   • Service-specific configurations

Total deployment time: 8-14 minutes for complete stack

Incremental Deployment

For subsequent deployments or updates:

# Deploy only changed components
ansible-playbook site.yml --tags authentik --ask-vault-pass

# Deploy with check mode first (dry run)
ansible-playbook site.yml --check --ask-vault-pass

# Deploy with verbose output for debugging
ansible-playbook site.yml -v --ask-vault-pass

Authentication Deployment

Authentik SSO Server

Rick-infra includes comprehensive authentication via Authentik. For detailed deployment instructions, see the Authentik Deployment Guide.

Quick Authentik Deployment

# Deploy authentik and all dependencies
ansible-playbook site.yml --tags authentik --ask-vault-pass

# Verify authentik deployment
curl -I https://auth.jnss.me/
ssh root@your-vps "systemctl --user -M authentik@ status authentik-pod"

Authentik Integration Pattern

All services in rick-infra use forward authentication through Caddy:

# Example service with authentik protection
myservice.jnss.me {
    forward_auth https://auth.jnss.me {
        uri /outpost.goauthentik.io/auth/caddy
        copy_headers Remote-User Remote-Name Remote-Email Remote-Groups
    }
    
    reverse_proxy localhost:8080
}

Post-Deployment Authentik Setup

1. Access Admin Interface: https://auth.jnss.me/if/admin/
2. Default Admin: Email from authentik_default_admin_email variable
3. Initial Configuration: Follow Authentik Deployment Guide

For complete authentik architecture details, see Architecture Decisions.

Configuration Management

Variable Organization

Rick-infra uses a hybrid approach for variable management:

Group Variables (group_vars/):

• production/main.yml - Production environment configuration
• production/vault.yml - Production secrets (encrypted)

Host Variables (host_vars/):

• arch-vps/main.yml - Homelab configuration
• arch-vps/vault.yml - Homelab secrets (encrypted)

Note: Due to variable loading issues in Ansible 2.20, playbooks manually load variables using include_vars. This ensures reliable variable resolution during execution.

Example: Homelab Configuration

Core infrastructure settings in host_vars/arch-vps/main.yml:

# TLS Configuration
caddy_tls_enabled: true
caddy_domain: "jnss.me"
caddy_tls_email: "{{ vault_caddy_tls_email }}"

# DNS Challenge
caddy_dns_provider: "cloudflare"
cloudflare_api_token: "{{ vault_cloudflare_api_token }}"

# API Configuration
caddy_api_enabled: true
caddy_server_name: "main"

# Logging
caddy_log_level: "INFO"
caddy_log_format: "json"
caddy_systemd_security: true

Vault Variables

Sensitive data in host_vars/arch-vps/vault.yml (encrypted):

# Infrastructure secrets
vault_caddy_tls_email: "admin@jnss.me"
vault_cloudflare_api_token: "your-cloudflare-token"

# Database passwords  
vault_postgresql_password: "secure-postgres-password"
vault_valkey_password: "secure-valkey-password"

# Authentik authentication secrets
vault_authentik_secret_key: "base64-encoded-secret-key"
vault_authentik_admin_password: "secure-admin-password"  
vault_authentik_db_password: "authentik-database-password"

# Application secrets
vault_gitea_secret_key: "gitea-secret-key"
vault_gitea_jwt_secret: "gitea-jwt-secret"

Generate Vault Secrets

# Generate secure passwords and keys
openssl rand -base64 32  # For secret keys
openssl rand -base64 20  # For passwords

# Encrypt vault file
ansible-vault encrypt host_vars/arch-vps/vault.yml

# Edit vault file  
ansible-vault edit host_vars/arch-vps/vault.yml

Monitoring and Verification

Service Health Checks

# Infrastructure services
ssh root@your-vps "systemctl status postgresql valkey caddy podman"

# Authentication services
ssh root@your-vps "systemctl --user -M authentik@ status authentik-pod"

# Application services
ssh root@your-vps "systemctl --user -M gitea@ status gitea"

# Web service accessibility  
curl -I https://auth.jnss.me/
curl -I https://git.jnss.me/

Log Monitoring

# Infrastructure logs
ssh root@your-vps "journalctl -u caddy -f"
ssh root@your-vps "journalctl -u postgresql -f"
ssh root@your-vps "journalctl -u valkey -f"

# Authentication logs
ssh root@your-vps "journalctl --user -M authentik@ -u authentik-server -f"

# Application logs
ssh root@your-vps "journalctl --user -M gitea@ -u gitea -f"

Performance Monitoring

# Container resource usage
ssh root@your-vps "sudo -u authentik podman stats"
ssh root@your-vps "sudo -u gitea podman stats"

# Database performance
ssh root@your-vps "sudo -u postgres psql -h /var/run/postgresql -c 'SELECT * FROM pg_stat_activity;'"

# System resources
ssh root@your-vps "htop"
ssh root@your-vps "df -h"
ssh root@your-vps "free -h"

Security Best Practices

Deployment Security

• Always use vault: Never commit secrets to version control
• Verify connectivity: Test Ansible connection before deployment
• Monitor logs: Watch deployment logs for errors or warnings
• Validate services: Verify all services start correctly after deployment
• Check certificates: Ensure HTTPS certificates are issued correctly

Operational Security

• Regular updates: Keep system packages and container images updated
• Access monitoring: Monitor authentication logs for suspicious activity
• Backup procedures: Regular backups of databases and configurations
• Certificate monitoring: Monitor TLS certificate expiration
• Security scans: Regular security assessments of deployed services

Network Security

• Firewall verification: Ensure only ports 80/443 are exposed externally
• Unix socket usage: Verify database/cache communications use Unix sockets
• TLS encryption: Confirm all external communications use HTTPS
• Access logs: Monitor Caddy access logs for unusual patterns

Troubleshooting Common Issues

Deployment Failures

Issue: Ansible vault decryption fails

# Solution: Verify vault password and file encryption
ansible-vault view host_vars/arch-vps/vault.yml
ansible-vault decrypt host_vars/arch-vps/vault.yml  # Edit, then re-encrypt
ansible-vault encrypt host_vars/arch-vps/vault.yml

Issue: SSH connection failures

# Solution: Verify SSH configuration and keys
ssh-add -l  # List SSH keys
ssh root@your-vps  # Test direct connection
ansible arch-vps -m ping  # Test Ansible connectivity

Issue: Container services fail to start

# Check container user sessions
ssh root@your-vps "loginctl list-users"
ssh root@your-vps "systemctl --user -M authentik@ status"

# Verify container images are available
ssh root@your-vps "sudo -u authentik podman images"

# Check container logs
ssh root@your-vps "sudo -u authentik podman logs authentik-server"

Service Connectivity Issues

Issue: Database connection failures

# Verify PostgreSQL socket exists and is accessible
ssh root@your-vps "ls -la /var/run/postgresql/"
ssh root@your-vps "sudo -u authentik psql -h /var/run/postgresql -U authentik"

# Check PostgreSQL service status  
ssh root@your-vps "systemctl status postgresql"

Issue: Authentication not working

# Check authentik service status
ssh root@your-vps "systemctl --user -M authentik@ status authentik-pod authentik-server"

# Test authentik HTTP endpoint
ssh root@your-vps "curl -I http://127.0.0.1:9000/"

# Check Caddy forward auth configuration
ssh root@your-vps "caddy validate --config /etc/caddy/Caddyfile"

Issue: HTTPS certificate problems

# Check certificate status
curl -vI https://auth.jnss.me/ 2>&1 | grep -E "(certificate|expire)"

# Check Caddy certificate management
ssh root@your-vps "journalctl -u caddy | grep -i cert"

# Verify DNS records
dig +short auth.jnss.me

Performance Issues

Issue: Slow database queries

# Monitor active connections
ssh root@your-vps "sudo -u postgres psql -h /var/run/postgresql -c 'SELECT * FROM pg_stat_activity;'"

# Check database performance
ssh root@your-vps "sudo -u postgres psql -h /var/run/postgresql -c 'SELECT * FROM pg_stat_database;'"

Issue: High memory usage

# Check system memory
ssh root@your-vps "free -h"

# Check container memory usage  
ssh root@your-vps "sudo -u authentik podman stats"

# Check service memory limits
ssh root@your-vps "systemctl --user -M authentik@ show authentik-server --property=MemoryCurrent"

Maintenance Procedures

Regular Maintenance Tasks

System Updates

# Update system packages
ansible arch-vps -m pacman -a "update_cache=yes upgrade=yes" --become

# Update container images
ansible-playbook site.yml --tags containers,image-pull --ask-vault-pass

Certificate Renewal

# Certificates renew automatically, but monitor logs
ssh root@your-vps "journalctl -u caddy | grep -i 'certificate\|renewal'"

# Force certificate renewal if needed
ssh root@your-vps "systemctl reload caddy"

Database Maintenance

# PostgreSQL maintenance
ssh root@your-vps "sudo -u postgres psql -h /var/run/postgresql -c 'VACUUM ANALYZE;'"

# Database backups
ssh root@your-vps "sudo -u postgres pg_dumpall -h /var/run/postgresql > /backup/postgres-backup-\$(date +%Y%m%d).sql"

Backup Procedures

Configuration Backup

# Backup Ansible configurations (run from control machine)
tar -czf rick-infra-backup-$(date +%Y%m%d).tar.gz \
    inventory/ host_vars/ roles/ docs/ *.yml

# Backup vault files separately and securely
ansible-vault view host_vars/arch-vps/vault.yml > secure-backup/vault-$(date +%Y%m%d).yml

Data Backup

# Database backup
ssh root@your-vps "sudo -u postgres pg_dump -h /var/run/postgresql authentik > /backup/authentik-$(date +%Y%m%d).sql"
ssh root@your-vps "sudo -u postgres pg_dump -h /var/run/postgresql gitea > /backup/gitea-$(date +%Y%m%d).sql"

# Application data backup
ssh root@your-vps "tar -czf /backup/authentik-media-$(date +%Y%m%d).tar.gz -C /opt/authentik media"
ssh root@your-vps "tar -czf /backup/gitea-data-$(date +%Y%m%d).tar.gz -C /opt/gitea data"

Documentation References

Core Documentation

• Setup Guide - Initial VPS and Ansible setup
• Authentik Deployment Guide - Detailed authentik deployment
• Architecture Decisions - Technical decision rationale
• Security Hardening - Security configuration details

Service Integration

• Service Integration Guide - Adding new services
• Caddy Configuration - Reverse proxy patterns

Role Documentation

• Authentik Role - Authentication service details
• PostgreSQL Role - Database service details
• Valkey Role - Cache service details
• Caddy Role - Reverse proxy details

---

This comprehensive deployment guide provides the foundation for deploying and maintaining the complete rick-infra stack with emphasis on security, performance, and operational excellence through native database services and Unix socket IPC architecture.