rick-infra/roles/devigo/README.md

# Devigo Ansible Role

Deploys the Devigo sales training website (devigo.no) to production infrastructure using Podman Quadlet and Caddy reverse proxy.

## Description

Devigo is a Hugo-based static marketing website deployed via a hybrid approach:
- **Infrastructure**: Managed by Ansible (this role)
- **Application**: Built and deployed via GitHub Actions CI/CD
- **Runtime**: Podman Quadlet containers with systemd integration
- **Web Server**: Caddy reverse proxy with automatic HTTPS

This role deploys two services:
1. **devigo-site**: The main website container (Hugo + Nginx)
2. **devigo-decap-oauth**: OAuth2 proxy for Decap CMS content management

## Architecture

```
GitHub Actions (CI/CD)
    ↓
Build Hugo site → Docker image → GHCR
    ↓
SSH to mini-vps → systemctl restart devigo-site
    ↓
Podman Quadlet pulls latest image
    ↓
Systemd manages container lifecycle
    ↓
Caddy reverse proxy (HTTPS)
    ↓
Internet (devigo.no)
```

## Requirements

### Control Machine
- Ansible 2.20+
- Vault password for encrypted credentials
- SSH access to production host

### Target Host
- Arch Linux (or compatible)
- Podman 5.7+
- Caddy with Cloudflare DNS plugin
- Systemd with Quadlet support

## Role Variables

### Required Variables (set in group_vars/production/main.yml)

```yaml
# Domains
devigo_domain: "devigo.no"                    # Apex domain (primary)
devigo_www_domain: "www.devigo.no"            # WWW subdomain (redirects to apex)
devigo_primary_domain: "devigo.no"            # Which domain is primary

# OAuth Service
devigo_oauth_domain: "decap.jnss.me"          # OAuth proxy domain
devigo_oauth_client_id: "{{ vault_devigo_oauth_client_id }}"
devigo_oauth_client_secret: "{{ vault_devigo_oauth_client_secret }}"

# IMPORTANT: Must be comma-separated string, NOT YAML array
# The decapcms-oauth2 service requires TRUSTED_ORIGINS (plural) as a simple string
devigo_oauth_trusted_origins: "https://devigo.no,https://www.devigo.no"

# Container Registry
devigo_ghcr_image: "ghcr.io/jnschaffer/rustan:prod"
```

### Optional Variables (defaults/main.yml)

```yaml
devigo_container_name: "devigo-site"          # Container name
devigo_container_port: 80                      # Nginx port inside container
devigo_oauth_user: "devigo-oauth"             # OAuth service user
devigo_oauth_home: "/opt/devigo-oauth"        # OAuth config directory
devigo_oauth_container_name: "devigo-decap-oauth"
devigo_oauth_container_port: 12000            # OAuth service port
```

### Vault Variables (group_vars/production/vault.yml - encrypted)

```yaml
vault_devigo_oauth_client_id: "Ov23liNvuDONNB0dhwj6"
vault_devigo_oauth_client_secret: "5dc4d17cfa7d550dd1b5ade0e33d1c9689bbbfad"
```

## Dependencies

This role depends on:
- `podman` - Container runtime
- `caddy` - Reverse proxy with TLS

These are automatically included via `meta/main.yml`.

## Deployed Services

### 1. devigo-site (Main Website)

**Systemd Unit**: `devigo-site.service`  
**Quadlet File**: `/etc/containers/systemd/devigo-site.container`  
**Container**: `ghcr.io/jnschaffer/rustan:prod`  
**Network**: `caddy` (Podman network)  
**Health Check**: Verifies `/usr/share/nginx/html/index.html` exists

**Features**:
- Auto-update support via `podman auto-update`
- Systemd managed (auto-restart on failure)
- Health monitoring
- Connected to Caddy network for reverse proxy

### 2. devigo-decap-oauth (CMS OAuth Proxy)

**Systemd Unit**: `devigo-decap-oauth.service`  
**Quadlet File**: `/etc/containers/systemd/devigo-decap-oauth.container`  
**Container**: `docker.io/alukovenko/decapcms-oauth2:latest`  
**Port**: `127.0.0.1:12000:12000`  
**Purpose**: OAuth2 proxy for Decap CMS GitHub authentication

**Features**:
- Enables content editing via Decap CMS at `/admin`
- Authenticates with GitHub OAuth app
- Proxied via Caddy at `decap.jnss.me`

## Caddy Configuration

### Site Configuration
- **devigo.no** - Primary domain, serves content
- **www.devigo.no** - Redirects to apex domain
- Reverse proxy to `devigo-site` container via Podman network
- Security headers (X-Frame-Options, CSP, etc.)
- JSON logging to `/var/log/caddy/devigo.log`

### OAuth Configuration
- **decap.jnss.me** - OAuth proxy endpoint
- Reverse proxy to `localhost:12000`
- Required for Decap CMS `/admin` functionality

## GitHub Actions Integration

### Workflow Overview

The site is deployed via GitHub Actions when code is pushed to the `prod` branch:

1. **Build Phase**: Hugo site built in Docker container
2. **Push Phase**: Image pushed to `ghcr.io/jnschaffer/rustan:prod`
3. **Deploy Phase**: SSH to mini-vps, restart systemd service

### Deploy Script (in .github/workflows/deploy.yml)

```yaml
deploy:
  needs: build-and-push
  runs-on: ubuntu-latest
  if: github.ref == 'refs/heads/prod'
  
  steps:
    - name: Deploy to mini-vps
      uses: appleboy/ssh-action@v1.0.0
      with:
        host: ${{ secrets.VPS_HOST }}
        username: ${{ secrets.VPS_USER }}
        key: ${{ secrets.VPS_SSH_KEY }}
        script: |
          # Check for updates
          podman auto-update --dry-run devigo-site
          
          # Restart service to pull and use new image
          systemctl restart devigo-site
          
          # Verify deployment
          sleep 5
          systemctl is-active devigo-site
          podman ps | grep devigo-site
          
          # Cleanup
          podman system prune -f
```

### Required GitHub Secrets

Configure in repository settings (`https://github.com/jnschaffer/rustan/settings/secrets/actions`):

- `VPS_HOST`: `72.62.91.251` (mini-vps IP)
- `VPS_USER`: `root`
- `VPS_SSH_KEY`: SSH private key for mini-vps access

## Usage

### Deploy Infrastructure

```bash
# Deploy all devigo infrastructure
ansible-playbook playbooks/production.yml --tags devigo

# Deploy all production services
ansible-playbook playbooks/production.yml
```

### Manual Operations

```bash
# Check service status
ansible mini-vps -m shell -a "systemctl status devigo-site"
ansible mini-vps -m shell -a "systemctl status devigo-decap-oauth"

# View logs
ansible mini-vps -m shell -a "journalctl -u devigo-site -n 50"
ansible mini-vps -m shell -a "journalctl -u devigo-decap-oauth -n 50"

# Check containers
ansible mini-vps -m shell -a "podman ps"

# Manually restart services
ansible mini-vps -m shell -a "systemctl restart devigo-site"
ansible mini-vps -m shell -a "systemctl restart devigo-decap-oauth"

# Check for available updates
ansible mini-vps -m shell -a "podman auto-update --dry-run"

# Force pull latest image
ansible mini-vps -m shell -a "systemctl restart devigo-site"
```

### Testing Deployment

```bash
# Test site (after DNS update)
curl -I https://devigo.no
curl -I https://www.devigo.no  # Should redirect to apex

# Test OAuth proxy
curl -I https://decap.jnss.me

# Test from control machine with IP
curl -I -k -H "Host: devigo.no" https://72.62.91.251
```

## Content Management

### Decap CMS Access

1. Navigate to `https://devigo.no/admin`
2. Click "Login with GitHub"
3. Authenticate via OAuth proxy (`decap.jnss.me`)
4. Edit content via CMS interface
5. Changes commit to GitHub → triggers rebuild → auto-deploy

### Direct Git Workflow

```bash
cd ~/rustan
# Make changes to content/norwegian/
git add .
git commit -m "Update content"
git push origin prod
# GitHub Actions builds and deploys automatically
```

## DNS Configuration

Required DNS records:

```
devigo.no        A      72.62.91.251
www.devigo.no    CNAME  devigo.no.
decap.jnss.me    A      72.62.91.251
```

TLS certificates are automatically obtained via Caddy + Cloudflare DNS challenge.

## Troubleshooting

### Container Won't Start

```bash
# Check Quadlet file syntax
ansible mini-vps -m shell -a "cat /etc/containers/systemd/devigo-site.container"

# Check systemd status
ansible mini-vps -m shell -a "systemctl status devigo-site"

# View detailed logs
ansible mini-vps -m shell -a "journalctl -u devigo-site -n 100"

# Check if image exists
ansible mini-vps -m shell -a "podman images | grep rustan"
```

### OAuth Not Working

```bash
# Verify OAuth container running
ansible mini-vps -m shell -a "systemctl status devigo-decap-oauth"

# Check OAuth logs
ansible mini-vps -m shell -a "journalctl -u devigo-decap-oauth -n 50"

# Test OAuth endpoint locally
ansible mini-vps -m shell -a "curl -I http://localhost:12000"

# Verify environment variables
ansible mini-vps -m shell -a "cat /opt/devigo-oauth/decap-oauth.env"
```

### TLS Certificate Issues

```bash
# Check Caddy logs
ansible mini-vps -m shell -a "journalctl -u caddy -n 100"

# Verify certificate obtained
ansible mini-vps -m shell -a "ls -la /var/lib/caddy/.local/share/caddy/certificates"

# Force certificate renewal
ansible mini-vps -m shell -a "systemctl reload caddy"
```

### GitHub Actions Deployment Fails

1. Verify secrets are set in GitHub repository
2. Check SSH connectivity: `ssh -i /path/to/key root@72.62.91.251`
3. Verify image was pushed to GHCR: `https://github.com/jnschaffer/rustan/pkgs/container/rustan`
4. Check GitHub Actions logs for detailed error messages

## Automatic Updates (Future Enhancement)

For fully automatic updates without GitHub Actions triggering:

### Systemd Timer Approach

Create a systemd timer to run `podman auto-update` daily:

```ini
# /etc/systemd/system/podman-auto-update.timer
[Unit]
Description=Podman Auto Update Timer

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target
```

```ini
# /etc/systemd/system/podman-auto-update.service  
[Unit]
Description=Podman Auto Update Service

[Service]
Type=oneshot
ExecStart=/usr/bin/podman auto-update
ExecStartPost=/usr/bin/systemctl restart devigo-site
```

Enable timer:
```bash
systemctl enable --now podman-auto-update.timer
```

This provides automatic daily checks for new images independent of GitHub Actions.

## Files Deployed

```
/etc/containers/systemd/
├── devigo-site.container           # Website Quadlet
└── devigo-decap-oauth.container    # OAuth Quadlet

/etc/caddy/sites-enabled/
├── devigo.caddy                    # Site reverse proxy config
└── devigo-decap-oauth.caddy        # OAuth reverse proxy config

/opt/devigo-oauth/
└── decap-oauth.env                 # OAuth environment variables

/var/log/caddy/
├── devigo.log                      # Site access logs
└── devigo-decap-oauth.log          # OAuth access logs
```

## Security Considerations

1. **OAuth Credentials**: Stored encrypted in Ansible Vault
2. **Container Security**: Runs with `NoNewPrivileges=true`
3. **Network Isolation**: Containers only on `caddy` network
4. **TLS**: All traffic encrypted via Cloudflare DNS challenge
5. **SSH Keys**: GitHub Actions uses dedicated SSH key
6. **Image Verification**: Images pulled from trusted GHCR

## Notes

- Hugo version on control machine (v0.152.2) is newer than package.json (v0.144.2) - builds work fine
- Contact form uses Airform.io service (patrick@devigo.no endpoint)
- Build time: ~3-5 seconds locally, ~30-60 seconds in GitHub Actions
- Image size: ~50MB (Hugo + Nginx Alpine-based)
- The role does not build the Hugo site - that's handled by GitHub Actions
- DNS must point to mini-vps (72.62.91.251) before site will be accessible

## Example Playbook

```yaml
---
- name: Deploy Production Services
  hosts: production
  become: true
  roles:
    - role: devigo
      tags: ['devigo', 'website', 'sales', 'oauth']
```

## License

MIT

## Author

Created for rick-infra project by Claude Code

## Troubleshooting

### OAuth "Origin not trusted" Error

**Symptom**: Browser console shows `Origin not trusted: https://devigo.no` when clicking "Login with GitHub"

**Cause**: The `TRUSTED_ORIGINS` environment variable is incorrectly formatted or using the wrong variable name.

**Solution**:
1. Check the deployed env file:
   ```bash
   ansible mini-vps -m shell -a "cat /opt/devigo-oauth/decap-oauth.env"
   ```

2. Verify it contains:
   ```
   TRUSTED_ORIGINS=https://devigo.no,https://www.devigo.no
   ```
   
   **NOT**:
   - `TRUSTED_ORIGIN=` (singular - only supports one origin)
   - `TRUSTED_ORIGINS=['https://...']` (YAML array - incorrect format)

3. If incorrect, ensure `roles/devigo/defaults/main.yml` contains:
   ```yaml
   devigo_oauth_trusted_origins: "https://devigo.no,https://www.devigo.no"
   ```
   
4. Redeploy and restart:
   ```bash
   ansible-playbook playbooks/production.yml --tags devigo
   ansible mini-vps -m shell -a "systemctl restart devigo-decap-oauth"
   ```

### Decap CMS "Response was not yaml" Error

**Symptom**: Decap CMS shows error: `Response for config.yml was not yaml. (Content-Type: application/octet-stream)`

**Cause**: The `/admin/config.yml` file is not being served with `Content-Type: text/yaml`

**Solution**:
1. Verify nginx configuration in the container has the location block:
   ```bash
   ansible mini-vps -m shell -a "podman exec devigo-site grep -A 3 'location.*yml' /etc/nginx/nginx.conf"
   ```

2. Should contain:
   ```nginx
   location ~ \.(yml|yaml)$ {
       default_type text/yaml;
   }
   ```

3. Test the MIME type:
   ```bash
   curl -I https://devigo.no/admin/config.yml | grep content-type
   # Expected: content-type: text/yaml
   ```

4. If missing, check the `rustan` repository's `nginx.conf` file and redeploy.

### HTML/CSS Files Show as "application/octet-stream"

**Symptom**: Site doesn't render correctly, browser shows files as `application/octet-stream`

**Cause**: Server-level `types {}` block in nginx.conf overriding all MIME types

**Solution**:
1. Check nginx.conf in the container:
   ```bash
   ansible mini-vps -m shell -a "podman exec devigo-site cat /etc/nginx/nginx.conf | head -20"
   ```

2. Ensure there is **NO** `types { ... }` block at the server level
3. YAML MIME type should be handled by location block only (see above)
4. Fix in `rustan` repository, commit, and push to trigger rebuild

### Cannot Pull Private GHCR Images

**Symptom**: `podman pull` fails with 401 Unauthorized or authentication errors

**Cause**: Missing or incorrect GitHub Container Registry authentication

**Solution**:
1. Verify auth.json exists:
   ```bash
   ansible mini-vps -m shell -a "ls -la /etc/containers/auth.json"
   # Expected: -rw------- 1 root root
   ```

2. Test authentication:
   ```bash
   ansible mini-vps -m shell -a "podman login ghcr.io --get-login"
   # Expected: jnschaffer (your GitHub username)
   ```

3. If fails, verify the Quadlet has the environment variable:
   ```bash
   ansible mini-vps -m shell -a "systemctl cat devigo-site.service | grep REGISTRY_AUTH_FILE"
   # Expected: Environment=REGISTRY_AUTH_FILE=/etc/containers/auth.json
   ```

4. If missing, redeploy the role:
   ```bash
   ansible-playbook playbooks/production.yml --tags devigo,podman
   ```

### Service Won't Start After GitHub Actions Deployment

**Symptom**: `systemctl status devigo-site` shows failed or inactive

**Causes and Solutions**:

1. **Image pull failed**:
   ```bash
   ansible mini-vps -m shell -a "journalctl -u devigo-site -n 50"
   # Look for "401 Unauthorized" - see authentication troubleshooting above
   ```

2. **Port already in use**:
   ```bash
   ansible mini-vps -m shell -a "ss -tlnp | grep 9080"
   # If occupied, stop conflicting service or change port
   ```

3. **Corrupted image**:
   ```bash
   ansible mini-vps -m shell -a "podman pull --force ghcr.io/jnschaffer/rustan:prod"
   ansible mini-vps -m shell -a "systemctl restart devigo-site"
   ```

## Architecture Decisions

### Why Apex Domain (devigo.no) Over WWW?

**Decision**: Use `devigo.no` as primary, redirect `www.devigo.no` → `devigo.no`

**Rationale**:
- Shorter, cleaner URLs
- Better for SEO (single canonical domain)
- Simpler TLS certificate management
- Modern web convention (www is legacy)

### Why JavaScript Initialization for Decap CMS?

**Decision**: Use `CMS.init()` with manual config loading instead of `<link>` tag

**Rationale**:
- Eliminates dependency on YAML MIME type for `<link>` element
- More portable across different server configurations
- Explicit and debuggable (clear in source what's happening)
- Matches working configuration from previous deployment

**Trade-off**: Slightly more verbose HTML (3 extra lines of JavaScript)

### Why Location Block for YAML MIME Type?

**Decision**: Use `location ~ \.(yml|yaml)$ { default_type text/yaml; }` instead of server-level `types {}` block

**Rationale**:
- Surgical approach - only affects YAML files
- Doesn't override nginx's default MIME types for HTML/CSS/JS
- Easier to understand and maintain
- Prevents accidental breaking of other file types

**Trade-off**: Requires location block in nginx.conf (but this is minimal)

### Why Quadlet Over Docker Compose?

**Decision**: Use Podman Quadlet with systemd integration instead of docker-compose

**Rationale**:
- Native systemd integration (journald logs, dependency management)
- Better security (rootless capable, no daemon required)
- Automatic service management via systemd generators
- Auto-update support via `podman auto-update`
- Aligns with Ansible infrastructure-as-code approach

**Trade-off**: Less familiar to developers used to Docker Compose (but well-documented)

## Related Documentation

- [DEVIGO Deployment Guide](../../docs/devigo-deployment-guide.md) - Complete deployment walkthrough
- [Podman Role](../podman/README.md) - Container runtime configuration
- [Caddy Role](../caddy/README.md) - Reverse proxy and TLS
- [Service Integration Guide](../../docs/service-integration-guide.md) - How services interact

## License

Part of the rick-infra project. See repository root for license information.