rick-infra/docs/devigo-deployment-guide.md

Devigo Deployment Guide

Status: ✅ DEPLOYED AND OPERATIONAL
Environment: Production (mini-vps)
Last Updated: 2025-12-16
Deployment Date: 2025-12-15

Overview

This guide documents the complete deployment of devigo.no (sales training website) from the legacy arch-vps server to the production mini-vps environment. The deployment uses:

• Infrastructure Management: Ansible (rick-infra repository)
• Application Code: Hugo static site (rustan repository)
• CI/CD: GitHub Actions
• Container Runtime: Podman with Quadlet/systemd integration
• Web Server: Caddy (reverse proxy with automatic HTTPS)
• Content Management: Decap CMS with GitHub OAuth

Architecture

Request Flow

Internet → Caddy (443) → nginx container (9080) → Static files
           ↓
       TLS termination
       Security headers
       Access logging

Services

Service	Domain	Container	Purpose
devigo-site	devigo.no	ghcr.io/jnschaffer/rustan:prod	Main website
devigo-site	www.devigo.no	(redirects to apex)	WWW redirect
devigo-decap-oauth	decap.jnss.me	alukovenko/decapcms-oauth2	OAuth proxy for CMS

OAuth Flow

1. User visits devigo.no/admin
2. Clicks "Login with GitHub"
3. Popup opens → decap.jnss.me/auth
4. Redirects to GitHub OAuth
5. GitHub redirects → decap.jnss.me/callback
6. Callback validates origin (TRUSTED_ORIGINS)
7. postMessage sends token to parent window
8. Popup closes, CMS authenticated ✅

Deployment Flow

Developer → git push origin/prod
           ↓
      GitHub Actions
           ↓
    Build Hugo site → Docker image
           ↓
    Push to GHCR (ghcr.io/jnschaffer/rustan:prod)
           ↓
    SSH to mini-vps → podman pull (uses auth.json)
           ↓
    systemctl restart devigo-site
           ↓
    Podman Quadlet pulls latest image
           ↓
    Container starts with new code ✅

Completed Deployment Steps ✅

Infrastructure Configuration

• ✅ Created Ansible role roles/devigo/ with Podman Quadlet support
• ✅ Updated group_vars/production/main.yml with devigo configuration
• ✅ Encrypted OAuth credentials in group_vars/production/vault.yml
• ✅ Updated playbooks/production.yml to include devigo role
• ✅ Deployed infrastructure to mini-vps

Application Configuration

• ✅ Changed rustan/hugo.toml baseURL from https://www.devigo.no/ to https://devigo.no/
• ✅ Updated rustan/.github/workflows/deploy.yml for Podman + systemctl deployment
• ✅ Fixed nginx.conf MIME type handling (location-specific YAML override)
• ✅ Implemented JavaScript-based Decap CMS initialization

Authentication & Security

• ✅ Created GitHub Personal Access Token for GHCR
• ✅ Configured Podman authentication (/etc/containers/auth.json)
• ✅ Added REGISTRY_AUTH_FILE environment to Quadlet
• ✅ Fixed OAuth TRUSTED_ORIGINS environment variable (plural form)
• ✅ Generated SSH key for GitHub Actions deployment
• ✅ Updated GitHub repository secrets

DNS & Networking

• ✅ Updated DNS records to point to mini-vps (72.62.91.251)
• ✅ Configured Caddy reverse proxy
• ✅ Obtained TLS certificates via Let's Encrypt
• ✅ Configured GitHub OAuth app callback URLs

Testing & Verification

• ✅ Verified MIME types (HTML: text/html, CSS: text/css, YAML: text/yaml)
• ✅ Tested Decap CMS initialization
• ✅ Verified OAuth authentication flow
• ✅ Confirmed container image pulls work
• ✅ Tested automated GitHub Actions deployment
• ✅ Confirmed site accessible at devigo.no

Lessons Learned

Critical Fixes Applied

1. TRUSTED_ORIGINS Variable Name (OAuth)

Issue: OAuth showing "Origin not trusted: https://devigo.no" error

Root Cause: The decapcms-oauth2 service source code expects:

• TRUSTED_ORIGIN (singular) for a single origin
• TRUSTED_ORIGINS (plural) for comma-separated multiple origins

We were using TRUSTED_ORIGIN=https://devigo.no,https://www.devigo.no which treated the entire string as ONE origin.

Solution: Changed to TRUSTED_ORIGINS=https://devigo.no,https://www.devigo.no (plural)

Files Modified:

• roles/devigo/templates/devigo-decap-oauth.env.j2
• roles/devigo/defaults/main.yml

2. MIME Type Handling Strategy (nginx)

Issue: Adding types { text/yaml yml yaml; } in nginx.conf broke all MIME types:

• HTML → application/octet-stream
• CSS → application/octet-stream
• JS → application/octet-stream

Root Cause: Server-level types {} block replaces all MIME types, not extends them.

Solution: Use location-specific override instead:

location ~ \.(yml|yaml)$ {
    default_type text/yaml;
}

Outcome: Standard MIME types work, YAML gets special handling.

3. Decap CMS Initialization Method

Issue: Decap CMS failing to load config with <link href="config.yml" type="text/yaml"> approach

Root Cause: Even with JavaScript initialization, Decap CMS validates Content-Type header when fetching via link tag.

Solution: Switched to JavaScript manual initialization:

window.CMS_MANUAL_INIT = true;
CMS.init({ config: { load_config_file: '/admin/config.yml' } });

Benefits:

• Eliminates YAML MIME type dependency for link element
• More portable across server configurations
• Matches working configuration from previous deployment

4. Podman Authentication for systemd Services

Issue: podman pull worked manually but failed when systemd restarted service

Root Cause: Podman wasn't finding /etc/containers/auth.json when run by systemd

Solution: Added environment variable to Quadlet:

[Service]
Environment=REGISTRY_AUTH_FILE=/etc/containers/auth.json

Outcome: Automated deployments now pull private images successfully.

Current Infrastructure State

Services Status

Service	Status	Container ID	Image Version
devigo-site	✅ Running	a61788a8a673	2025-12-15T23:02:XX
devigo-decap-oauth	✅ Running	91fc8c4b21e3	latest
caddy	✅ Running	N/A	Native service

Verification Commands

# Check all services
ansible mini-vps -m shell -a "systemctl status devigo-site devigo-decap-oauth caddy --no-pager"

# View recent logs
ansible mini-vps -m shell -a "journalctl -u devigo-site -n 20 --no-pager"

# Check containers
ansible mini-vps -m shell -a "podman ps"

# Test endpoints
curl -I https://devigo.no
curl -I https://www.devigo.no  # Should 301 redirect to apex
curl -I https://decap.jnss.me

# Verify MIME types
curl -I https://devigo.no/ | grep content-type  # Should be text/html
curl -I https://devigo.no/admin/config.yml | grep content-type  # Should be text/yaml

Maintenance Guide

Updating Content via Decap CMS

1. Visit https://devigo.no/admin
2. Click "Login with GitHub"
3. Edit content in the CMS interface
4. Changes automatically commit to GitHub
5. GitHub Actions builds and deploys new version
6. Site updates within 2-3 minutes

Updating Application Code

# In rustan repository
cd ~/rustan
git checkout prod
# Make changes to code
git add .
git commit -m "Description of changes"
git push origin prod

# GitHub Actions automatically:
# 1. Builds Hugo site
# 2. Creates Docker image
# 3. Pushes to GHCR
# 4. SSHs to mini-vps
# 5. Pulls latest image
# 6. Restarts service

Rotating GitHub PAT for GHCR

When the Personal Access Token expires:

# 1. Create new token at https://github.com/settings/tokens/new?scopes=read:packages

# 2. Update Ansible vault
cd ~/rick-infra
ansible-vault edit group_vars/production/vault.yml
# Update: vault_github_token: "ghp_NEW_TOKEN_HERE"

# 3. Deploy updated authentication
ansible-playbook playbooks/production.yml --tags podman

# 4. Verify
ansible mini-vps -m shell -a "podman login ghcr.io --get-login"
# Should show: jnschaffer

Updating OAuth Trusted Origins

If you need to add/change trusted origins for OAuth:

# 1. Update defaults
cd ~/rick-infra
# Edit roles/devigo/defaults/main.yml
# Update: devigo_oauth_trusted_origins: "https://devigo.no,https://new-domain.com"

# 2. Deploy changes
ansible-playbook playbooks/production.yml --tags devigo

# 3. Restart OAuth service
ansible mini-vps -m shell -a "systemctl restart devigo-decap-oauth"

# 4. Test OAuth flow at https://devigo.no/admin

Manual Container Updates

# Pull latest image manually
ansible mini-vps -m shell -a "podman pull ghcr.io/jnschaffer/rustan:prod"

# Restart service to use new image
ansible mini-vps -m shell -a "systemctl restart devigo-site"

# Check auto-update is working
ansible mini-vps -m shell -a "podman auto-update --dry-run"

Troubleshooting

See the Devigo Role README for comprehensive troubleshooting steps.

Quick Reference

OAuth not working?

# Check TRUSTED_ORIGINS format
ansible mini-vps -m shell -a "cat /opt/devigo-oauth/decap-oauth.env | grep TRUSTED"
# Should be: TRUSTED_ORIGINS=https://devigo.no,https://www.devigo.no

MIME types wrong?

# Check config.yml MIME type
curl -I https://devigo.no/admin/config.yml | grep content-type
# Should be: content-type: text/yaml

# Check homepage MIME type  
curl -I https://devigo.no/ | grep content-type
# Should be: content-type: text/html

Can't pull images?

# Test authentication
ansible mini-vps -m shell -a "podman login ghcr.io --get-login"
# Should show: jnschaffer (your GitHub username)

Security Considerations

Credentials Storage

All sensitive credentials are encrypted in Ansible Vault:

• GitHub OAuth Client ID & Secret
• GitHub Personal Access Token for GHCR
• Deployed files have restrictive permissions (0600 for secrets)

Network Security

• TLS certificates automatically renewed via Caddy + Let's Encrypt
• All HTTP traffic redirected to HTTPS
• Security headers applied by Caddy:
  • X-Frame-Options: SAMEORIGIN
  • X-Content-Type-Options: nosniff
  • X-XSS-Protection: 1; mode=block
  • Referrer-Policy: strict-origin-when-cross-origin
  • Permissions-Policy: geolocation=(), microphone=(), camera=()

Container Security

• Containers run as non-root where possible
• NoNewPrivileges=true in Quadlet files
• Minimal attack surface (nginx:alpine base image)
• Regular updates via GitHub Actions rebuild

Related Documentation

• Devigo Role README - Role configuration and usage
• Podman Role - Container runtime setup
• Caddy Role - Reverse proxy configuration
• Service Integration Guide - How services work together

Migration Notes

Differences from arch-vps Deployment

Aspect	arch-vps (old)	mini-vps (new)
Container	Docker + docker-compose	Podman + Quadlet
Service Mgmt	docker-compose	systemd
Deployment	Manual / docker-compose pull	GitHub Actions + systemctl
OAuth	Same service	Same, but TRUSTED_ORIGINS fixed
CMS Init	Link tag (fragile)	JavaScript (reliable)
MIME Types	Not documented	Location-specific override

Why We Migrated

1. Better systemd integration - Quadlet generates native systemd units
2. Ansible consistency - Declarative infrastructure management
3. Improved security - Podman rootless capable, no daemon
4. Auto-updates - Podman auto-update support
5. Better logging - Native journald integration

Success Metrics

• ✅ Site accessible at https://devigo.no
• ✅ WWW redirect working (www.devigo.no → devigo.no)
• ✅ TLS certificates valid and auto-renewing
• ✅ OAuth authentication functional
• ✅ Decap CMS loads and allows content editing
• ✅ GitHub Actions automated deployment working
• ✅ All MIME types correct (HTML, CSS, JS, YAML)
• ✅ Zero downtime during migration
• ✅ All content preserved from previous deployment

Future Improvements

Potential Enhancements

1. Monitoring: Add Prometheus metrics collection
2. Backups: Automated backup of Decap CMS content (GitHub is primary backup)
3. Staging Environment: Deploy to separate staging domain for testing
4. CDN: Consider Cloudflare CDN for improved global performance
5. Image Optimization: Automated image optimization in build pipeline
6. A/B Testing: Framework for testing landing page variations

Technical Debt

None identified - deployment is clean and well-documented.

Contact & Support

For issues or questions:

1. Check this guide's troubleshooting section
2. Review role README files
3. Check systemd/container logs
4. Consult Ansible playbook output

---

Deployment Status: ✅ COMPLETE AND OPERATIONAL
Next Review: When rotating GitHub PAT (1 year from creation)