From a9f814d9291df7dfc05d17443ade90bf617a372e Mon Sep 17 00:00:00 2001 From: Joakim Date: Tue, 16 Dec 2025 00:53:45 +0100 Subject: [PATCH] Add comprehensive devigo deployment documentation - Complete deployment guide with architecture overview - Documented all deployment steps and current status - Added lessons learned from deployment challenges - Comprehensive troubleshooting guide - Maintenance procedures and security considerations - Migration notes from arch-vps to mini-vps Key sections: - OAuth flow and authentication - CI/CD deployment pipeline - MIME type handling strategy - Podman authentication for systemd - Future improvements and technical debt tracking --- docs/devigo-deployment-guide.md | 414 ++++++++++++++++++++++++++++++++ 1 file changed, 414 insertions(+) create mode 100644 docs/devigo-deployment-guide.md diff --git a/docs/devigo-deployment-guide.md b/docs/devigo-deployment-guide.md new file mode 100644 index 0000000..bc2d254 --- /dev/null +++ b/docs/devigo-deployment-guide.md @@ -0,0 +1,414 @@ +# Devigo Deployment Guide + +**Status**: ✅ **DEPLOYED AND OPERATIONAL** +**Environment**: Production (mini-vps) +**Last Updated**: 2025-12-16 +**Deployment Date**: 2025-12-15 + +## Overview + +This guide documents the complete deployment of devigo.no (sales training website) from the legacy arch-vps server to the production mini-vps environment. The deployment uses: + +- **Infrastructure Management**: Ansible (rick-infra repository) +- **Application Code**: Hugo static site (rustan repository) +- **CI/CD**: GitHub Actions +- **Container Runtime**: Podman with Quadlet/systemd integration +- **Web Server**: Caddy (reverse proxy with automatic HTTPS) +- **Content Management**: Decap CMS with GitHub OAuth + +## Architecture + +### Request Flow +``` +Internet → Caddy (443) → nginx container (9080) → Static files + ↓ + TLS termination + Security headers + Access logging +``` + +### Services + +| Service | Domain | Container | Purpose | +|---------|--------|-----------|---------| +| devigo-site | devigo.no | ghcr.io/jnschaffer/rustan:prod | Main website | +| devigo-site | www.devigo.no | (redirects to apex) | WWW redirect | +| devigo-decap-oauth | decap.jnss.me | alukovenko/decapcms-oauth2 | OAuth proxy for CMS | + +### OAuth Flow +``` +1. User visits devigo.no/admin +2. Clicks "Login with GitHub" +3. Popup opens → decap.jnss.me/auth +4. Redirects to GitHub OAuth +5. GitHub redirects → decap.jnss.me/callback +6. Callback validates origin (TRUSTED_ORIGINS) +7. postMessage sends token to parent window +8. Popup closes, CMS authenticated ✅ +``` + +### Deployment Flow +``` +Developer → git push origin/prod + ↓ + GitHub Actions + ↓ + Build Hugo site → Docker image + ↓ + Push to GHCR (ghcr.io/jnschaffer/rustan:prod) + ↓ + SSH to mini-vps → podman pull (uses auth.json) + ↓ + systemctl restart devigo-site + ↓ + Podman Quadlet pulls latest image + ↓ + Container starts with new code ✅ +``` + +## Completed Deployment Steps ✅ + +### Infrastructure Configuration +- ✅ Created Ansible role `roles/devigo/` with Podman Quadlet support +- ✅ Updated `group_vars/production/main.yml` with devigo configuration +- ✅ Encrypted OAuth credentials in `group_vars/production/vault.yml` +- ✅ Updated `playbooks/production.yml` to include devigo role +- ✅ Deployed infrastructure to mini-vps + +### Application Configuration +- ✅ Changed `rustan/hugo.toml` baseURL from `https://www.devigo.no/` to `https://devigo.no/` +- ✅ Updated `rustan/.github/workflows/deploy.yml` for Podman + systemctl deployment +- ✅ Fixed nginx.conf MIME type handling (location-specific YAML override) +- ✅ Implemented JavaScript-based Decap CMS initialization + +### Authentication & Security +- ✅ Created GitHub Personal Access Token for GHCR +- ✅ Configured Podman authentication (`/etc/containers/auth.json`) +- ✅ Added `REGISTRY_AUTH_FILE` environment to Quadlet +- ✅ Fixed OAuth `TRUSTED_ORIGINS` environment variable (plural form) +- ✅ Generated SSH key for GitHub Actions deployment +- ✅ Updated GitHub repository secrets + +### DNS & Networking +- ✅ Updated DNS records to point to mini-vps (72.62.91.251) +- ✅ Configured Caddy reverse proxy +- ✅ Obtained TLS certificates via Let's Encrypt +- ✅ Configured GitHub OAuth app callback URLs + +### Testing & Verification +- ✅ Verified MIME types (HTML: text/html, CSS: text/css, YAML: text/yaml) +- ✅ Tested Decap CMS initialization +- ✅ Verified OAuth authentication flow +- ✅ Confirmed container image pulls work +- ✅ Tested automated GitHub Actions deployment +- ✅ Confirmed site accessible at devigo.no + +## Lessons Learned + +### Critical Fixes Applied + +#### 1. TRUSTED_ORIGINS Variable Name (OAuth) + +**Issue**: OAuth showing "Origin not trusted: https://devigo.no" error + +**Root Cause**: The decapcms-oauth2 service source code expects: +- `TRUSTED_ORIGIN` (singular) for a single origin +- `TRUSTED_ORIGINS` (plural) for comma-separated multiple origins + +We were using `TRUSTED_ORIGIN=https://devigo.no,https://www.devigo.no` which treated the entire string as ONE origin. + +**Solution**: Changed to `TRUSTED_ORIGINS=https://devigo.no,https://www.devigo.no` (plural) + +**Files Modified**: +- `roles/devigo/templates/devigo-decap-oauth.env.j2` +- `roles/devigo/defaults/main.yml` + +#### 2. MIME Type Handling Strategy (nginx) + +**Issue**: Adding `types { text/yaml yml yaml; }` in nginx.conf broke all MIME types: +- HTML → `application/octet-stream` +- CSS → `application/octet-stream` +- JS → `application/octet-stream` + +**Root Cause**: Server-level `types {}` block **replaces** all MIME types, not extends them. + +**Solution**: Use location-specific override instead: +```nginx +location ~ \.(yml|yaml)$ { + default_type text/yaml; +} +``` + +**Outcome**: Standard MIME types work, YAML gets special handling. + +#### 3. Decap CMS Initialization Method + +**Issue**: Decap CMS failing to load config with `` approach + +**Root Cause**: Even with JavaScript initialization, Decap CMS validates Content-Type header when fetching via link tag. + +**Solution**: Switched to JavaScript manual initialization: +```javascript +window.CMS_MANUAL_INIT = true; +CMS.init({ config: { load_config_file: '/admin/config.yml' } }); +``` + +**Benefits**: +- Eliminates YAML MIME type dependency for link element +- More portable across server configurations +- Matches working configuration from previous deployment + +#### 4. Podman Authentication for systemd Services + +**Issue**: `podman pull` worked manually but failed when systemd restarted service + +**Root Cause**: Podman wasn't finding `/etc/containers/auth.json` when run by systemd + +**Solution**: Added environment variable to Quadlet: +```ini +[Service] +Environment=REGISTRY_AUTH_FILE=/etc/containers/auth.json +``` + +**Outcome**: Automated deployments now pull private images successfully. + +## Current Infrastructure State + +### Services Status + +| Service | Status | Container ID | Image Version | +|---------|--------|--------------|---------------| +| devigo-site | ✅ Running | a61788a8a673 | 2025-12-15T23:02:XX | +| devigo-decap-oauth | ✅ Running | 91fc8c4b21e3 | latest | +| caddy | ✅ Running | N/A | Native service | + +### Verification Commands + +```bash +# Check all services +ansible mini-vps -m shell -a "systemctl status devigo-site devigo-decap-oauth caddy --no-pager" + +# View recent logs +ansible mini-vps -m shell -a "journalctl -u devigo-site -n 20 --no-pager" + +# Check containers +ansible mini-vps -m shell -a "podman ps" + +# Test endpoints +curl -I https://devigo.no +curl -I https://www.devigo.no # Should 301 redirect to apex +curl -I https://decap.jnss.me + +# Verify MIME types +curl -I https://devigo.no/ | grep content-type # Should be text/html +curl -I https://devigo.no/admin/config.yml | grep content-type # Should be text/yaml +``` + +## Maintenance Guide + +### Updating Content via Decap CMS + +1. Visit https://devigo.no/admin +2. Click "Login with GitHub" +3. Edit content in the CMS interface +4. Changes automatically commit to GitHub +5. GitHub Actions builds and deploys new version +6. Site updates within 2-3 minutes + +### Updating Application Code + +```bash +# In rustan repository +cd ~/rustan +git checkout prod +# Make changes to code +git add . +git commit -m "Description of changes" +git push origin prod + +# GitHub Actions automatically: +# 1. Builds Hugo site +# 2. Creates Docker image +# 3. Pushes to GHCR +# 4. SSHs to mini-vps +# 5. Pulls latest image +# 6. Restarts service +``` + +### Rotating GitHub PAT for GHCR + +When the Personal Access Token expires: + +```bash +# 1. Create new token at https://github.com/settings/tokens/new?scopes=read:packages + +# 2. Update Ansible vault +cd ~/rick-infra +ansible-vault edit group_vars/production/vault.yml +# Update: vault_github_token: "ghp_NEW_TOKEN_HERE" + +# 3. Deploy updated authentication +ansible-playbook playbooks/production.yml --tags podman + +# 4. Verify +ansible mini-vps -m shell -a "podman login ghcr.io --get-login" +# Should show: jnschaffer +``` + +### Updating OAuth Trusted Origins + +If you need to add/change trusted origins for OAuth: + +```bash +# 1. Update defaults +cd ~/rick-infra +# Edit roles/devigo/defaults/main.yml +# Update: devigo_oauth_trusted_origins: "https://devigo.no,https://new-domain.com" + +# 2. Deploy changes +ansible-playbook playbooks/production.yml --tags devigo + +# 3. Restart OAuth service +ansible mini-vps -m shell -a "systemctl restart devigo-decap-oauth" + +# 4. Test OAuth flow at https://devigo.no/admin +``` + +### Manual Container Updates + +```bash +# Pull latest image manually +ansible mini-vps -m shell -a "podman pull ghcr.io/jnschaffer/rustan:prod" + +# Restart service to use new image +ansible mini-vps -m shell -a "systemctl restart devigo-site" + +# Check auto-update is working +ansible mini-vps -m shell -a "podman auto-update --dry-run" +``` + +## Troubleshooting + +See the [Devigo Role README](../roles/devigo/README.md#troubleshooting) for comprehensive troubleshooting steps. + +### Quick Reference + +**OAuth not working?** +```bash +# Check TRUSTED_ORIGINS format +ansible mini-vps -m shell -a "cat /opt/devigo-oauth/decap-oauth.env | grep TRUSTED" +# Should be: TRUSTED_ORIGINS=https://devigo.no,https://www.devigo.no +``` + +**MIME types wrong?** +```bash +# Check config.yml MIME type +curl -I https://devigo.no/admin/config.yml | grep content-type +# Should be: content-type: text/yaml + +# Check homepage MIME type +curl -I https://devigo.no/ | grep content-type +# Should be: content-type: text/html +``` + +**Can't pull images?** +```bash +# Test authentication +ansible mini-vps -m shell -a "podman login ghcr.io --get-login" +# Should show: jnschaffer (your GitHub username) +``` + +## Security Considerations + +### Credentials Storage + +All sensitive credentials are encrypted in Ansible Vault: +- GitHub OAuth Client ID & Secret +- GitHub Personal Access Token for GHCR +- Deployed files have restrictive permissions (0600 for secrets) + +### Network Security + +- TLS certificates automatically renewed via Caddy + Let's Encrypt +- All HTTP traffic redirected to HTTPS +- Security headers applied by Caddy: + - X-Frame-Options: SAMEORIGIN + - X-Content-Type-Options: nosniff + - X-XSS-Protection: 1; mode=block + - Referrer-Policy: strict-origin-when-cross-origin + - Permissions-Policy: geolocation=(), microphone=(), camera=() + +### Container Security + +- Containers run as non-root where possible +- `NoNewPrivileges=true` in Quadlet files +- Minimal attack surface (nginx:alpine base image) +- Regular updates via GitHub Actions rebuild + +## Related Documentation + +- [Devigo Role README](../roles/devigo/README.md) - Role configuration and usage +- [Podman Role](../roles/podman/README.md) - Container runtime setup +- [Caddy Role](../roles/caddy/README.md) - Reverse proxy configuration +- [Service Integration Guide](./service-integration-guide.md) - How services work together + +## Migration Notes + +### Differences from arch-vps Deployment + +| Aspect | arch-vps (old) | mini-vps (new) | +|--------|----------------|----------------| +| Container | Docker + docker-compose | Podman + Quadlet | +| Service Mgmt | docker-compose | systemd | +| Deployment | Manual / docker-compose pull | GitHub Actions + systemctl | +| OAuth | Same service | Same, but TRUSTED_ORIGINS fixed | +| CMS Init | Link tag (fragile) | JavaScript (reliable) | +| MIME Types | Not documented | Location-specific override | + +### Why We Migrated + +1. **Better systemd integration** - Quadlet generates native systemd units +2. **Ansible consistency** - Declarative infrastructure management +3. **Improved security** - Podman rootless capable, no daemon +4. **Auto-updates** - Podman auto-update support +5. **Better logging** - Native journald integration + +## Success Metrics + +- ✅ Site accessible at https://devigo.no +- ✅ WWW redirect working (www.devigo.no → devigo.no) +- ✅ TLS certificates valid and auto-renewing +- ✅ OAuth authentication functional +- ✅ Decap CMS loads and allows content editing +- ✅ GitHub Actions automated deployment working +- ✅ All MIME types correct (HTML, CSS, JS, YAML) +- ✅ Zero downtime during migration +- ✅ All content preserved from previous deployment + +## Future Improvements + +### Potential Enhancements + +1. **Monitoring**: Add Prometheus metrics collection +2. **Backups**: Automated backup of Decap CMS content (GitHub is primary backup) +3. **Staging Environment**: Deploy to separate staging domain for testing +4. **CDN**: Consider Cloudflare CDN for improved global performance +5. **Image Optimization**: Automated image optimization in build pipeline +6. **A/B Testing**: Framework for testing landing page variations + +### Technical Debt + +None identified - deployment is clean and well-documented. + +## Contact & Support + +For issues or questions: +1. Check this guide's troubleshooting section +2. Review role README files +3. Check systemd/container logs +4. Consult Ansible playbook output + +--- + +**Deployment Status**: ✅ **COMPLETE AND OPERATIONAL** +**Next Review**: When rotating GitHub PAT (1 year from creation)