rick-infra/docs/devigo-deployment-guide.md

# Devigo Deployment Guide

**Status**: ✅ **DEPLOYED AND OPERATIONAL**  
**Environment**: Production (mini-vps)  
**Last Updated**: 2025-12-16  
**Deployment Date**: 2025-12-15

## Overview

This guide documents the complete deployment of devigo.no (sales training website) from the legacy arch-vps server to the production mini-vps environment. The deployment uses:

- **Infrastructure Management**: Ansible (rick-infra repository)
- **Application Code**: Hugo static site (rustan repository)  
- **CI/CD**: GitHub Actions
- **Container Runtime**: Podman with Quadlet/systemd integration
- **Web Server**: Caddy (reverse proxy with automatic HTTPS)
- **Content Management**: Decap CMS with GitHub OAuth

## Architecture

### Request Flow
```
Internet → Caddy (443) → nginx container (9080) → Static files
           ↓
       TLS termination
       Security headers
       Access logging
```

### Services

| Service | Domain | Container | Purpose |
|---------|--------|-----------|---------|
| devigo-site | devigo.no | ghcr.io/jnschaffer/rustan:prod | Main website |
| devigo-site | www.devigo.no | (redirects to apex) | WWW redirect |
| devigo-decap-oauth | decap.jnss.me | alukovenko/decapcms-oauth2 | OAuth proxy for CMS |

### OAuth Flow
```
1. User visits devigo.no/admin
2. Clicks "Login with GitHub"
3. Popup opens → decap.jnss.me/auth
4. Redirects to GitHub OAuth
5. GitHub redirects → decap.jnss.me/callback
6. Callback validates origin (TRUSTED_ORIGINS)
7. postMessage sends token to parent window
8. Popup closes, CMS authenticated ✅
```

### Deployment Flow
```
Developer → git push origin/prod
           ↓
      GitHub Actions
           ↓
    Build Hugo site → Docker image
           ↓
    Push to GHCR (ghcr.io/jnschaffer/rustan:prod)
           ↓
    SSH to mini-vps → podman pull (uses auth.json)
           ↓
    systemctl restart devigo-site
           ↓
    Podman Quadlet pulls latest image
           ↓
    Container starts with new code ✅
```

## Completed Deployment Steps ✅

### Infrastructure Configuration
- ✅ Created Ansible role `roles/devigo/` with Podman Quadlet support
- ✅ Updated `group_vars/production/main.yml` with devigo configuration
- ✅ Encrypted OAuth credentials in `group_vars/production/vault.yml`
- ✅ Updated `playbooks/production.yml` to include devigo role
- ✅ Deployed infrastructure to mini-vps

### Application Configuration  
- ✅ Changed `rustan/hugo.toml` baseURL from `https://www.devigo.no/` to `https://devigo.no/`
- ✅ Updated `rustan/.github/workflows/deploy.yml` for Podman + systemctl deployment
- ✅ Fixed nginx.conf MIME type handling (location-specific YAML override)
- ✅ Implemented JavaScript-based Decap CMS initialization

### Authentication & Security
- ✅ Created GitHub Personal Access Token for GHCR
- ✅ Configured Podman authentication (`/etc/containers/auth.json`)
- ✅ Added `REGISTRY_AUTH_FILE` environment to Quadlet
- ✅ Fixed OAuth `TRUSTED_ORIGINS` environment variable (plural form)
- ✅ Generated SSH key for GitHub Actions deployment
- ✅ Updated GitHub repository secrets

### DNS & Networking
- ✅ Updated DNS records to point to mini-vps (72.62.91.251)
- ✅ Configured Caddy reverse proxy
- ✅ Obtained TLS certificates via Let's Encrypt
- ✅ Configured GitHub OAuth app callback URLs

### Testing & Verification
- ✅ Verified MIME types (HTML: text/html, CSS: text/css, YAML: text/yaml)
- ✅ Tested Decap CMS initialization
- ✅ Verified OAuth authentication flow
- ✅ Confirmed container image pulls work
- ✅ Tested automated GitHub Actions deployment
- ✅ Confirmed site accessible at devigo.no

## Lessons Learned

### Critical Fixes Applied

#### 1. TRUSTED_ORIGINS Variable Name (OAuth)

**Issue**: OAuth showing "Origin not trusted: https://devigo.no" error

**Root Cause**: The decapcms-oauth2 service source code expects:
- `TRUSTED_ORIGIN` (singular) for a single origin
- `TRUSTED_ORIGINS` (plural) for comma-separated multiple origins

We were using `TRUSTED_ORIGIN=https://devigo.no,https://www.devigo.no` which treated the entire string as ONE origin.

**Solution**: Changed to `TRUSTED_ORIGINS=https://devigo.no,https://www.devigo.no` (plural)

**Files Modified**:
- `roles/devigo/templates/devigo-decap-oauth.env.j2`
- `roles/devigo/defaults/main.yml`

#### 2. MIME Type Handling Strategy (nginx)

**Issue**: Adding `types { text/yaml yml yaml; }` in nginx.conf broke all MIME types:
- HTML → `application/octet-stream`
- CSS → `application/octet-stream`  
- JS → `application/octet-stream`

**Root Cause**: Server-level `types {}` block **replaces** all MIME types, not extends them.

**Solution**: Use location-specific override instead:
```nginx
location ~ \.(yml|yaml)$ {
    default_type text/yaml;
}
```

**Outcome**: Standard MIME types work, YAML gets special handling.

#### 3. Decap CMS Initialization Method

**Issue**: Decap CMS failing to load config with `<link href="config.yml" type="text/yaml">` approach

**Root Cause**: Even with JavaScript initialization, Decap CMS validates Content-Type header when fetching via link tag.

**Solution**: Switched to JavaScript manual initialization:
```javascript
window.CMS_MANUAL_INIT = true;
CMS.init({ config: { load_config_file: '/admin/config.yml' } });
```

**Benefits**:
- Eliminates YAML MIME type dependency for link element
- More portable across server configurations
- Matches working configuration from previous deployment

#### 4. Podman Authentication for systemd Services

**Issue**: `podman pull` worked manually but failed when systemd restarted service

**Root Cause**: Podman wasn't finding `/etc/containers/auth.json` when run by systemd

**Solution**: Added environment variable to Quadlet:
```ini
[Service]
Environment=REGISTRY_AUTH_FILE=/etc/containers/auth.json
```

**Outcome**: Automated deployments now pull private images successfully.

## Current Infrastructure State

### Services Status

| Service | Status | Container ID | Image Version |
|---------|--------|--------------|---------------|
| devigo-site | ✅ Running | a61788a8a673 | 2025-12-15T23:02:XX |
| devigo-decap-oauth | ✅ Running | 91fc8c4b21e3 | latest |
| caddy | ✅ Running | N/A | Native service |

### Verification Commands

```bash
# Check all services
ansible mini-vps -m shell -a "systemctl status devigo-site devigo-decap-oauth caddy --no-pager"

# View recent logs
ansible mini-vps -m shell -a "journalctl -u devigo-site -n 20 --no-pager"

# Check containers
ansible mini-vps -m shell -a "podman ps"

# Test endpoints
curl -I https://devigo.no
curl -I https://www.devigo.no  # Should 301 redirect to apex
curl -I https://decap.jnss.me

# Verify MIME types
curl -I https://devigo.no/ | grep content-type  # Should be text/html
curl -I https://devigo.no/admin/config.yml | grep content-type  # Should be text/yaml
```

## Maintenance Guide

### Updating Content via Decap CMS

1. Visit https://devigo.no/admin
2. Click "Login with GitHub"
3. Edit content in the CMS interface
4. Changes automatically commit to GitHub
5. GitHub Actions builds and deploys new version
6. Site updates within 2-3 minutes

### Updating Application Code

```bash
# In rustan repository
cd ~/rustan
git checkout prod
# Make changes to code
git add .
git commit -m "Description of changes"
git push origin prod

# GitHub Actions automatically:
# 1. Builds Hugo site
# 2. Creates Docker image
# 3. Pushes to GHCR
# 4. SSHs to mini-vps
# 5. Pulls latest image
# 6. Restarts service
```

### Rotating GitHub PAT for GHCR

When the Personal Access Token expires:

```bash
# 1. Create new token at https://github.com/settings/tokens/new?scopes=read:packages

# 2. Update Ansible vault
cd ~/rick-infra
ansible-vault edit group_vars/production/vault.yml
# Update: vault_github_token: "ghp_NEW_TOKEN_HERE"

# 3. Deploy updated authentication
ansible-playbook playbooks/production.yml --tags podman

# 4. Verify
ansible mini-vps -m shell -a "podman login ghcr.io --get-login"
# Should show: jnschaffer
```

### Updating OAuth Trusted Origins

If you need to add/change trusted origins for OAuth:

```bash
# 1. Update defaults
cd ~/rick-infra
# Edit roles/devigo/defaults/main.yml
# Update: devigo_oauth_trusted_origins: "https://devigo.no,https://new-domain.com"

# 2. Deploy changes
ansible-playbook playbooks/production.yml --tags devigo

# 3. Restart OAuth service
ansible mini-vps -m shell -a "systemctl restart devigo-decap-oauth"

# 4. Test OAuth flow at https://devigo.no/admin
```

### Manual Container Updates

```bash
# Pull latest image manually
ansible mini-vps -m shell -a "podman pull ghcr.io/jnschaffer/rustan:prod"

# Restart service to use new image
ansible mini-vps -m shell -a "systemctl restart devigo-site"

# Check auto-update is working
ansible mini-vps -m shell -a "podman auto-update --dry-run"
```

## Troubleshooting

See the [Devigo Role README](../roles/devigo/README.md#troubleshooting) for comprehensive troubleshooting steps.

### Quick Reference

**OAuth not working?**
```bash
# Check TRUSTED_ORIGINS format
ansible mini-vps -m shell -a "cat /opt/devigo-oauth/decap-oauth.env | grep TRUSTED"
# Should be: TRUSTED_ORIGINS=https://devigo.no,https://www.devigo.no
```

**MIME types wrong?**
```bash
# Check config.yml MIME type
curl -I https://devigo.no/admin/config.yml | grep content-type
# Should be: content-type: text/yaml

# Check homepage MIME type  
curl -I https://devigo.no/ | grep content-type
# Should be: content-type: text/html
```

**Can't pull images?**
```bash
# Test authentication
ansible mini-vps -m shell -a "podman login ghcr.io --get-login"
# Should show: jnschaffer (your GitHub username)
```

## Security Considerations

### Credentials Storage

All sensitive credentials are encrypted in Ansible Vault:
- GitHub OAuth Client ID & Secret
- GitHub Personal Access Token for GHCR
- Deployed files have restrictive permissions (0600 for secrets)

### Network Security

- TLS certificates automatically renewed via Caddy + Let's Encrypt
- All HTTP traffic redirected to HTTPS
- Security headers applied by Caddy:
  - X-Frame-Options: SAMEORIGIN
  - X-Content-Type-Options: nosniff
  - X-XSS-Protection: 1; mode=block
  - Referrer-Policy: strict-origin-when-cross-origin
  - Permissions-Policy: geolocation=(), microphone=(), camera=()

### Container Security

- Containers run as non-root where possible
- `NoNewPrivileges=true` in Quadlet files
- Minimal attack surface (nginx:alpine base image)
- Regular updates via GitHub Actions rebuild

## Related Documentation

- [Devigo Role README](../roles/devigo/README.md) - Role configuration and usage
- [Podman Role](../roles/podman/README.md) - Container runtime setup
- [Caddy Role](../roles/caddy/README.md) - Reverse proxy configuration
- [Service Integration Guide](./service-integration-guide.md) - How services work together

## Migration Notes

### Differences from arch-vps Deployment

| Aspect | arch-vps (old) | mini-vps (new) |
|--------|----------------|----------------|
| Container | Docker + docker-compose | Podman + Quadlet |
| Service Mgmt | docker-compose | systemd |
| Deployment | Manual / docker-compose pull | GitHub Actions + systemctl |
| OAuth | Same service | Same, but TRUSTED_ORIGINS fixed |
| CMS Init | Link tag (fragile) | JavaScript (reliable) |
| MIME Types | Not documented | Location-specific override |

### Why We Migrated

1. **Better systemd integration** - Quadlet generates native systemd units
2. **Ansible consistency** - Declarative infrastructure management
3. **Improved security** - Podman rootless capable, no daemon
4. **Auto-updates** - Podman auto-update support
5. **Better logging** - Native journald integration

## Success Metrics

- ✅ Site accessible at https://devigo.no
- ✅ WWW redirect working (www.devigo.no → devigo.no)
- ✅ TLS certificates valid and auto-renewing
- ✅ OAuth authentication functional
- ✅ Decap CMS loads and allows content editing
- ✅ GitHub Actions automated deployment working
- ✅ All MIME types correct (HTML, CSS, JS, YAML)
- ✅ Zero downtime during migration
- ✅ All content preserved from previous deployment

## Future Improvements

### Potential Enhancements

1. **Monitoring**: Add Prometheus metrics collection
2. **Backups**: Automated backup of Decap CMS content (GitHub is primary backup)
3. **Staging Environment**: Deploy to separate staging domain for testing
4. **CDN**: Consider Cloudflare CDN for improved global performance
5. **Image Optimization**: Automated image optimization in build pipeline
6. **A/B Testing**: Framework for testing landing page variations

### Technical Debt

None identified - deployment is clean and well-documented.

## Contact & Support

For issues or questions:
1. Check this guide's troubleshooting section
2. Review role README files
3. Check systemd/container logs
4. Consult Ansible playbook output

---

**Deployment Status**: ✅ **COMPLETE AND OPERATIONAL**  
**Next Review**: When rotating GitHub PAT (1 year from creation)