Migrate sigvild-gallery to production environment

- Add multi-environment architecture (homelab + production) - Create production environment (mini-vps) for client projects - Create homelab playbook for arch-vps services - Create production playbook for mini-vps services - Move sigvild-gallery from homelab to production - Restructure variables: group_vars/production + host_vars/arch-vps - Add backup-sigvild.yml playbook with auto-restore functionality - Fix restore logic to check for data before creating directories - Add manual variable loading workaround for Ansible 2.20 - Update all documentation for multi-environment setup - Add ADR-007 documenting multi-environment architecture decision
2025-12-15 16:33:33 +01:00
parent e8b76c6a72
commit ecbeb07ba2
18 changed files with 553 additions and 213 deletions
--- a/docs/architecture-decisions.md
+++ b/docs/architecture-decisions.md
@@ -1127,3 +1127,120 @@ curl -I http://127.0.0.1:9000/
 3. **Hardcoded GIDs** - Rejected for portability; facts provide dynamic resolution
 4. **Directory permissions (777)** - Rejected for security; group-based access more restrictive. This is then later changed again to 777, due to Nextcloud switching from root to www-data, breaking group-based permissions.
 ---
+
+---
+## ADR-007: Multi-Environment Infrastructure Architecture
+
+**Date**: December 2025  
+**Status**: Accepted  
+**Context**: Separation of homelab services from production client projects
+
+### Decision
+
+Rick-infra will manage two separate environments with different purposes and uptime requirements:
+
+1. **Homelab Environment** (arch-vps)
+   - Purpose: Personal services and experimentation
+   - Infrastructure: Full stack (PostgreSQL, Valkey, Podman, Caddy)
+   - Services: Authentik, Nextcloud, Gitea
+   - Uptime requirement: Best effort
+
+2. **Production Environment** (mini-vps)
+   - Purpose: Client projects requiring high uptime
+   - Infrastructure: Minimal (Caddy only)
+   - Services: Sigvild Gallery
+   - Uptime requirement: High availability
+
+### Rationale
+
+**Separation of Concerns**:
+- Personal experiments don't affect client services
+- Client services isolated from homelab maintenance
+- Clear distinction between environments in code
+
+**Infrastructure Optimization**:
+- Production runs minimal services (no PostgreSQL/Valkey overhead)
+- Homelab can be rebooted/upgraded without affecting clients
+- Cost optimization: smaller VPS for production
+
+**Operational Flexibility**:
+- Different backup strategies per environment
+- Different monitoring/alerting levels
+- Independent deployment schedules
+
+### Implementation
+
+**Variable Organization**:
+```
+rick-infra/
+├── group_vars/
+│   └── production/        # Production environment config
+│       ├── main.yml
+│       └── vault.yml
+├── host_vars/
+│   └── arch-vps/          # Homelab host config
+│       ├── main.yml
+│       └── vault.yml
+└── playbooks/
+    ├── homelab.yml        # Homelab deployment
+    ├── production.yml     # Production deployment
+    └── site.yml           # Orchestrates both
+```
+
+**Playbook Structure**:
+- `site.yml` imports both homelab.yml and production.yml
+- Each playbook manually loads variables (Ansible 2.20 workaround)
+- Services deploy only to their designated environment
+
+**Inventory Groups**:
+```yaml
+homelab:
+  hosts:
+    arch-vps:
+      ansible_host: 69.62.119.31
+
+production:
+  hosts:
+    mini-vps:
+      ansible_host: 72.62.91.251
+```
+
+### Migration Example
+
+**Sigvild Gallery Migration** (December 2025):
+- **From**: arch-vps (homelab)
+- **To**: mini-vps (production)
+- **Reason**: Client project requiring higher uptime
+- **Process**: 
+  1. Created backup on arch-vps
+  2. Deployed to mini-vps with automatic restore
+  3. Updated DNS (5 min downtime)
+  4. Removed from arch-vps configuration
+
+### Consequences
+
+**Positive**:
+- Clear separation of personal vs. client services
+- Reduced blast radius for experiments
+- Optimized resource usage per environment
+- Independent scaling and management
+
+**Negative**:
+- Increased complexity in playbook organization
+- Need to manage multiple VPS instances
+- Ansible 2.20 variable loading requires workarounds
+- Duplicate infrastructure code (Caddy on both)
+
+**Neutral**:
+- Services can be migrated between environments with minimal friction
+- Backup/restore procedures work across environments
+- Group_vars vs. host_vars hybrid approach
+
+### Future Considerations
+
+- Consider grouping multiple client projects on production VPS
+- Evaluate if homelab needs full infrastructure stack
+- Monitor for opportunities to share infrastructure between environments
+- Document migration procedures for moving services between environments
+
+---
--- a/docs/deployment-guide.md
+++ b/docs/deployment-guide.md
@@ -43,21 +43,45 @@ The rick-infra deployment system provides:
 └─────────────────────────────────────────────────────────────┘
 ```

+## Infrastructure Overview
+
+Rick-infra now manages **two separate environments**:
+
+### Homelab (arch-vps)
+Personal services and experimentation platform at **jnss.me**:
+- PostgreSQL, Valkey, Podman infrastructure
+- Caddy reverse proxy with auto-HTTPS
+- Nextcloud (cloud.jnss.me)
+- Authentik SSO (auth.jnss.me)
+- Gitea (git.jnss.me)
+
+### Production (mini-vps)
+Client projects requiring high uptime:
+- Caddy reverse proxy with auto-HTTPS
+- Sigvild Gallery (sigvild.no, api.sigvild.no)
+
 ## Available Deployments

-### 1. `site.yml` - Complete Infrastructure Stack
-Deploys the full rick-infra stack with role dependencies automatically managed.
+### 1. `site.yml` - Deploy All Environments
+Deploys both homelab and production infrastructure.

 ```bash
-ansible-playbook -i inventory/hosts.yml site.yml --ask-vault-pass
+ansible-playbook site.yml --ask-vault-pass
 ```

-**What it deploys:**
- **Security Foundation**: SSH hardening, firewall, fail2ban, system updates
- **Infrastructure Services**: PostgreSQL, Valkey, Podman container runtime
- **Reverse Proxy**: Caddy with automatic HTTPS and Cloudflare DNS integration
- **Authentication**: Authentik SSO server with forward auth integration
- **Applications**: Gitea, Gallery, and other configured services
+### 2. Environment-Specific Deployments
+
+```bash
+# Deploy only homelab services
+ansible-playbook playbooks/homelab.yml --ask-vault-pass
+
+# Deploy only production services
+ansible-playbook playbooks/production.yml --ask-vault-pass
+
+# Or use site.yml with limits
+ansible-playbook site.yml -l homelab --ask-vault-pass
+ansible-playbook site.yml -l production --ask-vault-pass
+```

 ### 2. Service-Specific Deployments
 Deploy individual components using tags:
@@ -193,7 +217,21 @@ For complete authentik architecture details, see [Architecture Decisions](archit

 ## Configuration Management

-### Host Variables
+### Variable Organization
+
+Rick-infra uses a hybrid approach for variable management:
+
+**Group Variables** (`group_vars/`):
+- `production/main.yml` - Production environment configuration  
+- `production/vault.yml` - Production secrets (encrypted)
+
+**Host Variables** (`host_vars/`):
+- `arch-vps/main.yml` - Homelab configuration
+- `arch-vps/vault.yml` - Homelab secrets (encrypted)
+
+**Note:** Due to variable loading issues in Ansible 2.20, playbooks manually load variables using `include_vars`. This ensures reliable variable resolution during execution.
+
+### Example: Homelab Configuration

 Core infrastructure settings in `host_vars/arch-vps/main.yml`:

--- a/docs/sigvild-gallery-deployment.md
+++ b/docs/sigvild-gallery-deployment.md
@@ -1,39 +1,50 @@
 # Sigvild Gallery Deployment Guide

+## Overview
+
+Sigvild Wedding Gallery is deployed on **mini-vps** (production environment) for high uptime and reliability. The gallery uses PocketBase API backend with SvelteKit frontend.
+
+**Production Host**: mini-vps (72.62.91.251)  
+**Domains**: sigvild.no (frontend), api.sigvild.no (API)
+
 ## Quick Start

-Deploy the complete Sigvild Wedding Gallery with PocketBase API and SvelteKit frontend.
+Deploy Sigvild Gallery to production:
+
+```bash
+ansible-playbook playbooks/production.yml
+```

 ## Prerequisites Setup

 ### 1. Vault Password Configuration

-Create encrypted passwords for the gallery authentication:
+Encrypted passwords are stored in `group_vars/production/vault.yml`:

 ```bash
-# Create vault passwords (run from rick-infra directory)
-ansible-vault encrypt_string 'your-host-password-here' --name 'vault_sigvild_host_password'
-ansible-vault encrypt_string 'your-guest-password-here' --name 'vault_sigvild_guest_password'
+# Edit production vault (run from rick-infra directory)
+ansible-vault edit group_vars/production/vault.yml
 ```

-Add the encrypted strings to `host_vars/arch-vps/main.yml`:
+Add these variables:

 ```yaml
-# Add to host_vars/arch-vps/main.yml
-vault_sigvild_host_password: !vault |
-          $ANSIBLE_VAULT;1.1;AES256
-          66386439653765386...
-          
-vault_sigvild_guest_password: !vault |
-          $ANSIBLE_VAULT;1.1;AES256  
-          33663065383834313...
+# Production vault variables
+vault_cloudflare_api_token: "your-cloudflare-token"
+vault_caddy_tls_email: "admin@example.com"
+vault_sigvild_host_password: "host-user-password"
+vault_sigvild_guest_password: "guest-user-password"
+vault_pb_su_email: "admin@sigvild.no"
+vault_pb_su_password: "admin-password"
 ```

+**Note**: Use `ansible-vault encrypt group_vars/production/vault.yml` after editing.
+
 ### 2. DNS Configuration

-Ensure these domains point to your server:
- `sigvild.no` → Frontend static site
- `api.sigvild.no` → API backend proxy
+Point these domains to **mini-vps** (72.62.91.251):
+- `sigvild.no` A record → 72.62.91.251
+- `api.sigvild.no` A record → 72.62.91.251

 ### 3. Project Structure

@@ -50,70 +61,92 @@ Ensure the sigvild-gallery project is adjacent to rick-infra:

 ## Deployment Commands

-### Full Infrastructure + Gallery
+### Production Deployment

-Deploy everything including Sigvild Gallery:
+Deploy to production environment (mini-vps):

 ```bash
-ansible-playbook site.yml
-```
+# Deploy complete production stack (Caddy + Sigvild Gallery)
+ansible-playbook playbooks/production.yml

-### Gallery Only
-
-Deploy just the Sigvild Gallery service:
-
-```bash
-ansible-playbook playbooks/deploy-sigvild.yml
+# Or deploy everything with production limit
+ansible-playbook site.yml -l production
 ```

 ### Selective Updates

-Update specific components:
+Update specific components using tags:

 ```bash
 # Frontend only (quick static file updates)
-ansible-playbook site.yml --tags="frontend"
+ansible-playbook playbooks/production.yml --tags="frontend"

 # Backend only (API service updates)  
-ansible-playbook site.yml --tags="backend"
+ansible-playbook playbooks/production.yml --tags="backend"

 # Caddy configuration only
-ansible-playbook site.yml --tags="caddy"
+ansible-playbook playbooks/production.yml --tags="caddy"

 # Just build process (development)
-ansible-playbook site.yml --tags="build"
+ansible-playbook playbooks/production.yml --tags="build"
 ```

+### Backup and Restore
+
+Create backups before major changes:
+
+```bash
+# Create backup (works on any host running sigvild-gallery)
+ansible-playbook playbooks/backup-sigvild.yml -l mini-vps
+
+# Backup is saved to: ~/sigvild-gallery-backup/
+```
+
+**Automatic Restore**: When deploying to a fresh server, the role automatically detects and restores from the latest backup if available.
+
 ## Architecture Overview

+**Production Environment**: mini-vps (72.62.91.251)
+
 ```
 Internet
    ↓
-Caddy (Auto HTTPS)
+Cloudflare DNS → mini-vps
+    ↓
+Caddy (Auto HTTPS with DNS Challenge)
    ├── sigvild.no → /var/www/sigvild-gallery/ (Static Files)
    └── api.sigvild.no → localhost:8090 (PocketBase API)
            ↓
-        Go Binary (sigvild-gallery-server)
+        Go Binary (/opt/sigvild-gallery/sigvild-gallery)
            ↓
-        SQLite Database + File Storage
+        SQLite Database (/opt/sigvild-gallery/pb_data/)
+            └── File Storage (wedding photos)
 ```

+**Key Features**:
+- Automatic HTTPS with Let's Encrypt
+- Cloudflare DNS challenge for certificate validation
+- Security headers and CORS protection
+- SystemD service management
+- Automatic backup/restore capability
+
 ## Service Management

 ### Status Checks

 ```bash
-# Gallery API service
-systemctl status sigvild-gallery
-
-# Caddy web server  
-systemctl status caddy
+# Check services on mini-vps
+ansible mini-vps -a "systemctl status sigvild-gallery"
+ansible mini-vps -a "systemctl status caddy"

 # View gallery logs
-journalctl -u sigvild-gallery -f
+ansible mini-vps -a "journalctl -u sigvild-gallery -n 50 --no-pager"

 # View Caddy logs
-journalctl -u caddy -f
+ansible mini-vps -a "journalctl -u caddy -n 20 --no-pager"
+
+# Check data directory
+ansible mini-vps -a "ls -lh /opt/sigvild-gallery/pb_data/"
 ```

 ### Manual Operations
@@ -216,23 +249,30 @@ journalctl -u caddy | grep -i "acme\|certificate"
 - **CORS restrictions**: API access limited to frontend domain
 - **Rate limiting**: API endpoint protection

-## File Locations
+## File Locations (on mini-vps)

 ### Application Files
- **Binary**: `/opt/sigvild-gallery/sigvild-gallery-server`
- **Database**: `/opt/sigvild-gallery/data/data.db`
- **File uploads**: `/opt/sigvild-gallery/data/storage/`
+- **Binary**: `/opt/sigvild-gallery/sigvild-gallery`
+- **Database**: `/opt/sigvild-gallery/pb_data/data.db`
+- **File uploads**: `/opt/sigvild-gallery/pb_data/storage/`
 - **Frontend**: `/var/www/sigvild-gallery/`
+- **User/Group**: `sigvild:sigvild`

 ### Configuration Files  
 - **Service**: `/etc/systemd/system/sigvild-gallery.service`
 - **Caddy frontend**: `/etc/caddy/sites-enabled/sigvild-frontend.caddy`
 - **Caddy API**: `/etc/caddy/sites-enabled/sigvild-api.caddy`

+### Local Files (on control machine)
+- **Configuration**: `group_vars/production/main.yml`
+- **Secrets**: `group_vars/production/vault.yml` (encrypted)
+- **Backups**: `~/sigvild-gallery-backup/`
+- **Source code**: `~/sigvild-gallery/`
+
 ### Log Files
 - **Service logs**: `journalctl -u sigvild-gallery`
 - **Caddy logs**: `journalctl -u caddy`
- **Access logs**: `/var/log/caddy/sigvild-*.log`
+- **Access logs**: `/var/log/caddy/access.log`

 ## Next Steps After Deployment

@@ -248,15 +288,28 @@ For ongoing development:

 ```bash
 # 1. Make changes to sigvild-gallery project
-cd ../sigvild-gallery
+cd ~/sigvild-gallery

 # 2. Test locally  
 go run . serve &
 cd sigvild-kit && npm run dev

-# 3. Deploy updates
-cd ../rick-infra
-ansible-playbook site.yml --tags="sigvild"
+# 3. Deploy updates to production
+cd ~/rick-infra
+ansible-playbook playbooks/production.yml --tags="sigvild"
 ```

-The deployment system builds locally and transfers assets, so you don't need build tools on the server.
+**Build Process**:
+- Backend: Built locally with `GOOS=linux GOARCH=amd64 go build`
+- Frontend: Built locally with `npm run build` in sigvild-kit/
+- Assets transferred to mini-vps via Ansible
+- No build tools required on the server
+
+## Migration History
+
+**December 2025**: Migrated from arch-vps (homelab) to mini-vps (production)
+- **Reason**: Client project requiring higher uptime reliability
+- **Method**: Backup from arch-vps, automatic restore to mini-vps
+- **Downtime**: ~5 minutes during DNS propagation
+- **Previous host**: arch-vps (69.62.119.31)
+- **Current host**: mini-vps (72.62.91.251)