insertr/TODO.md

Insertr Development TODO

🔍 Architecture Analysis Complete (Dec 2024)

Key Discovery: The architecture is already 90% complete and brilliantly designed! The missing piece is not LocalStorage persistence, but the HTTP server application that implements the API contract both clients expect.

✅ What's Already Built & Working

Complete Foundation

• ✅ Go Content Client - Full REST API client with all CRUD operations (internal/content/client.go)
• ✅ JavaScript API Client - Browser client with same API endpoints (lib/src/core/api-client.js)
• ✅ Content Types - Well-defined data structures (ContentItem, ContentResponse)
• ✅ Mock Backend - Working development server with realistic test data
• ✅ Build-Time Enhancement - Content injection from database → HTML during builds
• ✅ Authentication System - Complete auth flow ready for server integration
• ✅ Professional Editor - Modal forms, validation, smart field detection

Architecture Philosophy Preserved

• ✅ "Tailwind of CMS" - Zero configuration, build-time optimization
• ✅ Framework Agnostic - Works with any static site generator
• ✅ Performance First - Regular visitors get pure static HTML
• ✅ Editor Progressive Enhancement - Editing assets only load when needed

🚨 CRITICAL MISSING PIECE

HTTP Server Application (90% of work remaining)

The CLI client and JavaScript client both expect a server at /api/content/*, but no server exists!

Required API Endpoints:

GET  /api/content/{id}?site_id={site}           # Get single content
GET  /api/content?site_id={site}                # Get all content for site  
GET  /api/content/bulk?site_id={site}&ids[]=... # Bulk get content
PUT  /api/content/{id}                          # Update existing content
POST /api/content                               # Create new content

Current State: Both clients make HTTP calls to these endpoints, but they 404 because no server implements them.

🎯 Immediate Implementation Plan

🔴 Phase 1: HTTP Server (CRITICAL)

Goal: Build the missing server application that implements the API contract

1.1 Go HTTP Server ⭐ HIGHEST PRIORITY

• REST API Server - Implement all 5 required endpoints
• Database Layer - SQLite for development, PostgreSQL for production
• Authentication Middleware - JWT/OAuth integration
• CORS & Security - Proper headers for browser integration
• Content Validation - Input sanitization and type checking

1.2 Integration Testing

• CLI Client Integration - Test all CLI commands work with real server
• JavaScript Client Integration - Test browser editor saves to real server
• End-to-End Workflow - Edit → Save → Build → Deploy cycle

🟡 Phase 2: Production Polish (IMPORTANT)

2.1 Client-Side Enhancements

• Editor-Server Integration - Wire up handleSave to use ApiClient
• Optimistic Updates - Show immediate feedback, sync in background
• Offline Support - LocalStorage cache + sync when online
• Loading States - Professional feedback during saves

2.2 Deployment Pipeline

• Build Triggers - Auto-rebuild sites when content changes
• Multi-Site Support - Handle multiple domains/site IDs
• CDN Integration - Host insertr.js library on CDN
• Database Migrations - Schema versioning and updates

🟢 Phase 3: Advanced Features (NICE-TO-HAVE)

3.1 Content Management Enhancements

• Content Versioning - Track edit history and allow rollbacks
• Content Validation - Advanced validation rules per content type
• Markdown Enhancements - Live preview, toolbar, syntax highlighting
• Media Management - Image upload and asset management

3.2 Developer Experience

• Development Tools - Better debugging and development workflow
• Configuration API - Extensible field type system
• Testing Suite - Comprehensive test coverage
• Documentation - API reference and integration guides

💡 Key Architectural Insights

Why This Architecture is Brilliant

1. Performance First: Regular visitors get pure static HTML with zero CMS overhead
2. Separation of Concerns: Content editing completely separate from site performance
3. Build-Time Optimization: Database content gets "baked into" static HTML during builds
4. Progressive Enhancement: Sites work without JavaScript, editing enhances with JavaScript
5. Framework Agnostic: Works with Hugo, Next.js, Jekyll, Gatsby, vanilla HTML, etc.

Production Flow

Content Edits → HTTP API Server → Database
                     ↓
Static Site Build ← CLI Enhancement ← Database Content
                     ↓  
              Enhanced HTML → CDN/Deploy

Current Working Flow

✅ Browser Editor → (404) Missing Server → ❌
✅ CLI Enhancement ← Mock Data ← ✅
✅ Static HTML Generation ← ✅

Gap: The HTTP server that connects editor saves to database storage.

🗂️ Next Steps: Server Implementation

✅ Implemented - Unified Binary Architecture

✅ COMPLETED: All server functionality integrated into unified binary
cmd/
├── serve.go                             # Runtime API server command
└── enhance.go                          # Build-time enhancement command
internal/
├── api/                                # HTTP handlers and middleware
├── db/                                 # Multi-database layer with sqlc
└── content/                            # Content management logic

Files to Modify

• lib/src/core/editor.js:91 - Wire up ApiClient to handleSave method
• README.md - Add server setup instructions
• docker-compose.yml - Add server for development stack

🎯 Success Criteria

Phase 1 Complete When:

• ✅ HTTP server running on localhost:8080 (or configurable port)
• ✅ All 5 API endpoints returning proper JSON responses
• ✅ JavaScript editor successfully saves edits to database
• ✅ CLI enhancement pulls latest content from database
• ✅ Full edit → save → build → view cycle working end-to-end

Production Ready When:

• ✅ Multi-site support with proper site isolation
• ✅ Authentication and authorization working
• ✅ Database migrations and backup strategy
• ✅ CDN hosting for insertr.js library
• ✅ Deployment documentation and examples

Current Architecture Status

✅ What's Working Well

• CLI Parser: Detects 41+ elements across demo site, generates stable IDs
• Authentication System: Professional login/edit mode toggle with visual states
• Form System: Dynamic modal forms with smart field detection
• Build Pipeline: Automated library building and copying to demo site
• Development Experience: Hot reload with Air integration

🔍 Investigation Results

• File Analysis: All legacy code removed, clean single implementation
• Integration Testing: CLI ↔ Library integration works seamlessly
• Demo Site: Both index.html and about.html use modern library correctly
• Content Detection: CLI successfully identifies text/markdown/link content types

Immediate Next Steps

🎯 Priority 1: Content Persistence

Goal: Make edits survive page reload

• Create lib/src/core/content-manager.js for LocalStorage operations
• Integrate with existing form system for automatic save/restore
• Add change tracking and storage management

🎯 Priority 2: Server Application

Goal: Backend API for real content storage

• Design REST API for content CRUD operations
• Add authentication integration (OAuth/JWT)
• Consider database choice (SQLite for simplicity vs PostgreSQL for production)

---

🏁 Ready to Build

The analysis is complete. The architecture is sound and 90% implemented.

Next Action: Create the HTTP server application that implements the API contract both clients expect.

This will immediately unlock:

• ✅ Real content persistence (not just LocalStorage)
• ✅ Multi-user editing capabilities
• ✅ Production-ready content management
• ✅ Full integration between browser editor and CLI enhancement

Let's build the missing server!

🏗️ Database Schema Architecture Decision (Sept 2025)

Issue: Inlined SQLite schema in database.go creates multiple sources of truth, same problem we just solved with model duplication.

Recommended Solutions (in order of preference):

🎯 Option 1: Schema-as-Query Pattern ⭐ RECOMMENDED

db/queries/
├── content.sql          # CRUD queries
├── versions.sql         # Version control queries  
├── schema_setup.sql     # Schema initialization as named query
└── indexes_setup.sql    # Index creation as named query

Benefits:

• ✅ Single source of truth (schema files)
• ✅ sqlc generates type-safe setup functions
• ✅ Consistent with existing sqlc workflow
• ✅ Database-agnostic parameter syntax (sqlc.arg())

Implementation:

-- name: InitializeSchema :exec
CREATE TABLE IF NOT EXISTS content (...);
CREATE TABLE IF NOT EXISTS content_versions (...);

-- name: CreateIndexes :exec  
CREATE INDEX IF NOT EXISTS idx_content_site_id ON content(site_id);

🔧 Option 2: Migration Tool Integration

• Use goose, golang-migrate, or dbmate
• sqlc natively supports parsing migration directories
• Professional database management with up/down migrations
• Trade-off: Additional dependency and complexity

🗂️ Option 3: Embedded Schema Files

• Use go:embed to read schema files at compile time
• Keep schema in separate .sql files
• Trade-off: Not processed by sqlc, less type safety

✅ COMPLETED: Implemented Option 1 (Schema-as-Query) with important discovery:

sqlc Limitations Discovered:

• ✅ sqlc generates functions for CREATE TABLE and ALTER TABLE statements
• ❌ sqlc does NOT generate functions for CREATE INDEX or CREATE TRIGGER statements
• 🔧 Solution: Handle table creation via sqlc-generated functions, handle indexes/triggers manually in Go initialization code

Final Implementation:

• db/*/setup.sql - Contains table creation queries (sqlc generates type-safe functions)
• internal/db/database.go - Manual index/trigger creation using raw SQL
• Best Practice: Use sqlc for what it supports, manual SQL for what it doesn't

🔄 Editor Cache Architecture & Content State Management (Dec 2024)

Current Architecture Issues Identified

Problem: Conflict between preview system and content persistence after save operations.

Root Cause: The unified Editor system was designed with preview-first architecture:

• originalContent stores DOM state when editing begins
• clearPreview() always restores to originalContent
• This creates race condition: applyContent() → clearPreview() → content reverted

✅ Implemented Solution: Post-Save Baseline Update

Strategy: After successful save, update the stored originalContent to match the new saved state.

Implementation (lib/src/ui/Editor.js):

// In save handler:
context.applyContent(content);        // Apply new content to DOM
context.updateOriginalContent();      // Update baseline to match current DOM  
this.previewer.clearPreview();        // Clear preview (won't revert since baseline matches)

Benefits:

• ✅ Content persists after save operations
• ✅ Future cancellations restore to saved state, not pre-edit state
• ✅ Maintains clean preview functionality
• ✅ No breaking changes to existing architecture

Future Considerations: Draft System Architecture

Current State Management:

• Browser Cache: DOM elements store current content state
• Server Cache: Database stores persisted content
• No Intermediate Cache: Edits are either preview (temporary) or saved (permanent)

Potential Draft System Design:

Option 1: LocalStorage Drafts

// Auto-save drafts locally during editing
const draftKey = `insertr_draft_${contentId}_${siteId}`;
localStorage.setItem(draftKey, JSON.stringify({
    content: currentContent,
    timestamp: Date.now(),
    originalContent: baseline
}));

Benefits: Offline support, immediate feedback, no server load Drawbacks: Per-browser, no cross-device sync, storage limits

Option 2: Server-Side Drafts

// Auto-save drafts to server with special draft flag
PUT /api/content/{id}/draft
{
    "value": "Draft content...",
    "type": "markdown",
    "is_draft": true
}

Benefits: Cross-device sync, unlimited storage, collaborative editing potential Drawbacks: Server complexity, authentication requirements, network dependency

Option 3: Hybrid Draft System

• LocalStorage: Immediate draft saving during typing
• Server Sync: Periodic sync of drafts (every 30s or on significant changes)
• Conflict Resolution: Handle cases where server content changed while editing draft

Cache Invalidation Strategy

Current Behavior:

• Content is cached in DOM until page reload
• No automatic refresh when content changes on server
• No awareness of multi-user editing scenarios

Recommended Enhancements:

Option A: Manual Refresh Strategy

• Add "Refresh Content" button to editor interface
• Check server content before editing (warn if changed)
• Simple conflict resolution (server wins vs local wins vs merge)

Option B: Polling Strategy

• Poll server every N minutes for content changes
• Show notification if content was updated by others
• Allow user to choose: keep editing, reload, or merge

Option C: WebSocket Strategy

• Real-time content change notifications
• Live collaborative editing indicators
• Automatic conflict resolution

Implementation Priority

Phase 1 (Immediate): ✅ COMPLETED

• Fix content persistence after save (baseline update approach)

Phase 2 (Short-term):

• Add LocalStorage draft auto-save during editing
• Implement draft recovery on page reload
• Basic conflict detection (server timestamp vs local timestamp)

Phase 3 (Long-term):

• Server-side draft support
• Real-time collaboration features
• Advanced conflict resolution

Design Principles for Draft System

1. Progressive Enhancement: Site works without drafts, drafts enhance UX
2. Data Safety: Never lose user content, even in edge cases
3. Performance First: Drafts shouldn't impact site loading for regular visitors
4. Conflict Transparency: Always show user what's happening with their content
5. Graceful Degradation: Fallback to basic editing if draft system fails

Note: Current architecture already supports foundation for all these enhancements through the unified Editor system and API client pattern.

🏗️ Server-Hosted Static Sites Implementation (Sept 2025)

✅ COMPLETED: Unified Binary HTTP Server

The HTTP server has been successfully implemented and is production-ready:

• ✅ Full REST API - All content CRUD operations (internal/api/handlers.go)
• ✅ Multi-Database Support - SQLite + PostgreSQL with sqlc-generated queries
• ✅ Authentication System - JWT + mock tokens, ready for authentik integration
• ✅ Bulk Operations - Efficient bulk content retrieval (injector.go:56-96)
• ✅ Content Enhancement - Build-time content injection from database

🎯 NEW PRIORITY: Server-Hosted File Enhancement

Goal: Enable real-time content updates for server-hosted static sites through database-driven file enhancement.

Architecture: Database as source of truth → Content changes → File updates → Immediate deployment

Phase 1: Core File Enhancement System

• Extend enhancer.go - Add EnhanceInPlace() method for in-place file modification
• Site Manager - Create registration system for static site paths
• Handler Integration - Hook file enhancement into UpdateContent() API calls
• Configuration Extension - Add sites configuration to insertr.yaml

Files to modify:

• internal/content/enhancer.go - Add in-place enhancement methods
• internal/api/handlers.go:225 - Integrate file enhancement trigger
• insertr.yaml - Add sites configuration section
• cmd/serve.go - Initialize site manager in server

Phase 2: Multi-Site Server Management

• CLI Site Commands - insertr sites register/list/enhance
• File Backup System - Backup original files before enhancement
• Error Handling - Graceful file permission and disk space handling
• Performance Optimization - Selective file updates, change detection

Phase 3: Production Features

• Authentik Integration - OIDC authentication for production deployment
• Caddy Integration - Reverse proxy configuration templates
• Monitoring & Logging - Track enhancement operations and errors
• Multi-Environment - Development vs production configuration

Implementation Approach: Leverage Existing Architecture

Reuse Existing Components:

• ✅ Bulk Injection - injector.go:56-96 already handles efficient content injection
• ✅ Multi-Site Support - siteID parameter used throughout API and database
• ✅ Authentication - auth.go:47-67 supports JWT tokens ready for authentik
• ✅ Content Client Interface - types.go:18-28 extensible for new features

New Components (minimal additions):

• Site Manager - Register static site paths and manage file enhancement
• In-Place Enhancer - Extend existing enhancer for file modification
• Configuration - Extend YAML config for site registration

Target Deployment Architecture

┌─────────────────────────────────────┐
│           Caddy Reverse Proxy        │
├─────────────────┬───────────────────┤
│  Static Sites   │  Insertr Server   │
│  /site1/* ────→ │  :8080/api/*     │
│  /site2/* ────→ │  Authentication   │
│  /admin/* ────→ │  File Enhancement │
└─────────────────┴───────────────────┘
        │                    │
        ▼                    ▼
  ┌─────────────┐    ┌─────────────────┐
  │Static Files │    │   Database +    │
  │/var/www/    │◄───┤   Enhancement   │
  │  site1/     │    │   Engine        │
  │  site2/     │    └─────────────────┘
  └─────────────┘

Content Update Workflow:

1. Editor makes content change → API call to /api/content/{id}
2. Content saved to database (source of truth)
3. File enhancement triggered → Static files updated in-place
4. Changes immediately visible to visitors (served by Caddy)

Future CI/CD Integration Features (Post-Server Implementation)

External Build Trigger System

• Webhook Integration: Trigger external CI/CD pipelines when content changes
• GitHub Actions Templates: Pre-built workflows for Hugo, Next.js, Jekyll + Insertr
• Platform Integrations: Native Netlify, Vercel, CloudFlare Pages integration
• Build Status Tracking: Monitor external build progress and deployment status

Hybrid Deployment Models

• File + Pipeline Mode: Server-hosted sites with optional CI/CD rebuilds
• Content API Mode: Headless CMS for external static site generators
• Multi-Environment: Staging/production deployment coordination
• Build Caching: Intelligent rebuilds based on content vs template changes

Advanced Content Workflows

• Scheduled Publishing: Time-based content activation
• Content Approval: Multi-user approval workflows before deployment
• A/B Testing: Content variations for different user segments
• Rollback Integration: Coordinate content rollbacks with deployment rollbacks

Draft System Foundation (Ready for Extension)

Current Architecture Supports:

• ✅ Bulk Operations - injector.go:56-96 efficient for batch content updates
• ✅ Version Control - Full version history with rollback capabilities
• ✅ Content Types - Extensible type system ready for draft fields

Future Draft Features:

• Server-Side Drafts - Extend ContentItem with is_draft field
• Batch Publishing - Publish multiple draft items simultaneously
• Preview Mode - View sites with draft content before publishing
• Auto-Save Drafts - Background saving during editing sessions

Benefits of Server-Hosted Approach

1. ✅ Immediate Deployment - Content changes are live instantly
2. ✅ Database Source of Truth - No git-based workflow complexity
3. ✅ Content-Only Updates - No template rebuilds required
4. ✅ Existing Infrastructure - Leverages Caddy + authentik setup
5. ✅ Performance - Static file serving with dynamic content injection
6. ✅ Multi-Site - Single server manages multiple enhanced sites
7. ✅ Zero Downtime - Updates don't require site rebuilds or deploys