insertr/TODO.md

Insertr Development TODO

🔍 Architecture Analysis Complete (Dec 2024)

Key Discovery: The architecture is already 90% complete and brilliantly designed! The missing piece is not LocalStorage persistence, but the HTTP server application that implements the API contract both clients expect.

✅ What's Already Built & Working

Complete Foundation

• ✅ Go Content Client - Full REST API client with all CRUD operations (internal/content/client.go)
• ✅ JavaScript API Client - Browser client with same API endpoints (lib/src/core/api-client.js)
• ✅ Content Types - Well-defined data structures (ContentItem, ContentResponse)
• ✅ Mock Backend - Working development server with realistic test data
• ✅ Build-Time Enhancement - Content injection from database → HTML during builds
• ✅ Authentication System - Complete auth flow ready for server integration
• ✅ Professional Editor - Modal forms, validation, smart field detection

Architecture Philosophy Preserved

• ✅ "Tailwind of CMS" - Zero configuration, build-time optimization
• ✅ Framework Agnostic - Works with any static site generator
• ✅ Performance First - Regular visitors get pure static HTML
• ✅ Editor Progressive Enhancement - Editing assets only load when needed

🚨 CRITICAL MISSING PIECE

HTTP Server Application (90% of work remaining)

The CLI client and JavaScript client both expect a server at /api/content/*, but no server exists!

Required API Endpoints:

GET  /api/content/{id}?site_id={site}           # Get single content
GET  /api/content?site_id={site}                # Get all content for site  
GET  /api/content/bulk?site_id={site}&ids[]=... # Bulk get content
PUT  /api/content/{id}                          # Update existing content
POST /api/content                               # Create new content

Current State: Both clients make HTTP calls to these endpoints, but they 404 because no server implements them.

🎯 Immediate Implementation Plan

🔴 Phase 1: HTTP Server (CRITICAL)

Goal: Build the missing server application that implements the API contract

1.1 Go HTTP Server ⭐ HIGHEST PRIORITY

• REST API Server - Implement all 5 required endpoints
• Database Layer - SQLite for development, PostgreSQL for production
• Authentication Middleware - JWT/OAuth integration
• CORS & Security - Proper headers for browser integration
• Content Validation - Input sanitization and type checking

1.2 Integration Testing

• CLI Client Integration - Test all CLI commands work with real server
• JavaScript Client Integration - Test browser editor saves to real server
• End-to-End Workflow - Edit → Save → Build → Deploy cycle

🟡 Phase 2: Production Polish (IMPORTANT)

2.1 Client-Side Enhancements

• Editor-Server Integration - Wire up handleSave to use ApiClient
• Optimistic Updates - Show immediate feedback, sync in background
• Offline Support - LocalStorage cache + sync when online
• Loading States - Professional feedback during saves

2.2 Deployment Pipeline

• Build Triggers - Auto-rebuild sites when content changes
• Multi-Site Support - Handle multiple domains/site IDs
• CDN Integration - Host insertr.js library on CDN
• Database Migrations - Schema versioning and updates

🟢 Phase 3: Advanced Features (NICE-TO-HAVE)

3.1 Content Management Enhancements

• Content Versioning - Track edit history and allow rollbacks
• Content Validation - Advanced validation rules per content type
• Markdown Enhancements - Live preview, toolbar, syntax highlighting
• Media Management - Image upload and asset management

3.2 Developer Experience

• Development Tools - Better debugging and development workflow
• Configuration API - Extensible field type system
• Testing Suite - Comprehensive test coverage
• Documentation - API reference and integration guides

💡 Key Architectural Insights

Why This Architecture is Brilliant

1. Performance First: Regular visitors get pure static HTML with zero CMS overhead
2. Separation of Concerns: Content editing completely separate from site performance
3. Build-Time Optimization: Database content gets "baked into" static HTML during builds
4. Progressive Enhancement: Sites work without JavaScript, editing enhances with JavaScript
5. Framework Agnostic: Works with Hugo, Next.js, Jekyll, Gatsby, vanilla HTML, etc.

Production Flow

Content Edits → HTTP API Server → Database
                     ↓
Static Site Build ← CLI Enhancement ← Database Content
                     ↓  
              Enhanced HTML → CDN/Deploy

Current Working Flow

✅ Browser Editor → (404) Missing Server → ❌
✅ CLI Enhancement ← Mock Data ← ✅
✅ Static HTML Generation ← ✅

Gap: The HTTP server that connects editor saves to database storage.

🗂️ Next Steps: Server Implementation

✅ Implemented - Unified Binary Architecture

✅ COMPLETED: All server functionality integrated into unified binary
cmd/
├── serve.go                             # Runtime API server command
└── enhance.go                          # Build-time enhancement command
internal/
├── api/                                # HTTP handlers and middleware
├── db/                                 # Multi-database layer with sqlc
└── content/                            # Content management logic

Files to Modify

• lib/src/core/editor.js:91 - Wire up ApiClient to handleSave method
• README.md - Add server setup instructions
• docker-compose.yml - Add server for development stack

🎯 Success Criteria

Phase 1 Complete When:

• ✅ HTTP server running on localhost:8080 (or configurable port)
• ✅ All 5 API endpoints returning proper JSON responses
• ✅ JavaScript editor successfully saves edits to database
• ✅ CLI enhancement pulls latest content from database
• ✅ Full edit → save → build → view cycle working end-to-end

Production Ready When:

• ✅ Multi-site support with proper site isolation
• ✅ Authentication and authorization working
• ✅ Database migrations and backup strategy
• ✅ CDN hosting for insertr.js library
• ✅ Deployment documentation and examples

Current Architecture Status

✅ What's Working Well

• CLI Parser: Detects 41+ elements across demo site, generates stable IDs
• Authentication System: Professional login/edit mode toggle with visual states
• Form System: Dynamic modal forms with smart field detection
• Build Pipeline: Automated library building and copying to demo site
• Development Experience: Hot reload with Air integration

🔍 Investigation Results

• File Analysis: All legacy code removed, clean single implementation
• Integration Testing: CLI ↔ Library integration works seamlessly
• Demo Site: Both index.html and about.html use modern library correctly
• Content Detection: CLI successfully identifies text/markdown/link content types

Immediate Next Steps

🎯 Priority 1: Content Persistence

Goal: Make edits survive page reload

• Create lib/src/core/content-manager.js for LocalStorage operations
• Integrate with existing form system for automatic save/restore
• Add change tracking and storage management

🎯 Priority 2: Server Application

Goal: Backend API for real content storage

• Design REST API for content CRUD operations
• Add authentication integration (OAuth/JWT)
• Consider database choice (SQLite for simplicity vs PostgreSQL for production)

---

🏁 Ready to Build

The analysis is complete. The architecture is sound and 90% implemented.

Next Action: Create the HTTP server application that implements the API contract both clients expect.

This will immediately unlock:

• ✅ Real content persistence (not just LocalStorage)
• ✅ Multi-user editing capabilities
• ✅ Production-ready content management
• ✅ Full integration between browser editor and CLI enhancement

Let's build the missing server!

🏗️ Database Schema Architecture Decision (Sept 2025)

Issue: Inlined SQLite schema in database.go creates multiple sources of truth, same problem we just solved with model duplication.

Recommended Solutions (in order of preference):

🎯 Option 1: Schema-as-Query Pattern ⭐ RECOMMENDED

db/queries/
├── content.sql          # CRUD queries
├── versions.sql         # Version control queries  
├── schema_setup.sql     # Schema initialization as named query
└── indexes_setup.sql    # Index creation as named query

Benefits:

• ✅ Single source of truth (schema files)
• ✅ sqlc generates type-safe setup functions
• ✅ Consistent with existing sqlc workflow
• ✅ Database-agnostic parameter syntax (sqlc.arg())

Implementation:

-- name: InitializeSchema :exec
CREATE TABLE IF NOT EXISTS content (...);
CREATE TABLE IF NOT EXISTS content_versions (...);

-- name: CreateIndexes :exec  
CREATE INDEX IF NOT EXISTS idx_content_site_id ON content(site_id);

🔧 Option 2: Migration Tool Integration

• Use goose, golang-migrate, or dbmate
• sqlc natively supports parsing migration directories
• Professional database management with up/down migrations
• Trade-off: Additional dependency and complexity

🗂️ Option 3: Embedded Schema Files

• Use go:embed to read schema files at compile time
• Keep schema in separate .sql files
• Trade-off: Not processed by sqlc, less type safety

✅ COMPLETED: Implemented Option 1 (Schema-as-Query) with important discovery:

sqlc Limitations Discovered:

• ✅ sqlc generates functions for CREATE TABLE and ALTER TABLE statements
• ❌ sqlc does NOT generate functions for CREATE INDEX or CREATE TRIGGER statements
• 🔧 Solution: Handle table creation via sqlc-generated functions, handle indexes/triggers manually in Go initialization code

Final Implementation:

• db/*/setup.sql - Contains table creation queries (sqlc generates type-safe functions)
• internal/db/database.go - Manual index/trigger creation using raw SQL
• Best Practice: Use sqlc for what it supports, manual SQL for what it doesn't