insertr/TODO.md

Insertr Development TODO

🔍 Architecture Analysis Complete (Dec 2024)

Key Discovery: The architecture is already 90% complete and brilliantly designed! The missing piece is not LocalStorage persistence, but the HTTP server application that implements the API contract both clients expect.

✅ What's Already Built & Working

Complete Foundation

• ✅ Go Content Client - Full REST API client with all CRUD operations (internal/content/client.go)
• ✅ JavaScript API Client - Browser client with same API endpoints (lib/src/core/api-client.js)
• ✅ Content Types - Well-defined data structures (ContentItem, ContentResponse)
• ✅ Mock Backend - Working development server with realistic test data
• ✅ Build-Time Enhancement - Content injection from database → HTML during builds
• ✅ Authentication System - Complete auth flow ready for server integration
• ✅ Professional Editor - Modal forms, validation, smart field detection

Architecture Philosophy Preserved

• ✅ "Tailwind of CMS" - Zero configuration, build-time optimization
• ✅ Framework Agnostic - Works with any static site generator
• ✅ Performance First - Regular visitors get pure static HTML
• ✅ Editor Progressive Enhancement - Editing assets only load when needed

🚨 CRITICAL MISSING PIECE

HTTP Server Application (90% of work remaining)

The CLI client and JavaScript client both expect a server at /api/content/*, but no server exists!

Required API Endpoints:

GET  /api/content/{id}?site_id={site}           # Get single content
GET  /api/content?site_id={site}                # Get all content for site  
GET  /api/content/bulk?site_id={site}&ids[]=... # Bulk get content
PUT  /api/content/{id}                          # Update existing content
POST /api/content                               # Create new content

Current State: Both clients make HTTP calls to these endpoints, but they 404 because no server implements them.

🎯 Immediate Implementation Plan

🔴 Phase 1: HTTP Server (CRITICAL)

Goal: Build the missing server application that implements the API contract

1.1 Go HTTP Server ⭐ HIGHEST PRIORITY

• REST API Server - Implement all 5 required endpoints
• Database Layer - SQLite for development, PostgreSQL for production
• Authentication Middleware - JWT/OAuth integration
• CORS & Security - Proper headers for browser integration
• Content Validation - Input sanitization and type checking

1.2 Integration Testing

• CLI Client Integration - Test all CLI commands work with real server
• JavaScript Client Integration - Test browser editor saves to real server
• End-to-End Workflow - Edit → Save → Build → Deploy cycle

🟡 Phase 2: Production Polish (IMPORTANT)

2.1 Client-Side Enhancements

• Editor-Server Integration - Wire up handleSave to use ApiClient
• Optimistic Updates - Show immediate feedback, sync in background
• Offline Support - LocalStorage cache + sync when online
• Loading States - Professional feedback during saves

2.2 Deployment Pipeline

• Build Triggers - Auto-rebuild sites when content changes
• Multi-Site Support - Handle multiple domains/site IDs
• CDN Integration - Host insertr.js library on CDN
• Database Migrations - Schema versioning and updates

🟢 Phase 3: Advanced Features (NICE-TO-HAVE)

3.1 Content Management Enhancements

• Content Versioning - Track edit history and allow rollbacks
• Content Validation - Advanced validation rules per content type
• Markdown Enhancements - Live preview, toolbar, syntax highlighting
• Media Management - Image upload and asset management

3.2 Developer Experience

• Development Tools - Better debugging and development workflow
• Configuration API - Extensible field type system
• Testing Suite - Comprehensive test coverage
• Documentation - API reference and integration guides

💡 Key Architectural Insights

Why This Architecture is Brilliant

1. Performance First: Regular visitors get pure static HTML with zero CMS overhead
2. Separation of Concerns: Content editing completely separate from site performance
3. Build-Time Optimization: Database content gets "baked into" static HTML during builds
4. Progressive Enhancement: Sites work without JavaScript, editing enhances with JavaScript
5. Framework Agnostic: Works with Hugo, Next.js, Jekyll, Gatsby, vanilla HTML, etc.

Production Flow

Content Edits → HTTP API Server → Database
                     ↓
Static Site Build ← CLI Enhancement ← Database Content
                     ↓  
              Enhanced HTML → CDN/Deploy

Current Working Flow

✅ Browser Editor → (404) Missing Server → ❌
✅ CLI Enhancement ← Mock Data ← ✅
✅ Static HTML Generation ← ✅

Gap: The HTTP server that connects editor saves to database storage.

🗂️ Next Steps: Server Implementation

✅ Implemented - Unified Binary Architecture

✅ COMPLETED: All server functionality integrated into unified binary
cmd/
├── serve.go                             # Runtime API server command
└── enhance.go                          # Build-time enhancement command
internal/
├── api/                                # HTTP handlers and middleware
├── db/                                 # Multi-database layer with sqlc
└── content/                            # Content management logic

Files to Modify

• lib/src/core/editor.js:91 - Wire up ApiClient to handleSave method
• README.md - Add server setup instructions
• docker-compose.yml - Add server for development stack

🎯 Success Criteria

Phase 1 Complete When:

• ✅ HTTP server running on localhost:8080 (or configurable port)
• ✅ All 5 API endpoints returning proper JSON responses
• ✅ JavaScript editor successfully saves edits to database
• ✅ CLI enhancement pulls latest content from database
• ✅ Full edit → save → build → view cycle working end-to-end

Production Ready When:

• ✅ Multi-site support with proper site isolation
• ✅ Authentication and authorization working
• ✅ Database migrations and backup strategy
• ✅ CDN hosting for insertr.js library
• ✅ Deployment documentation and examples

Current Architecture Status

✅ What's Working Well

• CLI Parser: Detects 41+ elements across demo site, generates stable IDs
• Authentication System: Professional login/edit mode toggle with visual states
• Form System: Dynamic modal forms with smart field detection
• Build Pipeline: Automated library building and copying to demo site
• Development Experience: Hot reload with Air integration

🔍 Investigation Results

• File Analysis: All legacy code removed, clean single implementation
• Integration Testing: CLI ↔ Library integration works seamlessly
• Demo Site: Both index.html and about.html use modern library correctly
• Content Detection: CLI successfully identifies text/markdown/link content types

Immediate Next Steps

🎯 Priority 1: Content Persistence

Goal: Make edits survive page reload

• Create lib/src/core/content-manager.js for LocalStorage operations
• Integrate with existing form system for automatic save/restore
• Add change tracking and storage management

🎯 Priority 2: Server Application

Goal: Backend API for real content storage

• Design REST API for content CRUD operations
• Add authentication integration (OAuth/JWT)
• Consider database choice (SQLite for simplicity vs PostgreSQL for production)

---

🏁 Ready to Build

The analysis is complete. The architecture is sound and 90% implemented.

Next Action: Create the HTTP server application that implements the API contract both clients expect.

This will immediately unlock:

• ✅ Real content persistence (not just LocalStorage)
• ✅ Multi-user editing capabilities
• ✅ Production-ready content management
• ✅ Full integration between browser editor and CLI enhancement

Let's build the missing server!

🏗️ Database Schema Architecture Decision (Sept 2025)

Issue: Inlined SQLite schema in database.go creates multiple sources of truth, same problem we just solved with model duplication.

Recommended Solutions (in order of preference):

🎯 Option 1: Schema-as-Query Pattern ⭐ RECOMMENDED

db/queries/
├── content.sql          # CRUD queries
├── versions.sql         # Version control queries  
├── schema_setup.sql     # Schema initialization as named query
└── indexes_setup.sql    # Index creation as named query

Benefits:

• ✅ Single source of truth (schema files)
• ✅ sqlc generates type-safe setup functions
• ✅ Consistent with existing sqlc workflow
• ✅ Database-agnostic parameter syntax (sqlc.arg())

Implementation:

-- name: InitializeSchema :exec
CREATE TABLE IF NOT EXISTS content (...);
CREATE TABLE IF NOT EXISTS content_versions (...);

-- name: CreateIndexes :exec  
CREATE INDEX IF NOT EXISTS idx_content_site_id ON content(site_id);

🔧 Option 2: Migration Tool Integration

• Use goose, golang-migrate, or dbmate
• sqlc natively supports parsing migration directories
• Professional database management with up/down migrations
• Trade-off: Additional dependency and complexity

🗂️ Option 3: Embedded Schema Files

• Use go:embed to read schema files at compile time
• Keep schema in separate .sql files
• Trade-off: Not processed by sqlc, less type safety

✅ COMPLETED: Implemented Option 1 (Schema-as-Query) with important discovery:

sqlc Limitations Discovered:

• ✅ sqlc generates functions for CREATE TABLE and ALTER TABLE statements
• ❌ sqlc does NOT generate functions for CREATE INDEX or CREATE TRIGGER statements
• 🔧 Solution: Handle table creation via sqlc-generated functions, handle indexes/triggers manually in Go initialization code

Final Implementation:

• db/*/setup.sql - Contains table creation queries (sqlc generates type-safe functions)
• internal/db/database.go - Manual index/trigger creation using raw SQL
• Best Practice: Use sqlc for what it supports, manual SQL for what it doesn't

🔄 Editor Cache Architecture & Content State Management (Dec 2024)

Current Architecture Issues Identified

Problem: Conflict between preview system and content persistence after save operations.

Root Cause: The unified Editor system was designed with preview-first architecture:

• originalContent stores DOM state when editing begins
• clearPreview() always restores to originalContent
• This creates race condition: applyContent() → clearPreview() → content reverted

✅ Implemented Solution: Post-Save Baseline Update

Strategy: After successful save, update the stored originalContent to match the new saved state.

Implementation (lib/src/ui/Editor.js):

// In save handler:
context.applyContent(content);        // Apply new content to DOM
context.updateOriginalContent();      // Update baseline to match current DOM  
this.previewer.clearPreview();        // Clear preview (won't revert since baseline matches)

Benefits:

• ✅ Content persists after save operations
• ✅ Future cancellations restore to saved state, not pre-edit state
• ✅ Maintains clean preview functionality
• ✅ No breaking changes to existing architecture

Future Considerations: Draft System Architecture

Current State Management:

• Browser Cache: DOM elements store current content state
• Server Cache: Database stores persisted content
• No Intermediate Cache: Edits are either preview (temporary) or saved (permanent)

Potential Draft System Design:

Option 1: LocalStorage Drafts

// Auto-save drafts locally during editing
const draftKey = `insertr_draft_${contentId}_${siteId}`;
localStorage.setItem(draftKey, JSON.stringify({
    content: currentContent,
    timestamp: Date.now(),
    originalContent: baseline
}));

Benefits: Offline support, immediate feedback, no server load Drawbacks: Per-browser, no cross-device sync, storage limits

Option 2: Server-Side Drafts

// Auto-save drafts to server with special draft flag
PUT /api/content/{id}/draft
{
    "value": "Draft content...",
    "type": "markdown",
    "is_draft": true
}

Benefits: Cross-device sync, unlimited storage, collaborative editing potential Drawbacks: Server complexity, authentication requirements, network dependency

Option 3: Hybrid Draft System

• LocalStorage: Immediate draft saving during typing
• Server Sync: Periodic sync of drafts (every 30s or on significant changes)
• Conflict Resolution: Handle cases where server content changed while editing draft

Cache Invalidation Strategy

Current Behavior:

• Content is cached in DOM until page reload
• No automatic refresh when content changes on server
• No awareness of multi-user editing scenarios

Recommended Enhancements:

Option A: Manual Refresh Strategy

• Add "Refresh Content" button to editor interface
• Check server content before editing (warn if changed)
• Simple conflict resolution (server wins vs local wins vs merge)

Option B: Polling Strategy

• Poll server every N minutes for content changes
• Show notification if content was updated by others
• Allow user to choose: keep editing, reload, or merge

Option C: WebSocket Strategy

• Real-time content change notifications
• Live collaborative editing indicators
• Automatic conflict resolution

Implementation Priority

Phase 1 (Immediate): ✅ COMPLETED

• Fix content persistence after save (baseline update approach)

Phase 2 (Short-term):

• Add LocalStorage draft auto-save during editing
• Implement draft recovery on page reload
• Basic conflict detection (server timestamp vs local timestamp)

Phase 3 (Long-term):

• Server-side draft support
• Real-time collaboration features
• Advanced conflict resolution

Design Principles for Draft System

1. Progressive Enhancement: Site works without drafts, drafts enhance UX
2. Data Safety: Never lose user content, even in edge cases
3. Performance First: Drafts shouldn't impact site loading for regular visitors
4. Conflict Transparency: Always show user what's happening with their content
5. Graceful Degradation: Fallback to basic editing if draft system fails

Note: Current architecture already supports foundation for all these enhancements through the unified Editor system and API client pattern.