insertr/TODO.md

# Insertr Development TODO

## 🔍 Architecture Analysis Complete (Dec 2024)

**Key Discovery**: The architecture is already 90% complete and brilliantly designed! The missing piece is not LocalStorage persistence, but the **HTTP server application** that implements the API contract both clients expect.

## ✅ What's Already Built & Working

### **Complete Foundation**
- ✅ **Go Content Client** - Full REST API client with all CRUD operations (`internal/content/client.go`)
- ✅ **JavaScript API Client** - Browser client with same API endpoints (`lib/src/core/api-client.js`)
- ✅ **Content Types** - Well-defined data structures (`ContentItem`, `ContentResponse`)
- ✅ **Mock Backend** - Working development server with realistic test data
- ✅ **Build-Time Enhancement** - Content injection from database → HTML during builds
- ✅ **Authentication System** - Complete auth flow ready for server integration
- ✅ **Professional Editor** - Modal forms, validation, smart field detection

### **Architecture Philosophy Preserved**
- ✅ **"Tailwind of CMS"** - Zero configuration, build-time optimization
- ✅ **Framework Agnostic** - Works with any static site generator  
- ✅ **Performance First** - Regular visitors get pure static HTML
- ✅ **Editor Progressive Enhancement** - Editing assets only load when needed

## 🚨 **CRITICAL MISSING PIECE**

### **HTTP Server Application** (90% of work remaining)
The CLI client and JavaScript client both expect a server at `/api/content/*`, but **no server exists**!

**Required API Endpoints**:
```
GET  /api/content/{id}?site_id={site}           # Get single content
GET  /api/content?site_id={site}                # Get all content for site  
GET  /api/content/bulk?site_id={site}&ids[]=... # Bulk get content
PUT  /api/content/{id}                          # Update existing content
POST /api/content                               # Create new content
```

**Current State**: Both clients make HTTP calls to these endpoints, but they 404 because no server implements them.

## 🎯 **Immediate Implementation Plan**

### **🔴 Phase 1: HTTP Server (CRITICAL)**
**Goal**: Build the missing server application that implements the API contract

#### 1.1 **Go HTTP Server** ⭐ **HIGHEST PRIORITY**
- [ ] **REST API Server** - Implement all 5 required endpoints
- [ ] **Database Layer** - SQLite for development, PostgreSQL for production
- [ ] **Authentication Middleware** - JWT/OAuth integration
- [ ] **CORS & Security** - Proper headers for browser integration
- [ ] **Content Validation** - Input sanitization and type checking

#### 1.2 **Integration Testing**
- [ ] **CLI Client Integration** - Test all CLI commands work with real server
- [ ] **JavaScript Client Integration** - Test browser editor saves to real server
- [ ] **End-to-End Workflow** - Edit → Save → Build → Deploy cycle

### **🟡 Phase 2: Production Polish (IMPORTANT)**

#### 2.1 **Client-Side Enhancements**
- [ ] **Editor-Server Integration** - Wire up `handleSave` to use `ApiClient` 
- [ ] **Optimistic Updates** - Show immediate feedback, sync in background
- [ ] **Offline Support** - LocalStorage cache + sync when online
- [ ] **Loading States** - Professional feedback during saves

#### 2.2 **Deployment Pipeline**
- [ ] **Build Triggers** - Auto-rebuild sites when content changes
- [ ] **Multi-Site Support** - Handle multiple domains/site IDs
- [ ] **CDN Integration** - Host insertr.js library on CDN
- [ ] **Database Migrations** - Schema versioning and updates

### **🟢 Phase 3: Advanced Features (NICE-TO-HAVE)**

#### 3.1 **Content Management Enhancements**
- [ ] **Content Versioning** - Track edit history and allow rollbacks
- [ ] **Content Validation** - Advanced validation rules per content type
- [ ] **Markdown Enhancements** - Live preview, toolbar, syntax highlighting  
- [ ] **Media Management** - Image upload and asset management

#### 3.2 **Developer Experience**
- [ ] **Development Tools** - Better debugging and development workflow
- [ ] **Configuration API** - Extensible field type system
- [ ] **Testing Suite** - Comprehensive test coverage
- [ ] **Documentation** - API reference and integration guides

## 💡 **Key Architectural Insights**

### **Why This Architecture is Brilliant**
1. **Performance First**: Regular visitors get pure static HTML with zero CMS overhead
2. **Separation of Concerns**: Content editing completely separate from site performance  
3. **Build-Time Optimization**: Database content gets "baked into" static HTML during builds
4. **Progressive Enhancement**: Sites work without JavaScript, editing enhances with JavaScript
5. **Framework Agnostic**: Works with Hugo, Next.js, Jekyll, Gatsby, vanilla HTML, etc.

### **Production Flow**
```
Content Edits → HTTP API Server → Database
                     ↓
Static Site Build ← CLI Enhancement ← Database Content
                     ↓  
              Enhanced HTML → CDN/Deploy
```

### **Current Working Flow** 
```
✅ Browser Editor → (404) Missing Server → ❌
✅ CLI Enhancement ← Mock Data ← ✅
✅ Static HTML Generation ← ✅
```

**Gap**: The HTTP server that connects editor saves to database storage.

## 🗂️ **Next Steps: Server Implementation**

### **✅ Implemented - Unified Binary Architecture**
```
✅ COMPLETED: All server functionality integrated into unified binary
cmd/
├── serve.go                             # Runtime API server command
└── enhance.go                          # Build-time enhancement command
internal/
├── api/                                # HTTP handlers and middleware
├── db/                                 # Multi-database layer with sqlc
└── content/                            # Content management logic
```

### **Files to Modify**
- `lib/src/core/editor.js:91` - Wire up `ApiClient` to `handleSave` method
- `README.md` - Add server setup instructions
- `docker-compose.yml` - Add server for development stack

## 🎯 **Success Criteria**

### **Phase 1 Complete When**:
- ✅ HTTP server running on `localhost:8080` (or configurable port)
- ✅ All 5 API endpoints returning proper JSON responses
- ✅ JavaScript editor successfully saves edits to database
- ✅ CLI enhancement pulls latest content from database
- ✅ Full edit → save → build → view cycle working end-to-end

### **Production Ready When**:
- ✅ Multi-site support with proper site isolation
- ✅ Authentication and authorization working
- ✅ Database migrations strategy
- ✅ CDN hosting for insertr.js library
- ✅ Deployment documentation and examples

## Current Architecture Status

### ✅ **What's Working Well**
- **CLI Parser**: Detects 41+ elements across demo site, generates stable IDs
- **Authentication System**: Professional login/edit mode toggle with visual states
- **Form System**: Dynamic modal forms with smart field detection
- **Build Pipeline**: Automated library building and copying to demo site
- **Development Experience**: Hot reload with Air integration

### 🔍 **Investigation Results**
- **File Analysis**: All legacy code removed, clean single implementation
- **Integration Testing**: CLI ↔ Library integration works seamlessly  
- **Demo Site**: Both index.html and about.html use modern library correctly
- **Content Detection**: CLI successfully identifies text/markdown/link content types

## Immediate Next Steps

### 🎯 **Priority 1: Content Persistence** 
**Goal**: Make edits survive page reload
- Create `lib/src/core/content-manager.js` for LocalStorage operations
- Integrate with existing form system for automatic save/restore
- Add change tracking and storage management

### 🎯 **Priority 2: Server Application**
**Goal**: Backend API for real content storage
- Design REST API for content CRUD operations
- Add authentication integration (OAuth/JWT)
- Consider database choice (SQLite for simplicity vs PostgreSQL for production)

---

## 🏁 **Ready to Build**

The analysis is complete. The architecture is sound and 90% implemented. 

**Next Action**: Create the HTTP server application that implements the API contract both clients expect.

This will immediately unlock:
- ✅ Real content persistence (not just LocalStorage)
- ✅ Multi-user editing capabilities  
- ✅ Production-ready content management
- ✅ Full integration between browser editor and CLI enhancement

*Let's build the missing server!*

## 🏗️ **Database Schema Architecture Decision** (Sept 2025)

**Issue**: Inlined SQLite schema in `database.go` creates multiple sources of truth, same problem we just solved with model duplication.

### **Recommended Solutions** (in order of preference):

#### **🎯 Option 1: Schema-as-Query Pattern** ⭐ **RECOMMENDED**
```
db/queries/
├── content.sql          # CRUD queries
├── versions.sql         # Version control queries  
├── schema_setup.sql     # Schema initialization as named query
└── indexes_setup.sql    # Index creation as named query
```

**Benefits**:
- ✅ Single source of truth (schema files)
- ✅ sqlc generates type-safe setup functions  
- ✅ Consistent with existing sqlc workflow
- ✅ Database-agnostic parameter syntax (`sqlc.arg()`)

**Implementation**:
```sql
-- name: InitializeSchema :exec
CREATE TABLE IF NOT EXISTS content (...);
CREATE TABLE IF NOT EXISTS content_versions (...);

-- name: CreateIndexes :exec  
CREATE INDEX IF NOT EXISTS idx_content_site_id ON content(site_id);
```

#### **🔧 Option 2: Migration Tool Integration**
- Use `goose`, `golang-migrate`, or `dbmate`
- sqlc natively supports parsing migration directories
- Professional database management with up/down migrations
- **Trade-off**: Additional dependency and complexity

#### **🗂️ Option 3: Embedded Schema Files**
- Use `go:embed` to read schema files at compile time
- Keep schema in separate `.sql` files
- **Trade-off**: Not processed by sqlc, less type safety

**✅ COMPLETED**: Implemented Option 1 (Schema-as-Query) with important discovery:

**sqlc Limitations Discovered**:
- ✅ sqlc generates functions for `CREATE TABLE` and `ALTER TABLE` statements
- ❌ sqlc does **NOT** generate functions for `CREATE INDEX` or `CREATE TRIGGER` statements  
- 🔧 **Solution**: Handle table creation via sqlc-generated functions, handle indexes/triggers manually in Go initialization code

**Final Implementation**:
- `db/*/setup.sql` - Contains table creation queries (sqlc generates type-safe functions)
- `internal/db/database.go` - Manual index/trigger creation using raw SQL
- **Best Practice**: Use sqlc for what it supports, manual SQL for what it doesn't

## 🔄 **Editor Cache Architecture & Content State Management** (Dec 2024)

### **Current Architecture Issues Identified**

**Problem**: Conflict between preview system and content persistence after save operations.

**Root Cause**: The unified Editor system was designed with preview-first architecture:
- `originalContent` stores DOM state when editing begins
- `clearPreview()` always restores to `originalContent` 
- This creates race condition: `applyContent()` → `clearPreview()` → content reverted

### **✅ Implemented Solution: Post-Save Baseline Update**

**Strategy**: After successful save, update the stored `originalContent` to match the new saved state.

**Implementation** (`lib/src/ui/Editor.js`):
```js
// In save handler:
context.applyContent(content);        // Apply new content to DOM
context.updateOriginalContent();      // Update baseline to match current DOM  
this.previewer.clearPreview();        // Clear preview (won't revert since baseline matches)
```

**Benefits**:
- ✅ Content persists after save operations
- ✅ Future cancellations restore to saved state, not pre-edit state
- ✅ Maintains clean preview functionality
- ✅ No breaking changes to existing architecture

### **Future Considerations: Draft System Architecture**

**Current State Management**:
- **Browser Cache**: DOM elements store current content state
- **Server Cache**: Database stores persisted content
- **No Intermediate Cache**: Edits are either preview (temporary) or saved (permanent)

**Potential Draft System Design**:

#### **Option 1: LocalStorage Drafts**
```js
// Auto-save drafts locally during editing
const draftKey = `insertr_draft_${contentId}_${siteId}`;
localStorage.setItem(draftKey, JSON.stringify({
    content: currentContent,
    timestamp: Date.now(),
    originalContent: baseline
}));
```

**Benefits**: Offline support, immediate feedback, no server load
**Drawbacks**: Per-browser, no cross-device sync, storage limits

#### **Option 2: Server-Side Drafts**
```js
// Auto-save drafts to server with special draft flag
PUT /api/content/{id}/draft
{
    "value": "Draft content...",
    "type": "markdown",
    "is_draft": true
}
```

**Benefits**: Cross-device sync, unlimited storage, collaborative editing potential
**Drawbacks**: Server complexity, authentication requirements, network dependency

#### **Option 3: Hybrid Draft System**
- **LocalStorage**: Immediate draft saving during typing
- **Server Sync**: Periodic sync of drafts (every 30s or on significant changes)
- **Conflict Resolution**: Handle cases where server content changed while editing draft

### **Cache Invalidation Strategy**

**Current Behavior**:
- Content is cached in DOM until page reload
- No automatic refresh when content changes on server
- No awareness of multi-user editing scenarios

**Recommended Enhancements**:

#### **Option A: Manual Refresh Strategy**
- Add "Refresh Content" button to editor interface
- Check server content before editing (warn if changed)
- Simple conflict resolution (server wins vs local wins vs merge)

#### **Option B: Polling Strategy**  
- Poll server every N minutes for content changes
- Show notification if content was updated by others
- Allow user to choose: keep editing, reload, or merge

#### **Option C: WebSocket Strategy**
- Real-time content change notifications
- Live collaborative editing indicators
- Automatic conflict resolution

### **Implementation Priority**

**Phase 1** (Immediate): ✅ **COMPLETED**
- Fix content persistence after save (baseline update approach)

**Phase 2** (Short-term): 
- Add LocalStorage draft auto-save during editing
- Implement draft recovery on page reload
- Basic conflict detection (server timestamp vs local timestamp)

**Phase 3** (Long-term):
- Server-side draft support
- Real-time collaboration features
- Advanced conflict resolution

### **Design Principles for Draft System**

1. **Progressive Enhancement**: Site works without drafts, drafts enhance UX
2. **Data Safety**: Never lose user content, even in edge cases
3. **Performance First**: Drafts shouldn't impact site loading for regular visitors
4. **Conflict Transparency**: Always show user what's happening with their content
5. **Graceful Degradation**: Fallback to basic editing if draft system fails

**Note**: Current architecture already supports foundation for all these enhancements through the unified Editor system and API client pattern.

## 🏗️ **Server-Hosted Static Sites Implementation** (Sept 2025)

### **✅ COMPLETED: Unified Binary HTTP Server**
The HTTP server has been successfully implemented and is production-ready:
- ✅ **Full REST API** - All content CRUD operations (`internal/api/handlers.go`)
- ✅ **Multi-Database Support** - SQLite + PostgreSQL with sqlc-generated queries
- ✅ **Authentication System** - JWT + mock tokens, ready for authentik integration
- ✅ **Bulk Operations** - Efficient bulk content retrieval (`injector.go:56-96`)
- ✅ **Content Enhancement** - Build-time content injection from database

### **🎯 NEW PRIORITY: Server-Hosted File Enhancement**

**Goal**: Enable real-time content updates for server-hosted static sites through database-driven file enhancement.

**Architecture**: Database as source of truth → Content changes → File updates → Immediate deployment

### **Phase 1: Core File Enhancement System**
- [ ] **Extend enhancer.go** - Add `EnhanceInPlace()` method for in-place file modification
- [ ] **Site Manager** - Create registration system for static site paths
- [ ] **Handler Integration** - Hook file enhancement into `UpdateContent()` API calls
- [ ] **Configuration Extension** - Add sites configuration to `insertr.yaml`

**Files to modify**:
- `internal/content/enhancer.go` - Add in-place enhancement methods
- `internal/api/handlers.go:225` - Integrate file enhancement trigger
- `insertr.yaml` - Add sites configuration section
- `cmd/serve.go` - Initialize site manager in server

### **Phase 2: Multi-Site Server Management**
- [ ] **CLI Site Commands** - `insertr sites register/list/enhance`
- [ ] **File Backup System** - Backup original files before enhancement
- [ ] **Error Handling** - Graceful file permission and disk space handling
- [ ] **Performance Optimization** - Selective file updates, change detection

### **Phase 3: Production Features**
- [ ] **Authentik Integration** - OIDC authentication for production deployment
- [ ] **Caddy Integration** - Reverse proxy configuration templates
- [ ] **Monitoring & Logging** - Track enhancement operations and errors
- [ ] **Multi-Environment** - Development vs production configuration

### **Implementation Approach: Leverage Existing Architecture**

**Reuse Existing Components**:
- ✅ **Bulk Injection** - `injector.go:56-96` already handles efficient content injection
- ✅ **Multi-Site Support** - `siteID` parameter used throughout API and database
- ✅ **Authentication** - `auth.go:47-67` supports JWT tokens ready for authentik
- ✅ **Content Client Interface** - `types.go:18-28` extensible for new features

**New Components** (minimal additions):
- **Site Manager** - Register static site paths and manage file enhancement
- **In-Place Enhancer** - Extend existing enhancer for file modification
- **Configuration** - Extend YAML config for site registration

### **Target Deployment Architecture**
```
┌─────────────────────────────────────┐
│           Caddy Reverse Proxy        │
├─────────────────┬───────────────────┤
│  Static Sites   │  Insertr Server   │
│  /site1/* ────→ │  :8080/api/*     │
│  /site2/* ────→ │  Authentication   │
│  /admin/* ────→ │  File Enhancement │
└─────────────────┴───────────────────┘
        │                    │
        ▼                    ▼
  ┌─────────────┐    ┌─────────────────┐
  │Static Files │    │   Database +    │
  │/var/www/    │◄───┤   Enhancement   │
  │  site1/     │    │   Engine        │
  │  site2/     │    └─────────────────┘
  └─────────────┘
```

**Content Update Workflow**:
1. Editor makes content change → API call to `/api/content/{id}`
2. Content saved to database (source of truth)
3. File enhancement triggered → Static files updated in-place
4. Changes immediately visible to visitors (served by Caddy)

### **Future CI/CD Integration Features** (Post-Server Implementation)

#### **External Build Trigger System**
- **Webhook Integration**: Trigger external CI/CD pipelines when content changes
- **GitHub Actions Templates**: Pre-built workflows for Hugo, Next.js, Jekyll + Insertr
- **Platform Integrations**: Native Netlify, Vercel, CloudFlare Pages integration
- **Build Status Tracking**: Monitor external build progress and deployment status

#### **Hybrid Deployment Models**
- **File + Pipeline Mode**: Server-hosted sites with optional CI/CD rebuilds
- **Content API Mode**: Headless CMS for external static site generators
- **Multi-Environment**: Staging/production deployment coordination
- **Build Caching**: Intelligent rebuilds based on content vs template changes

#### **Advanced Content Workflows**
- **Scheduled Publishing**: Time-based content activation
- **Content Approval**: Multi-user approval workflows before deployment
- **A/B Testing**: Content variations for different user segments
- **Rollback Integration**: Coordinate content rollbacks with deployment rollbacks

### **Draft System Foundation** (Ready for Extension)

**Current Architecture Supports**:
- ✅ **Bulk Operations** - `injector.go:56-96` efficient for batch content updates
- ✅ **Version Control** - Full version history with rollback capabilities
- ✅ **Content Types** - Extensible type system ready for draft fields

**Future Draft Features**:
- **Server-Side Drafts** - Extend `ContentItem` with `is_draft` field
- **Batch Publishing** - Publish multiple draft items simultaneously
- **Preview Mode** - View sites with draft content before publishing
- **Auto-Save Drafts** - Background saving during editing sessions

### **Benefits of Server-Hosted Approach**

1. **✅ Immediate Deployment** - Content changes are live instantly
2. **✅ Database Source of Truth** - No git-based workflow complexity
3. **✅ Content-Only Updates** - No template rebuilds required
4. **✅ Existing Infrastructure** - Leverages Caddy + authentik setup
5. **✅ Performance** - Static file serving with dynamic content injection
6. **✅ Multi-Site** - Single server manages multiple enhanced sites
7. **✅ Zero Downtime** - Updates don't require site rebuilds or deploys

## 🎯 **Future Class Extensions: Advanced Access Control** (Planned)

### **Granular Permission System for `.insertr-gate`**

**Current Implementation**: Simple boolean authentication gate for page-level editing access.

**Future Enhancement Concept**: Role-based access control and section-level permissions for enterprise applications.

#### **Potential Extended Gate Classes**
```html
<!-- Current: Simple page-level auth -->
<div class="insertr-gate"></div>

<!-- Future: Role-based section permissions -->
<div class="admin-content insertr-gate-admin">
  <div class="insertr-add">Admin-only dynamic content</div>
</div>

<div class="editor-section insertr-gate-editor">
  <div class="insertr-content">Editor-level rich content</div>
</div>

<div class="premium-content insertr-gate-premium" data-auth-level="subscription">
  <div class="insertr">Premium subscriber content</div>
</div>
```

#### **Enterprise Use Cases**
- **Multi-tenant Applications**: Different organizations editing separate content areas
- **Editorial Workflows**: Writers, editors, and admins with different capabilities
- **Subscription Content**: Different content areas for different subscription tiers
- **Department Permissions**: Marketing vs Engineering vs Sales content areas
- **Geographic Restrictions**: Region-specific editing permissions

#### **Implementation Considerations**
- **Configuration Complexity**: How to maintain zero-config philosophy while supporting complex permissions
- **Role Definition**: Where and how to define roles and permissions
- **Authentication Integration**: Extending current auth system vs external RBAC systems
- **Fallback Behavior**: Graceful degradation for unauthorized users
- **Performance Impact**: Permission checks shouldn't slow down regular site visitors

#### **Integration Points**
- **Authentik OIDC**: Leverage existing auth provider for role information
- **Database Schema**: Extend current content system with permission metadata
- **API Endpoints**: New endpoints for permission management and validation
- **Editor Interface**: Dynamic interface based on user permissions

**Priority**: Low - implement after core functionality is stable and enterprise customers request advanced permissions.

**Design Principle**: Keep simple `.insertr-gate` as default, add optional complexity only when needed.