insertr/TODO.md

# Insertr Development TODO

## 🔍 Architecture Analysis Complete (Dec 2024)

**Key Discovery**: The architecture is already 90% complete and brilliantly designed! The missing piece is not LocalStorage persistence, but the **HTTP server application** that implements the API contract both clients expect.

## ✅ What's Already Built & Working

### **Complete Foundation**
- ✅ **Go Content Client** - Full REST API client with all CRUD operations (`internal/content/client.go`)
- ✅ **JavaScript API Client** - Browser client with same API endpoints (`lib/src/core/api-client.js`)
- ✅ **Content Types** - Well-defined data structures (`ContentItem`, `ContentResponse`)
- ✅ **Mock Backend** - Working development server with realistic test data
- ✅ **Build-Time Enhancement** - Content injection from database → HTML during builds
- ✅ **Authentication System** - Complete auth flow ready for server integration
- ✅ **Professional Editor** - Modal forms, validation, smart field detection

### **Architecture Philosophy Preserved**
- ✅ **"Tailwind of CMS"** - Zero configuration, build-time optimization
- ✅ **Framework Agnostic** - Works with any static site generator  
- ✅ **Performance First** - Regular visitors get pure static HTML
- ✅ **Editor Progressive Enhancement** - Editing assets only load when needed

## 🚨 **CRITICAL MISSING PIECE**

### **HTTP Server Application** (90% of work remaining)
The CLI client and JavaScript client both expect a server at `/api/content/*`, but **no server exists**!

**Required API Endpoints**:
```
GET  /api/content/{id}?site_id={site}           # Get single content
GET  /api/content?site_id={site}                # Get all content for site  
GET  /api/content/bulk?site_id={site}&ids[]=... # Bulk get content
PUT  /api/content/{id}                          # Update existing content
POST /api/content                               # Create new content
```

**Current State**: Both clients make HTTP calls to these endpoints, but they 404 because no server implements them.

## 🎯 **Immediate Implementation Plan**

### **🔴 Phase 1: HTTP Server (CRITICAL)**
**Goal**: Build the missing server application that implements the API contract

#### 1.1 **Go HTTP Server** ⭐ **HIGHEST PRIORITY**
- [ ] **REST API Server** - Implement all 5 required endpoints
- [ ] **Database Layer** - SQLite for development, PostgreSQL for production
- [ ] **Authentication Middleware** - JWT/OAuth integration
- [ ] **CORS & Security** - Proper headers for browser integration
- [ ] **Content Validation** - Input sanitization and type checking

#### 1.2 **Integration Testing**
- [ ] **CLI Client Integration** - Test all CLI commands work with real server
- [ ] **JavaScript Client Integration** - Test browser editor saves to real server
- [ ] **End-to-End Workflow** - Edit → Save → Build → Deploy cycle

### **🟡 Phase 2: Production Polish (IMPORTANT)**

#### 2.1 **Client-Side Enhancements**
- [ ] **Editor-Server Integration** - Wire up `handleSave` to use `ApiClient` 
- [ ] **Optimistic Updates** - Show immediate feedback, sync in background
- [ ] **Offline Support** - LocalStorage cache + sync when online
- [ ] **Loading States** - Professional feedback during saves

#### 2.2 **Deployment Pipeline**
- [ ] **Build Triggers** - Auto-rebuild sites when content changes
- [ ] **Multi-Site Support** - Handle multiple domains/site IDs
- [ ] **CDN Integration** - Host insertr.js library on CDN
- [ ] **Database Migrations** - Schema versioning and updates

### **🟢 Phase 3: Advanced Features (NICE-TO-HAVE)**

#### 3.1 **Content Management Enhancements**
- [ ] **Content Versioning** - Track edit history and allow rollbacks
- [ ] **Content Validation** - Advanced validation rules per content type
- [ ] **Markdown Enhancements** - Live preview, toolbar, syntax highlighting  
- [ ] **Media Management** - Image upload and asset management

#### 3.2 **Developer Experience**
- [ ] **Development Tools** - Better debugging and development workflow
- [ ] **Configuration API** - Extensible field type system
- [ ] **Testing Suite** - Comprehensive test coverage
- [ ] **Documentation** - API reference and integration guides

## 💡 **Key Architectural Insights**

### **Why This Architecture is Brilliant**
1. **Performance First**: Regular visitors get pure static HTML with zero CMS overhead
2. **Separation of Concerns**: Content editing completely separate from site performance  
3. **Build-Time Optimization**: Database content gets "baked into" static HTML during builds
4. **Progressive Enhancement**: Sites work without JavaScript, editing enhances with JavaScript
5. **Framework Agnostic**: Works with Hugo, Next.js, Jekyll, Gatsby, vanilla HTML, etc.

### **Production Flow**
```
Content Edits → HTTP API Server → Database
                     ↓
Static Site Build ← CLI Enhancement ← Database Content
                     ↓  
              Enhanced HTML → CDN/Deploy
```

### **Current Working Flow** 
```
✅ Browser Editor → (404) Missing Server → ❌
✅ CLI Enhancement ← Mock Data ← ✅
✅ Static HTML Generation ← ✅
```

**Gap**: The HTTP server that connects editor saves to database storage.

## 🗂️ **Next Steps: Server Implementation**

### **✅ Implemented - Unified Binary Architecture**
```
✅ COMPLETED: All server functionality integrated into unified binary
cmd/
├── serve.go                             # Runtime API server command
└── enhance.go                          # Build-time enhancement command
internal/
├── api/                                # HTTP handlers and middleware
├── db/                                 # Multi-database layer with sqlc
└── content/                            # Content management logic
```

### **Files to Modify**
- `lib/src/core/editor.js:91` - Wire up `ApiClient` to `handleSave` method
- `README.md` - Add server setup instructions
- `docker-compose.yml` - Add server for development stack

## 🎯 **Success Criteria**

### **Phase 1 Complete When**:
- ✅ HTTP server running on `localhost:8080` (or configurable port)
- ✅ All 5 API endpoints returning proper JSON responses
- ✅ JavaScript editor successfully saves edits to database
- ✅ CLI enhancement pulls latest content from database
- ✅ Full edit → save → build → view cycle working end-to-end

### **Production Ready When**:
- ✅ Multi-site support with proper site isolation
- ✅ Authentication and authorization working
- ✅ Database migrations and backup strategy
- ✅ CDN hosting for insertr.js library
- ✅ Deployment documentation and examples

## Current Architecture Status

### ✅ **What's Working Well**
- **CLI Parser**: Detects 41+ elements across demo site, generates stable IDs
- **Authentication System**: Professional login/edit mode toggle with visual states
- **Form System**: Dynamic modal forms with smart field detection
- **Build Pipeline**: Automated library building and copying to demo site
- **Development Experience**: Hot reload with Air integration

### 🔍 **Investigation Results**
- **File Analysis**: All legacy code removed, clean single implementation
- **Integration Testing**: CLI ↔ Library integration works seamlessly  
- **Demo Site**: Both index.html and about.html use modern library correctly
- **Content Detection**: CLI successfully identifies text/markdown/link content types

## Immediate Next Steps

### 🎯 **Priority 1: Content Persistence** 
**Goal**: Make edits survive page reload
- Create `lib/src/core/content-manager.js` for LocalStorage operations
- Integrate with existing form system for automatic save/restore
- Add change tracking and storage management

### 🎯 **Priority 2: Server Application**
**Goal**: Backend API for real content storage
- Design REST API for content CRUD operations
- Add authentication integration (OAuth/JWT)
- Consider database choice (SQLite for simplicity vs PostgreSQL for production)

---

## 🏁 **Ready to Build**

The analysis is complete. The architecture is sound and 90% implemented. 

**Next Action**: Create the HTTP server application that implements the API contract both clients expect.

This will immediately unlock:
- ✅ Real content persistence (not just LocalStorage)
- ✅ Multi-user editing capabilities  
- ✅ Production-ready content management
- ✅ Full integration between browser editor and CLI enhancement

*Let's build the missing server!*

## 🏗️ **Database Schema Architecture Decision** (Sept 2025)

**Issue**: Inlined SQLite schema in `database.go` creates multiple sources of truth, same problem we just solved with model duplication.

### **Recommended Solutions** (in order of preference):

#### **🎯 Option 1: Schema-as-Query Pattern** ⭐ **RECOMMENDED**
```
db/queries/
├── content.sql          # CRUD queries
├── versions.sql         # Version control queries  
├── schema_setup.sql     # Schema initialization as named query
└── indexes_setup.sql    # Index creation as named query
```

**Benefits**:
- ✅ Single source of truth (schema files)
- ✅ sqlc generates type-safe setup functions  
- ✅ Consistent with existing sqlc workflow
- ✅ Database-agnostic parameter syntax (`sqlc.arg()`)

**Implementation**:
```sql
-- name: InitializeSchema :exec
CREATE TABLE IF NOT EXISTS content (...);
CREATE TABLE IF NOT EXISTS content_versions (...);

-- name: CreateIndexes :exec  
CREATE INDEX IF NOT EXISTS idx_content_site_id ON content(site_id);
```

#### **🔧 Option 2: Migration Tool Integration**
- Use `goose`, `golang-migrate`, or `dbmate`
- sqlc natively supports parsing migration directories
- Professional database management with up/down migrations
- **Trade-off**: Additional dependency and complexity

#### **🗂️ Option 3: Embedded Schema Files**
- Use `go:embed` to read schema files at compile time
- Keep schema in separate `.sql` files
- **Trade-off**: Not processed by sqlc, less type safety

**✅ COMPLETED**: Implemented Option 1 (Schema-as-Query) with important discovery:

**sqlc Limitations Discovered**:
- ✅ sqlc generates functions for `CREATE TABLE` and `ALTER TABLE` statements
- ❌ sqlc does **NOT** generate functions for `CREATE INDEX` or `CREATE TRIGGER` statements  
- 🔧 **Solution**: Handle table creation via sqlc-generated functions, handle indexes/triggers manually in Go initialization code

**Final Implementation**:
- `db/*/setup.sql` - Contains table creation queries (sqlc generates type-safe functions)
- `internal/db/database.go` - Manual index/trigger creation using raw SQL
- **Best Practice**: Use sqlc for what it supports, manual SQL for what it doesn't

## 🔄 **Editor Cache Architecture & Content State Management** (Dec 2024)

### **Current Architecture Issues Identified**

**Problem**: Conflict between preview system and content persistence after save operations.

**Root Cause**: The unified Editor system was designed with preview-first architecture:
- `originalContent` stores DOM state when editing begins
- `clearPreview()` always restores to `originalContent` 
- This creates race condition: `applyContent()` → `clearPreview()` → content reverted

### **✅ Implemented Solution: Post-Save Baseline Update**

**Strategy**: After successful save, update the stored `originalContent` to match the new saved state.

**Implementation** (`lib/src/ui/Editor.js`):
```js
// In save handler:
context.applyContent(content);        // Apply new content to DOM
context.updateOriginalContent();      // Update baseline to match current DOM  
this.previewer.clearPreview();        // Clear preview (won't revert since baseline matches)
```

**Benefits**:
- ✅ Content persists after save operations
- ✅ Future cancellations restore to saved state, not pre-edit state
- ✅ Maintains clean preview functionality
- ✅ No breaking changes to existing architecture

### **Future Considerations: Draft System Architecture**

**Current State Management**:
- **Browser Cache**: DOM elements store current content state
- **Server Cache**: Database stores persisted content
- **No Intermediate Cache**: Edits are either preview (temporary) or saved (permanent)

**Potential Draft System Design**:

#### **Option 1: LocalStorage Drafts**
```js
// Auto-save drafts locally during editing
const draftKey = `insertr_draft_${contentId}_${siteId}`;
localStorage.setItem(draftKey, JSON.stringify({
    content: currentContent,
    timestamp: Date.now(),
    originalContent: baseline
}));
```

**Benefits**: Offline support, immediate feedback, no server load
**Drawbacks**: Per-browser, no cross-device sync, storage limits

#### **Option 2: Server-Side Drafts**
```js
// Auto-save drafts to server with special draft flag
PUT /api/content/{id}/draft
{
    "value": "Draft content...",
    "type": "markdown",
    "is_draft": true
}
```

**Benefits**: Cross-device sync, unlimited storage, collaborative editing potential
**Drawbacks**: Server complexity, authentication requirements, network dependency

#### **Option 3: Hybrid Draft System**
- **LocalStorage**: Immediate draft saving during typing
- **Server Sync**: Periodic sync of drafts (every 30s or on significant changes)
- **Conflict Resolution**: Handle cases where server content changed while editing draft

### **Cache Invalidation Strategy**

**Current Behavior**:
- Content is cached in DOM until page reload
- No automatic refresh when content changes on server
- No awareness of multi-user editing scenarios

**Recommended Enhancements**:

#### **Option A: Manual Refresh Strategy**
- Add "Refresh Content" button to editor interface
- Check server content before editing (warn if changed)
- Simple conflict resolution (server wins vs local wins vs merge)

#### **Option B: Polling Strategy**  
- Poll server every N minutes for content changes
- Show notification if content was updated by others
- Allow user to choose: keep editing, reload, or merge

#### **Option C: WebSocket Strategy**
- Real-time content change notifications
- Live collaborative editing indicators
- Automatic conflict resolution

### **Implementation Priority**

**Phase 1** (Immediate): ✅ **COMPLETED**
- Fix content persistence after save (baseline update approach)

**Phase 2** (Short-term): 
- Add LocalStorage draft auto-save during editing
- Implement draft recovery on page reload
- Basic conflict detection (server timestamp vs local timestamp)

**Phase 3** (Long-term):
- Server-side draft support
- Real-time collaboration features
- Advanced conflict resolution

### **Design Principles for Draft System**

1. **Progressive Enhancement**: Site works without drafts, drafts enhance UX
2. **Data Safety**: Never lose user content, even in edge cases
3. **Performance First**: Drafts shouldn't impact site loading for regular visitors
4. **Conflict Transparency**: Always show user what's happening with their content
5. **Graceful Degradation**: Fallback to basic editing if draft system fails

**Note**: Current architecture already supports foundation for all these enhancements through the unified Editor system and API client pattern.