summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorBen Sima <ben@bsima.me>2025-11-20 19:11:00 -0500
committerBen Sima <ben@bsima.me>2025-11-20 19:11:00 -0500
commitb69682dd5e6ee374d066dcbfc90e96b47fe341cf (patch)
tree04af90d5e212c1d135c0fc5731d2b17943012f4b
parent0d554cc268343687406c1ea1a87d35d1595e2503 (diff)
doc: add design docs for open epics
- Omni/Task/DESIGN.md: Task manager improvements - Biz/PodcastItLater/DESIGN.md: Architecture and features - Biz/PodcastItLater/TESTING.md: Test strategy - Network/Wai/Middleware/Braid/DESIGN.md: Keep-alive design
-rw-r--r--Biz/PodcastItLater/DESIGN.md43
-rw-r--r--Biz/PodcastItLater/TESTING.md45
-rw-r--r--Network/Wai/Middleware/Braid/DESIGN.md27
-rw-r--r--Omni/Task/DESIGN.md232
4 files changed, 347 insertions, 0 deletions
diff --git a/Biz/PodcastItLater/DESIGN.md b/Biz/PodcastItLater/DESIGN.md
new file mode 100644
index 0000000..29c4837
--- /dev/null
+++ b/Biz/PodcastItLater/DESIGN.md
@@ -0,0 +1,43 @@
+# PodcastItLater Design & Architecture
+
+## Overview
+Service converting web articles to podcast episodes via email/web submission.
+
+## Architecture
+- **Web**: `Biz/PodcastItLater/Web.py` (Ludic + HTMX + Starlette)
+- **Worker**: `Biz/PodcastItLater/Worker.py` (Background processing)
+- **Core**: `Biz/PodcastItLater/Core.py` (DB & Shared Logic)
+- **Billing**: `Biz/PodcastItLater/Billing.py` (Stripe Integration)
+
+## Key Features
+1. **User Management**: Email-based magic links, RSS tokens.
+2. **Article Processing**: Trafilatura extraction -> LLM cleanup -> TTS.
+3. **Billing (In Progress)**: Stripe Checkout/Portal, Freemium model.
+
+## Path to Paid Product (Epic: t-143KQl2)
+
+### 1. Billing Infrastructure
+- **Stripe**: Use Stripe Checkout for subs, Portal for management.
+- **Webhooks**: Handle `checkout.session.completed`, `customer.subscription.*`.
+- **Tiers**:
+ - `Free`: 10 articles/month.
+ - `Paid`: Unlimited (initially).
+
+### 2. Usage Tracking
+- **Table**: `users` table needs `plan_tier`, `subscription_status`, `stripe_customer_id`.
+- **Logic**: Check usage count vs tier limit before allowing submission.
+- **Reset**: Usage counters reset at billing period boundary.
+
+### 3. Admin Dashboard
+- View all users and their status.
+- Manually retry/delete jobs.
+- View metrics (signups, conversions).
+
+## UX Polish (Epic: t-1vIPJYG)
+- **Mobile**: Ensure all pages work on mobile.
+- **Feedback**: Better error messages for failed URLs.
+- **Navigation**: Clean up navbar, account management access.
+
+## Audio Improvements
+- **Intro/Outro**: Add metadata-rich intro ("Title by Author").
+- **Sound Design**: Crossfade intro music.
diff --git a/Biz/PodcastItLater/TESTING.md b/Biz/PodcastItLater/TESTING.md
new file mode 100644
index 0000000..2911610
--- /dev/null
+++ b/Biz/PodcastItLater/TESTING.md
@@ -0,0 +1,45 @@
+# PodcastItLater Testing Strategy
+
+## Overview
+We use `pytest` with `Omni.Test` integration. Tests are co-located with code or in `Biz/PodcastItLater/Test.py` for E2E.
+
+## Test Categories
+
+### 1. Core (Database/Logic)
+- **Location**: `Biz/PodcastItLater/Core.py`
+- **Scope**: User creation, Job queue ops, Episode management.
+- **Key Tests**:
+ - `test_create_user`: Unique tokens.
+ - `test_queue_isolation`: Users see only their jobs.
+
+### 2. Web (HTTP/UI)
+- **Location**: `Biz/PodcastItLater/Web.py`
+- **Scope**: Routes, Auth, HTMX responses.
+- **Key Tests**:
+ - `test_submit_requires_auth`.
+ - `test_rss_feed_xml`.
+ - `test_admin_access_control`.
+
+### 3. Worker (Processing)
+- **Location**: `Biz/PodcastItLater/Worker.py`
+- **Scope**: Extraction, TTS, S3 upload.
+- **Key Tests**:
+ - `test_extract_content`: Mocked network calls.
+ - `test_tts_chunking`: Handle long text.
+ - **Error Handling**: Ensure retries work and errors are logged.
+
+### 4. Billing (Stripe)
+- **Location**: `Biz/PodcastItLater/Billing.py`
+- **Scope**: Webhook processing, Entitlement checks.
+- **Key Tests**:
+ - `test_webhook_subscription_update`: Update local DB.
+ - `test_enforce_limits`: Block submission if over limit.
+
+## Running Tests
+```bash
+# Run all
+bild --test Biz/PodcastItLater.hs
+
+# Run specific file
+./Biz/PodcastItLater/Web.py test
+```
diff --git a/Network/Wai/Middleware/Braid/DESIGN.md b/Network/Wai/Middleware/Braid/DESIGN.md
new file mode 100644
index 0000000..90c6016
--- /dev/null
+++ b/Network/Wai/Middleware/Braid/DESIGN.md
@@ -0,0 +1,27 @@
+# Braid Middleware Design
+
+**Goal**: Implement HTTP Keep-Alive mechanism for Braid updates to support real-time streams.
+
+## Current State
+`Network/Wai/Middleware/Braid.hs` implements the Braid protocol headers but lacks a robust mechanism to keep the connection open and push updates.
+
+## Design Requirements
+
+1. **Connection Management**:
+ - Identify Braid subscriptions via headers.
+ - Keep the response body open (streaming response).
+ - Handle client disconnects gracefully.
+
+2. **Update Channel**:
+ - Use a `TChan` or `BroadcastChan` to signal updates to the connection handler.
+ - When a resource changes, push a new Braid frame to the open stream.
+
+3. **Frame Format**:
+ - Adhere to Braid spec for patch frames.
+ - `Content-Type: application/json` (or relevant type).
+ - `Merge-Type: braid`.
+
+## Implementation Plan
+1. Modify middleware to hijack response for Braid requests.
+2. Spawn a lightweight thread to listen on an update channel.
+3. Stream chunks to the client.
diff --git a/Omni/Task/DESIGN.md b/Omni/Task/DESIGN.md
new file mode 100644
index 0000000..0dbf3b5
--- /dev/null
+++ b/Omni/Task/DESIGN.md
@@ -0,0 +1,232 @@
+# Task Manager Improvement Plan
+
+Based on beads project planning patterns, here are proposed improvements for Omni/Task.hs.
+
+## Current State
+
+**What we have:**
+- ✅ Basic CRUD operations (create, list, update, ready)
+- ✅ Dependency tracking (--deps for blocking)
+- ✅ JSONL storage with git sync
+- ✅ Short base62 task IDs
+- ✅ Optional namespace field
+- ✅ Project field for grouping
+- ✅ Three status levels (open, in-progress, done)
+
+**What we're missing:**
+- ❌ Multiple dependency types (blocks, discovered-from, parent-child, related)
+- ❌ Hierarchical task IDs (parent.1, parent.2)
+- ❌ Task types (epic vs task) - epics will replace "project"
+- ❌ Dependency tree visualization
+- ❌ Work discovery tracking
+- ❌ Epic/child task relationships
+
+## Proposed Improvements (Priority Order)
+
+### Phase 1: Core Features (High Priority)
+
+#### 1.1 Add Task Types (Epic vs Task)
+```haskell
+data TaskType = Epic | Task
+ deriving (Show, Eq, Generic)
+```
+
+**Benefits:**
+- Epics are containers for related tasks (replace "project" concept)
+- Tasks are the actual work items
+- Simple two-level hierarchy
+- Epic-based planning support
+
+**Schema Changes:**
+- Replace `taskProject :: Text` with `taskType :: TaskType`
+- Add `taskParent :: Maybe Text` for parent epic
+- Epics can contain tasks or other epics (for nested structure)
+
+**Commands:**
+```bash
+# Create an epic (container)
+task create "User Authentication System" --type=epic
+
+# Create tasks within an epic
+task create "Design API" --type=task --parent=t-abc123
+task create "Implement JWT" --type=task --parent=t-abc123
+
+# Create a sub-epic (optional, for complex projects)
+task create "OAuth Integration" --type=epic --parent=t-abc123
+```
+
+#### 1.2 Enhanced Dependency Types
+```haskell
+data DependencyType = Blocks | DiscoveredFrom | ParentChild | Related
+ deriving (Show, Eq, Generic)
+```
+
+**Benefits:**
+- Track work discovery context
+- Maintain audit trail
+- Support epic hierarchies
+
+**Commands:**
+```bash
+task create "Fix bug" project --discovered-from=t-abc123
+task create "Subtask 1" project --parent=t-epic-id
+task dep add t-123 t-124 --type=related
+```
+
+### Phase 2: Hierarchical Tasks (Medium Priority)
+
+#### 2.1 Parent-Child Task IDs
+**Pattern:** `t-abc123.1`, `t-abc123.2`, `t-abc123.3`
+
+**Benefits:**
+- Human-friendly sequential IDs under epic
+- Natural work breakdown
+- Up to 3 levels of nesting
+
+**Schema Changes:**
+```haskell
+data Task = Task
+ { ...
+ taskParent :: Maybe Text -- Parent task ID
+ ...
+ }
+
+-- New table for child counters
+CREATE TABLE child_counters (
+ parent_id TEXT PRIMARY KEY,
+ last_child INTEGER NOT NULL DEFAULT 0,
+ FOREIGN KEY (parent_id) REFERENCES tasks(id) ON DELETE CASCADE
+);
+```
+
+**Commands:**
+```bash
+task create "Design auth API" project --parent=t-abc123
+# Creates: t-abc123.1
+
+task create "Implement JWT" project --parent=t-abc123
+# Creates: t-abc123.2
+```
+
+#### 2.2 Dependency Tree Visualization
+```bash
+task tree t-epic-id
+# Shows:
+# t-abc123 [Epic] User Authentication System
+# t-abc123.1 [Task] Design auth API
+# t-abc123.2 [Task] Implement JWT
+# t-abc123.2.1 [Task] Add token generation
+# t-abc123.2.2 [Task] Add token validation
+# t-abc123.3 [Task] Add password hashing
+```
+
+### Phase 3: Project Management (Lower Priority)
+
+#### 3.1 Task Filtering and Queries
+```bash
+task list --type=epic
+task list --status=open
+task list --parent=t-epic-id # List all children
+```
+
+#### 3.2 Epic Statistics
+```bash
+task stats # Overall stats
+task stats --epic=t-abc123 # Epic-specific
+task progress t-epic-id # Epic completion %
+```
+
+#### 3.3 Discovered Work Tracking
+```bash
+task create "Found memory leak" project --discovered-from=t-abc123
+# Automatically links context
+# Shows in dependency tree as "discovered during t-abc123"
+```
+
+## Implementation Strategy
+
+### Milestone 1: Type System Foundations
+- [ ] Add TaskType enum (Epic | Task)
+- [ ] Add DependencyType enum
+- [ ] Update Task data structure (replace project with type and parent)
+- [ ] Update CLI commands
+- [ ] Update tests
+- [ ] Update AGENTS.md
+- [ ] Migration: existing tasks default to type=Task, project becomes epic name
+
+### Milestone 2: Enhanced Dependencies
+- [ ] Add `discovered-from` support
+- [ ] Add `related` dependency type
+- [ ] Track dependency metadata (who, when, why)
+- [ ] Update ready work algorithm to respect dependency types
+
+### Milestone 3: Hierarchical Structure
+- [ ] Add parent field to Task
+- [ ] Implement child ID generation (t-abc123.1)
+- [ ] Add child_counters table/storage
+- [ ] Update createTask to handle --parent flag
+- [ ] Implement parent-child dependency creation
+
+### Milestone 4: Visualization & Reporting
+- [ ] Implement `task tree` command
+- [ ] Implement `task stats` command
+- [ ] Implement `task progress` for epics
+- [ ] Add filtering by type, priority
+- [ ] Improve task list display with colors/formatting
+
+## Open Questions
+
+1. **Storage Format**: Should we keep JSONL or move to SQLite like beads?
+ - JSONL: Simple, git-friendly, human-readable
+ - SQLite: Fast queries, complex relationships, beads-compatible
+ - **Recommendation**: Start with JSONL, can add SQLite later for caching
+
+2. **Child Counter Storage**: Where to store child counters?
+ - Option A: Separate .tasks/counters.jsonl file
+ - Option B: In-memory during session, persist to JSONL
+ - Option C: Add SQLite just for this
+ - **Recommendation**: Option A - separate JSONL file
+
+3. **Dependency Storage**: How to store complex dependencies?
+ - Current: List of text IDs in task
+ - Beads: Separate dependencies table
+ - **Recommendation**: Add dependencies field with type info:
+ ```haskell
+ data Dependency = Dependency
+ { depId :: Text
+ , depType :: DependencyType
+ }
+ ```
+
+4. **Backward Compatibility**: How to handle existing tasks?
+ - Add sensible defaults (type=Task, priority=Medium)
+ - Migration script or auto-upgrade on load?
+ - **Recommendation**: Auto-upgrade with defaults on import
+
+## Benefits Summary
+
+**For AI Agents:**
+- Better work discovery and context tracking
+- Clearer project structure
+- Easier to understand what work is related
+- Natural way to break down large tasks
+
+**For Humans:**
+- Epic-based planning for large features
+- Priority-driven work queue
+- Visual dependency trees
+- Better project tracking and reporting
+
+**For Collaboration:**
+- Discovered work maintains context
+- Related work is easily found
+- Epic structure provides clear organization
+- Dependency tracking prevents duplicate work
+
+## Next Steps
+
+1. Create tasks for each milestone
+2. Start with Milestone 1 (Type System Foundations)
+3. Get feedback on hierarchical ID format
+4. Implement incrementally, test thoroughly
+5. Update AGENTS.md with new patterns