Commit graph

129 commits

Author SHA1 Message Date
Donna
ab52debf8b Add v0 prompt for frontend generation 2026-01-27 05:14:56 +00:00
Donna
f9b873d331 Add Premium Librarian spec — RLM-powered codebase intelligence
Features:
- Ingest entire codebases (millions of tokens)
- 4 built-in chunkers: code, prose, structured, tabular
- Custom WASM chunker escape hatch
- eXist-db storage with versioning
- RLM query processing for natural language questions
- Structural indexing (call graph, type hierarchy, symbols)

Use cases:
- Legacy code understanding
- API discovery
- Impact analysis
- Developer onboarding

Premium tier pricing model included.

Co-authored-by: Dan
2026-01-26 07:32:07 +00:00
Donna
1e2073a81c Add edge_mappings table for AI-assisted field mapping
New table stores:
- Edge identification (from_node → to_node)
- Analysis results (confidence, level, method)
- Proposed mapping (AI-generated)
- User mapping (overrides)
- Confirmation status

Indexes:
- By flow_id for listing
- Unique on (flow_id, from_node, to_node) for upsert

This supports the edge analysis API for visual wiring in the canvas.

Co-authored-by: Dan
2026-01-26 07:23:03 +00:00
Donna
a6f44b1fc9 Set up Alembic database migrations
- Initialize Alembic in bloxserver/
- Configure for async SQLAlchemy (converts async URLs to sync for migrations)
- Generate initial migration with all existing tables:
  - users (Clerk sync, Stripe billing)
  - flows (workflow definitions)
  - triggers (webhook, schedule, manual, event)
  - executions (run history)
  - user_api_keys (BYOK encrypted storage)
  - usage_records (billing metrics)
  - stripe_events (webhook idempotency)

Usage:
  cd bloxserver
  alembic upgrade head    # Apply migrations
  alembic revision --autogenerate -m 'description'  # New migration

Co-authored-by: Dan
2026-01-26 07:21:56 +00:00
Donna
cf49154ddd Add LLM-created sequences at runtime to edge analysis spec
- Runtime policy: green-only (no YOLO on yellow)
- LLM clarification flow when wiring fails
- Edge hints payload: map, constant, drop, expression
- Structured error response for LLM to resolve issues

Conservative but flexible: LLM can provide explicit instructions
to turn yellow into green.

Co-authored-by: Dan
2026-01-26 06:59:51 +00:00
Donna
ca7e426d57 Add parallelism-by-topology documentation
Key insight: parallelism is a wiring decision, not a config option.
- Direct wire = sequential execution
- Buffer node = parallel execution (spawns independent threads)

Includes:
- Visual diagrams comparing approaches
- Real-world video processing example
- Advanced patterns (throttled buffer, fan-out/fan-in, conditional)
- Implementation notes for buffer node contract
- Comparison with n8n/Zapier/Temporal

Co-authored-by: Dan
2026-01-26 06:52:15 +00:00
Donna
4aa40ed29b Add edge analysis API spec - AI-assisted field mapping
- POST /api/v1/flows/{id}/analyze-edge endpoint spec
- Confidence levels: high (green), medium (yellow), low (red)
- Heuristic + LLM analysis paths
- Database schema for edge_mappings
- Sequencer integration notes
- Future enhancements roadmap

Co-authored-by: Dan
2026-01-26 06:46:17 +00:00
dullfig
a623c534d5 Add Sequence and Buffer orchestration primitives
Implement two virtual node patterns for message flow orchestration:

- Sequence: Chains listeners in order (A→B→C), feeding each step's
  output as input to the next. Uses ephemeral listeners to intercept
  step results without modifying core pump behavior.

- Buffer: Fan-out to parallel worker threads with optional result
  collection. Supports fire-and-forget mode (collect=False) for
  non-blocking dispatch.

New files:
- sequence_registry.py / buffer_registry.py: State tracking
- sequence.py / buffer.py: Payloads and handlers
- test_sequence.py / test_buffer.py: 52 new tests

Pump additions:
- register_generic_listener(): Accept any payload type
- unregister_listener(): Cleanup ephemeral listeners
- Global singleton accessors for pump instance

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-25 14:56:15 -08:00
dullfig
a69eae79c5 Add FlowRunner - execution bridge for BloxServer flows
FlowRunner connects Flow domain model to xml-pipeline's StreamPump:

- Lifecycle management: start(), stop() with state tracking
- Flow → OrganismConfig conversion: nodes become listeners
- Trigger handling: webhook, schedule, manual triggers
- Message injection: inject() to specific nodes, trigger() by trigger ID
- Execution tracking: events log, status reporting, message counting

States: CREATED → STARTING → RUNNING → STOPPING → STOPPED (or ERROR)

Usage:
    runner = FlowRunner(flow)
    await runner.start()
    thread_id = await runner.trigger("webhook_id", payload)
    await runner.stop()

Known limitation: Tool nodes need adapter wrappers to convert
@tool decorator (ToolResult) to handler pattern (HandlerResponse).

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-22 22:20:59 -08:00
dullfig
9ba77b843d Add Flow domain model for BloxServer
Domain model bridges frontend canvas, database, and xml-pipeline:

- nodes.py: AgentNode, ToolNode, GatewayNode with serialization
  - Built-in tool mappings (calculate, fetch, shell, etc.)
  - Agent config (prompt, model, temperature)
  - Gateway config (federation + REST API)

- edges.py: Edge connections with conditions
  - Auto-compute peers from edges
  - Cycle detection, entry/exit node finding

- triggers.py: Webhook, Schedule, Manual, Event triggers
  - Config dataclasses for each type
  - Factory functions for common patterns

- flow.py: Main Flow class aggregating all components
  - to_organism_yaml(): Generate xml-pipeline config
  - from_canvas_json() / to_canvas_json(): React Flow compat
  - validate(): Check for errors before execution
  - to_db_dict(): Database serialization

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-22 22:15:51 -08:00
dullfig
a5c00c1e90 Add BloxServer API scaffold + architecture docs
BloxServer API (FastAPI + SQLAlchemy async):
- Database models: users, flows, triggers, executions, usage tracking
- Clerk JWT auth with dev mode bypass for local testing
- SQLite support for local dev, PostgreSQL for production
- CRUD routes for flows, triggers, executions
- Public webhook endpoint with token auth
- Health/readiness endpoints
- Pydantic schemas with camelCase aliases for frontend
- Docker + docker-compose setup

Architecture documentation:
- Librarian architecture with RLM-powered query engine
- Stripe billing integration (usage-based, trials, webhooks)
- LLM abstraction layer (rate limiting, semantic cache, failover)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-22 22:04:25 -08:00
dullfig
d184d22c60 Add Monitor AI (shadow watchdog) to security model
Invisible AI watchdog for every flow:
- Read-only access to context buffer
- Cannot emit messages to pipeline
- Agents have no way to detect or probe it
- Alerts via control plane (email, UI, auto-stop)
- Runs on cheap models (Mistral/Mixtral)

Watches for: endless loops, goal drift, prompt injection,
sandbox escape attempts, token budget exhaustion.

Added to Phase 2 (core safety feature).

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-22 20:54:51 -08:00
dullfig
d682c677db Rename Nextra → BloxServer (OpenBlox.ai)
- Rename all nextra-* files to bloxserver-*
- Replace all "Nextra" references with "BloxServer"
- Update copyright year to 2026
- Add domain: OpenBlox.ai

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-22 20:43:43 -08:00
dullfig
9bfbb5d8c7 Add AI-assisted WASM coding feature (Pro)
Monaco editor with AI assistance for writing AssemblyScript:
- Inline completion (like Copilot)
- Chat panel (like Claude Code)
- Frontend calls LLM API directly
- Human reviews before building

Ships with Phase 3 (Monaco integration).

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-20 22:42:47 -08:00
dullfig
f275382922 Add AI Flow Builder Assistant architecture
The assistant is itself a Nextra flow (dogfooding):
- Builder agent with catalog, validator, examples tools
- Queries real available nodes dynamically
- Self-validates generated YAML before returning
- Uses marketplace flows as few-shot examples
- Same billing model (LLM tokens)

Added Phase 4.5 to implementation roadmap.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-20 22:35:10 -08:00
dullfig
1beabd5736 Document Flow Lifecycle Controls (Run/Stop only)
- Simple two-state model: Stopped ↔ Running
- Edit only allowed when stopped
- No pause (simpler, matches Zapier/n8n/Make)
- No hot-edit (unsafe for mid-execution swarms)
- Future consideration: Graceful Stop for Pro users

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-20 22:27:49 -08:00
dullfig
51cd94f0f2 Document Monaco TypeScript mode for AssemblyScript editing
- Monaco's built-in TS language service provides IDE features
- AS type definitions loaded at startup for autocomplete
- Real errors come from `asc` compiler at build time
- No separate LSP server (asls) needed = zero infra cost

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-20 22:23:37 -08:00
dullfig
3a128d4d1f Fix line endings in wiki docs
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-20 22:16:26 -08:00
dullfig
03516bb9f7 Add Nextra API contract (TypeScript + Pydantic)
Defines shared API contract between frontend and backend:
- types.ts: TypeScript interfaces for Next.js frontend
- models.py: Matching Pydantic models for FastAPI backend

Covers: User, Flow, Trigger, Execution, WasmModule, Marketplace,
ProjectMemory, and pagination/error types.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-20 22:16:20 -08:00
dullfig
a6cd3ce7e8 Add v0 prompt for Nextra landing page
Detailed prompt for generating the SaaS landing page with Vercel v0.
Includes: hero, features, pricing, testimonials, FAQ, and styling specs.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-20 22:09:49 -08:00
dullfig
2d05fd3656 Add Nextra SaaS platform architecture document
Comprehensive design doc covering:
- System overview and architecture diagrams
- Tier model (Free/Paid/Pro/Enterprise)
- Component architecture (Next.js, FastAPI, Pump containers)
- Database schema (PostgreSQL)
- Trigger system (webhooks, schedules, manual)
- Security model (tenant isolation, WASM sandboxing, memory safety)
- Canvas ↔ YAML synchronization
- Marketplace design
- Monitoring and observability
- Scaling considerations
- Implementation phases

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-20 21:40:33 -08:00
dullfig
515c738abb Add wiki documentation for xml-pipeline.org
Comprehensive documentation set for XWiki:
- Home, Installation, Quick Start guides
- Writing Handlers and LLM Router guides
- Architecture docs (Overview, Message Pump, Thread Registry, Shared Backend)
- Reference docs (Configuration, Handler Contract, CLI)
- Hello World tutorial
- Why XML rationale
- Pandoc conversion scripts (bash + PowerShell)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-20 20:40:47 -08:00
dullfig
c01428260c Archive obsolete docs and misc cleanup
- Move lsp-integration.md and secure-console-v3.md to docs/archive-obsolete/
  (these features are now in the Nextra SaaS product)
- Update CLAUDE.md with current project state
- Simplify run_organism.py
- Fix test fixtures for shared backend compatibility
- Minor handler and llm_api cleanups

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-20 20:20:10 -08:00
dullfig
6790c7a46c Add shared backend for multiprocess pipeline support
Introduces SharedBackend Protocol for cross-process state sharing:
- InMemoryBackend: default single-process storage
- ManagerBackend: multiprocessing.Manager for local multi-process
- RedisBackend: distributed deployments with TTL auto-GC

Adds ProcessPoolExecutor support for CPU-bound handlers:
- worker.py: worker process entry point
- stream_pump.py: cpu_bound handler dispatch
- Config: backend and process_pool sections in organism.yaml

ContextBuffer and ThreadRegistry now accept optional backend
parameter while maintaining full backward compatibility.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-20 20:18:22 -08:00
dullfig
f87d9f80e9 Move console, auth, server to Nextra (v0.4.0)
These modules are now proprietary and live in the Nextra SaaS product.
xml-pipeline remains the OSS core with:
- Message pump and pipeline steps
- Handler contract and responses
- LLM router abstraction
- Native tools
- Config loading
- Memory/context buffer

Removed:
- xml_pipeline/console/ → nextra/console/
- xml_pipeline/auth/ → nextra/auth/
- xml_pipeline/server/ → nextra/server/
- Legacy files: agentserver.py, main.py, xml_listener.py

The simple console example remains in examples/console/.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-19 22:37:21 -08:00
dullfig
d53bc1dfbe Fix XSD path for installed package (v0.3.1)
Use __file__-based path resolution for envelope.xsd so the schema
loads correctly when xml-pipeline is installed via pip.

Also:
- Add build artifacts to .gitignore
- Bump version to 0.3.1

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-19 22:24:36 -08:00
dullfig
6b9c378e52 Bump version to 0.3.0
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-19 21:54:58 -08:00
dullfig
947cd5838a Fix termcolor compatibility in vendored xmlable
Remove unused Color type import from termcolor.termcolor which doesn't
exist in newer termcolor versions. Change type hints from Color to str.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-19 21:53:32 -08:00
dullfig
0553ec294f Update README and pyproject.toml for PyPI release
README.md:
- Rebrand from AgentServer to xml-pipeline
- Library-focused introduction with pip install
- Quick start guide with code examples
- Console example documentation
- Concise feature overview

pyproject.toml:
- Update authors to "xml-pipeline contributors"
- Update URLs to xml-pipeline.org

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-19 21:48:52 -08:00
dullfig
3cae862d7a Update .gitignore for package rename
- Add .claude/ (Claude Code config)
- Update paths from agentserver/ to xml_pipeline/

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-19 21:44:44 -08:00
dullfig
e653d63bc1 Rename agentserver to xml_pipeline, add console example
OSS restructuring for open-core model:
- Rename package from agentserver/ to xml_pipeline/
- Update all imports (44 Python files, 31 docs/configs)
- Update pyproject.toml for OSS distribution (v0.3.0)
- Move prompt_toolkit from core to optional [console] extra
- Remove auth/server/lsp from core optional deps (-> Nextra)

New console example in examples/console/:
- Self-contained demo with handlers and config
- Uses prompt_toolkit (optional, falls back to input())
- No password auth, no TUI, no LSP — just the basics
- Shows how to use xml-pipeline as a library

Import changes:
- from agentserver.* -> from xml_pipeline.*
- CLI entry points updated: xml_pipeline.cli:main

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-19 21:41:19 -08:00
dullfig
3ffab8a3dd fixing docs 2026-01-18 18:00:11 -08:00
dullfig
0fb35da5dd Add /configure command with full-screen editor
- AgentConfigStore: Per-agent YAML configs in ~/.xml-pipeline/agents/
- Full-screen editor using prompt_toolkit with YAML highlighting
- /configure: Edit organism.yaml (swarm wiring)
- /configure @agent: Edit agent config (prompt, model, temperature)

Agent configs separate behavior (prompt, model) from wiring (peers, routing).

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-17 21:17:43 -08:00
dullfig
0796e45412 Add SystemPipeline for external message injection
- SystemPipeline: Entry point for console/webhook/API messages
- TextInput/TextOutput: Generic primitives for human text I/O
- Server: WebSocket "send" command routes through SystemPipeline
- Console: @target message now injects into pipeline

Flow: Console → WebSocket → SystemPipeline → XML envelope → pump

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-17 21:07:28 -08:00
dullfig
ebf72c1f8c Add authentication system and HTTP/WebSocket server
- auth/users.py: User store with Argon2id password hashing
- auth/sessions.py: Token-based session management with expiry
- server/app.py: aiohttp server with auth middleware and WebSocket
- console/client.py: SSH-style login console client

Server endpoints: /auth/login, /auth/logout, /auth/me, /health, /ws

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-17 20:50:53 -08:00
dullfig
986db2e79b Implement native tools and infrastructure
Tools (18 total):
- calculate: Safe AST-based math expression evaluator
- fetch: Async HTTP with SSRF protection
- files: Sandboxed read/write/list/delete
- shell: Command execution with blocklist
- search: Web search (SerpAPI, Google, Bing)
- keyvalue: In-memory key-value store
- librarian: exist-db XML database integration
- convert: XML↔JSON conversion + XPath extraction

Infrastructure:
- CLI with run/init/check/version commands
- Config loader for organism.yaml
- Feature detection for optional dependencies
- Optional extras in pyproject.toml

LLM:
- Fixed llm_connection.py to wrap working router

WASM:
- Documented WASM listener interface
- Stub implementation for future work

MCP:
- Reddit sentiment MCP server example

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-17 20:25:48 -08:00
dullfig
3764c30628 Add tool stubs for native agent tools
Stub implementations for:
- base.py: Tool, ToolResult, @tool decorator, registry
- calculate.py: Math expressions (simpleeval)
- fetch.py: HTTP requests (aiohttp)
- files.py: read_file, write_file, list_dir
- shell.py: run_command (sandboxed)
- search.py: web_search
- keyvalue.py: key_value_get/set/delete (in-memory stub)
- librarian.py: exist-db integration (store, get, query, search)

All stubs return "Not implemented" - ready for real implementation.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-12 22:40:58 -08:00
dullfig
7950be66f3 Add native tools specification
Core tools:
- calculate (simpleeval) - math expressions in Python syntax
- fetch_url - HTTP requests with security controls
- read_file / write_file / list_dir - sandboxed file ops
- run_command - sandboxed shell execution
- web_search - web search integration
- key_value_store - persistent agent state
- send_email / webhook - notifications

Librarian tools (exist-db):
- librarian_store - store XML documents
- librarian_get - retrieve by path
- librarian_query - XQuery execution
- librarian_search - full-text search

Includes permission model and implementation checklist.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-12 22:37:40 -08:00
dullfig
1de6a1249e fixing docs 2026-01-12 22:31:06 -08:00
dullfig
398fd5ac23 Add Lua listeners proposal and message log stream
Lua proposal covers:
- Sandboxed Lua handlers for SaaS custom logic
- XML <-> Lua table bridging with XSD validation
- Security sandbox (restricted stdlib, resource limits)
- Schema DSL alternative to raw XSD
- Hot reload mechanism

API spec updated with:
- Dedicated /ws/messages endpoint for message log streaming
- Filter by agent/thread/payload type
- Example for reconstructing agent buffers client-side

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-12 22:30:45 -08:00
dullfig
a29c6f8321 Add security considerations document
Covers authentication (TOTP, WebAuthn, mTLS, OAuth), authorization
(RBAC, resource permissions), transport security, secrets management,
audit logging, prompt injection defense, and rate limiting.

Includes open questions and implementation checklist.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 22:52:55 -08:00
dullfig
5ccffece60 Add agentserver API specification
Defines the GUI-agnostic API for monitoring and controlling the organism:
- REST endpoints for queries and control
- WebSocket for real-time push events
- Data models for agents, messages, threads
- Example usage for 3D office GUI

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 22:47:57 -08:00
dullfig
a6b83f2dbc fixing docs 2026-01-11 16:16:25 -08:00
dullfig
a9035d34d8 fixing docs 2026-01-11 16:01:04 -08:00
dullfig
fff43f1a78 fixing docs 2026-01-11 16:00:03 -08:00
dullfig
573caaa681 Fix bottom detection - consider last 3 lines as "at bottom"
- is_at_bottom() now checks if on last 3 lines (not exact end)
- Page Down snaps to bottom when within 3 lines of end
- Added scroll_to_bottom() helper method

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 15:30:53 -08:00
dullfig
45d6eea4ac Dynamic spacer and silent append when scrolled up
- Spacer only shows when output is at bottom (disappears when scrolled up)
- New content appends silently when user has scrolled up (preserves position)
- is_at_bottom() helper to check scroll state

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 15:27:21 -08:00
dullfig
4dcc726280 Fix command history with history_backward/forward methods
auto_up/auto_down were wrong - use history_backward/history_forward
for proper command history navigation.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 15:23:41 -08:00
dullfig
8e98e98bf1 Add command history (Up/Down) and spacer above separator
- Up/Down arrows now navigate command history via auto_up/auto_down
- Added blank line spacer between output and separator for clarity

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 15:21:07 -08:00
dullfig
35b3381b7c Simplify scroll: Page Up/Down always, Up/Down for history
Standard CLI behavior:
- Up/Down = command history (always)
- Page Up/Down = scroll output (no focus needed)
- Ctrl+Home/End = jump to top/bottom
- Mouse/scrollbar works too

Removed Tab focus switching - not intuitive for CLI users.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 15:15:53 -08:00