Commit graph

17 commits

Author SHA1 Message Date
dullfig
860395cd58 Add usage/gas tracking REST API endpoints
Endpoints:
- GET /api/v1/usage - Overview with totals, per-agent, per-model breakdown
- GET /api/v1/usage/threads - List all thread budgets sorted by usage
- GET /api/v1/usage/threads/{id} - Single thread budget details
- GET /api/v1/usage/agents/{id} - Usage totals for specific agent
- GET /api/v1/usage/models/{model} - Usage totals for specific model
- POST /api/v1/usage/reset - Reset all usage tracking

Models:
- UsageTotals, UsageOverview, UsageResponse
- ThreadBudgetInfo, ThreadBudgetListResponse
- AgentUsageInfo, ModelUsageInfo

Also adds has_budget() method to ThreadBudgetRegistry for checking
if a thread exists without auto-creating it.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-27 21:20:36 -08:00
dullfig
8b11323a8b Add token budget enforcement and usage tracking
Token Budget System:
- ThreadBudgetRegistry tracks per-thread token usage with configurable limits
- BudgetExhaustedError raised when thread exceeds max_tokens_per_thread
- Integrates with LLMRouter to block LLM calls when budget exhausted
- Automatic cleanup when threads are pruned

Usage Tracking (for production billing):
- UsageTracker emits events after each LLM completion
- Subscribers receive UsageEvent with tokens, latency, estimated cost
- Cost estimation for common models (Grok, Claude, GPT, etc.)
- Aggregate stats by agent, model, and totals

Configuration:
- max_tokens_per_thread in organism.yaml (default 100k)
- LLMRouter.complete() accepts thread_id and metadata parameters

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-27 21:07:43 -08:00
dullfig
4530c06835 Add capability introspection REST endpoints
Adds operator-only endpoints for discovering organism capabilities:
- GET /api/v1/capabilities - list all listeners
- GET /api/v1/capabilities/{name} - detailed info with schema/example

These are REST-only for operators. Agents cannot access them -
they only know their declared peers (peer constraint isolation).

10 new tests for introspection functionality.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-27 20:51:17 -08:00
dullfig
3ff399e849 Add hot-reload for organism configuration
Implements runtime configuration reload via POST /api/v1/organism/reload:
- StreamPump.reload_config() re-reads organism.yaml
- Adds new listeners, removes old ones, updates changed ones
- System listeners (system.*) are protected from removal
- ReloadEvent emitted to notify WebSocket subscribers
- ServerState.reload_config() refreshes agent runtime state

14 new tests covering add/remove/update scenarios.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-27 20:38:48 -08:00
dullfig
bf31b0d14e Add AgentServer REST/WebSocket API
Implements the AgentServer API from docs/agentserver_api_spec.md:

REST API (/api/v1):
- Organism info and config endpoints
- Agent listing, details, config, schema
- Thread and message history with filtering
- Control endpoints (inject, pause, resume, kill, stop)

WebSocket:
- /ws: Main control channel with state snapshot + real-time events
- /ws/messages: Dedicated message stream with filtering

Infrastructure:
- Pydantic models with camelCase serialization
- ServerState bridges StreamPump to API
- Pump event hooks for real-time updates
- CLI 'serve' command: xml-pipeline serve [config] --port 8080

35 new tests for models, state, REST, and WebSocket.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-27 20:22:58 -08:00
dullfig
809862af35 Add Ed25519 envelope signing infrastructure
Implement cryptographic signing for message envelopes using Ed25519:

- Identity module: Generate, load, save Ed25519 keypairs
- Signing module: Sign/verify envelopes using Exclusive C14N
- Envelope utilities: Build envelopes with optional signing
- CLI keygen command: xml-pipeline keygen [-o path]
- Pump integration: Auto-sign when identity configured

Signature is embedded in <meta> block using namespace
https://xml-pipeline.org/ns/sig/v1, fitting existing xs:any in envelope.xsd.

Usage:
  xml-pipeline keygen -o config/identity.key

  # organism.yaml
  organism:
    identity: "config/identity.key"

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-27 20:00:35 -08:00
dullfig
a623c534d5 Add Sequence and Buffer orchestration primitives
Implement two virtual node patterns for message flow orchestration:

- Sequence: Chains listeners in order (A→B→C), feeding each step's
  output as input to the next. Uses ephemeral listeners to intercept
  step results without modifying core pump behavior.

- Buffer: Fan-out to parallel worker threads with optional result
  collection. Supports fire-and-forget mode (collect=False) for
  non-blocking dispatch.

New files:
- sequence_registry.py / buffer_registry.py: State tracking
- sequence.py / buffer.py: Payloads and handlers
- test_sequence.py / test_buffer.py: 52 new tests

Pump additions:
- register_generic_listener(): Accept any payload type
- unregister_listener(): Cleanup ephemeral listeners
- Global singleton accessors for pump instance

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-25 14:56:15 -08:00
dullfig
c01428260c Archive obsolete docs and misc cleanup
- Move lsp-integration.md and secure-console-v3.md to docs/archive-obsolete/
  (these features are now in the Nextra SaaS product)
- Update CLAUDE.md with current project state
- Simplify run_organism.py
- Fix test fixtures for shared backend compatibility
- Minor handler and llm_api cleanups

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-20 20:20:10 -08:00
dullfig
6790c7a46c Add shared backend for multiprocess pipeline support
Introduces SharedBackend Protocol for cross-process state sharing:
- InMemoryBackend: default single-process storage
- ManagerBackend: multiprocessing.Manager for local multi-process
- RedisBackend: distributed deployments with TTL auto-GC

Adds ProcessPoolExecutor support for CPU-bound handlers:
- worker.py: worker process entry point
- stream_pump.py: cpu_bound handler dispatch
- Config: backend and process_pool sections in organism.yaml

ContextBuffer and ThreadRegistry now accept optional backend
parameter while maintaining full backward compatibility.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-20 20:18:22 -08:00
dullfig
e653d63bc1 Rename agentserver to xml_pipeline, add console example
OSS restructuring for open-core model:
- Rename package from agentserver/ to xml_pipeline/
- Update all imports (44 Python files, 31 docs/configs)
- Update pyproject.toml for OSS distribution (v0.3.0)
- Move prompt_toolkit from core to optional [console] extra
- Remove auth/server/lsp from core optional deps (-> Nextra)

New console example in examples/console/:
- Self-contained demo with handlers and config
- Uses prompt_toolkit (optional, falls back to input())
- No password auth, no TUI, no LSP — just the basics
- Shows how to use xml-pipeline as a library

Import changes:
- from agentserver.* -> from xml_pipeline.*
- CLI entry points updated: xml_pipeline.cli:main

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-19 21:41:19 -08:00
dullfig
f75aa3eee6 Add SecureConsole v3.0 with password authentication
- SecureConsole with Argon2id password hashing
- Password stored in ~/.xml-pipeline/console.key
- Protected commands require password re-entry
- Attach/detach model with idle timeout
- Fallback to simple input for incompatible terminals
- @listener message injection into pump
- Boot handler no longer sends to old console
- Response handler prints to stdout with prompt refresh

Dependencies: argon2-cffi, prompt_toolkit

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-10 18:24:24 -08:00
dullfig
fc7170a02e Add context buffer - virtual memory manager for AI agents
Implements thread-scoped, append-only storage for validated payloads.
Handlers receive immutable references; messages cannot be modified
after insertion.

Core components:
- BufferSlot: frozen dataclass holding payload + metadata
- ThreadContext: append-only buffer per thread
- ContextBuffer: global manager with GC and limits

Design parallels OS virtual memory:
- Thread ID = virtual address space
- Buffer slot = memory page
- Immutable reference = read-only mapping
- Thread isolation = process isolation

Integration:
- Incoming messages appended after pipeline validation
- Outgoing responses appended before serialization
- Full audit trail preserved

This is incremental - handlers still receive copies for backward
compatibility. Next step: skip serialization for internal routing.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-10 17:20:43 -08:00
dullfig
a5e2ab22da Add thread registry, LLM router, console handler, and docs updates
Thread Registry:
- Root thread initialization at boot
- Thread chain tracking for message flow
- register_thread() for external message UUIDs

LLM Router:
- Multi-backend support with failover strategy
- Token bucket rate limiting per backend
- Async completion API with retries

Console Handler:
- Message-driven REPL (not separate async loop)
- ConsolePrompt/ConsoleInput payloads
- Handler returns None to disconnect

Boot System:
- System primitives module
- Boot message injected at startup
- Initializes root thread context

Documentation:
- Updated v2.1 docs for new architecture
- LLM router documentation
- Gap analysis cross-check

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-10 16:53:38 -08:00
dullfig
8aa58715df Add TodoUntil watcher system for async confirmation tracking
Implements an observer pattern where agents can register watchers
for conditions on their thread. When the condition is met, the
agent gets "nagged" on subsequent invocations until it explicitly
closes the todo.

Key components:
- TodoRegistry: thread-scoped watcher tracking with eyebrow state
- TodoUntil/TodoComplete payloads and system handlers
- HandlerMetadata.todo_nudge for delivering raised eyebrow notices
- Integration in StreamPump dispatch to check and nudge

Greeter now demonstrates the pattern:
1. Registers watcher for ShoutedResponse from shouter
2. On next invocation, sees nudge and closes completed todos
3. Includes nudge in LLM prompt for awareness

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-10 16:51:59 -08:00
dullfig
8fa92b8f56 Add multi-agent flow demo with shouter listener
Demonstrates message chaining: user -> greeter -> shouter -> user

- Add ShoutedResponse payload class (ALL CAPS response)
- Add handle_shout handler that forwards to original sender
- Update GreetingResponse to track original_sender
- Fix wrap_in_envelope to add xmlns="" to payloads
- Update organism.yaml with shouter listener
- Update tests for 2-listener configuration

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-10 10:47:09 -08:00
dullfig
82b5fcdd78 Replace MessageBus with aiostream-based StreamPump
Major refactor of the message pump architecture:

- Replace bus.py with stream_pump.py using aiostream for composable
  stream processing with natural fan-out via flatmap
- Add to_id field to MessageState for explicit routing
- Fix routing to use to_id.class format (e.g., "greeter.greeting")
- Generate XSD schemas from xmlified payload classes
- Fix xmlable imports (absolute -> relative) and parse_element ctx

New features:
- handlers/hello.py: Sample Greeting/GreetingResponse handler
- config/organism.yaml: Sample organism configuration
- 41 tests (31 unit + 10 integration) all passing

Schema changes:
- envelope.xsd: Allow any namespace payloads (##other -> ##any)

Dependencies added to pyproject.toml:
- aiostream>=0.5 (core dependency)
- pyhumps, termcolor (for xmlable)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-10 10:41:17 -08:00
dullfig
20c11cb2c2 initial dir structure 2025-12-25 21:52:36 -08:00