diff --git a/README_v2.0.md b/README_v2.0.md deleted file mode 100644 index b7d3f97..0000000 --- a/README_v2.0.md +++ /dev/null @@ -1,94 +0,0 @@ -# AgentServer — The Living Substrate (v2.0) -***"It just works... safely."*** - -**January 03, 2026** -**Architecture: Autonomous Schema-Driven, Turing-Complete Multi-Agent Organism** - -## The Rant -**Why XML?** -[Why not JSON?](docs/why-not-json.md) - -XML is the sovereign wire format — standards-based, self-describing, attack-resistant, and evolvable without drift. JSON was a quick hack that escaped into the wild and became the default for everything, including AI tool calling, where its brittleness causes endless prompt surgery and validation headaches. - -This project chooses XML deliberately. The organism enforces contracts exactly (XSD validation, no transcription bugs), tolerates dirty streams (repair + dummy extraction), and keeps reasoning visible. No fragile conventions. No escaping hell. Just bounded, auditable computation. - -Read the full rant [here](docs/why-not-json.md) for the history, pitfalls, and why XML wins permanently. - -## What It Is -AgentServer is a production-ready substrate for the `xml-pipeline` nervous system. Version 2.0 stabilizes the design around exact XSD validation, typed dataclass handlers, mandatory hierarchical threading, and strict out-of-band privileged control. - -See [Core Architectural Principles](docs/core-principles-v2.0.md) for the single canonical source of truth. - -## Core Philosophy -- **Autonomous DNA:** Listeners declare their contract via `@xmlify` dataclasses; the organism auto-generates XSDs, examples, and tool prompts. -- **Schema-Locked Intelligence:** Payloads validated directly against XSD (lxml) → deserialized to typed instances → pure handlers. -- **Multi-Response Tolerance:** Handlers return raw bytes; bus wraps in `` and extracts multiple payloads (perfect for parallel tool calls or dirty LLM output). -- **Computational Sovereignty:** Turing-complete via self-calls, subthreading primitives, and visible reasoning — all bounded by thread hierarchy and local-only control. - -## Developer Experience — Create a Listener in 12 Lines -**No manual schemas. No brittle JSON conventions. No hand-written prompts.** -Just declare a dataclass contract and a one-line human description. The organism handles validation, XSD, examples, and tool prompts automatically. - -```python -from xmlable import xmlify -from dataclasses import dataclass -from xml_pipeline import Listener, bus # bus is the global MessageBus - -@xmlify -@dataclass -class AddPayload: - a: int - b: int - -def add_handler(payload: AddPayload) -> bytes: - result = payload.a + payload.b - return f"{result}".encode("utf-8") - -Listener( - payload_class=AddPayload, - handler=add_handler, - name="calculator.add", - description="Adds two integers and returns their sum." -).register() # ← Boom: XSD, example, prompt auto-generated + registered -``` - -The organism now speaks `` — fully validated, typed, and discoverable.
-Unlike rigid platforms requiring custom mappings or fragile item structures, this is pure Python — typed, testable, and sovereign. - -## Key Features -### 1. The Autonomous Schema Layer -- Dataclass → cached XSD + example + rich tool prompt (mandatory description + field docs). -- Namespaces: `https://xml-pipeline.org/ns///v1` (served live via domain for discoverability). - -### 2. Thread-Based Lifecycle & Reasoning -- Mandatory `` with hierarchical IDs for reliable subthreading and audit trails. -- LLM agents reason via open self-calls and optional ``. -- All thought steps visible as messages — no hidden state. - -### 3. Message Pump -- Single linear pipeline with repair, C14N, XSD validation, deserialization, handler execution, and multi-payload extraction. -- Supports clean tools and forgiving LLM streams alike. -- Thread-base message queue with bounded memory. - -### 4. Structural Control -- Bootstrap from `organism.yaml`. -- Runtime changes (hot-reload, add/remove listeners) via local-only OOB channel (localhost WSS or Unix socket — GUI-ready). -- Main bus oblivious to privileged ops. - -### 5. Federation & Introspection -- YAML-declared gateways with trusted keys. -- Controlled meta queries (schema/example/prompt/capability list). - -## Technical Stack -- **Validation & Parsing:** lxml (XSD, C14N, repair) + xmlable (round-trip). -- **Protocol:** Mandatory WSS (TLS) + TOTP on main port. -- **Identity:** Ed25519 (signing, federation, privileged). -- **Format:** Exclusive C14N XML (wire sovereign). - -## Why This Matters -AgentServer v2.0 is a bounded, auditable, owner-controlled organism where the **XSD is the security**, the **thread is the memory**, and the **OOB channel is the sovereignty**. - -One port. Many bounded minds. Autonomous yet obedient evolution. 🚀 - ---- -*XML wins. Safely. Permanently.* \ No newline at end of file diff --git a/docs/core-principles-v2.0.md b/docs/core-principles-v2.0.md deleted file mode 100644 index d940b2a..0000000 --- a/docs/core-principles-v2.0.md +++ /dev/null @@ -1,103 +0,0 @@ -# AgentServer v2.0 — Core Architectural Principles -**January 03, 2026** -**Architecture: Autonomous Schema-Driven, Turing-Complete Multi-Agent Organism** - -These principles are the single canonical source of truth for the project. All documentation, code, and future decisions must align with this file. - -## Identity & Communication -- All traffic uses the universal `` envelope defined in `envelope.xsd` (namespace `https://xml-pipeline.org/ns/envelope/v1`). -- Mandatory `` and `` (convo_id string, supports hierarchical dot notation for subthreading, e.g., "root.1.research"). -- Optional `` (rare direct routing; most flows use payload namespace/root). -- Exclusive C14N on ingress and egress. -- Malformed XML repaired on ingress; repairs logged in `` metadata. - -## Identity Injection & Handler Purity -- Handlers are pure, stateless functions with no knowledge of routing, thread context, or their own registered name. -- On ingress (external or gateway messages): is provided and authenticated by the client/gateway (enforced by envelope validation). -- On response generation (after handler execution and multi-payload extraction): - - The dispatcher injects using the executing listener's registered name (e.g., "calculator.add" or "researcher"). - - For meta/primitive responses: is injected as "core". -- <thread> is inherited from the incoming message (or assigned/updated for primitives like spawn-thread). -- <to> remains optional and rarely used. -- This ensures every enveloped message has a trustworthy, auditable without handler involvement, preventing spoofing and keeping capability code minimal/testable. - -## Configuration & Composition -- YAML file (`organism.yaml`) is the bootstrap source of truth, loaded at startup. -- Defines initial listeners, agents, gateways, meta privileges, and OOB channel configuration. -- Runtime structural changes (add/remove listeners, rewire agents, etc.) via local-only privileged commands on the dedicated OOB channel (hot-reload capability). -- No remote or unprivileged structural changes ever. - -## Autonomous Schema Layer -- Listeners defined by `@xmlify`-decorated dataclass (payload contract) + pure handler function. -- Mandatory human-readable description string (short "what this does" blurb for tool prompt lead-in). -- Registration (at startup or via hot-reload) automatically generates: - - XSD cached on disk (`schemas//v1.xsd`) - - Example XML - - Tool description prompt fragment (includes description, params with field docs if present, example input) -- All capability namespaces under `https://xml-pipeline.org/ns///v1`. -- Root element derived from payload class name (lowercase) or explicit. - -## Message Pump -- Single linear pipeline on main port: ingress → repair → C14N → envelope validation → payload routing. -- Routing key = (payload namespace, root element); unique per listener. -- Meta requests (`https://xml-pipeline.org/ns/meta/v1`) handled by privileged core handler. -- User payloads: - - Validated directly against listener's cached XSD (lxml) - - On success → deserialized to typed dataclass instance (`xmlable.from_xml`) - - Handler called with instance → returns raw bytes (XML fragment, possibly dirty/multi-root) - - Bytes wrapped in `` → repaired/parsed → all top-level payload elements extracted - - Each extracted payload wrapped in separate response envelope (inherits thread/from, optional new subthread if primitive used) - - Enveloped responses buffered and sent sequentially -- Supports single clean response, multi-payload emission (parallel tools/thoughts), and dirty LLM output tolerance. -- Message pump tracks token budgets per agent and thread, enforcing token limits and preventing abuse. The LLM abstraction layer informs the message bus on the actual token usage. -- Message pump uses asynchronous non-blocking I/O for maximum throughput. - -## Reasoning & Iteration -- LLM agents iterate via open self-calls (same root tag, same thread ID). -- Conversation thread = complete memory and audit trail (all messages logged). -- Subthreading natively supported via hierarchical thread IDs and primitives (e.g., reserved payload to spawn "parent.sub1"). -- Optional structured constructs like `` for visible planning. -- No hidden loops or state machines; all reasoning steps are visible messages. -- Thread management follows the dynamic call tracing model (see thread-management.md). Paths are built by appending target listener names on emission, with automatic popping on responses. Agents remain oblivious, enabling natural delegation and parallelism. - -## Security & Sovereignty -- Privileged messages (per `privileged-msg.xsd`) handled exclusively on dedicated OOB channel. -- OOB channel bound to localhost by default (safe for local GUI); separate port/socket from main bus. -- Main MessageBus and pump oblivious to privileged operations — no routing or handling for privileged roots. -- Remote privileged attempts impossible (channel not exposed); any leak to main port logged as security event and dropped. -- Ed25519 identity key used for envelope signing, federation auth, and privileged command verification. -- No agent may modify organism structure, register listeners, or access host resources beyond declared scope. -- “No Paperclippers” manifesto injected as first system message for every LLM-based listener. - -## Federation -- Gateways declared in YAML with trusted remote public key. -- Remote tools referenced by gateway name in agent tool lists. -- Regular messages flow bidirectionally; privileged messages never forwarded or accepted. - -## Introspection (Meta) -- Controlled via YAML flags (`allow_list_capabilities`, `allow_schema_requests`, etc.). -- Supports `request-schema`, `request-example`, `request-prompt`, `list-capabilities`. -- Remote meta queries optionally allowed per YAML (federation peers). - -## Technical Constraints -- Mandatory WSS (TLS) + TOTP on main port. -- OOB channel WSS or Unix socket, localhost-default. -- Internal: lxml trees → XSD validation → xmlable deserialization → dataclass → handler → bytes → dummy extraction. -- Single process, async non-blocking. -- XML is the sovereign wire format; everything else is implementation detail. - -## Scheduled Computation -- Timers and delays implemented as normal listeners using async sleeps. -- Caller idles naturally; wakeup messages bubble back via standard tracing. -- Enables recurrent tasks (e.g., periodic monitoring) without blocking or external schedulers. - -## Bounded Stateful Listeners -- Pure tools remain stateless. -- Stateful capabilities (e.g., calculator memory, game state) store data per thread path UUID. -- Ensures isolation across conversations, automatic cleanup on idle, and minimal mutable state. -- Handler closes over or receives UUID for access — still oblivious to readable path. - -## Resource Stewardship -- The Message Pump ensures fair execution and prevents "Paperclip" runaway scenarios via internal Thread-Level Scheduling. Every thread is subject to Token-Rate Monitoring and Fair-Share Queuing, ensuring that a high-volume agent (like a deep-thinking LLM) cannot block high-priority system events or starve simpler organs (like tools). - -These principles are now locked. All existing docs will be updated to match this file exactly. Future changes require explicit discussion and amendment here first. \ No newline at end of file diff --git a/docs/message-pump-v1.0.md b/docs/message-pump-v1.0.md deleted file mode 100644 index 37357e5..0000000 --- a/docs/message-pump-v1.0.md +++ /dev/null @@ -1,60 +0,0 @@ -# Message Pump — End-to-End Flow (v2.0) - -The AgentServer message pump processes individual messages through a single, linear, attack-resistant pipeline. The outer dispatcher runs a continuous async loop, draining per-thread message buffers (queues) until empty — enabling persistent, branched reasoning without artificial limits. - - -```mermaid -flowchart TD - A["WebSocket Ingress\n(enqueue to thread buffer)"] --> B["Dispatcher Loop:\nSelect next message\n(per thread_scheduling strategy)"] - B --> C["Repair + Exclusive C14N"] - C --> D["Envelope Validation (lxml)"] - D --> E["Extract Payload Tree"] - E --> F{"Payload Namespace?"} - F -->|meta/v1| G["Core Meta Handler\n(introspection & reserved primitives)"] - F -->|capability| H["Route by (namespace, root)"] - H --> I["Validate Payload vs Listener XSD (lxml)"] - I --> J["Deserialize to Dataclass Instance (xmlable)"] - J --> K["Call handler(instance) → raw bytes"] - K --> L["Wrap bytes in "] - L --> M["Repair/Parse → Extract all top-level payloads"] - M --> N["Wrap each payload in separate envelope\n(enqueue to target thread buffers)"] - G --> N - N --> O["Exclusive C14N + Sign"] - O --> P["WebSocket Egress\n(sequential per connection)"] - P --> B["Continue dispatcher loop if buffers non-empty"] -``` - -## Detailed Stages (Per-Message) - -1. **Ingress/Enqueue**: Raw bytes → repair → preliminary tree → enqueue to target thread buffer. - -2. **Dispatcher Loop**: Single async non-blocking loop selects next message from per-thread queues (breadth-first default for fairness). - -3. **Processing**: - - Full repair + C14N. - - Envelope validation. - - Routing decision: - - **Meta Branch** (`https://xml-pipeline.org/ns/meta/v1` namespace): Handled directly by privileged core handler (no listener lookup or XSD validation needed). - - Purpose: Introspection and reserved organism primitives. - - Examples: - - `request-schema`, `request-example`, `request-prompt`, `list-capabilities` (returns XSD bytes, example XML, prompt fragment, or capability list). - - Thread primitives like `spawn-thread`, `clear-context`. - - Privileged: Controlled via YAML `meta` flags (e.g., `allow_schema_requests: "admin"` or "none"). Remote queries optional. - - Why separate: Faster, safer (no user listener involved), topology privacy preserved. - - Capability namespace → normal listener route (XSD validation + deserialization). - - - Typed handler call → raw bytes. - -4. **Response Handling**: - - Dummy wrap → extract multi-payloads. - - Each enqueued as new message(s) in appropriate thread buffer(s). - -5. **Egress**: Dequeue → C14N/sign → send. - -## Key Properties -- Continuous looping until all thread buffers empty — natural iteration/subthreading without one-shot constraints. -- Multi-payload enqueues enable parallel branches/thoughts. -- Scheduling balances deep dives vs fair exploration. -- Attack-resistant at every step. - -XML in → queued → processed → multi-out → re-queued. Loops forever if needed. Safely. Permanently. \ No newline at end of file diff --git a/docs/message-pump-v2.0.md b/docs/message-pump-v2.0.md deleted file mode 100644 index 5b36fb3..0000000 --- a/docs/message-pump-v2.0.md +++ /dev/null @@ -1,139 +0,0 @@ -# Message Pump — End-to-End Flow (v2.0) - -The AgentServer message pump processes individual messages through a single, linear, attack-resistant pipeline. The outer dispatcher runs a continuous async loop, draining per-thread message buffers (queues) until empty — enabling persistent, branched reasoning without artificial limits. - -```mermaid -flowchart TD - subgraph Ingress - A[Raw Bytes] --> B[Repair + C14N] - B --> C[Enqueue to Thread Queue] - end - - subgraph DispatcherLoop - D[Dequeue Next Message] --> E{Envelope Valid?} - E -->|No| F[Discard / System Error] - E -->|Yes| G{Payload Namespace?} - G -->|Meta| H["Core Handler
(raw payload)"] - G -->|Normal| I[Validate Payload
vs Cached XSD] - I -->|Fail| F - I -->|Pass| J["Deserialize to
@xmlify Dataclass"] - J --> K["Call Handler
(typed instance → bytes)"] - H --> L["Wrap bytes in
<dummy></dummy>"] - K --> L - end - - subgraph Response - L --> M[Repair + Extract
Multi-Payloads] - M --> N{Extracted Payloads?} - N -->|0| O["Optional: Inject
<agent-error> or Idle"] - N -->|1 or more| P[For Each Payload:] - P --> Q[Determine Target Listener] - Q --> R["Append Target Name
to Current Path
(new thread ID)"] - R --> S["Inject
(listener name)"] - S --> T["Enqueue New Message(s)
to Deepened Path(s)
(parallel if multi)"] - T --> U[On Response Bubbling:
Pop Last Segment
Route to Parent Path] - end - - C --> DispatcherLoop - DispatcherLoop --> Response - Response --> DispatcherLoop - style Ingress fill:#f0f8ff - style DispatcherLoop fill:#f0fff0 - style Response fill:#fff0f0 -``` - -```mermaid -flowchart TD - subgraph MessagePump - subgraph Init - start([Start]) - wrapstart["Boot Msg
<message>{...}</message>"] - end - enq1([QUEUE 1]) - rawwaiting{Raw
Msg
Waiting?} - waitRaw([Wait]) - subgraph Process - extract["Extract
Tree"] - split["Split
Tree"] - subgraph foreach [For Each Message] - getmsg[Get Msg] - badTo{<To> Missing?} - endnoto([Discard]) - addfrom["Add .from"] - repair[Repair + C14N] - validate[Validate] - invalidMsg{Bad
Message?} - badmsg([Discard]) - more{More?} - end - enqueue([QUEUE 2]) - xmlWaiting{XML
Waiting?} - waitXml([Wait]) - subgraph Async - lookup[Lookup Listener] - route[Route] - wait[await Response] - wrap["Wrap
<from>{...}</from>"] - end - end - end - - start --> wrapstart --> enq1 --> rawwaiting - rawwaiting --> |NO| waitRaw - rawwaiting ---> |YES| extract - extract --> split --> foreach - getmsg --> badTo - badTo --> |YES| endnoto - badTo --> |NO| addfrom --> repair --> validate --> invalidMsg - invalidMsg ---> |NO| more --> |Yes| getmsg - invalidMsg --> |YES| badmsg - more --> |NO| enqueue - enqueue --> xmlWaiting - xmlWaiting --> |NO| waitXml - xmlWaiting ---> |YES| lookup --> route --> wait --> wrap --> enq1 - -``` -## Detailed Stages (Per-Message) - -### Boot Message - - Since all agents are listeners, there would be no way for a client to initiate a message pump. - - The boot message is a dummy message that is enqueued to the root thread buffer. Any listener may chose to register a root tag for it. - - The root thread buffer is the only one that is drained by the dispatcher loop. - - if a listener (like a human agent) is registered for the boot message, it will receive the boot message and then async await for keyboard input. -### Queue 1 - - The first buffer holds raw unprocessed messages from the network. -### Queue 2 - - The second buffer holds messages that have been processed and are ready to be sent back to the network. - - -1. **Ingress/Enqueue**: Raw bytes → repair → preliminary tree → enqueue to target thread buffer. - -2. **Dispatcher Loop**: Single async non-blocking loop selects next message from per-thread queues (breadth-first default for fairness). - -3. **Processing**: - - Full repair + C14N. - - Envelope validation. - - Routing decision: - - **Meta Branch** (`https://xml-pipeline.org/ns/meta/v1` namespace): Handled directly by privileged core handler (no listener lookup or XSD validation needed). - - Purpose: Introspection and reserved organism primitives. - - Examples: - - `request-schema`, `request-example`, `request-prompt`, `list-capabilities` (returns XSD bytes, example XML, prompt fragment, or capability list). - - Thread primitives like `spawn-thread`, `clear-context`. - - Privileged: Controlled via YAML `meta` flags (e.g., `allow_schema_requests: "admin"` or "none"). Remote queries optional. - - Why separate: Faster, safer (no user listener involved), topology privacy preserved. - - Capability namespace → normal listener route (XSD validation + deserialization). - - - Typed handler call → raw bytes. - -4. **Response Handling**: - - Dummy wrap → extract multi-payloads. - - Each enqueued as new message(s) in appropriate thread buffer(s). - -5. **Egress**: Dequeue → C14N/sign → send. - -## Key Properties -- Continuous looping until all thread buffers empty — natural iteration/subthreading without one-shot constraints. -- Multi-payload enqueues enable parallel branches/thoughts. -- Scheduling balances deep dives vs fair exploration. -- Attack-resistant at every step. - -XML in → queued → processed → multi-out → re-queued. Loops forever if needed. Safely. Permanently. \ No newline at end of file