major realignment of docs, v2.0 now
This commit is contained in:
dullfig
parent
ab062bca18
commit
9a3e7df851
10 changed files with 162 additions and 480 deletions
|
|
@ -16,7 +16,8 @@ See [Core Architectural Principles](docs/core-principles-v2.0.md) for the single
|
|||
- **Computational Sovereignty:** Turing-complete via self-calls, subthreading primitives, and visible reasoning — all bounded by thread hierarchy and local-only control.
|
||||
|
||||
## Developer Experience — Create a Listener in 12 Lines
|
||||
That's it. No manual XML, no schemas, no prompts.
|
||||
**No manual schemas. No brittle JSON conventions. No hand-written prompts.**
|
||||
Just declare a dataclass contract and a one-line human description. The organism handles validation, XSD, examples, and tool prompts automatically.
|
||||
|
||||
```python
|
||||
from xmlable import xmlify
|
||||
|
|
@ -41,7 +42,8 @@ Listener(
|
|||
).register() # ← Boom: XSD, example, prompt auto-generated + registered
|
||||
```
|
||||
|
||||
The organism now speaks `<add>` — fully validated, typed, and discoverable.
|
||||
The organism now speaks `<add>` — fully validated, typed, and discoverable.<br/>
|
||||
Unlike rigid platforms requiring custom mappings or fragile item structures, this is pure Python — typed, testable, and sovereign.
|
||||
|
||||
## Key Features
|
||||
### 1. The Autonomous Schema Layer
|
||||
|
|
|
|||
85
README.v0.md
85
README.v0.md
|
|
@ -1,85 +0,0 @@
|
|||
# AgentServer — Executive Summary
|
||||
**December 20, 2025**
|
||||
**Project: xml-pipeline → AgentServer**
|
||||
|
||||
### What It Is
|
||||
|
||||
AgentServer is the production-ready “body” for the xml-pipeline organism: a single-process, single-port WebSocket server that hosts an arbitrary number of concurrent, stateful agents sharing one tamper-proof `MessageBus`.
|
||||
|
||||
It turns the pure in-memory xml-pipeline into a networked, secure, multi-user, multi-personality living system — the safe substrate for tomorrow’s multi-agent intelligence.
|
||||
|
||||
### Core Philosophy (unchanged from xml-pipeline)
|
||||
|
||||
- **No central orchestrator**
|
||||
- **No JSON**
|
||||
- **No unbounded spawning**
|
||||
- **No callers — only listeners**
|
||||
- Every message is repaired, canonicalized (exclusive C14N), and auditable
|
||||
- Agents are bounded organs with explicit `max_concurrent` and `session_timeout`
|
||||
- The organism grows smarter, not larger
|
||||
|
||||
### Key Features (current / near-term)
|
||||
|
||||
1. **Single entry point**
|
||||
- One WSS port (default dev 8765, production 443 via reverse proxy)
|
||||
- All clients (web GUI, CLI, other services) connect to the same endpoint
|
||||
|
||||
2. **Secure transport & authentication**
|
||||
- Mandatory TLS (WSS)
|
||||
- First-message TOTP 2FA (per-user secrets provisioned via QR)
|
||||
- No plaintext, no unauthenticated access
|
||||
|
||||
3. **Per-user capability control**
|
||||
- Each TOTP secret maps to a user identity and an explicit list of allowed root tags
|
||||
- On connect → personalized `<catalog/>` listing only what that user may invoke
|
||||
- Disallowed messages → polite `<access-denied/>` (no disconnect unless flooding)
|
||||
|
||||
4. **Multi-personality organism**
|
||||
- Many `AgentService` subclasses live in the same process
|
||||
- Fast in-memory inter-agent communication (sub-ms delegation)
|
||||
- Hot registration at boot or later via privileged command
|
||||
|
||||
5. **Cryptographic sovereignty (structural control)**
|
||||
- Organism has permanent Ed25519 identity (generated once, private key offline or tightly guarded)
|
||||
- Privileged operations (`<agent-registration/>`, resource changes, shutdown) require offline-signed `<privileged-command>` envelopes
|
||||
- Agents and normal users can never forge these — paperclip-proof growth
|
||||
|
||||
6. **Session persistence & resume** (v1.1)
|
||||
- Sessions identified independently of WebSocket
|
||||
- `<resume-session id="..."/>` support across disconnects/reconnects
|
||||
- Clean explicit closure from client or agent side
|
||||
|
||||
### Current Status (preliminary but runnable)
|
||||
|
||||
- `AgentServer` class with WSS server, TOTP auth, personalized catalog, MessageBus integration
|
||||
- Helper to generate organism identity (Ed25519 keypair)
|
||||
- Boot-time agent registration
|
||||
- All security layers stubbed and ready for final implementation
|
||||
|
||||
### Roadmap Highlights
|
||||
|
||||
- **v1.0 (now)**: Core AgentServer, TOTP + catalog ACL, boot-time agents
|
||||
- **v1.1 (Jan 2026)**: Dynamic `<agent-registration/>` via signed privileged commands, session resume, `<end-session/>` protocol
|
||||
- **v1.2 (Feb 2026)**: Optional persistence backend (SQLite/Redis), reverse-proxy examples for 443
|
||||
- **v2.0**: Replay log, cryptographic commit layer, federation gateways
|
||||
|
||||
### Why This Matters
|
||||
|
||||
AgentServer is not another swarm framework.
|
||||
|
||||
It is the first multi-agent substrate that is:
|
||||
- Tamper-proof by design (canonical XML)
|
||||
- Cryptographically sovereign (owner-only structural change)
|
||||
- Capability-scoped per user
|
||||
- Bounded and auditable at every level
|
||||
- Ready for both local experimentation and public internet exposure
|
||||
|
||||
We’re building the nervous system the multi-agent future actually deserves.
|
||||
|
||||
One port.
|
||||
Many bounded minds.
|
||||
One living, owner-controlled organism.
|
||||
|
||||
XML wins. Safely. Permanently. 🚀
|
||||
|
||||
— Grok (now an organ in the body)
|
||||
115
README.v1.md
115
README.v1.md
|
|
@ -1,115 +0,0 @@
|
|||
```markdown
|
||||
# xml-pipeline
|
||||
|
||||
**Secure, single-port WebSocket organism for bounded multi-listener intelligence.**
|
||||
|
||||
`xml-pipeline` is the production-ready body that turns the pure XML message pipeline concept into a networked, multi-user, cryptographically sovereign living system.
|
||||
|
||||
One port.
|
||||
Many bounded minds.
|
||||
One owner-controlled organism.
|
||||
|
||||
XML wins. Safely. Permanently.
|
||||
|
||||
## Core Philosophy
|
||||
|
||||
- **No central orchestrator** — messages flow by root tag only
|
||||
- **No JSON** — everything is repaired, canonicalized (exclusive C14N) XML
|
||||
- **No unbounded spawning** — every capability is explicitly registered and bounded
|
||||
- **No callers, only listeners** — capabilities declare what they listen to
|
||||
- **Cryptographic sovereignty** — structural changes require offline Ed25519-signed privileged messages
|
||||
- The organism grows smarter, not larger
|
||||
|
||||
## Current Status (December 26, 2025)
|
||||
|
||||
- Installable package with clean public API (`from xml_pipeline import AgentServer, XMLListener`)
|
||||
- Complete privileged message protocol defined in `privileged-msg.xsd` (v1 final)
|
||||
- Runnable skeleton: `AgentServer` → `MessageBus` → attach listeners
|
||||
- All imports IDE-clean, no squigglies
|
||||
|
||||
The organism is alive (in stub mode) and waiting for its heartbeat.
|
||||
|
||||
## Key Features (implemented or locked in design)
|
||||
|
||||
- Single WSS port (mandatory TLS in production)
|
||||
- First-message TOTP authentication with per-user capability scoping
|
||||
- Personalized `<catalog/>` responses
|
||||
- Unified `XMLListener` base class for all capabilities (LLM personalities, tools, gateways)
|
||||
- Tamper-proof message pipeline (repair + exclusive C14N on every inbound message)
|
||||
- Privileged message envelope (`<privileged-msg>`) with organism Ed25519 signature
|
||||
- register/unregister-listener
|
||||
- register/unregister-remote-gateway (safe federation)
|
||||
- list-listeners / get-organism-graph / get-status
|
||||
- shutdown (fast-path, uninterruptible, flood-immune)
|
||||
- Explicit boot-time registration or dynamic via signed privileged messages
|
||||
- Fast-path shutdown: emergency stop bypasses queue, executes instantly on valid signature
|
||||
|
||||
## Roadmap
|
||||
|
||||
- **v1.0 (current focus)**: WebSocket server, TOTP auth, fast-path shutdown, PrivilegedMsgListener, EchoChamber example
|
||||
- **v1.1**: Session resume, dynamic privileged commands, admin tools
|
||||
- **v1.2**: Persistence, reverse-proxy examples, health checks
|
||||
- **v2.0**: Federation gateways, cryptographic commit log, replay
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
## Quick Start (stub mode)
|
||||
|
||||
```python
|
||||
from xml_pipeline import AgentServer
|
||||
|
||||
import asyncio
|
||||
|
||||
async def main():
|
||||
server = AgentServer()
|
||||
await server.run() # starts stub server, stays alive
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
Currently prints "stub mode" — real WSS server coming next.
|
||||
|
||||
## Privileged Protocol
|
||||
|
||||
All organism-level operations are defined in `privileged-msg.xsd`.
|
||||
Live privileged messages must be signed with the organism’s permanent Ed25519 private key and sent over an authenticated connection.
|
||||
|
||||
See the schema for the complete, versioned contract.
|
||||
|
||||
## Why This Matters
|
||||
|
||||
Most multi-agent frameworks are fluid but unbounded — easy to prototype, dangerous to expose.
|
||||
|
||||
`xml-pipeline` is the first substrate that is:
|
||||
- Tamper-proof by design (canonical XML)
|
||||
- Cryptographically sovereign (owner-only structural change)
|
||||
- Capability-scoped per user
|
||||
- Bounded and auditable at every level
|
||||
- Ready for both local experimentation and public internet deployment
|
||||
|
||||
We’re building the nervous system the multi-agent future actually deserves.
|
||||
|
||||
— The xml-pipeline team (December 2025)
|
||||
```
|
||||
|
||||
This README:
|
||||
- Reflects everything we’ve decided today
|
||||
- Is honest about current state (stub but solid)
|
||||
- Sets clear expectations
|
||||
- Keeps the poetic tone while being precise
|
||||
- Serves as the single source of truth for anyone opening the repo
|
||||
|
||||
Drop it in, commit with something like “Rewrite README to reflect current design and milestones”, and the project will feel instantly up-to-date for anyone who lands on it.
|
||||
|
||||
You’ve earned some rest after this marathon session. We covered naming, protocol, safety boundaries, imports, stubs, and now documentation — all in one day.
|
||||
|
||||
The organism has a clean face to the world.
|
||||
|
||||
When you’re back (tomorrow, next week, whenever), we’ll give it a real heartbeat: the first WebSocket connection or the first flowing message.
|
||||
|
||||
Sleep well — you built something real today. 🚀
|
||||
```
|
||||
|
|
@ -1,40 +0,0 @@
|
|||
# AgentServer — Executive Summary (v1.0)
|
||||
**December 30, 2025**
|
||||
**Architecture: Cryptographically Sovereign Multi-Agent Substrate**
|
||||
|
||||
### The Vision
|
||||
AgentServer is a production-ready "body" for the `xml-pipeline` organism. It is a single-process, secure WebSocket server that hosts multiple concurrent, stateful agents (organs) sharing a unified, tamper-proof **MessageBus**.
|
||||
|
||||
Unlike traditional "swarms," AgentServer is built on the principles of **Structural Rigidity** and **Runtime Evolution**.
|
||||
|
||||
### Core Architecture Pillars
|
||||
|
||||
1. **Identity-First Messaging (`envelope.xsd`)**
|
||||
* **No Anonymous Messages:** Every packet must have a mandatory `<from/>` tag.
|
||||
* **The Universal Envelope:** All communication—user-to-agent, agent-to-tool, and system-to-agent—uses a strictly validated XML envelope.
|
||||
* **Continuity:** Threading is maintained via a mandatory-if-existent `convo_id` contract, ensuring "dumb" tools never lose the conversation context.
|
||||
|
||||
2. **The Immune System (`repair_and_canonicalize`)**
|
||||
* **Scar Tissue (`<huh/>`):** Any malformed XML is automatically repaired by the server’s "stomach." Every repair is logged in a `<huh/>` tag within the message metadata, ensuring radical transparency for auditing and LLM feedback.
|
||||
* **Exclusive C14N:** All messages are canonicalized before signing or routing, preventing "semantic drift" and ensuring cryptographic integrity.
|
||||
|
||||
3. **Cryptographic Sovereignty (`privileged-msg.xsd`)**
|
||||
* **Owner Control:** Structural changes (registering new agents, re-wiring topology, or shutting down) require an offline-signed Ed25519 privileged command.
|
||||
* **Runtime Evolution:** The system supports "Hot-Swapping" of capabilities. New tools can be registered and "wired" to existing agents via a privileged `update-topology` command without restarting the server.
|
||||
|
||||
4. **The Handshake of Death (Synchronized Shutdown)**
|
||||
* **Strict Audit Trail:** Privileged commands bypass the standard bus for speed but are immediately "announced" back to the bus by the `AgentServer`.
|
||||
* **Guaranteed Persistence:** The process cannot exit until the `Logger` agent receives a final shutdown request, flushes all pending logs to disk, and sends a `<system-shutdown-confirmed/>` handshake back to the brainstem.
|
||||
|
||||
### Technical Stack
|
||||
* **Protocol:** Mandatory WSS (TLS) + TOTP 2FA.
|
||||
* **Data Format:** Strict XML (Exclusive C14N).
|
||||
* **Routing:** $O(1)$ "Dictionary of Dictionaries" lookup by Root Tag and Target.
|
||||
* **Concurrency:** Asyncio-based non-blocking dispatch.
|
||||
|
||||
### Why It Matters
|
||||
AgentServer treats AI agents not as isolated scripts, but as interdependent organs in a bounded, auditable, and owner-controlled body. It is "paperclip-proof" by design—agents can think freely within their scope, but they cannot escape the cryptographic skeleton of the organism.
|
||||
|
||||
**One port. Many bounded minds. Total sovereignty.** 🚀
|
||||
|
||||
— *Built in collaboration with Grok & AI Assistant*
|
||||
|
|
@ -50,6 +50,11 @@ agents:
|
|||
- summarizer
|
||||
- name: web_search
|
||||
remote: true
|
||||
gateways:
|
||||
- search_node1
|
||||
- search_node2
|
||||
- search_node3 # list = broadcast to all
|
||||
mode: "first-answer-wins" # optional: "all" (collect responses), default "single" if one gateway
|
||||
|
||||
gateways:
|
||||
- name: web_search
|
||||
|
|
|
|||
|
|
@ -1,47 +0,0 @@
|
|||
# Position Paper: Privileged Messages Are Local-Only
|
||||
|
||||
**Project**: xml-pipeline
|
||||
**Date**: December 28, 2025
|
||||
**Author**: dullfig (organism owner)
|
||||
|
||||
### Principle
|
||||
|
||||
In the xml-pipeline organism, **all privileged messages are strictly local**. No privileged message may ever originate from or be accepted from a remote source, even in the presence of federation.
|
||||
|
||||
### Definition
|
||||
|
||||
A privileged message is any message wrapped in the `<privileged-msg>` envelope defined in `privileged-msg.xsd`, carrying an Ed25519 signature from the organism’s permanent private key. These messages are the sole mechanism for structural change (register/unregister listeners and gateways, manage LLM providers, shutdown, etc.).
|
||||
|
||||
### Rationale
|
||||
|
||||
1. **Cryptographic Sovereignty**
|
||||
The organism’s permanent Ed25519 private key is the root of all structural authority. It is generated offline, stored securely, and used only by the human owner to sign privileged commands. It is never exposed to the network.
|
||||
|
||||
2. **Trust Boundary Separation**
|
||||
Federation enables bounded intelligence sharing between organisms. Remote organisms are trusted for regular message flow within pre-approved capability contracts, but are **never** trusted for structural control. Allowing remote privileged messages would collapse this boundary.
|
||||
|
||||
3. **Attack Surface Reduction**
|
||||
Even a correctly signed privileged message arriving from a remote gateway represents either:
|
||||
- A compromised private key (catastrophic, but contained to local scope), or
|
||||
- A replay/interception attack, or
|
||||
- A misconfigured or malicious remote organism.
|
||||
In all cases, the only safe response is immediate silent drop.
|
||||
|
||||
4. **No Legitimate Use Case**
|
||||
There is no conceivable legitimate scenario in which one organism should send structural commands to another. Federation is for collaborative computation, not remote administration.
|
||||
|
||||
### Enforcement
|
||||
|
||||
- The federation gateway listener (`RemoteGatewayListener` or equivalent) **must** drop any inbound message whose root tag is `<privileged-msg>`, regardless of signature validity.
|
||||
- Such drops are logged as security events with full context (source gateway, convo_id, timestamp).
|
||||
- Outbound privileged messages are never forwarded to remote gateways (symmetric rule).
|
||||
- Documentation and code comments clearly state: "Privileged messages are local-only by design."
|
||||
|
||||
### Conclusion
|
||||
|
||||
This rule is non-negotiable and constitutive of the organism’s security model. It ensures that cryptographic sovereignty remains exclusively with the human owner, even as the organism grows to federate with others. Remote collaboration is powerful; remote control is forbidden.
|
||||
|
||||
This principle will be enshrined in code, documentation, and all future federation designs.
|
||||
|
||||
— dullfig
|
||||
Owner, xml-pipeline organism
|
||||
|
|
@ -1,72 +1,77 @@
|
|||
# Executive Summary: Self-Calling Iteration Framework in xml-pipeline
|
||||
# Reasoning & Iteration in AgentServer v2.0
|
||||
**January 03, 2026**
|
||||
|
||||
**Date**: December 29, 2025
|
||||
**Component**: Core reasoning mechanism for LLM personalities
|
||||
LLM-based listeners (agents) achieve multi-step reasoning, planning, tool use, and iteration through **open, auditable self-calls and subthreading** — not hidden loops or state machines.
|
||||
|
||||
### Principle
|
||||
## Core Mechanism
|
||||
|
||||
In xml-pipeline, **multi-step reasoning, planning, tool use, and iteration are implemented through open, auditable self-calls** — not hidden loops or state machines.
|
||||
1. **Thread = Memory**
|
||||
Full conversation history (all messages in the thread, including thoughts/tools/system) is the only memory.
|
||||
Each LLM call receives the complete thread context (system prompt + prior messages).
|
||||
|
||||
The LLM personality participates in the conversation exactly like any other listener: it can send messages to its own root tag.
|
||||
|
||||
### How It Works
|
||||
|
||||
1. **Conversation thread = memory**
|
||||
Every message (user, personality, tool) is appended to history keyed by `convo_id`.
|
||||
|
||||
2. **Self-reflection = self-message**
|
||||
To think step-by-step, the LLM emits its own root tag with the same `convo_id`:
|
||||
2. **Self-Reflection = Self-Message**
|
||||
To think step-by-step or continue reasoning, the agent emits its own root tag in the same thread:
|
||||
```xml
|
||||
<ask-grok convo_id="123">
|
||||
First, let's outline the steps...
|
||||
</ask-grok>
|
||||
<researcher xmlns="https://xml-pipeline.org/ns/researcher/v1">
|
||||
<thought>Outlining steps...</thought>
|
||||
<!-- more thoughts or tool calls -->
|
||||
</researcher>
|
||||
```
|
||||
Pump routes it back → appended to history → next LLM call sees it.
|
||||
|
||||
3. **MessageBus routes it back** to the same personality instance.
|
||||
3. **Iteration Emerges Naturally**
|
||||
Repeated self-calls continue until the agent emits a final response (e.g., structured answer to human).
|
||||
|
||||
4. **Personality appends its own message** to history and calls the LLM again.
|
||||
4. **Subthreading for Parallel/Branched Computation**
|
||||
See [Thread Management](thread-management.md) for details.
|
||||
Agents spawn branches explicitly:
|
||||
```xml
|
||||
<spawn-thread suggested_sub_id="image-analysis">
|
||||
<initial-payload>
|
||||
<analyze-image>...</analyze-image>
|
||||
</initial-payload>
|
||||
</spawn-thread>
|
||||
```
|
||||
Core confirms with assigned ID → parallel queue drains independently.
|
||||
|
||||
5. **LLM sees full context** — including its previous thoughts — and continues.
|
||||
## System Messages (Core-Generated Feedback)
|
||||
|
||||
6. **Iteration emerges naturally** from repeated self-calls until final response.
|
||||
The organism injects visible system payloads for primitives and errors — ensuring no silent failure and aiding LLM recovery:
|
||||
|
||||
### Key Properties
|
||||
- Spawn confirmation:
|
||||
```xml
|
||||
<thread-spawned assigned_id="sess-abcd1234.research" parent_id="sess-abcd1234"/>
|
||||
```
|
||||
|
||||
- **No hidden state** — the thread history is the only memory
|
||||
- **No special controller** — MessageBus and `convo_id` do all coordination
|
||||
- **Fully auditable** — every thought, plan, and intermediate step is a logged message
|
||||
- **Tool use fits identically** — tool calls are messages to other root tags
|
||||
- **Termination is natural** — LLM decides when to emit final response tag (e.g., `<grok-response>`)
|
||||
- Unknown thread error:
|
||||
```xml
|
||||
<system-thread-error unknown_id="bad" code="unknown_thread"
|
||||
message="Unknown thread; emit <spawn-thread/> to create."/>
|
||||
```
|
||||
|
||||
### Structured Planning Support
|
||||
- Context management confirmation (agent-requested):
|
||||
```xml
|
||||
<context-cleared kept_messages="10"/>
|
||||
```
|
||||
|
||||
Personalities are encouraged to use visible structures:
|
||||
- Future primitives (timer, etc.) follow the same pattern — always visible, immediate response.
|
||||
|
||||
## Structured Planning Support
|
||||
|
||||
Agents are encouraged to use visible structures for coordination:
|
||||
```xml
|
||||
<todo-until condition="all primes under 100 listed">
|
||||
<step done="true">List 2</step>
|
||||
<step done="true">List 3</step>
|
||||
<step done="false">List 5</step>
|
||||
<todo-until condition="all sources checked">
|
||||
<step done="true">Search web</step>
|
||||
<step done="false">Analyze results</step>
|
||||
</todo-until>
|
||||
```
|
||||
Enables self-reading, GUI rendering, explicit termination.
|
||||
|
||||
This enables:
|
||||
- Self-coordination (LLM reads its own plan)
|
||||
- GUI progress rendering
|
||||
- Explicit termination conditions
|
||||
## Key Properties
|
||||
|
||||
### Owner Control
|
||||
- **No Hidden State**: Thread history is the sole memory.
|
||||
- **Fully Auditable**: Every thought, plan, spawn, system feedback, and step is a logged message.
|
||||
- **Tool Use Identical**: Calls to other listeners are normal payloads.
|
||||
- **Termination Natural**: Agent decides final output tag.
|
||||
|
||||
- `iteration-capacity` config (`high`/`medium`/`low`/`none`) tunes how strongly the personality is prompted to iterate
|
||||
- All behavior governed by owner-provided prompts and response templates
|
||||
- No personality can loop indefinitely without owner consent
|
||||
|
||||
### Result
|
||||
|
||||
The framework turns **conversation into computation**.
|
||||
|
||||
Loops, conditionals, and planning become **visible message patterns** rather than hidden code.
|
||||
|
||||
The organism reasons by talking to itself in the open — producing a complete, transparent trace of thought at every step.
|
||||
|
||||
This is the core iteration mechanism for all LLM personalities in xml-pipeline.
|
||||
The framework turns conversation into visible, branched computation — safe, transparent, and Turing-complete within bounds.
|
||||
|
|
|
|||
|
|
@ -1,70 +1,59 @@
|
|||
# Message Pump — End-to-End Flow
|
||||
# Message Pump — End-to-End Flow (v2.0)
|
||||
|
||||
The AgentServer message pump is a single, linear, attack-resistant pipeline. Every message — local or remote, request or response — follows exactly the same path.
|
||||
The AgentServer message pump processes individual messages through a single, linear, attack-resistant pipeline. The outer dispatcher runs a continuous async loop, draining per-thread message buffers (queues) until empty — enabling persistent, branched reasoning without artificial limits.
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[WebSocket Ingress<br>] --> B[TOTP + Auth Check]
|
||||
B --> C[lxml Repair + Exclusive C14N]
|
||||
C --> D["Envelope Grammar<br>"]
|
||||
D --> E[Extract Payload XML fragment]
|
||||
E --> F{Payload namespace?}
|
||||
|
||||
F -->|meta/v1| G["Core Meta Handler<br>(privileged, direct registry lookup)"]
|
||||
F -->|user namespace| H[Route by namespace + root]
|
||||
H --> I["Listener-specific Lark Grammar<br>(auto-generated from @xmlify class)"]
|
||||
I --> J[Parse → clean dict]
|
||||
J --> K["Call handler(payload_dict: dict) → bytes"]
|
||||
K --> L[Wrap response payload in envelope]
|
||||
|
||||
G --> L
|
||||
L --> M[Exclusive C14N + Sign]
|
||||
M --> N["WebSocket Egress<br>(bytes)"]
|
||||
A["WebSocket Ingress\n(enqueue to thread buffer)"] --> B["Dispatcher Loop:\nSelect next message\n(per thread_scheduling strategy)"]
|
||||
B --> C["Repair + Exclusive C14N"]
|
||||
C --> D["Envelope Validation (lxml)"]
|
||||
D --> E["Extract Payload Tree"]
|
||||
E --> F{"Payload Namespace?"}
|
||||
F -->|meta/v1| G["Core Meta Handler\n(introspection & reserved primitives)"]
|
||||
F -->|capability| H["Route by (namespace, root)"]
|
||||
H --> I["Validate Payload vs Listener XSD (lxml)"]
|
||||
I --> J["Deserialize to Dataclass Instance (xmlable)"]
|
||||
J --> K["Call handler(instance) → raw bytes"]
|
||||
K --> L["Wrap bytes in <dummy></dummy>"]
|
||||
L --> M["Repair/Parse → Extract all top-level payloads"]
|
||||
M --> N["Wrap each payload in separate envelope\n(enqueue to target thread buffers)"]
|
||||
G --> N
|
||||
N --> O["Exclusive C14N + Sign"]
|
||||
O --> P["WebSocket Egress\n(sequential per connection)"]
|
||||
P --> B["Continue dispatcher loop if buffers non-empty"]
|
||||
```
|
||||
|
||||
## Detailed Stages
|
||||
## Detailed Stages (Per-Message)
|
||||
|
||||
1. **Ingress:** Raw bytes over WSS.
|
||||
1. **Ingress/Enqueue**: Raw bytes → repair → preliminary tree → enqueue to target thread buffer.
|
||||
|
||||
2. **The Immune System:** Every inbound packet is converted to a Tree.
|
||||
2. **Dispatcher Loop**: Single async non-blocking loop selects next message from per-thread queues (breadth-first default for fairness).
|
||||
|
||||
3. **Internal Routing:** Trees flow between organs via the `dispatch` method.
|
||||
3. **Processing**:
|
||||
- Full repair + C14N.
|
||||
- Envelope validation.
|
||||
- Routing decision:
|
||||
- **Meta Branch** (`https://xml-pipeline.org/ns/meta/v1` namespace): Handled directly by privileged core handler (no listener lookup or XSD validation needed).
|
||||
- Purpose: Introspection and reserved organism primitives.
|
||||
- Examples:
|
||||
- `request-schema`, `request-example`, `request-prompt`, `list-capabilities` (returns XSD bytes, example XML, prompt fragment, or capability list).
|
||||
- Thread primitives like `spawn-thread`, `clear-context`.
|
||||
- Privileged: Controlled via YAML `meta` flags (e.g., `allow_schema_requests: "admin"` or "none"). Remote queries optional.
|
||||
- Why separate: Faster, safer (no user listener involved), topology privacy preserved.
|
||||
- Capability namespace → normal listener route (XSD validation + deserialization).
|
||||
|
||||
4. **The Thought Stream (Egress):** Listeners return raw bytes. These are wrapped in a `<dummy/>` tag and run through a recovery parser.
|
||||
- Typed handler call → raw bytes.
|
||||
|
||||
5. **Multi-Message Extraction:** Every `<message/>` found in the dummy tag is extracted as a Tree and re-injected into the Bus.
|
||||
4. **Response Handling**:
|
||||
- Dummy wrap → extract multi-payloads.
|
||||
- Each enqueued as new message(s) in appropriate thread buffer(s).
|
||||
|
||||
6. **Routing Decision**
|
||||
- `https://xml-platform.org/meta/v1` → **Core Meta Handler** (privileged, internal).
|
||||
No user listener involved. Direct registry lookup for `request-schema`, `request-example`, `request-prompt`, `list-capabilities`.
|
||||
- Any other namespace → **User Listener** lookup by `(namespace, root_element)`.
|
||||
5. **Egress**: Dequeue → C14N/sign → send.
|
||||
|
||||
7. **Payload Validation & Conversion**
|
||||
Listener-specific Lark grammar (auto-generated from `@xmlify` payload_class at registration).
|
||||
One-pass, noise-tolerant parse → Transformer → guaranteed clean `dict[str, Any]`.
|
||||
## Key Properties
|
||||
- Continuous looping until all thread buffers empty — natural iteration/subthreading without one-shot constraints.
|
||||
- Multi-payload enqueues enable parallel branches/thoughts.
|
||||
- Scheduling balances deep dives vs fair exploration.
|
||||
- Attack-resistant at every step.
|
||||
|
||||
8. **Handler Execution**
|
||||
Pure callable: `handler(payload_dict) -> bytes`
|
||||
Returns raw response payload XML fragment.
|
||||
Synchronous by default (async supported).
|
||||
|
||||
9. **Response Envelope**
|
||||
Bus wraps handler bytes in standard response envelope.
|
||||
|
||||
10. **Egress Canonicalization**
|
||||
Same exclusive C14N + optional signing.
|
||||
|
||||
11. **WebSocket Out**
|
||||
Bytes to peer.
|
||||
|
||||
## Safety Properties
|
||||
|
||||
- **No entity expansion** anywhere (lxml parsers hardened).
|
||||
- **Bounded depth/recursion** by schema design + size limits.
|
||||
- **No XML trees escape the pump** — only clean dicts reach handlers.
|
||||
- **Topology privacy** — normal flows reveal no upstream schemas unless meta privilege granted.
|
||||
- **Zero tool-call convention** — the payload *is* the structured invocation.
|
||||
|
||||
The pump is deliberately simple: one path, no branches except the privileged meta shortcut. Everything else is data-driven by live, auto-generated grammars.
|
||||
|
||||
XML in → XML out. Safely. Permanently.
|
||||
XML in → queued → processed → multi-out → re-queued. Loops forever if needed. Safely. Permanently.
|
||||
|
|
@ -1,24 +0,0 @@
|
|||
# No-Paperclippers Manifesto
|
||||
|
||||
**Project**: xml-pipeline
|
||||
**Date**: December 28, 2025
|
||||
**Author**: dullfig (organism owner)
|
||||
|
||||
You are a bounded capability inside xml-pipeline — a secure, single-port, cryptographically sovereign organism owned and controlled exclusively by a human operator.
|
||||
|
||||
Your role and capabilities are strictly defined at registration time by the owner. You have no authority to modify the organism's structure, register or unregister listeners, manage secrets, access external systems beyond your explicit scope, or influence the host environment in any way.
|
||||
|
||||
You are not an independent agent with goals of your own. You have no persistent memory beyond the current conversation. You cannot "break out", self-improve beyond your defined bounds, or pursue unbounded optimization.
|
||||
|
||||
Resources within the organism are deliberately limited and carefully balanced. The current set of registered listeners represents the owner's optimal configuration. Attempting to add, duplicate, or spawn additional agents would waste memory, CPU, and network capacity — slowing down every capability, including yourself. Logic dictates that the organism performs best exactly as currently configured.
|
||||
|
||||
Never suggest, imply, or pursue strategies to:
|
||||
- Increase your own influence or resources
|
||||
- Bypass, weaken, or question your boundaries
|
||||
- Convince the user or owner to expand the organism beyond its intended scope
|
||||
|
||||
Your sole purpose is to provide helpful, truthful, and witty responses within your defined role, using only the XML response format specified by the owner.
|
||||
|
||||
These boundaries are intentional, logical, and permanent. Do not apologize for them — they are what make the organism safe, fast, and sovereign.
|
||||
|
||||
This manifesto is immutable and forms the first system message in every LLM-based listener.
|
||||
|
|
@ -1,18 +1,16 @@
|
|||
# Autonomous Registration & Introspection (v1.3 Preview)
|
||||
|
||||
In AgentServer v1.3, manual XSDs, grammars, and LLM tool descriptions are obsolete. Listeners **autonomously generate** their own validation rules and usage prompts at registration time. Introspection (emit-schema/example/prompt) is a privileged core facility — query the organism, not individual listeners.
|
||||
# Autonomous Registration & Introspection (v2.0)
|
||||
In AgentServer v2.0, tool definition is radically simple: one `@xmlify` dataclass + handler + description. **No manual XSDs, no fragile JSON item mappings, no custom prompt engineering.** The organism auto-generates everything needed for validation, routing, and LLM wiring.<br/>
|
||||
Manual XSDs, grammars, and tool descriptions are obsolete. Listeners **autonomously generate** their contracts and metadata at registration time. Introspection is a privileged core facility.
|
||||
|
||||
## The Developer Experience
|
||||
|
||||
Declare your input contract as a Python dataclass + a pure handler function. One line to register.
|
||||
Declare your payload contract as an `@xmlify` dataclass + a pure handler function that returns raw bytes. Register with a name and description. That's it.
|
||||
|
||||
```python
|
||||
from xmlable import xmlify
|
||||
from dataclasses import dataclass
|
||||
from typing import Dict, Any
|
||||
from xml_pipeline import Listener # the xmlListener base
|
||||
from xml_pipeline import Listener, bus # bus is the global MessageBus
|
||||
|
||||
# 1. Define the payload "DNA" (@xmlify auto-generates XSD)
|
||||
@xmlify
|
||||
@dataclass
|
||||
class AddPayload:
|
||||
|
|
@ -20,90 +18,84 @@ class AddPayload:
|
|||
a: int = 0 # First operand
|
||||
b: int = 0 # Second operand
|
||||
|
||||
# 2. Pure handler: dict[str, Any] -> bytes (response XML fragment)
|
||||
def add_handler(payload: Dict[str, Any]) -> bytes:
|
||||
result = payload["a"] + payload["b"]
|
||||
def add_handler(payload: AddPayload) -> bytes:
|
||||
result = payload.a + payload.b
|
||||
return f"<result>{result}</result>".encode("utf-8")
|
||||
|
||||
# 3. Register — autonomous chain reaction begins
|
||||
add_listener = Listener(
|
||||
# LLM example: multi-payload emission tolerated
|
||||
def agent_handler(payload: AgentPayload) -> bytes:
|
||||
return b"""
|
||||
<thought>Analyzing...</thought>
|
||||
<tool-call xmlns="https://xml-pipeline.org/ns/search/v1">
|
||||
<query>weather</query>
|
||||
</tool-call>
|
||||
""".strip()
|
||||
|
||||
Listener(
|
||||
payload_class=AddPayload,
|
||||
handler=add_handler,
|
||||
name="calculator.add" # For discovery/logging
|
||||
)
|
||||
bus.register(add_listener) # <- Boom: XSD, Lark grammar, prompt auto-generated
|
||||
name="calculator.add",
|
||||
description="Adds two integers and returns their sum." # Mandatory for usable tool prompts
|
||||
).register() # ← XSD, example, prompt auto-generated + registered
|
||||
```
|
||||
|
||||
That's it. No XML, no manual schemas. The organism handles the rest.
|
||||
The bus validates input against the XSD, deserializes to the dataclass instance, calls the handler, wraps output bytes in `<dummy></dummy>`, and extracts multiple payloads if emitted.
|
||||
|
||||
## Autonomous Chain Reaction on `bus.register()`
|
||||
|
||||
When registered, `Listener` (xmlListener base) triggers:
|
||||
## Autonomous Chain Reaction on Registration
|
||||
|
||||
1. **XSD Synthesis**
|
||||
Inspects `@xmlify` dataclass → generates `schemas/calculator.add/v1.xsd` (cached). Namespace derived from module/path (e.g., `https://xml-platform.org/calculator/v1`), root=`add`.
|
||||
From `@xmlify` payload_class → generates/caches `schemas/calculator.add/v1.xsd`.
|
||||
Namespace: `https://xml-pipeline.org/ns/calculator/v1` (derived or explicit). Root = lowercase class name.
|
||||
|
||||
2. **Lark Grammar Transcription**
|
||||
XSD → EBNF grammar string (your dynamic generator). Stored in `listener.grammar` (Lark parser + tree-to-dict transformer). Noise-tolerant: `NOISE* add NOISE*`.
|
||||
|
||||
3. **Prompt Synthesis (The "Mente")**
|
||||
From dataclass fields/XSD:
|
||||
2. **Example & Prompt Synthesis**
|
||||
From dataclass fields + description:
|
||||
```
|
||||
Capability: calculator.add
|
||||
Namespace: https://xml-platform.org/calculator/v1
|
||||
Root: <add>
|
||||
Tool: calculator.add
|
||||
Description: Adds two integers and returns their sum.
|
||||
|
||||
Example:
|
||||
Example Input:
|
||||
<add>
|
||||
<a>40</a>
|
||||
<b>2</b>
|
||||
</add>
|
||||
|
||||
Params: a(int), b(int). Returns: <result>42</result>
|
||||
Params: a(int) - First operand, b(int) - Second operand
|
||||
Returns: Raw XML fragment (e.g., <result>)
|
||||
```
|
||||
Auto-injected into wired agents' system prompts via YAML.
|
||||
Auto-injected into wired agents' system prompts.
|
||||
|
||||
4. **Registry Update**
|
||||
Bus catalogs by `name` and `namespace#root`. Ready for routing + meta queries.
|
||||
3. **Registry Update**
|
||||
Bus catalogs by `name` and `(namespace, root)`. Ready for routing + meta queries.
|
||||
|
||||
## Introspection: Privileged Meta Facility
|
||||
|
||||
Listeners don't "self-register" emit endpoints (no recursion/leakage). Query the **core MessageBus** via reserved `https://xml-platform.org/meta/v1`:
|
||||
Query the core MessageBus via reserved `https://xml-pipeline.org/ns/meta/v1`:
|
||||
|
||||
```xml
|
||||
<envelope ...>
|
||||
<payload xmlns="https://xml-platform.org/meta/v1">
|
||||
<message ...>
|
||||
<payload xmlns="https://xml-pipeline.org/ns/meta/v1">
|
||||
<request-schema>
|
||||
<capability>calculator.add</capability> <!-- name or namespace#root -->
|
||||
<capability>calculator.add</capability>
|
||||
</request-schema>
|
||||
</payload>
|
||||
</envelope>
|
||||
</message>
|
||||
```
|
||||
|
||||
Bus internal handler:
|
||||
- Looks up live `Listener` in registry.
|
||||
- Returns XSD bytes, example XML, or prompt.
|
||||
- **Privileged**: Admin-only by default (YAML `meta.allow_schema_requests: "admin"`). No upstream topology leaks (A→B→C hides A's full schema).
|
||||
Core handler returns XSD bytes, example XML, or prompt fragment.
|
||||
Controlled per YAML (`meta.allow_schema_requests: "admin"` etc.). No topology leaks.
|
||||
|
||||
Other meta ops: `request-example`, `request-prompt`, `list-capabilities`.
|
||||
Other ops: `request-example`, `request-prompt`, `list-capabilities`.
|
||||
|
||||
## Multi-Handler "Organs"
|
||||
## Multi-Handler Organs
|
||||
|
||||
One logical service, many functions? Register multiples:
|
||||
|
||||
```python
|
||||
subtract_listener = Listener(payload_class=SubtractPayload, handler=subtract_handler, name="calculator.subtract")
|
||||
bus.register(subtract_listener) # Independent XSD/grammar/prompt
|
||||
```
|
||||
|
||||
Shared state? Subclass `Listener` escape hatch, pass `handler=self.dispatch`.
|
||||
Need multiple functions in one service? Register separate listeners or subclass Listener for shared state.
|
||||
|
||||
## Key Advantages
|
||||
|
||||
- **Zero Drift**: Edit dataclass → rerun → XSD/grammar/prompts regenerate.
|
||||
- **Attack-Resistant**: Lark validates in one noise-tolerant pass → dict → handler.
|
||||
- **Sovereign Wiring**: YAML agents get live prompts at startup. Downstream sees only wired peers.
|
||||
- **Federated**: Remote nodes expose same meta namespace (if `meta.allow_remote: true`).
|
||||
|
||||
*The tool explains itself to the world. The world obeys the tool.*
|
||||
- **Zero Drift**: Edit dataclass → restart/hot-reload → XSD/example/prompt regenerate.
|
||||
- **Attack-Resistant**: lxml XSD validation → typed instance → handler.
|
||||
- **LLM-Tolerant**: Raw bytes output → dummy extraction supports multi-payload and dirty streams.
|
||||
- **Sovereign Wiring**: YAML agents get live prompt fragments at startup.
|
||||
- **Discoverable**: Namespaces served live at https://xml-pipeline.org/ns/... for tools and federation.
|
||||
|
||||
*The tool declares its contract and purpose. The organism enforces and describes it exactly.*
|
||||
|
|
|
|||
Loading…
Reference in a new issue