Add Premium Librarian spec — RLM-powered codebase intelligence
Features: - Ingest entire codebases (millions of tokens) - 4 built-in chunkers: code, prose, structured, tabular - Custom WASM chunker escape hatch - eXist-db storage with versioning - RLM query processing for natural language questions - Structural indexing (call graph, type hierarchy, symbols) Use cases: - Legacy code understanding - API discovery - Impact analysis - Developer onboarding Premium tier pricing model included. Co-authored-by: Dan
This commit is contained in:
Donna
parent
1e2073a81c
commit
f9b873d331
1 changed files with 323 additions and 0 deletions
323
docs/premium-librarian-spec.md
Normal file
323
docs/premium-librarian-spec.md
Normal file
|
|
@ -0,0 +1,323 @@
|
|||
# Premium Librarian — RLM-Powered Codebase Intelligence
|
||||
|
||||
**Status:** Design Spec
|
||||
**Author:** Dan & Donna
|
||||
**Date:** 2026-01-26
|
||||
|
||||
## Overview
|
||||
|
||||
The Premium Librarian is an RLM-powered (Recursive Language Model) tool that can ingest entire codebases or document corpora — millions of tokens — and answer natural language queries about them.
|
||||
|
||||
Unlike traditional code search (grep, ripgrep, Sourcegraph), the Premium Librarian *understands* structure, relationships, and intent. It can answer questions like:
|
||||
|
||||
- "Where are edges calculated in OpenCASCADE?"
|
||||
- "How does the authentication flow work?"
|
||||
- "What would break if I changed this interface?"
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ PREMIUM LIBRARIAN │
|
||||
│ │
|
||||
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────────┐ │
|
||||
│ │ Ingest │───▶│ Chunker │───▶│ eXist-db │ │
|
||||
│ │ (upload) │ │ (4 types │ │ (versioned storage) │ │
|
||||
│ │ │ │ + WASM) │ │ │ │
|
||||
│ └─────────────┘ └─────────────┘ └───────────┬─────────────┘ │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────────┐ │
|
||||
│ │ Query │───▶│ RLM │───▶│ Index / Map │ │
|
||||
│ │ (natural │ │ Processor │ │ (structure, relations) │ │
|
||||
│ │ language) │ │ │ │ │ │
|
||||
│ └─────────────┘ └──────┬──────┘ └─────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────┐ │
|
||||
│ │ Response │ │
|
||||
│ │ (answer + │ │
|
||||
│ │ sources) │ │
|
||||
│ └─────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Chunkers
|
||||
|
||||
The RLM algorithm needs content-aware chunking. Different content types have different natural boundaries.
|
||||
|
||||
### Built-in Chunkers
|
||||
|
||||
| Chunker | Content Types | Chunking Strategy |
|
||||
|---------|---------------|-------------------|
|
||||
| **Code** | .py, .js, .ts, .cpp, .c, .java, .go, .rs, etc. | Functions, classes, modules. Preserves imports, docstrings, signatures. |
|
||||
| **Prose** | .md, .txt, .rst, .adoc | Paragraphs, sections, chapters. Preserves headings, structure. |
|
||||
| **Structured** | .yaml, .json, .xml, .toml, .ini | Schema-aware. Preserves hierarchy, keys, nesting. |
|
||||
| **Tabular** | .csv, .tsv, .parquet | Row groups with headers. Preserves column semantics. |
|
||||
|
||||
### Content Type Detection
|
||||
|
||||
1. File extension mapping (fast path)
|
||||
2. MIME type detection
|
||||
3. Content sniffing (magic bytes, heuristics)
|
||||
4. User override via config
|
||||
|
||||
### Custom WASM Chunkers
|
||||
|
||||
For specialized formats, users can provide their own chunker:
|
||||
|
||||
```typescript
|
||||
// chunker.ts (AssemblyScript)
|
||||
import { Chunk, ChunkMetadata } from "@openblox/chunker-sdk";
|
||||
|
||||
export function chunk(content: string, metadata: ChunkMetadata): Chunk[] {
|
||||
// Custom logic for proprietary format
|
||||
const chunks: Chunk[] = [];
|
||||
|
||||
// Parse your format, emit chunks
|
||||
// Each chunk has: content, startLine, endLine, type, name
|
||||
|
||||
return chunks;
|
||||
}
|
||||
```
|
||||
|
||||
**Compile & upload:**
|
||||
```bash
|
||||
asc chunker.ts -o chunker.wasm
|
||||
# Upload via API or CLI
|
||||
```
|
||||
|
||||
**Security:**
|
||||
- WASM runs sandboxed (can't escape, can't access filesystem)
|
||||
- CPU/memory limits enforced
|
||||
- Chunker is pure function: string in, chunks out
|
||||
|
||||
## Ingest Pipeline
|
||||
|
||||
### 1. Upload
|
||||
|
||||
```
|
||||
POST /api/v1/libraries
|
||||
{
|
||||
"name": "opencascade",
|
||||
"source": {
|
||||
"type": "git",
|
||||
"url": "https://github.com/Open-Cascade-SAS/OCCT.git",
|
||||
"branch": "master"
|
||||
},
|
||||
// OR
|
||||
"source": {
|
||||
"type": "upload",
|
||||
"archive": "<base64 tarball>"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Chunking
|
||||
|
||||
For each file:
|
||||
1. Detect content type
|
||||
2. Select chunker (built-in or custom WASM)
|
||||
3. Chunk content
|
||||
4. Store chunks in eXist-db with metadata
|
||||
|
||||
**Chunk metadata:**
|
||||
```json
|
||||
{
|
||||
"id": "chunk-uuid",
|
||||
"library_id": "lib-uuid",
|
||||
"file_path": "src/BRepBuilderAPI/BRepBuilderAPI_MakeEdge.cxx",
|
||||
"start_line": 142,
|
||||
"end_line": 387,
|
||||
"type": "function",
|
||||
"name": "BRepBuilderAPI_MakeEdge::Build",
|
||||
"language": "cpp",
|
||||
"imports": ["gp_Pnt", "TopoDS_Edge", "BRepLib"],
|
||||
"calls": ["GCPnts_TangentialDeflection", "BRep_Builder"],
|
||||
"version": "v7.8.0",
|
||||
"indexed_at": "2026-01-26T..."
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Indexing (Background)
|
||||
|
||||
After chunking, RLM builds structural index:
|
||||
|
||||
- **Call graph**: What calls what
|
||||
- **Type hierarchy**: Classes, inheritance
|
||||
- **Module map**: How code is organized
|
||||
- **Symbol table**: Functions, classes, constants
|
||||
- **Dependency graph**: Imports, includes
|
||||
|
||||
This runs as a background job (can take hours for large codebases).
|
||||
|
||||
## Query Pipeline
|
||||
|
||||
### 1. Query
|
||||
|
||||
```
|
||||
POST /api/v1/libraries/{id}/query
|
||||
{
|
||||
"question": "Where are edges calculated?",
|
||||
"max_tokens": 8000,
|
||||
"include_sources": true
|
||||
}
|
||||
```
|
||||
|
||||
### 2. RLM Processing
|
||||
|
||||
The RLM receives:
|
||||
```
|
||||
You have access to a library "opencascade" with 2.3M lines of C++ code.
|
||||
|
||||
Structural index available:
|
||||
- 4,231 classes
|
||||
- 47,892 functions
|
||||
- 12 main modules: BRepBuilderAPI, BRepAlgoAPI, ...
|
||||
|
||||
User question: "Where are edges calculated?"
|
||||
|
||||
Available tools:
|
||||
- search(query) → relevant chunks
|
||||
- get_chunk(id) → full chunk content
|
||||
- get_structure(path) → module/class structure
|
||||
- recursive_query(sub_question) → ask yourself about a subset
|
||||
```
|
||||
|
||||
RLM then:
|
||||
1. Searches for "edge" in symbol table
|
||||
2. Finds `BRepBuilderAPI_MakeEdge`, `BRepAlgo_EdgeConnector`, etc.
|
||||
3. Recursively queries: "What does BRepBuilderAPI_MakeEdge do?"
|
||||
4. Retrieves relevant chunks, synthesizes answer
|
||||
|
||||
### 3. Response
|
||||
|
||||
```json
|
||||
{
|
||||
"answer": "Edge calculation in OpenCASCADE primarily happens in the BRepBuilderAPI module...",
|
||||
|
||||
"sources": [
|
||||
{
|
||||
"file": "src/BRepBuilderAPI/BRepBuilderAPI_MakeEdge.cxx",
|
||||
"lines": "142-387",
|
||||
"relevance": 0.95,
|
||||
"snippet": "void BRepBuilderAPI_MakeEdge::Build() { ... }"
|
||||
},
|
||||
{
|
||||
"file": "src/BRepAlgo/BRepAlgo_EdgeConnector.cxx",
|
||||
"lines": "89-201",
|
||||
"relevance": 0.82,
|
||||
"snippet": "..."
|
||||
}
|
||||
],
|
||||
|
||||
"tokens_used": 47832,
|
||||
"chunks_examined": 127,
|
||||
"cost": "$0.34"
|
||||
}
|
||||
```
|
||||
|
||||
## Storage (eXist-db)
|
||||
|
||||
### Why eXist-db?
|
||||
|
||||
- **Versioning**: Track codebase changes over time
|
||||
- **XML-native**: Fits with xml-pipeline philosophy
|
||||
- **XQuery**: Powerful querying for structured data
|
||||
- **Efficient**: Handles millions of documents
|
||||
|
||||
### Schema
|
||||
|
||||
```xml
|
||||
<library id="opencascade" version="v7.8.0">
|
||||
<metadata>
|
||||
<name>OpenCASCADE</name>
|
||||
<source>git:https://github.com/Open-Cascade-SAS/OCCT.git</source>
|
||||
<indexed_at>2026-01-26T08:00:00Z</indexed_at>
|
||||
<stats>
|
||||
<files>12847</files>
|
||||
<lines>2341892</lines>
|
||||
<chunks>89234</chunks>
|
||||
</stats>
|
||||
</metadata>
|
||||
|
||||
<structure>
|
||||
<module name="BRepBuilderAPI">
|
||||
<class name="BRepBuilderAPI_MakeEdge">
|
||||
<function name="Build" file="..." lines="142-387"/>
|
||||
...
|
||||
</class>
|
||||
</module>
|
||||
</structure>
|
||||
|
||||
<chunks>
|
||||
<chunk id="..." file="..." type="function" name="Build">
|
||||
<content>...</content>
|
||||
<relations>
|
||||
<calls>GCPnts_TangentialDeflection</calls>
|
||||
<imports>gp_Pnt</imports>
|
||||
</relations>
|
||||
</chunk>
|
||||
</chunks>
|
||||
</library>
|
||||
```
|
||||
|
||||
## Pricing (Premium Tier)
|
||||
|
||||
| Operation | Cost |
|
||||
|-----------|------|
|
||||
| Ingest (per 1M tokens) | $2.00 |
|
||||
| Index (per library) | $5.00 - $50.00 (depends on size) |
|
||||
| Query (per query) | $0.10 - $2.00 (depends on complexity) |
|
||||
| Storage (per GB/month) | $0.50 |
|
||||
|
||||
**Why premium:**
|
||||
- Compute-intensive (lots of LLM calls)
|
||||
- Storage-intensive (versioned codebases)
|
||||
- High value (saves weeks of manual exploration)
|
||||
|
||||
## Use Cases
|
||||
|
||||
### 1. Legacy Code Understanding
|
||||
"I inherited a 500K line Fortran codebase. Help me understand the data flow."
|
||||
|
||||
### 2. API Discovery
|
||||
"How do I create a NURBS surface with specific knot vectors?"
|
||||
|
||||
### 3. Impact Analysis
|
||||
"What would break if I deprecated this function?"
|
||||
|
||||
### 4. Onboarding
|
||||
"Explain the architecture of this codebase to a new developer."
|
||||
|
||||
### 5. Code Review Assistance
|
||||
"Does this change follow the patterns used elsewhere in the codebase?"
|
||||
|
||||
## Implementation Phases
|
||||
|
||||
### Phase 1: MVP
|
||||
- [ ] Basic ingest (git clone, tarball upload)
|
||||
- [ ] Code chunker (Python, JavaScript, C++)
|
||||
- [ ] eXist-db storage
|
||||
- [ ] Simple RLM query (search + retrieve)
|
||||
|
||||
### Phase 2: Full Chunkers
|
||||
- [ ] Prose chunker
|
||||
- [ ] Structured chunker (YAML/JSON/XML)
|
||||
- [ ] Tabular chunker
|
||||
- [ ] WASM chunker SDK
|
||||
|
||||
### Phase 3: Deep Indexing
|
||||
- [ ] Call graph extraction
|
||||
- [ ] Type hierarchy
|
||||
- [ ] Cross-reference index
|
||||
- [ ] Incremental re-indexing on changes
|
||||
|
||||
### Phase 4: Advanced Queries
|
||||
- [ ] Multi-turn conversations about code
|
||||
- [ ] "What if" analysis
|
||||
- [ ] Code generation informed by codebase patterns
|
||||
|
||||
---
|
||||
|
||||
*"Finally understand the codebase you inherited."*
|
||||
Loading…
Reference in a new issue