# Memory Capabilities Manifest A single, scannable map of **what the memory system can do and how the pieces connect**. This is the capability manifest — for the layer-by-layer build details (class signatures, storage, indexes) see [Memory Architecture](./memory.md); for the user-facing feature tour see [Memory (feature)](../features/memory.md). The system is deliberately interconnected: a conversation turn flows through extraction → consolidation → semantic facts/entities, all scoped by **channel**, and is later surfaced by multi-technique recall. The matrix indexes every capability; the sections below explain each one and how it feeds the others. ## Capability matrix | Capability | Layer | Status | Entrypoint (code) | Channel scope | |------------|-------|--------|-------------------|---------------| | Conversation capture | episodic | shipped | `AgentMemory.store_turn` (`memory/episodic.py`) | active (+ `agent_id` on assistant turns) | | Semantic facts & entities | semantic | shipped | `learn_fact`, `upsert_entity` (`memory/semantic.py`) | per subject | | Procedural strategies | procedural | shipped | `record_tool_usage`, `detect_patterns` (`memory/procedural.py`) | active / `_self_` | | Procedural distillation + reflex core | procedural | shipped | encode (`stage_procedure_candidate`) → distill (`distill_procedures`) → `learn_procedure` + `get_reflex_procedures` (`memory/procedural.py`) | `_self_{agent_id}` (corrections) / channel (rules) | | Working memory | working | shipped | `WorkingMemory` (Redis, TTL) | active | | Goal tracking | semantic | shipped | `add_goal` / `complete_goal` / `get_goals_for_conversation`; `TaskPlanner` | active, conversation-scoped | | Relevance + extraction | extraction | shipped | `ExtractionService.check_relevance_and_extract[_assistant]` | — | | Consolidation (episodic→semantic) | jobs | shipped | `consolidate_episodic_to_semantic()` (15-min) | global sweep | | Subject-aware attribution | extraction/jobs | shipped | `_resolve_agent_attribution`, `_resolve_subject_channel` | routes to subject's channel | | Multi-agent attribution | jobs | shipped | `_ensure_agent_entities`; Agent entities | `_self_{agent_id}` per agent | | Fact verification (3-layer) | jobs | shipped | hash → semantic → entity-scoped → `check_contradictions` | per channel | | Corrections / supersession | jobs | shipped | `check_correction`, `_handle_user_correction` | per channel | | Recall (5 techniques) | recall | shipped | `RecallLayer.recall` (hybrid, entity-centric, query-expansion, HyDE, self-query) | `[active, _self_, _global]` | | Two-stage rerank | recall | shipped | `RecallLayer._cross_encoder_stage` — 50-candidate pool → post-fusion cross-encoder, bounded demotion (cap 2); default-ON | `[active, _self_, _global]` | | Stable salient core | recall | shipped | `AgentMemory.get_salient_core` → `get_salient_facts`/`get_salient_entities` (`memory/semantic.py`); injected as the prio-70 ledger block (`agent/context_ledger.py`) | `[active, _self_, _global]` | | Episodic thread leads ("threads to pull") | recall | shipped | `AgentMemory.derive_thread_leads` (fact `source_turn_id` → origin turn) → prio-28 pointer block; `read_thread` pull tool (also `conversation_id="current"` — the digest-expandability path, served from the durable transcript so it works even with memory off) | leads: past conversations; pulls: any incl. current | | Context gating | context | shipped | `ToolOutputCompressor`, `tool_output_chunker`, trajectory compression | active | | Structured conversation state | working | shipped | `conversation_state_storage.py` (Redis hot cache + durable Postgres `conversation_state`); `update_conversation_state` tool + `GET/PATCH /api/conversations/{id}/state` | conversation (durable) | | Conversation context ledger | context | shipped | `assemble_ledger` (`agent/context_ledger.py`) — priority blocks + verbatim budget | per turn | | State compaction (rolling digest) | context | shipped | `SessionManager.maybe_compact_to_state` → `ConversationState.digest`; `_ensure_summary_coverage` (INV-CTX-1) | conversation | | Memory-tool coaching | prompts | shipped | `memory-tools` built-in prompt layer (`prompts/layers.py`) — ASSUME INTERRUPTION, recall-gating, leads-vs-transcript, untrusted content | — | | Poisoning defense (single-writer + provenance) | security | shipped | INV-8; `_coerce_author` (no forged `system` author), no auto-ingest of tool/web output; guard `MemoryPoisoningTest` | conversation state | | Project memory scoping | channels | shipped | `_resolve_project_channel_workspace` (`views.py`) → `AgentConfig.memory_channel = _project_{ws_id}` | `_project_{workspace_id}` (opt-out: `memory.project_channels`) | | Cross-channel promotion | lifecycle | shipped | `promote_to_global` | → `_global` | | Salience decay | lifecycle | shipped | decay job (`consolidation/`) | all | | Entity dedupe | lifecycle | shipped | `dedupe_entities` (mgmt cmd) | per/cross channel | | Deterministic link backfill | lifecycle | shipped | `link_facts_to_entities` (`entity_linking` job) | all | | Manual fact↔entity link | lifecycle | shipped | `link_fact_to_entity` / `unlink_fact_from_entity` (`memory/semantic.py`); `POST/DELETE /api/memory/facts/{id}/entities` | per fact | | Agent-attribution backfill | lifecycle | shipped | `backfill_agent_attribution` (mgmt cmd) | `_self_{agent_id}` | | Portability (export/import) | ops | shipped | `MemoryExporter` / `MemoryImporter` (v2 envelope: + procedures) | channel set / `_all` | | Extract (verified deletion) | ops | shipped | `extract_memory` (`portability/extract.py`); `POST /api/memory/extract` | exact channel set (never `_global`) | | Debug harness | ops | shipped | `debug_attribution` (mgmt cmd) | scenario-scoped | ## How the pieces connect ```mermaid graph TD WS[Project
workspace membership] -->|"_project_<ws_id> = active channel"| T T[Conversation turn
store_turn] --> EP[(Episodic
Neo4j + pgvector)] EP -->|15-min sweep| CON[Consolidation] CON --> EX[Extraction
relevance + facts + subject_agent] EX --> ATTR[Attribution
name → agent_id] ATTR --> ROUTE{Subject → channel} ROUTE -->|user / third_party| ACT[(active channel)] ROUTE -->|agent: subject_agent_id| SELF[(_self_<agent_id>)] EX --> VER[3-layer verification
hash → semantic → entity → LLM] VER --> SEM[(Semantic
facts + entities + Agent entities)] ACT --> SEM SELF --> SEM SEM -->|promote_to_global| GLOB[(_global)] SEM --> RECALL[RecallLayer
5 techniques] GLOB --> RECALL RECALL -->|"[active, _self_, _global]"| AGENT[Agent prompt] T -->|steer / explicit rule| PC[(procedure_candidates
encode)] PC -->|distill_procedures| PROC[(Procedures
trigger + body)] PROC -->|reflex core: top-strength| AGENT ``` ## Conversation context lifecycle (per turn) The model is **stateless between turns** — every turn the server rebuilds the whole prompt from scratch within a token budget (`assemble_ledger`, `agent/context_ledger.py`). Blocks compete by priority: the system prompt is mandatory, then memory (salient facts + the structured Conversation State), then the verbatim recent transcript, then the new message; low-priority blocks (query-driven recall) are dropped first under pressure. Per-turn context assembly and compaction The full conversation splits into recent turns kept verbatim and older turns compacted into the conversation-state digest; both feed the turn's prompt within the token budget, while every raw turn stays durable in Postgres. Full conversation — every turn, oldest → newest Older turns → Compaction one pass, rolled into the state digest Newest turns → kept verbatim the recent tail that fits the budget This turn's prompt · fits ≈ 90% of the model window System prompt · Memory (facts + Conversation State + digest) · Verbatim tail · Message Durability & the contract (INV-CTX-1) Every raw turn is also saved to Postgres — durable and searchable. A turn leaves the live window only once the digest (or a deterministic fallback) covers it. **What "aged out" means.** Before assembly, `_ensure_summary_coverage` walks the transcript newest→oldest, keeping the recent tail that fits the budget (always ≥ `recent_floor`). Older turns beyond that tail are *aged out* — dropped from the live window, but never from durable storage (`conversation_logs` in Postgres). **Compaction (Slice 1c, INV-CTX-1).** Aged-out turns are folded, in exactly one pass, into `ConversationState.digest` — a **rolling** summary re-summarized in place each time (bounded without dropping old coverage), rendered in the state block every turn (`SessionManager.maybe_compact_to_state`). A turn leaves the window only once the digest — or a deterministic `history_digest` fallback, if the summarizer is unavailable — covers it. Two triggers, both anchored to the turn's **real history budget** (what's left of the input budget after the preamble blocks): a post-turn pre-warm (`context.summary_trigger_ratio`, 0.85 — pays the summarizer call in idle time) and a just-in-time backstop right before assembly (within `context.verbatim_budget_ratio`, 0.9). One pass runs per over-budget turn (state compaction by default; the legacy prose summary is the flag-off fallback — no double-compression). Every compaction call site — streaming, post-turn, background chat — routes through one shared target gate, so they can never compact into different objects. **Why rolling, not per-conversation.** Re-summarizing only the newly-aged-out turns (rather than re-summarizing the whole window at a checkpoint) keeps compaction incremental, and — because the digest lives beside the goals / decisions / open-threads slots — it preserves the conversation's **active working state**, not just a narrative recap. The tradeoff is mild summary drift over very long runs, hedged by rolling the prior digest in and by keeping the raw turns durable. ### The techniques at a glance Every technique below is tunable in **Settings → Memory → Conversation Context**. "Between turns" techniques manage the conversation's transcript; "within a turn" techniques manage a single turn's working space (tool rounds never enter the transcript, so the two never double-compress the same content). | Technique | Scope | Fires when | Writes / renders | Key settings | |-----------|-------|-----------|------------------|--------------| | **Verbatim window** | between turns | every turn | the recent transcript, word-for-word, newest-first | `context.verbatim_budget_ratio`, `context.recent_floor` | | **Conversation-state slots** | between turns | agent tool call / user edit | goals · decisions · open threads · artifacts · narrative (provenance-stamped, bounded; Redis hot cache + durable Postgres) | `context.conversation_state_enabled` | | **State digest (compaction)** | between turns | history nears its budget (pre-warm at `summary_trigger_ratio`; JIT backstop before assembly) | `ConversationState.digest`, re-summarized in place — durable, and **expandable**: `read_thread("current")` pulls the verbatim turns behind it | `context.conversation_state_compaction_enabled`, `context.preassembly_summary_enabled`, `session.rolling_summary.*` | | **Legacy prose summary** | between turns | only when state compaction is off (+ read-back for old chats) | `session.summary` (Redis) | `session.rolling_summary.enabled` | | **`history_digest` fallback** | between turns | summarizer unavailable while over budget | deterministic one-line-per-turn digest block | (automatic) | | **Checkpoints / scratchpad** | between turns | agent tool call | Redis notes re-injected every turn (compression can't strip them) | (always on) | | **Rehydration** | conversation resume | cold session | reloads the durable transcript; beyond-cap turns → notice block pointing at `read_thread("current")` + recall | `context.rehydrate_max_turns` | | **Episodic thread leads** | between turns | episodic phrasing ("when did we…") | ≤5 pointers into past conversations; `read_thread` pulls detail | `memory.episodic_leads_enabled` | | **Trajectory compression** | within a turn | tool loop crosses `threshold_ratio` of the in-turn ceiling | older tool rounds → one `[KNOWLEDGE]` block | `trajectory_compression.*` | | **Tool-output compression** | within a turn | one tool result is oversized | task-aware summary + structure index; full output stays retrievable | `compression.*` | | **Input spend guard** | within a turn | set > 0 | caps the in-turn ceiling below the model window | `context.max_input_tokens` | ## Capabilities by area ### Storage layers - **Episodic** — every turn (`store_turn`) lands in Neo4j (graph) + pgvector (vectors) + PostgreSQL (audit log). Assistant turns carry `agent_id` + `agent_name`; this is the write-path contract the rest of attribution relies on. → feeds **Consolidation**. - **Semantic** — entities, facts, and relationships. Facts link to entities via `[:ABOUT]`; agents are first-class entities here (see **Multi-agent attribution**). → read by **Recall**. - **Procedural** — tool-usage trajectories and learned strategies (`detect_patterns`), plus **distilled procedures** (the "how we work here" delta): the encode loop stages `procedure_candidates` from steers/corrections + explicit user rules, the `distill_procedures` consolidation job mints scoped `Procedure` nodes (trigger + replayable body, strengthened not duplicated), and the **reflex core** (`get_reflex_procedures`) injects the top-strength ones into every prompt. → feeds the **Agent prompt** directly. - **Working** — short-lived Redis context with TTL refresh. - **Goals** — `add_goal`/`complete_goal`, linked from `TaskPlanner` so plans track intent. Each goal is stamped with the `conversation_id` that opened it, so `get_goals_for_conversation` attributes goals per conversation (surfaced in the ambassador's cross-conversation survey). Forward-looking: goals created before this carry no `conversation_id`. ### Extraction & consolidation - **Extraction** turns raw text into candidate entities/facts in one LLM call (`check_relevance_and_extract`), with a separate self-knowledge extractor for assistant turns (`..._assistant`). It is **roster-aware**: given the conversation's agents, it names the specific agent a fact concerns. - **Consolidation** (`consolidate_episodic_to_semantic`, every 15 min) sweeps unconsolidated turns through extraction → verification → storage. It is **global** (all pending conversations) — relevant when running the [debug harness](#debug-harness). ### Attribution & identity (interconnect: extraction ↔ channels ↔ semantic) - **Subject-aware attribution** tags each fact `user | agent | third_party` and routes it to the matching channel, so the user's memory and an agent's self-knowledge never blend. - **Multi-agent attribution** makes that agent-specific: the LLM names the agent (`subject_agent`), the server resolves the name → `agent_id` (the durable source of truth) in `_resolve_agent_attribution`, and `_resolve_subject_channel` homes the fact to that agent's `_self_{agent_id}`. Agents are registered as first-class `Entity(type="Agent")` in `_global` (`_ensure_agent_entities`), so facts link to them by name and survive renames (aliases). See [Multi-Agent](../features/multi-agent.md). ### Project scoping (interconnect: workspaces ↔ channels) - **Project memory channels** — a conversation that belongs to a project (workspace) stores and recalls on `_project_{workspace_id}`, the "project" tier of the channel scope hierarchy (`_global → user → project → _self_ → conversation`). Knowledge learned inside a project stays with it; recall still layers `_self_` + `_global`, and **cross-channel promotion** lifts durable facts to `_global` over time. Precedence: workflow shared channel > project channel > profile channel; the reserved Home media space never scopes memory. Opt-out: config `memory.project_channels`. Resolution is best-effort (`_resolve_project_channel_workspace`) — memory scoping can never fail a turn. ### Verification (interconnect: extraction → semantic) - **3-layer fact verification** gates every new fact: claim-hash → semantic-duplicate → entity-scoped candidates → LLM adjudication (`check_contradictions`). All checks run in the fact's *resolved* channel, so a correction to one agent's directive is scoped to that agent. - **Corrections** (`check_correction`) supersede prior facts rather than duplicating them. ### Recall & retrieval (interconnect: semantic → agent prompt) - **RecallLayer** offers five techniques (hybrid BM25+vector, entity-centric, query expansion, HyDE, self-query) over the channel list `[active, _self_{agent_id}, _global]`, so an agent sees the user's context, its own self-knowledge, and global facts together. - **Two-stage rerank** (default-ON): stage 1 over-fetches a ~50-candidate pool (`recall_candidate_pool`); stage 2 reranks the fused pool with a cross-encoder (`cross-encoder/ms-marco-MiniLM-L-6-v2`, lazy-loaded, failure-safe) and cuts back to top-k. The encoder promotes freely but demotes a fused candidate at most `recall_ce_max_demotion` (2) positions — a hedge against encoder blind spots. Eval: +20pp MRR on the `eval_recall` golden set. A first-person attribution guard exists behind `recall_first_person_guard` (default-OFF — failed its abstention gate; see Memory-Roadmap §2.11). - **Stable salient core** (`get_salient_core`) injects a high-priority, *maintained-not-searched* block of the top-salience facts/entities (cheap non-vector `ORDER BY salience`, excludes superseded/retired) every turn, so durable knowledge no longer depends on the current message matching it. The query-driven RecallLayer rides along as a lower-priority **supplement** (deduped against the core); the **Context Ledger** (`agent/context_ledger.py`) budgets both — see *Conversation Context* in `Development-Notes.md`. - Every knob named above is documented key-by-key — bounds, defaults, and when to move them — in the [Settings Reference](../getting-started/settings-reference.md), generated from the same declarations the settings screen reads. ### Context gating - Oversized tool outputs are compressed and indexed (`ToolOutputCompressor`), chunked and queried on demand (`tool_output_chunker`), and long tool loops are compressed in-trajectory. The in-turn ceiling these work within derives from the model's **real** context window (minus the output reservation) — see the technique matrix in [Conversation context lifecycle](#conversation-context-lifecycle-per-turn); all knobs live in Settings → Memory → Conversation Context. ### Lifecycle & operations - **Promotion** raises high-value facts to `_global`; **decay** ages salience; **dedupe** collapses duplicate entities (never agents — keyed by `agent_id`); **link backfill** repairs orphaned fact→entity edges; **agent-attribution backfill** renames legacy generic "Agent …" facts to the agent's name. **Portability** round-trips a user's memory as JSON — the v2 envelope carries every node family *including distilled procedures*, and exports/wipes accept a channel **set** (the substrate for Memory Cores). **Extract** takes a channel set *out* of the live system: export to the vault → verify the artifact → only then wipe (Neo4j + PG mirror together), with the channels consolidation-frozen for the duration. The **debug harness** (below) drives the whole pipeline end-to-end. ### Debug harness - `python manage.py debug_attribution --scenario directive --agents "Mobius,Jeff"` seeds a scripted multi-agent conversation, runs **real** consolidation, and reports which channel every fact landed in (+ `[:ABOUT]` links) with pass/fail expectations. Non-destructive by default (throwaway user, scoped cleanup); `--isolate` snapshots+wipes+restores for a sterile read. See [Task Commands](../development/tasks.md). ## Maintaining this manifest This manifest must stay current as mechanisms land. **When you add or change a memory capability, update its matrix row and its section in the same PR** — and add the new behavior to the interconnection diagram if it introduces a new edge. This page is part of the "keep docs updated alongside code" rule in `CLAUDE.md`. A future code-side capability registry may generate/validate this page automatically (see `Todo.md` backlog).