Memory System

AgentX features a sophisticated cognitive memory system inspired by human memory architecture, enabling the AI to remember past conversations, learn from interactions, and build a knowledge graph over time.

Overview

The memory system provides four types of memory:

Design Principles

• Extensibility: Easy to add new memory types, stores, or extraction methods
• Transparency: All memory operations traceable per session/conversation
• Auditability: Full query trace and operation audit trail in PostgreSQL
• Channel Scoping: Memory organized into channels for project-level organization

Memory Channels

Memory is organized into channels — traceable scopes that group related memories:

Key behaviors:

• Retrieval queries both the active channel AND _global, merging results
• Channels are traceable scopes, not isolation boundaries
• Cross-channel operations are logged in the audit trail
• Prominent project facts can be promoted to _global based on confidence/frequency thresholds

# Using channels
memory = AgentMemory(user_id="user123", channel="my-rust-project")

# Retrieval automatically merges project + global memories
context = memory.remember("What error handling pattern should I use?")
# Returns: project-specific Rust patterns + global user preferences

Key Features

Episodic Memory

• Stores complete conversation history with turns
• Vector-based semantic search across conversations
• Temporal filtering and recency boosting
• Automatic consolidation to semantic memory

Semantic Memory

• Entity recognition and tracking (people, organizations, concepts)
• Fact extraction with confidence scores
• Knowledge graph with relationships
• Entity salience scoring and decay

Procedural Memory

• Records tool usage patterns
• Learns successful strategies for tasks
• Reinforcement learning from outcomes
• Tool performance analytics

Working Memory

• Redis-based fast access for current session
• Maintains recent conversation context
• Active goal tracking
• Chain-of-thought reasoning steps

Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                         AGENT RUNTIME                                │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐                  │
│  │   LLM API   │  │  Tool Exec  │  │  Planning   │                  │
│  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘                  │
│         │                │                │                          │
│         └────────────────┼────────────────┘                          │
│                          ▼                                           │
│                 ┌─────────────────┐                                  │
│                 │ Memory Interface│                                  │
│                 └────────┬────────┘                                  │
└──────────────────────────┼──────────────────────────────────────────┘
                           │
        ┌──────────────────┼──────────────────┐
        ▼                  ▼                  ▼
┌───────────────┐  ┌───────────────┐  ┌───────────────┐
│    Neo4j      │  │  PostgreSQL   │  │    Redis      │
│ Graph+Vector  │  │  pgvector     │  │ Working Mem   │
│               │  │  Time-series  │  │ Cache Layer   │
│ - Semantic    │  │               │  │               │
│ - Episodic    │  │ - Raw logs    │  │ - Hot queries │
│ - Procedural  │  │ - Audit trail │  │ - Session     │
│ - Entities    │  │ - Timeline    │  │ - Rate limits │
└───────────────┘  └───────────────┘  └───────────────┘

Technology Stack

Usage

Basic Usage

from agentx_ai.kit.agent_memory import AgentMemory, Turn

# Initialize memory for a user
memory = AgentMemory(user_id="user123", conversation_id="conv456")

# Store a conversation turn
turn = Turn(
    conversation_id="conv456",
    index=0,
    role="user",
    content="What's the weather like today?"
)
memory.store_turn(turn)

# Retrieve relevant memories
context = memory.remember("What did we discuss about weather?")
print(context.to_context_string())

# Learn a new fact
memory.learn_fact(
    claim="User prefers concise responses",
    source="inferred",
    confidence=0.8
)

# Track a goal
from agentx_ai.kit.agent_memory import Goal
goal = Goal(
    description="Help user plan vacation",
    priority=4
)
memory.add_goal(goal)

# Complete a goal
memory.complete_goal(goal.id, status="completed", result="Vacation itinerary created")

# Or mark as abandoned/blocked
memory.complete_goal(goal.id, status="abandoned", result="User cancelled request")

# Record tool usage for learning
memory.record_tool_usage(
    tool_name="web_search",
    tool_input={"query": "weather"},
    tool_output={"results": [...]},
    success=True,
    latency_ms=250
)

Advanced Retrieval

# Retrieve with filters
context = memory.remember(
    query="Python programming discussion",
    top_k=15,
    include_episodic=True,
    include_semantic=True,
    include_procedural=True,
    time_window_hours=72  # Last 3 days only
)

# Find what worked for similar tasks
strategies = memory.what_worked_for("data analysis task")
for strategy in strategies:
    print(f"Strategy: {strategy.description}")
    print(f"Tools: {strategy.tool_sequence}")
    print(f"Success rate: {strategy.success_count / max(1, strategy.success_count + strategy.failure_count)}")

# Get active goals
goals = memory.get_active_goals()
for goal in goals:
    print(f"[P{goal.priority}] {goal.description} - {goal.status}")

Recall Layer

The RecallLayer augments base vector retrieval with multiple techniques:

graph LR
    Q[Query] --> BASE[Base Retrieval<br/>vector similarity]
    Q --> HYB[Hybrid Search<br/>BM25 + vector, RRF fusion]
    Q --> ENT[Entity-Centric<br/>graph traversal from matched entities]
    Q --> QE[Query Expansion<br/>question→statement transforms]
    Q --> HYDE[HyDE<br/>hypothetical document embedding]
    Q --> SQ[Self-Query<br/>LLM filter extraction]

    BASE & HYB & ENT & QE & HYDE & SQ --> MERGE[Merge + Deduplicate]
    MERGE --> RANK[Rerank<br/>salience, temporal, access boosts]
    RANK --> MB[MemoryBundle]

Each technique is independently toggleable via GET/POST /api/memory/recall-settings.

Consolidation Pipeline

The consolidation pipeline processes recent conversations into structured knowledge:

graph TD
    T[Recent Turns] --> RF[Relevance Filter<br/>skip trivial messages]
    RF --> EX[Combined Extraction<br/>entities + facts + relationships in one LLM call]
    EX --> EL[Entity Linking<br/>embedding-based entity resolution]
    EL --> CD[Contradiction Detection<br/>compare new facts against existing]
    CD --> |contradicts| RS[Resolution<br/>prefer_new / prefer_old / flag_review]
    CD --> |no conflict| ST[Store<br/>upsert_entity + learn_fact]
    RS --> ST

Phase 11.12 features:

• Confidence calibration — Certainty mapping: explicit=0.95, implied=0.85, inferred=0.70, uncertain=0.50
• Temporal reasoning — Facts tagged as current, past, or future with temporal boost in retrieval (current=1.2x, past=0.7x)
• Reinforcement signals — access_count and salience on facts, incremented on retrieval, used in reranking
• Source attribution — source_turn_id on extracted facts links back to the originating conversation turn
• User correction handling — Detects “actually…”, “no, I meant…” patterns; supersedes old facts via [:SUPERSEDES] relationship
• Contradiction detection — Compares new facts against recent existing facts; resolution strategies: prefer_new, prefer_old, flag_review

Background Processing

The JobRegistry manages background consolidation jobs:

• Consolidation (every 15 min): Extracts entities and facts from recent conversations
• Pattern Detection (hourly): Learns successful strategies from conversation outcomes
• Memory Decay (daily): Applies time-based decay to salience scores
• Cleanup (daily): Archives old conversations and removes low-salience entities

Jobs can be monitored and controlled via the Jobs API.

Memory API

See API Endpoints: Memory for full details.

Configuration

Configure the memory system via environment variables:

# Neo4j
NEO4J_URI=bolt://localhost:7687
NEO4J_USER=neo4j
NEO4J_PASSWORD=your_password

# PostgreSQL
POSTGRES_URI=postgresql://agent:password@localhost:5432/agent_memory

# Redis
REDIS_URI=redis://localhost:6379

# Embeddings
EMBEDDING_PROVIDER=openai  # or "local"
EMBEDDING_MODEL=text-embedding-3-small
OPENAI_API_KEY=sk-...

# Memory settings
EPISODIC_RETENTION_DAYS=90
FACT_CONFIDENCE_THRESHOLD=0.7
SALIENCE_DECAY_RATE=0.95
MAX_WORKING_MEMORY_ITEMS=50

Audit Logging

All memory operations are logged to a partitioned PostgreSQL table for traceability:

Logged information includes:

• Operation type, timestamp, user/session/conversation IDs
• Source channel and target channels (for cross-channel operations)
• Query text and result count for retrievals
• Latency per operation
• Promotion tracking (when facts are promoted from project to _global)
• Configuration snapshot (active thresholds at time of operation)

# Configure audit logging
AUDIT_LOG_LEVEL=writes
AUDIT_RETENTION_DAYS=30

Database Setup

Initialize the memory system schemas before first use:

# Start database services
task db:up

# Initialize all schemas (Neo4j, PostgreSQL, Redis)
task db:init:schemas

# Or verify existing schemas
task db:verify:schemas

This creates:

• Neo4j: Vector indexes, uniqueness constraints, channel indexes
• PostgreSQL: Memory tables with channel columns, partitioned audit log
• Redis: Verifies connectivity and documents key patterns

Related

• Memory System Architecture — Detailed technical architecture
• Database Stack — Infrastructure details
• API Models: Memory — Turn, Entity, Fact, Goal, Strategy schemas