Database Stack

AgentX uses a hybrid database architecture combining graph, relational, and in-memory databases.

Overview

graph TB
    API[Django API]
    Neo4j[Neo4j<br/>Graph Database]
    Postgres[PostgreSQL<br/>+ pgvector]
    Redis[Redis<br/>Cache]

    API -->|Relationships| Neo4j
    API -->|Embeddings| Postgres
    API -->|Cache| Redis

    style Neo4j fill:#4581C3
    style Postgres fill:#336791
    style Redis fill:#DC382D

Database Roles

Neo4j - Graph Database

Purpose: Store and analyze relationships between entities

Use Cases:

• Entity relationship mapping
• Knowledge graph construction
• Semantic connections
• Graph-based reasoning

Configuration:

• Port: 7474 (browser), 7687 (bolt)
• Data: ./data/neo4j/data
• Logs: ./data/neo4j/logs
• Plugins: APOC

Example Query:

// Create entity with relationships
CREATE (p:Person {name: 'Alice'})
CREATE (c:Concept {name: 'Machine Learning'})
CREATE (p)-[:INTERESTED_IN]->(c)

PostgreSQL + pgvector

Purpose: Store vector embeddings for semantic search

Use Cases:

• Embedding storage
• Similarity search
• Conversation history
• Structured data

Configuration:

• Port: 5432
• Database: agent_memory
• User: agent
• Data: ./data/postgres
• Extensions: pgvector

Example Usage:

-- Create embeddings table
CREATE TABLE embeddings (
    id SERIAL PRIMARY KEY,
    content TEXT,
    embedding vector(1024),
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Create vector index
CREATE INDEX ON embeddings
USING ivfflat (embedding vector_cosine_ops);

-- Find similar vectors
SELECT content,
       embedding <=> '[0.1, 0.2, ...]' AS distance
FROM embeddings
ORDER BY distance
LIMIT 10;

Redis - Cache Layer

Purpose: High-speed caching and temporary data storage

Use Cases:

• Translation cache
• Session storage
• Rate limiting
• Real-time data

Configuration:

• Port: 6379
• Data: ./data/redis
• Max Memory: 512MB
• Eviction: allkeys-lru

Example Usage:

import redis

r = redis.Redis(host='localhost', port=6379)

# Cache translation
r.setex('translate:en:fr:hello', 3600, 'bonjour')

# Retrieve cached value
cached = r.get('translate:en:fr:hello')

Data Flow

Translation Request

sequenceDiagram
    participant Client
    participant API
    participant Redis
    participant Model
    participant Neo4j

    Client->>API: POST /translate
    API->>Redis: Check cache

    alt Cache Hit
        Redis-->>API: Return cached
    else Cache Miss
        API->>Model: Translate
        Model-->>API: Result
        API->>Redis: Store cache
        API->>Neo4j: Store relationship
    end

    API-->>Client: Translation

Memory Storage

sequenceDiagram
    participant API
    participant Postgres
    participant Neo4j
    participant Redis

    API->>Postgres: Store embedding
    API->>Neo4j: Create relationships
    API->>Redis: Cache metadata

    Note over Postgres,Redis: Coordinated storage

Data Directories

All database data is stored in local bind mounts:

data/
├── neo4j/
│   ├── data/          # Graph database files
│   ├── logs/          # Application logs
│   └── plugins/       # APOC and other plugins
├── postgres/          # PostgreSQL data files
└── redis/             # Redis persistence files

Benefits

• Easy Backups: Simple directory copy
• Portability: Move between environments
• Debugging: Direct file access
• No Orphaned Volumes: Explicit data location

Migration from Docker Volumes

If you have existing data in Docker volumes:

# Migrate all databases
task db:migrate-volumes

# Or individually
task db:migrate-volumes:neo4j
task db:migrate-volumes:postgres
task db:migrate-volumes:redis

This copies data from Docker volumes to ./data/ directories.

Backup Strategies

PostgreSQL Backup

Automated backup:

task db:backup:postgres

Manual backup:

docker exec agent-postgres pg_dump -U agent agent_memory > backup.sql

Restore:

task db:restore:postgres BACKUP_FILE=backup.sql

Neo4j Backup

Using Neo4j dump:

docker exec agent-neo4j neo4j-admin database dump neo4j --to-path=/backups

Redis Backup

Redis automatically persists with AOF:

# Copy the appendonly.aof file
cp data/redis/appendonlydir/appendonly.aof.1.base.rdb backups/

Full Backup

Copy entire data directory:

tar -czf agentx-backup-$(date +%Y%m%d).tar.gz data/

Database Maintenance

PostgreSQL

Vacuum and analyze:

docker exec agent-postgres psql -U agent -d agent_memory -c "VACUUM ANALYZE;"

Neo4j

Check database size:

CALL dbms.queryJmx("org.neo4j:instance=kernel#0,name=Store sizes")
YIELD attributes
RETURN attributes.TotalStoreSize.value AS size

Redis

Get memory stats:

docker exec agent-redis redis-cli INFO memory

Connection Strings

Neo4j

bolt://localhost:7687

Python (neo4j driver):

from neo4j import GraphDatabase

driver = GraphDatabase.driver(
    "bolt://localhost:7687",
    auth=("neo4j", "your_secure_password")
)

PostgreSQL

postgresql://agent:your_secure_password@localhost:5432/agent_memory

Python (psycopg2):

import psycopg2

conn = psycopg2.connect(
    host="localhost",
    port=5432,
    database="agent_memory",
    user="agent",
    password="your_secure_password"
)

Redis

redis://localhost:6379

Python (redis-py):

import redis

r = redis.Redis(
    host='localhost',
    port=6379,
    decode_responses=True
)

Performance Considerations

Neo4j

• Default heap: 512MB initial, 2GB max
• Page cache: 1GB
• Increase for larger graphs

PostgreSQL

• Shared buffers: 25% of RAM
• Work memory: 4MB per connection
• Maintenance work memory: 64MB

Redis

• Max memory: 512MB (configurable)
• Eviction policy: allkeys-lru
• Persistence: AOF enabled

Security

Network Isolation

Databases are exposed on localhost only. For production:

# docker-compose.yml
services:
  postgres:
    ports: []  # Remove port mapping
    networks:
      - internal  # Use internal network

Authentication

Change default passwords in docker-compose.yml:

environment:
  - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}  # Use env var

Encryption

For production, enable:

• SSL/TLS for PostgreSQL
• Bolt encryption for Neo4j
• Redis AUTH password

Monitoring

Health Checks

All services have built-in health checks:

docker ps

Shows health status for each container.

Database Shells

Access databases directly:

# PostgreSQL
task db:shell:postgres

# Redis
task db:shell:redis

# Neo4j Browser
open http://localhost:7474

Troubleshooting

Connection Refused

Check if services are running:

docker-compose ps

Data Corruption

Restore from backup:

task db:restore:postgres BACKUP_FILE=backups/latest.sql

Disk Space

Check data directory sizes:

du -sh data/*

Performance Issues

Check resource usage:

docker stats

Next Steps

• Database Migration Guide - Data migration strategies
• API Reference - Database interactions via API
• Development Tasks - Database management commands