GPU Acceleration

AgentX runs two on-device model workloads: the local embedding model (BAAI/bge-m3 via sentence-transformers) and the translation models (NLLB-200 + language detection). Both honor a single compute-device setting, so enabling a GPU speeds up memory recall/consolidation and translation.

How device selection works

The device is resolved once at model load from the AGENTX_DEVICE environment variable:

The same resolved device is applied to the embedding model and to both translation models — there is no separate per-model knob.

Verifying GPU use

After starting the API, check the live device without needing a shell into the process:

curl -s localhost:12319/api/health | jq .compute
# { "device": "cuda", "cuda_available": true }

The startup logs also print the resolved device for each model:

INFO device Compute device resolved to 'cuda' (requested=auto, cuda_available=True).
INFO embeddings Local embedding model 'BAAI/bge-m3' loaded on device 'cuda'.
INFO translation TranslationKit models loaded on device 'cuda' (...).

Low-level check:

python -c "import torch; print(torch.cuda.is_available())"

Installing CUDA-enabled PyTorch (local dev)

GPU use depends entirely on having a CUDA build of torch, and the project installs the CPU build by default — uv sync, uv run and every task that wraps them pull torch from the cpu dependency group, which carries none of the ~5 GB CUDA payload. That is the right default for most machines; on an NVIDIA machine you have to opt out of it.

Linux with an NVIDIA GPU: disable the cpu group for good by exporting this in your shell profile, then sync:

export UV_NO_GROUP=cpu   # every uv sync / uv run now skips the CPU-only torch build
uv sync                  # PyPI's build of the pinned torch — for 2.9.1, it bundles CUDA 12.8

The variable must stay set: uv run re-syncs the environment on every call, and without it the next uv run (or task dev) quietly swaps the CPU build back in.

Windows: PyPI’s torch wheel is CPU-only on Windows, so the switch above does not help. Run the API under WSL2 instead. See Windows Setup.

The torch version lives in one place — constraint-dependencies in pyproject.toml — and applies to every build, so the CPU and CUDA environments can never drift apart.

Docker / production

For containerized deployments, GPU passthrough is handled by the docker-compose.gpu.yml overlay (applied via cluster:up CLUSTER=<name> NVIDIA=1). The overlay also switches the API to the qrmadness/agentx-api:latest-gpu image — the CUDA 12.8 build. The default :latest image is CPU-only, so on its own it cannot use a GPU no matter what the overlay reserves. Source clusters that build the image locally also need docker-compose.gpu.build.yml; the deployment manager adds it for you. See Clusters & Gateway → GPU Acceleration.