Erudi Documentation¶

Technical documentation for Erudi — a desktop application that runs open-source LLMs locally, with retrieval over your own documents.

Overview¶

Erudi lets you:

Run LLMs locally on Apple Silicon (MLX), NVIDIA GPUs (CUDA), or CPU (llama.cpp)
Ground a model on your documents by attaching a Knowledge Base (hybrid RAG)
Compare models side by side in the Arena
Manage the model lifecycle: catalog, download of pre-built quants, idle unloading

Erudi downloads pre-built MLX and GGUF artifacts. It never converts or quantizes weights locally.

Sections¶

Getting started¶

Getting Started — prerequisites, backend/frontend setup, first API calls
Backend Launcher — run.py, ports, lifecycle events

Concepts¶

Architecture — DDD layering, multi-engine, startup sequence
Logging & Traceability — log files, request-id correlation, tracing a bug
Internationalization — locale files, the no-literal-string rule

Guides¶

Conversations — sessions, streaming, parameters
LLMs — catalog, downloads, model management
Knowledge Base — ingestion, retrieval, attaching a KB
Hardware Detection — engine selection and performance scores

Development¶

Exception Handling — exception hierarchy and error wire format
Engines Architecture — engine hierarchy, llama.cpp builds
Database Migrations — Alembic workflow

API reference¶

Core — configuration, logging, health
Engines — MLX, CUDA, CPU
Agents — LangChain runner, prompts, middlewares, checkpointer
Ingestion — document reader, chunking, embeddings, vector store
Launcher — runtime paths, embedded PostgreSQL
Conversations — chat endpoints
LLMs — model management
Knowledge Base — ingestion and retrieval
Arena — model comparison
Hardware — system monitoring
User settings — global preferences (web search, language, reasoning effort, compaction threshold)
Startup — first-boot payload for the frontend
Entities — SQLAlchemy models
Database — seeding and DB access

Architecture at a glance¶

┌─────────────────────────────────────────────────────────────┐
│ Frontend (Electron + React)                                 │
└────────────────┬────────────────────────────────────────────┘
                 │ HTTP on 127.0.0.1:27182, routes under /erudi
┌────────────────▼────────────────────────────────────────────┐
│ Backend (FastAPI)                                           │
│  ├─ Domains: conversations, llms, knowledge_base, arena,    │
│  │            hardware, startup, user_settings              │
│  ├─ Agents:  LangChain/LangGraph runner + checkpointer      │
│  ├─ Engines: MLX_Engine / CUDA_Engine / CPU_Engine          │
│  ├─ Database: embedded PostgreSQL + pgvector (SQLAlchemy)   │
│  └─ Ingestion: document reader, chunking, e5 embeddings,    │
│                hybrid dense + keyword retrieval             │
└─────────────────────────────────────────────────────────────┘

Quick start¶

# Backend (from the repo root)
cd backend
source venv/bin/activate         # macOS/Linux; Windows: .\venv\Scripts\Activate
python run.py --port 27182

# Frontend (second terminal)
cd frontend
npm start

See Getting Started for the full setup, including the llama.cpp build step.