Architecture¶

Technical architecture of Erudi: layering, multi-engine inference, startup sequence, storage, and the main request flows.

Overview¶

The backend is a FastAPI application organised by business domain. Each domain owns its HTTP surface, its business logic, and its data access; infrastructure concerns (engines, database, ingestion, launcher) live outside the domains.

┌─────────────────────────────────────────────────────────────────┐
│ Frontend (Electron + React + Tailwind)                          │
│  - Contexts: KnowledgeBase, DownloadModal                       │
│  - services/api/client.js (retry, timeout, X-Request-ID)         │
└────────────────┬────────────────────────────────────────────────┘
                 │ HTTP on 127.0.0.1:27182, all routes under /erudi
┌────────────────▼────────────────────────────────────────────────┐
│ Backend FastAPI                                                 │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ endpoints.py — routes, Pydantic schemas, dependencies       │ │
│ ├─────────────────────────────────────────────────────────────┤ │
│ │ services.py — business logic, orchestration                 │ │
│ ├─────────────────────────────────────────────────────────────┤ │
│ │ repository.py — data access                                 │ │
│ ├─────────────────────────────────────────────────────────────┤ │
│ │ entities/*.py — SQLAlchemy models                           │ │
│ └─────────────────────────────────────────────────────────────┘ │
├─────────────────────────────────────────────────────────────────┤
│ Infrastructure                                                  │
│  - engines/    MLX / CUDA / CPU behind BaseEngine               │
│  - agents/     LangChain agent runner + LangGraph checkpointer  │
│  - ingestion/  document reader, chunking, embeddings, vectors   │
│  - database/   embedded PostgreSQL + pgvector, seeding          │
│  - launcher/   runtime paths, Postgres runtime, DB watchdog     │
└─────────────────────────────────────────────────────────────────┘

Domains¶

Routers are mounted under the /erudi prefix in register_routers (backend/src/core/api.py): llms, hardware, arena, knowledge_base, conversations, health, startup, user_settings.

1. Conversations¶

• Responsibility: chat sessions, streaming generation, message persistence
• Endpoints: POST /erudi/conversations/, GET /erudi/conversations/, POST /erudi/conversations/{id}/query, PATCH /erudi/conversations/{id}, GET /erudi/conversations/{id}/fetch_messages, POST /erudi/conversations/{id}/generate_title, POST /erudi/conversations/star_message and /unstar_message
• Entities: Conversation, Message
• Conversation state: held by the LangGraph checkpointer, not rebuilt from the message table; older turns are compacted by a summarization middleware when the conversation reaches its compaction threshold (a share of the working context window, 50-90 %, 80 % by default; the window is the smaller of the allocated window and the Apple-Silicon memory ceiling; see the Conversations guide); the thread shows each compaction while it runs and marks where it took over (compaction events)
• Reference

2. LLMs¶

• Responsibility: catalog, downloads, deletion, KB assistant binding
• Endpoints: GET /erudi/llms/, GET /erudi/llms/search?name=, GET /erudi/llms/search/huggingface, POST /erudi/llms/{llm_id}/download, POST /erudi/llms/download/huggingface, the downloads/* job routes, DELETE /erudi/llms/{id}, GET /erudi/llms/{llm_id}/dependents, POST /erudi/llms/{assistant_id}/rebind
• Entities: Llm, DownloadJob
• Features: Hugging Face integration, download of pre-built quants only, KB attachment
• Reference

3. Knowledge Base¶

• Responsibility: document ingestion and hybrid retrieval
• Endpoints: POST /erudi/knowledge_base/create, GET /erudi/knowledge_base/{llm_id}/status, GET|POST /erudi/knowledge_base/embedding-model/status|download
• Entities: KnowledgeBase, KnowledgeDocument, KBJob; chunks live in the rag.kb_chunks table
• Formats: .pdf, .docx, .xlsx, .csv, .txt, .md. Images are accepted and recorded as pending_vision — no OCR tier is bundled
• Deletion: there is no KB delete route; a KB assistant is removed through DELETE /erudi/llms/{llm_id}
• Reference

4. Arena¶

• Responsibility: comparing models on the same prompt
• Endpoint: POST /erudi/arena/{llm_id}/query (one call per model, plain-text stream)
• Reference

5. Hardware¶

• Responsibility: hardware profiling and performance scoring
• Endpoints: GET /erudi/hardware/app_startup, GET /erudi/hardware/detailed, POST /erudi/hardware/refresh
• Entities: HardwareProfile
• Reference · Hardware guide

6. Startup and user settings¶

• GET /erudi/startup/welcome-popup drives first-run UI state (StartupVariables).
• GET|PUT /erudi/user_settings/ reads and writes the UI language, the global web-search default, the automatic-update preference, the default reasoning effort and the default compaction threshold (UserSettings). See Internationalization.
• Connectivity for the status pill is read in the renderer from navigator.onLine and the online / offline events, corrected by requests that fail on the wire (frontend/src/utils/networkStatus.js). No request is made to answer it.

Multi-engine architecture¶

The uniform "subprocess + OpenAI-compatible HTTP" pattern¶

The three inference engines follow the same pattern: they spawn an OpenAI-compatible HTTP server in a child process and talk to it over SSE at http://127.0.0.1:<port>/v1/chat/completions. That uniformity is what lets the agent layer address all three through a single ChatOpenAI(base_url=...) client, with no per-backend code.

            ┌───────────────────────────────────┐
            │ FastAPI backend (parent process)  │
            │   src/engines/<engine>.py         │
            │      └─ requests.post(stream=True)│
            └────────────────┬──────────────────┘
                             │  HTTP SSE
                             ▼
            ┌───────────────────────────────────┐
            │ OpenAI-compatible HTTP server     │
            │   /v1/chat/completions  /health   │
            │   (child process)                 │
            └───────────────────────────────────┘

How the child is launched differs:

• CPU/CUDA: subprocess.Popen([llama-server, ...]) — a native binary at backend/artifacts/llama-cpp/<cpu|cuda>/bin/llama-server.
• MLX: multiprocessing.Process(target=run_mlx_vlm_server, args=(argv,)). There is no native binary on the MLX side, and a PyInstaller frozen build has no Python interpreter at sys.executable to pass -m to, so the process is spawned with mp.spawn (configured in backend/run.py via mp.freeze_support() and set_start_method("spawn", force=True)), which re-executes the launcher in child mode.

Shared lifecycle (port pick, two-stage /health + chat-ping probe, SSE byte-buffer parser, atexit storage, last-used stamp for the idle cleanup, kwarg translation) lives in BaseChatServerEngine; BaseLlamaCppEngine factors what is specific to llama-server. See Engines Architecture.

Available engines¶

The FastAPI backend itself binds 27182 and scans 27182-27199, stopping short of 27200 so the launcher and the inference pools can never fight over a port. Erudi's whole port footprint is 271xx-273xx. See Backend Launcher.

Automatic selection¶

At startup the engine class is selected by BaseEngine.get_engine() (backend/src/engines/base_engine.py):

1. macOS ARM (platform.system() == "Darwin" and "arm" in platform.machine()) → MLX_Engine
2. Linux/Windows with an NVIDIA GPU (pynvml.nvmlDeviceGetCount() > 0) → CUDA_Engine
3. Otherwise → CPU_Engine

ERUDI_FORCE_CPU=1 short-circuits detection entirely and returns CPU_Engine.

Detection is not the last word. Once the schema is migrated, the lifespan reads user_settings.inference_backend and replaces CUDA_Engine with CPU_Engine when the user pinned the processor — safe because both carry FORMAT_TAG = "gguf", so the catalog is identical either way. On the engine that actually wins, a CUDA pre-flight compares the GPU's compute capability and the driver's CUDA version to the floors of the bundled build (src/engines/cuda_compatibility.py) and emits an engine_notice event when the machine cannot run it. The event is a proposal, never an action. See Hardware Detection.

Engines are never instantiated: they expose classmethods only, and the selected class is stored on src.core.config.LLM_Engine.

Agent layer (src/agents)¶

Conversations and Arena share one streaming primitive, AgentRunner (backend/src/agents/runner.py), built on LangChain's create_agent:

• Conversations run with a thread_id, summarization enabled, and the LangGraph checkpointer, so history is restored from the checkpointer (only the new message is sent) and older turns are summarized in the agent state. Compaction fires on token pressure alone (the conversation's compaction threshold, 80 % of the working window by default), keeps a token budget of that window, and caps its summary; only when no window is known does a message count (20 messages, keep the last 10) stand in (see the Conversations guide). On Apple Silicon, when the memory ceiling is smaller than the model's trained window, the inference child itself enforces it (see Engines); on the CPU engine, the window Erudi passes to llama-server is already the one the machine's RAM affords. The amber memory notice about that bound appears only when memory is what bounds the window — and even the projected next request fills 85 % of it. Under memory pressure the memory brake (below) moves that bound, and its own notice takes precedence.
• Token counts are real tokens without a tokenizer in the backend (agents/token_accounting.py), costed message by message: every client streams the server's usage, and the chunk that carries it is stamped with the chars / 4 estimate of the request it answers (request_tokens_est, tool schemas included). What was inside the last measured request costs its measured ratio, the answers the server generated their exact output tokens, new text a script weight of its own (real_tokens_est). The output budget, the compaction counter and the amber-warning projection all read it; the first-chunk watchdog keeps its own one-token-per-byte upper bound (see Streaming budgets).
• Arena runs stateless: no thread_id, no summarization, no checkpointer.

Each conversation or Arena turn claims the engine's prefix cache for itself (claim_prefix); on Apple Silicon the cache holds one conversation at a time and is reset on a conversation change and after a compaction (see Engines). The generators between the service and LangGraph are consumed under contextlib.aclosing, and the runner iterates LangGraph itself from a child task (agents/isolated_stream.py). Starlette delivers a client disconnect as an anyio cancellation re-delivered at every await, which would abort LangGraph's own teardown and leave the model call streaming; the child task gets a single native cancellation instead, and the runner waits for it, shielded. Whether the consumer closes the stream or a disconnect cancels it, the in-flight model call is closed before the generation guard is released. Starlette never closes a body iterator itself: a disconnect noticed while a chunk is being sent leaves the service generator suspended, still holding the guard, until its finalization closes it the same way. Titles have no LangGraph: a title cut short flags the inference child, so the next prefix-cache reset sends a barrier first.

A turn is captured as structured events — answer, thinking, tool_call, tool_result — which the conversation service frames as NDJSON. Arena projects the same stream down to answer text only, preserving its plain-text contract.

Other modules in the layer:

Streaming budgets¶

A local model goes quiet for two unrelated reasons, so each model call carries two wall-clock budgets instead of langchain-openai's single stream_chunk_timeout (which the client switches off):

The ceiling derives from the loaded child's allocated context window (effective_context_tokens, stamped on the client per turn by the model factory): max(900 s, 30 s + window / 25), so a legitimately full window is never killed mid-prefill — with the dynamic context window, a flat 900 s would recreate the very kill the two budgets fixed. An unknown window keeps 900 s.

The prompt size is bounded, not averaged: one token per UTF-8 byte of text, plus a flat allowance per message and per image part. A byte-level tokenizer cannot emit more tokens than the text has bytes, so the bound holds for every language — where a character-based heuristic would under-count Chinese or Japanese threefold and end a healthy turn on exactly the machines this budget exists for. The bound is loose on English, so long histories reach the ceiling; that only delays the detection of a true hang. Both budgets apply per model call, so each hop of a tool-calling turn gets its own.

Either budget expiring raises GenerationTimeoutException, which the runner turns into one WARNING (phase, budget, estimated prompt size) and an error turn that names the cause: the model never started on a prompt this size, or it started and went silent.

Startup sequence (lifespan)¶

backend/src/core/api.py:lifespan runs, in order:

1. Embedded PostgreSQL — start_postgres(config.POSTGRES_DATA_DIR). On a first run this pays a one-time initdb; the phase event preparing_database is emitted so the Electron loader can say so.
2. Bind SQLAlchemy — init_database(app.state.postgres.sqlalchemy_url).
3. Engine selection — config.LLM_Engine = BaseEngine.get_engine(). Detection only; the persisted preference is applied at step 6, once the column it lives in exists.
4. Schema migration — Alembic, forward-only, run off the event loop (run_in_threadpool(run_migrations, ...)). Phase event: running_migrations. See Database migrations.
5. Seed and catalog — startup_populate_database() sets startup variables, the hardware profile, and reconciles the model catalog from the bundled snapshots. Phase event: loading_catalog.
6. Engine preference and CUDA pre-flight — apply_inference_backend_preference() swaps CUDA_Engine for CPU_Engine when the user pinned the processor, then emit_engine_notice(app) runs the pre-flight on the engine that won and emits an engine_notice event if the GPU cannot drive the bundled CUDA build.
7. KB vector store — init_kb_store(...), after the migration because its cross-schema foreign keys reference the business tables.
8. Checkpointer — open_checkpointer(...) on the same database, kept open for the app lifetime and published on app.state.checkpointer.
9. Idle cleanup task — config.LLM_Engine.start_cleanup_task(); the monitor ticks every 30 s and unloads the model once it has been idle longer than _max_idle_time (300 s), so about 300-330 s after its last use.
10. Database watchdog — start_watchdog(app) detects a dead embedded cluster, resurrects it, and exposes the state on /erudi/health/ as db.
11. Post-ready backfill — a background task verifies the tool-call wire capability of models downloaded before that column existed. It runs after readiness, in a threadpool, never inside the awaited boot sequence.

Shutdown reverses the order: cancel the backfill task, stop the watchdog, stop the cleanup task, clean up the engine, close the checkpointer, close the KB store, and stop the PostgreSQL cluster last.

Startup phase events are informational. Readiness is the launcher's ready event or a confirming health check — never a phase. engine_notice is informational too: it reports a machine that cannot run the GPU build, and the app is otherwise healthy, so the frontend holds it until after ready and shows it as a decision rather than an error screen.

Catalog snapshots¶

The remote catalog (which quant exists for each foundation model in a given engine format) is identical for every user, so it is resolved once at build time and shipped as JSON: backend/src/database/catalog_snapshot_mlx.json and catalog_snapshot_gguf.json. First boot loads the snapshot for the active engine's FORMAT_TAG with zero Hugging Face calls. Regeneration is a build-time CLI:

cd backend
python -m src.database.catalog_snapshot                     # active engine (mlx on Mac)
ERUDI_FORCE_CPU=1 python -m src.database.catalog_snapshot   # gguf

Code conventions¶

Naming¶

• snake_case for variables, functions, files, directories
• Capitalized_Snake_Case for classes (MLX_Engine, CUDA_Engine)
• UPPER_SNAKE_CASE for constants
• Absolute imports: always from src.core.config import ...

Domain structure¶

backend/src/domains/<domain>/
├── __init__.py
├── endpoints.py      # FastAPI routes
├── schemas.py        # Pydantic models
├── services.py       # business logic
└── repository.py     # data access (optional)

Request flow:

HTTP request
   ↓
endpoints.py (Pydantic validation)
   ↓
services.py (business logic)
   ↓
repository.py (DB queries)
   ↓
entities/*.py (SQLAlchemy models)

Error handling¶

All business exceptions inherit from AppBaseException (backend/src/core/exceptions.py), which carries a message, an HTTP status code, an erudi_code, an optional trace (logged, never returned), and an optional detail payload returned verbatim to the client.

from src.core.exceptions import ModelNotFoundException, StateConflictException

raise ModelNotFoundException("qwen2.5-7b")            # 404 MODEL_NOT_FOUND
raise StateConflictException(                          # 409 STATE_CONFLICT
    "Base model has dependent assistants",
    status_code=409,
    detail={"dependents": [...]},
)

Two handlers are registered in add_exception_handlers (core/api.py): app_base_exception_handler for AppBaseException, and unhandled_exception_handler for bare Exception. Both return the same shape:

{
  "success": false,
  "error": {
    "type": "MODEL_NOT_FOUND",
    "message": "Model 'qwen2.5-7b' not found",
    "detail": { "…": "present only when the exception carries one" }
  }
}

See Exception Handling.

Structured logging¶

from src.core.logging import logger

logger.info("Model loaded")

Log-message literals must stay ASCII-only. Logs are written to backend/logs/backend.log in development (rotating file, 10 MB, 10 backups). Packaged builds write to the OS log directory instead — macOS: ~/Library/Logs/erudi/backend.log, Windows: %LOCALAPPDATA%\erudi\logs\backend.log, Linux: ${XDG_STATE_HOME:-~/.local/state}/erudi/logs/backend.log. See Logging & Traceability.

Persistence¶

Storage is an embedded PostgreSQL cluster with pgvector, provided by pgserver as pip wheels — no Docker, no system install. The lifespan boots the cluster (src/launcher/postgres_runtime.py; data directory backend/data/postgres/ in development, a user-writable directory in packaged builds), creates the erudi database and the vector extension, then binds SQLAlchemy through init_database(url) over psycopg3 (postgresql+psycopg://).

Never import db_engine by value — read it through src.database.core attributes after initialization.

One database, three tenants:

Main entities¶

• Llm — catalog and local models (name, link, local, quantized, param_size, is_base, supports_tools, KB binding)
• Conversation — chat sessions (llm_id, temperature, top_p, max_tokens, custom_prompt)
• Message — individual messages (conversation_id, sender, content, starred)
• KnowledgeBase — KB metadata and status
• KnowledgeDocument — one row per ingested file, with its SHA-256 for dedup and a status (ingested, empty, pending_vision, …)
• KBJob — ingestion jobs
• DownloadJob — download jobs (status, progress, total bytes)
• HardwareProfile — detected hardware and its performance scores
• StartupVariables, UserSettings — first-run flags and UI language

See Entities Reference.

PostgreSQL sequences are non-transactional, so tests never assert absolute primary-key values.

Retrieval-augmented generation¶

Ingestion pipeline (src/ingestion)¶

Source files (.pdf .docx .xlsx .csv .txt .md; images → pending_vision)
   ↓  reader.py — DocumentReader routes to a per-format extractor
Markdown pivot
   ↓  cleaning.py — non-destructive cleaning (accents and casing preserved)
   ↓  chunking.py — 3-pass, token-accurate on the e5 tokenizer
Chunks (~180 tokens, ~15 % overlap)
   ↓  embeddings.py — intfloat/multilingual-e5-small, 384 dimensions
   ↓  vector_store.py — add_kb_chunks into rag.kb_chunks
KnowledgeDocument rows (SHA-256 dedup, per-file status)

Chunk defaults live in backend/src/ingestion/chunking.py (DEFAULT_TARGET_TOKENS = 180, DEFAULT_OVERLAP_TOKENS = 27).

The query: and passage: prefixes required by the e5 model family are mandatory and applied by E5Embeddings; skipping them silently degrades retrieval.

DocumentReader has a second caller: backend/src/utils/attachment_utils.py, which turns the documents and folders attached to a single chat or arena question into delimited text blocks for that turn. It stops at the reader — no chunking, no embedding, no storage — so the app keeps exactly one document parser.

Retrieval¶

rag.kb_chunks carries both a dense HNSW index (cosine over 384-dim embeddings) and a sparse tsvector column (pg_catalog.simple). A query runs both and fuses the two rankings with Reciprocal Rank Fusion (k = 60).

Always search through search_kb_chunks: langchain-postgres 0.0.17 freezes the first query's fts_query on the shared hybrid config, so each call needs a fresh config.

The conversation-side entry point is retrieve_kb_excerpts (backend/src/utils/kb_utils.py).

Whether the model retrieves by calling the search_knowledge_base tool (agentic) or the service retrieves on every turn (systematic) depends on the model's tool-calling support; ERUDI_KB_AGENTIC forces either mode.

See the Knowledge Base guide.

Context budget by model size¶

get_prompting_strategy(param_size) (backend/src/utils/prompt_utils.py) returns the system-prompt tier and the KB token budget — a ceiling in e5 tokens (roughly 180 per chunk), not a chunk count. The adaptive cut in kb_utils decides per query how much of the budget to consume.

Performance notes¶

Idle model cleanup¶

BaseEngine keeps a single model in memory as class state. The cleanup monitor started in the lifespan ticks every 30 seconds and unloads the model once it has been idle longer than _max_idle_time (300 seconds), so about 300 to 330 seconds after its last use. A tick with nothing to reap returns without touching the lock; one that finds an idle model takes the same lock as generation_guard, so it waits for a generation in flight to release it — and the guard refreshes _last_used on release — which is what keeps a model from being reaped mid-stream.

The memory brake (Apple Silicon and CPU)¶

Under memory pressure the inference child gives memory back, at the start and the end of every turn and every minute while it is idle (a second lifespan monitor, under the same lock), never during an answer. On Apple Silicon the MLX child shrinks the working window so the next turn compacts earlier, releases the prefix cache and MLX's buffer cache, and, under critical pressure while the user is away, unloads the model, which the next message reloads with the reduced window (memory_reload on the wire; the idle timeout's own unload sends the same event with reason: "idle"). On the CPU engine (llama-server, whose KV cache is allocated in full at load) the brake restarts the same model with a smaller window — at once under critical pressure, after two minutes of user idle under warning — or unloads it; the turn that re-reads the conversation shows the same reload state. The actions are bounded per pressure episode; see Engines and the CPU brake. On the CUDA engine the system's memory pressure does not drive the brake: only a Windows spill of the model out of the card's memory into system memory does, with the same restart or unload, and on Linux it never acts (see the CUDA brake).

Streaming¶

Conversation turns stream as NDJSON (application/x-ndjson): one JSON event per line, typed answer, thinking, tool_call, tool_result, compaction, memory_reload, memory_warning, error, or done. Title generation and Arena keep a plain-text stream.

Main flows¶

Conversation turn¶

POST /erudi/conversations/{id}/query  {"question": "..."}
  ↓
endpoints.query_and_respond → StreamingResponse(application/x-ndjson)
  ↓
ConversationService.query_and_respond_stream
  ↓
AgentRunner: create_agent(thread_id, checkpointer, summarization)
  ↓  history restored from the checkpointer; KB retrieval agentic or systematic
engine child server /v1/chat/completions (SSE)
  ↓
structured events → one JSON line each → renderer
  ↓
user + assistant messages persisted after the stream completes

Model download¶

POST /erudi/llms/{id}/download          (catalog)
  or POST /erudi/llms/download/huggingface {"link": ...}   (search / pasted link)
  ↓
Create DownloadJob (status=pending)
  ↓
Background task starts
  ↓
List repo files → gate: the repo MUST ship an artefact in the engine's format
(FORMAT_TAG: an MLX repo tag on Apple Silicon, a loadable .gguf file on CPU/CUDA;
GGUF weights published as raw byte chunks do not count)
  → otherwise InvalidInputException BEFORE a single byte is downloaded
  ↓
Download the pre-built quant from Hugging Face → update progress
  ↓
Move into place (no local conversion or quantization)
  ↓
Llm row local=1, DownloadJob status=completed

See the LLMs guide.

Quick reference¶

• Core Reference — config, logging, exceptions
• Engines Reference — MLX, CUDA, CPU
• Conversations Reference — chat endpoints
• LLMs Reference — model management
• KB Reference — ingestion and retrieval
• Entities Reference — SQLAlchemy models