Skip to content

model-replacement.md

Latest commit

 

History

History
74 lines (54 loc) · 2.63 KB

model-replacement.md

File metadata and controls

74 lines (54 loc) · 2.63 KB
Raw

Model Replacement

All model calls are created by the backend. The React application never selects a model, receives an API key, or receives a provider connection URL.

Current OpenRouter Configuration

The project uses these OpenRouter model identifiers by default:

Logical role Environment variable Model
Question analysis QUESTION_MODEL nex-agi/nex-n2.5-pro:free
SQL generation SQL_MODEL nex-agi/nex-n2.5-pro:free
SQL correction SQL_CORRECTION_MODEL nex-agi/nex-n2.5-pro:free
Answer generation ANSWER_MODEL nex-agi/nex-n2.5-pro:free
Schema embeddings EMBEDDING_MODEL nvidia/nemotron-3-embed-1b:free

The API key belongs only in the local ignored .env file or the deployment secret manager:

OPENROUTER_API_KEY=<backend-only-secret>
OPENROUTER_BASE_URL=https://openrouter.ai/api/v1

Never commit a real key. If a key is exposed, revoke it in OpenRouter and issue a replacement.

Replacing A Chat Model

Change one or more role variables in .env without changing Python code:

QUESTION_MODEL=provider/analysis-model
SQL_MODEL=provider/sql-model
SQL_CORRECTION_MODEL=provider/correction-model
ANSWER_MODEL=provider/answer-model

The same model may be assigned to multiple roles. Role boundaries remain explicit even when the configured identifiers are identical. Restart the backend after changing model settings.

The workflow still applies structured parsing, deterministic SQL validation, approval gates, retry bounds, and read-only database authorization regardless of which chat model is configured.

Replacing The Embedding Model

Change EMBEDDING_MODEL, then rebuild the local schema index:

docker compose run --rm backend index-schema

This is required because vectors generated by different embedding models are not interchangeable. The command introspects the external source database and writes only to the separate local index database. It never changes the source business schema.

Timeouts And Failures

Chat requests use MODEL_REQUEST_TIMEOUT_SECONDS. Embedding requests use EMBEDDING_REQUEST_TIMEOUT_SECONDS. Provider timeouts, unavailable models, and malformed structured responses become safe backend error categories. Raw provider responses, stack traces, credentials, and connection URLs are not returned to the frontend.

Verification

The model factory and service boundaries accept injected fake models. The backend test suite verifies distinct model assignments, shared model assignments, model role routing, correction routing, malformed output handling, and safe failures without making OpenRouter requests.