ensimpl

Fast local Ensembl gene annotation. Data is extracted from Ensembl BioMart and built into one self-contained SQLite database per (release, species), with an FTS5 index for search.

Built because Ensembl's own APIs are slow and require many calls; ensimpl returns exactly the data you need in one fast local query — as a library, an HTTP API, or a web UI.

Requirements

• Python 3.12+ and uv.
• uv sync installs everything. The extract + build core is dependency-free (stdlib urllib + csv + sqlite3); only the HTTP API adds fastapi + uvicorn.

Quick start

uv sync
uv run ensimpl build --release 116 --species mm --url https://jun2026.archive.ensembl.org -v
uv run ensimpl serve --dir dbs

Then open http://127.0.0.1:8000/ for the web UI, or /docs for the API reference.

Build databases

Each build extracts one (release, species) from a BioMart host and writes dbs/ensimpl.<release>.<species>.db. Nothing about releases is hardcoded — you pass the mart host with --url, so any release is buildable.

Ensembl retired BioMart in July 2026. https://www.ensembl.org/biomart/martservice briefly 404ed and now 308-redirects to jun2026.archive.ensembl.org, the last BioMart archive. Builds against www therefore still work — but they silently resolve to release 116 forever, whatever --release you pass. Name the archive host explicitly so the URL you record is the release you got. Everything through 116 is reproducible; 117 onward is not, by this method. What replaces it is in docs/DATA_SOURCES.md.

Current release (116) from the last BioMart archive:

uv run ensimpl build --release 116 --species mm --url https://jun2026.archive.ensembl.org -v

Species are mm (mouse), hs (human), rn (rat) — build each one you want to serve:

uv run ensimpl build -r 116 -s hs --url https://jun2026.archive.ensembl.org
uv run ensimpl build -r 116 -s rn --url https://jun2026.archive.ensembl.org

Older releases come from Ensembl archive hosts — the subdomain for that release:

# release 112 == the May 2024 archive
uv run ensimpl build -r 112 -s mm --url http://may2024.archive.ensembl.org -v

The protocol is load-bearing, and it is not the same for every archive. Releases up to 113 answer on http:// and hang on https://; the 2025 archives (114, 115) are https:// only and time out on http://. Using the wrong one looks like a network fault rather than a configuration error, so take the exact URL from docs/ensembl_assemblies.08202026.csv (archive_url, with valid_url recording whether it still answers) rather than assembling it by hand. The per-release URLs actually used are listed in docs/DATABASES.md.

For cross-release history, build two or more releases of a species into the same directory — e.g. 112 + 116 for mouse lets /{species}/history/{id} show Pcnx → Pcnx1.

Notes:

• Extraction is BioMart-bound — the exon download is the slow part (minutes); building from cached TSVs afterward takes seconds.
• A build either produces a complete database or none at all. BioMart can answer 200 OK with a truncated body, which would otherwise load a group as zero rows and still finish successfully — that is how one release once ended up with no orthologs while reporting a clean build. Before finalizing, the build compares what loaded against the extracted TSVs: if a group had usable rows and its table is empty, it raises IncompleteBuildError and deletes the partial file, so "the file exists" stays a signal you can trust. An absence the extract never supplied (release 113 has no rat dataset, so no rat orthologs) is a fact about the release, not a failure — the build succeeds and records it in missing_data.
• --refresh re-downloads cached TSVs; --extract-only downloads without building; --cache DIR and --out DIR change the download/output locations (defaults cache/, dbs/).
• Provenance for the databases already built — the URL and command behind each one, and the archive URL map — is in docs/DATABASES.md, with the machine-readable inventory dbs/ensimpl_databases.MMDDYYYY.tsv, generated from the databases themselves with uv run python scripts/dump_db_meta.py --dir dbs.
• The genome build behind every release — assembly and patch for human, mouse and rat across releases 54–116, which archive serves each one and whether it still answers — is docs/ensembl_assemblies.08202026.csv, parsed from Ensembl's table of assemblies. It is dated because none of it is permanent — Ensembl retires archives, which is what valid_url and last_checked record. GRCm39 is not an exact answer; GRCm38 vs GRCm38.p6 is the difference between two releases.
• Where annotation comes from now that BioMart is gone — the GraphQL evaluation, the FTP successor, and the recommended rewrite — is in docs/DATA_SOURCES.md.

Run the server

uv run ensimpl serve --dir dbs --host 127.0.0.1 --port 8000 -v

• / — the web UI: live typeahead search, gene detail (with an exon mini-track), the cross-release lineage timeline, a region/karyotype browser, a bulk-annotation download (CSV/TSV/JSON), and external-ID / ortholog resolvers. Served same-origin (no CORS needed) and base-path aware behind a proxy. It follows your system light/dark theme, with a manual toggle in the top bar (remembered per browser).
• /docs — interactive API reference (Scalar, vendored locally). /redoc — ReDoc. A static, browsable version — no server needed — is in docs/API.md (generated from the OpenAPI schema; regenerate with uv run python scripts/gen_api_docs.py).
• Availability is whatever .db files are present in --dir — no manifest. Add or remove a file and restart.

Endpoints are release-first — /{release}/{species}/…: search, gene/{id}, genes (POST batch), region/{chrom}/{start}/{end}, regions (POST batch), export, exon_info, chromosomes, external_sources, ortholog_species, externalids/resolve (POST), orthologs/resolve (POST) — plus the cross-release /{species}/history/{gene_id}, and the top-level /genomes and /health.

Response shaping, export & caching

• fields= — trim any gene response to just the keys you need, e.g. /116/mm/search?q=Naxd&fields=gene_id,symbol,start,end, or POST /genes with "fields": "gene_id,external_ids" for a compact id→external-id table. Nested blocks are opted in first (xrefs, subfeatures), then selected by name.
• region … ?exons=true — attach each gene's exon intervals, so a locus gene-track is one call instead of a region query plus a whole-chromosome exon export. Add cds=true for the coding intervals too, which is what lets a track draw translated stretches thick.
• Coding regions say when they are missing. A transcript's cds/utr are absent (not empty) on a database built without them, so an empty list means "this transcript does not code" and a missing key means "this build has no coding data". You do not have to infer that: gene, region and POST /genes all report meta.has_coding, /genomes carries the same flag per release, and the database's own meta.missing_data names cds when it lacks them — so the gap is visible from a response, from a listing, and from the file offline.
• format= — csv, tsv, or json on search, region, genes, and history downloads the rows as a file (the default, no format, returns the JSON envelope).
• Bulk export — GET /{release}/{species}/export downloads a whole release (or a chromosome, or a range via chrom/start/end) as a stream. detail=simple (default) is one flat row per gene — the spreadsheet shape, csv or tsv, ~14 MB for release 116 mouse. detail=full is the complete record per gene including transcripts, exons and proteins, as NDJSON (one gene per line, ~800 MB for a release) so it can be read incrementally. The response is streamed and un-enveloped: memory stays flat and the first byte arrives in milliseconds regardless of size.
• Bulk annotation by id — POST /genes is the annotation-download endpoint for a list of gene ids. format=csv/tsv returns a flat, spreadsheet-ready annotation table — one row per gene with the scalar fields plus synonyms and a column for each external-ID source (MGI, Entrez, UniProt, RefSeq, …) and ortholog species — to merge into a dataset. format=json (or the default envelope) returns the full nested record; add subfeatures=true for transcripts and exons.
• Bulk lookup by location — POST /regions takes a list of chrom:pos (a SNP) or chrom:start-end strings, mixed freely, and returns the genes for each, e.g. {"regions": ["8:70000000", "8:11500000-11600000"], "flank": 50000, "fields": "symbol"}. JSON groups genes under each input region; format=csv/tsv gives one row per region/gene pair. Regions with no genes are listed in meta.no_genes. flank widens each region, since most GWAS hits are intergenic. Max 10,000 regions per request; more returns a 422 stating the limit, so split into batches. Coordinates only — no rs numbers.
• Immutable caching — per-(release, species) data never changes, so those responses carry Cache-Control: immutable and a weak ETag. A conditional re-request (If-None-Match) gets 304 without touching the database. A rebuilt database gets a new created stamp, which changes the ETag and invalidates caches.

History (across releases)

GET /{species}/history/{gene_id} traces a gene through every release present in the serving directory (optionally bounded by start/end), reading the stable Ensembl ID from each. It surfaces the common case where the ID holds but the symbol moves — e.g. Carkd → Naxd — returning one record per release plus meta.symbols, the distinct symbols seen in order. More release DBs in the directory ⇒ deeper history; nothing else to configure. (It does not follow ID retirement/merge/split — the ID must be stable across the releases.)

Query a database directly (library)

Each .db is a self-contained, shippable artifact you can query without the server — the same engine the API uses:

from ensimpl import query

conn = query.connect("dbs/ensimpl.116.mm.db")     # opens read-only + immutable
hits = query.search(conn, "Naxd", limit=5)         # ranked FTS5 search -> list[dict]
gene = query.gene(conn, "ENSMUSG00000031505")      # one gene + xrefs + transcripts/exons
genes = query.region(conn, "8", 11_500_000, 11_600_000, exons=True)

Other functions: genes, chromosomes, external_sources, ortholog_species, resolve_external_ids, resolve_orthologs, exon_info, meta. All return plain dicts/lists.

Deploy (Docker behind a reverse proxy)

The image contains only the app; databases are mounted read-only at runtime, so one image serves any set of releases. Copy .env.example to .env and adjust, then:

docker compose up --build -d      # local: build from source and run

For a server, compose.prod.yaml pulls a published image and never builds, so the host needs no source checkout — just that file, a .env, and the databases:

docker compose -f compose.prod.yaml pull
docker compose -f compose.prod.yaml up -d

Behind a path-stripping proxy (e.g. Apache under /ensimpl), set ENSIMPL_BASE_PATH to that prefix so generated URLs stay correct; see apache-ensimpl.conf for the matching proxy config. Responses are gzipped and CORS is open by default (ENSIMPL_CORS_ORIGINS to restrict).

docs/DEPLOYMENT.md is the full walkthrough — local setup, building databases, testing, running the server (foreground, background, systemd), building and publishing multi-architecture images, copying databases to a server, the Apache configuration, start/stop and update/rollback, and troubleshooting.

Configuration (environment variables)

The server reads these at startup, so it is configurable without code changes. ensimpl serve sets ENSIMPL_DB_DIR from --dir (and ENSIMPL_LOG_LEVEL from -v); in Docker they come from .env / compose.yaml.

Variable	Default	Purpose
ENSIMPL_DB_DIR	dbs	Directory of .db files to serve (availability = files present).
ENSIMPL_BASE_PATH	(empty)	Reverse-proxy prefix the service runs under (sets FastAPI root_path); empty at the domain root.
ENSIMPL_CORS_ORIGINS	(empty = all)	Comma-separated allowed CORS origins for browser clients.
ENSIMPL_LOG_LEVEL	INFO	DEBUG / INFO / WARNING / … ; access logging emits at INFO.
ENSIMPL_CLIENT_HEADER	X-API-Key	Request header naming the caller in the access log (recorded, not validated).

CLI reference

ensimpl build   Extract one (release, species) from BioMart and build its database.
  -r, --release N     Ensembl release, e.g. 116            (required)
  -s, --species CODE  hs | mm | rn                         (required)
      --url URL       Mart host, e.g. https://www.ensembl.org
                      (archives: http://<month><year>.archive.ensembl.org)   (required)
      --cache DIR     Directory for downloaded TSVs         (default: ./cache)
      --out DIR       Directory for the built database       (default: ./dbs)
      --refresh       Re-download cached TSVs
      --extract-only  Download TSVs but do not build
      --ftp-url URL   FTP root for the coding-region GTF
                                          (default: https://ftp.ensembl.org/pub)
      --no-coding     Skip the coding-region extract; build without CDS
      --chrom NAME    Restrict coding regions to one chromosome (validation only)
  -v, --verbose       Increase log verbosity (repeatable)

ensimpl add-coding   Add coding regions (CDS) to an already-built database.
      --db PATH       Database to supplement                  (required)
      --out PATH      Write a supplemented copy here (directory or filename)
      --in-place      Modify --db itself (required to write to the input)
      --ftp-url URL   FTP root      (default: https://ftp.ensembl.org/pub)
      --cache DIR     Directory for downloaded files          (default: ./cache)
      --chrom NAME    Restrict to one chromosome (validation only)
      --refresh       Re-download cached files
  -v, --verbose       Increase log verbosity (repeatable)

ensimpl serve   Serve the databases (and web UI) over HTTP.
      --dir DIR       Directory of databases to serve        (default: ./dbs)
      --host HOST     Bind host                               (default: 127.0.0.1)
      --port PORT     Bind port                               (default: 8000)
      --reload        Auto-reload on code changes (development)
  -v, --verbose       Increase log verbosity (repeatable)

ensimpl sync-meta   REPAIR TOOL — not part of a normal workflow.
      --dir DIR       Directory of databases                  (default: ./dbs)
      --manifest TSV  Inventory to apply    (default: newest ensimpl_databases.*.tsv in --dir)
      --dry-run       Report changes without writing
  -v, --verbose       Increase log verbosity (repeatable)

build and the porting scripts already write complete metadata, so sync-meta exists only to repair a set whose metadata predates the current contract. If a newly created database needs it, the bug is in whatever created that database — fix it there.

add-coding writes to a copy unless --in-place is passed. The published set costs a lot of extraction time and is served from a read-only mount, so modifying one is an explicit choice rather than the default outcome of running a command. It is also idempotent: a second run replaces the coding tables rather than appending to them.

Coding regions are the one group that does not come from BioMart — they are parsed from Ensembl's FTP GTF dumps, which carry CDS with reading-frame phase, and (from release 75) ship CHECKSUMS so a truncated download is detected rather than guarded against later. GTF rather than the friendlier GFF3 because GFF3 dumps stop at release 84 while GTF exists for every release ensimpl has built, release 48 included — one format for all of them beats two parsers for one group of data. The price is the stop codon, which GTF publishes separately from the CDS; folding it back in reproduces GFF3's CDS exactly (all 9,227 chromosome-19 transcripts of mouse 116 match interval-for-interval and phase-for-phase).

The build then verifies what it loaded: every CDS interval must sit inside an exon BioMart supplied — the cross-service check — and each transcript's phases must form one reading frame. It refuses to finalize if either fails.

Only CDS is stored. UTRs are exactly "exon minus CDS", so they are derived on read. Derivation was verified against Ensembl's own five_prime_UTR/three_prime_UTR rows (which GFF3 publishes) for all 270,163 coding transcripts of mouse 116 — every one of the 1,273,235 pieces, intervals and 5'/3' labels — with zero disagreement, and measured both smaller and faster than storing them. See DATA_SOURCES.md.

Coding regions cost about 35% more on disk (mouse 116: 975 MB → 1,321 MB). On the read side they are opt-in: coding=false skips the work entirely, and asking for them roughly doubles a gene-detail call (0.172 ms → 0.287 ms, measured within one database over 8,000 random coding genes). Note that fields= does not avoid the cost — it trims after assembly.

Development

uv run pytest         # unit + API tests
uv run ruff check .   # lint

The tests build tiny fixture databases directly from schema.py and exercise the query layer and the full HTTP API (search, batch/missing, history across two releases, export, ETag caching, the served UI and docs) — no BioMart and no network. The BioMart extract/build path itself is not unit-tested (it is network-bound); it is exercised by actually building a database (see above).

Build and run the container image locally (same image used in production):

docker compose up --build -d      # serves on http://localhost:${HOST_PORT:-28222}
docker compose logs -f            # watch the access log
docker compose down

Logging

Every request logs one timed line — caller, client IP, method, path, status, duration:

access client=qtl2web ip=10.1.2.3 GET /116/mm/search -> 200 2.3ms

The caller comes from a request header (ENSIMPL_CLIENT_HEADER, default X-API-Key) that clients send to identify themselves — it is recorded, not validated. The IP is the real caller (read from X-Forwarded-For behind the proxy). Level is set by ENSIMPL_LOG_LEVEL (DEBUG/INFO/WARNING/…); access lines emit at INFO. Setting it to WARNING skips the per-request logging entirely, adding no overhead to queries.