Add babel-explorer: a CLI for querying Babel cross-references via DuckDB and NodeNorm

[gaurav]

Introduces babel-explorer, a CLI tool to query Babel intermediate files (Parquet) via DuckDB and NodeNorm. BabelDownloader handles caching and freshness, BabelXRefs handles querying, NodeNorm handles label enrichment, and cli.py wires them together with Click. Three commands: xrefs, ids and test-concord.

Endpoint configuration

Babel and NodeNorm endpoints are read from .env rather than hardcoded, so the repository ships only public URLs. BABEL_URL, BABEL_LOCAL_DIR, BABEL_CHECK_DOWNLOAD, NODENORM_URL and BABEL_ALLOW_VERSION_MISMATCH each have a matching command-line option, with precedence running flag > environment variable > .env > built-in default.

The committed .env.example carries the public Babel URL only, with a note telling Translator team members to contact the Babel developers for the Translator-specific URL.

Babel version handling

The release behind BABEL_URL is resolved from VERSION.txt, falling back to the final URL path segment for older trees that predate it, so latest/ resolves to whichever release it currently points at.

BABEL_LOCAL_DIR holds one Babel release at a time. When the release changes, last_checked is cleared from the .meta sidecars under <local_dir>/duckdb/ so the existing ETag path re-checks each cached file and re-downloads only what changed — the Parquet files themselves are never deleted, and the stored ETag is kept so an unchanged file costs one HEAD rather than a fresh multi-gigabyte download. Partial .tmp downloads are deleted, since they resume by byte offset with no If-Range validation and would otherwise splice two releases into one corrupt Parquet. This keeps Concord.parquet and Identifiers.parquet from being read together across two different Babel releases, which is the failure ETag alone does not prevent.

xrefs fails when NodeNorm's status endpoint reports a different babel_version than the Babel being queried, since labels and cliques would not match the cross-references. --allow-version-mismatch overrides it. The check runs only where NodeNorm is actually consulted — that is, under --labels. --recurse is served entirely by one WITH RECURSIVE DuckDB query and never touches NodeNorm, so it does not trigger the check.

WIP:

• We need to update the file names as per Export intermediate concords, identifiers, and metadata as Parquet files NCATSTranslator/Babel#704
• test-concord takes no --babel-url, so it cannot report which Babel release its NodeNorm is behind. Giving it one would make the before/after clique comparison self-describing, but needs a decision on what it should compare against — deferred rather than folded in here.

Blocked on Babel/NodeNorm deployments

The shipped default (BABEL_URL=https://stars.renci.org/var/babel/latest/) does not work end to end yet, because public Babel releases do not publish the DuckDB Parquet files. babel-explorer reports this explicitly rather than failing mid-download, but these still need doing:

• Build a new public Babel that includes the DuckDB files (duckdb/Concord.parquet, duckdb/Identifiers.parquet), then confirm the shipped BABEL_URL default works end to end.
• Publish the current Babel to its public endpoints.
• Update NodeNorm Dev (https://nodenormalization-sri.renci.org/) to the latest Babel. Its status endpoint currently reports 2025sep1, so xrefs --labels fails the version check against any current Babel unless --allow-version-mismatch is passed.
• Add a BABEL_URL repository secret so CI integration tests run against a Babel that publishes the Parquet files. Without it the 24 Parquet-dependent integration tests skip.

Linting

The repository had no [tool.ruff] section, so ruff ran with its default rule set and never checked import ordering. Rules are now E, F, I (import sorting) and UP (pyupgrade), with E501 left to the formatter and *.md excluded (ruff 0.16+ reformats Python inside Markdown code blocks). Line length stays at ruff's default of 88 rather than Babel's 120, which would have reflowed 12 of 15 files for no correctness gain.

CI passes --output-format github so lint failures annotate the diff inline, and keeps using uv run ruff rather than astral-sh/ruff-action — uv already resolves the ruff pinned in uv.lock, so CI and local runs share a version without extra plumbing. CLAUDE.md and README.md now say explicitly to run ruff check and ruff format before committing.

Smaller fixes folded in

• ids gains --labels, so identifier records can carry NodeNorm labels instead of only raw Parquet columns. The label lands in a nodenorm_label field rather than label, because Identifiers.parquet has a label column of its own that would otherwise overwrite it in json/tsv/csv output.
• --paths with --format json/tsv/csv is now rejected. It previously ignored the flag and emitted the full recursive cross-reference list, which looks like a successful --paths run but is not one.
• The test data directory is now removed once all xdist workers finish. addopts = "-n auto" made every run parallel, and the old teardown was guarded on a "master" worker that never exists under xdist, so data/test/ survived every run.
• .idea/ is gitignored.

Review fixes

A code review over the full branch turned up six defects, each fixed in its own commit with a regression test:

• --recurse triggered the NodeNorm version check it no longer needs. Recursion moved into a single DuckDB query, but the guard still read labels or recurse, so plain xrefs … --recurse failed outright against the public NodeNorm (still on 2025sep1) unless --allow-version-mismatch was passed.
• ids --labels silently dropped the NodeNorm label. Identifiers.parquet's own label column overwrote it when the record was flattened, so json/tsv/csv emitted the Babel label under label and lost the NodeNorm one; the console printed label= twice. The dataclass field is now nodenorm_label.
• A version change forced a full re-download instead of an ETag re-check. Deleting the .meta sidecar skips the conditional-GET path entirely, so every latest/ rollover pulled Concord (~626 MB) and Identifiers (2 GB+) in full even when byte-identical — the opposite of the documented intent. Only last_checked is cleared now.
• A stale .tmp could splice two releases into one corrupt Parquet. Resume is by byte offset with no If-Range; Ctrl-C is a BaseException and so escaped the tmp cleanup, leaving a partial file that the next run (after a release rollover) would append foreign bytes onto. .tmp files are now removed on a version change.
• The HTTP 416 fast path poisoned its own cache entry, persisting the 416 error response's headers as the file's metadata. It now HEADs for the real ones.
• The integration skip probe did not normalise BABEL_URL's trailing slash, so a slashless URL probed .../latestduckdb/Concord.parquet, 404'd, and silently skipped the whole integration suite — indistinguishable from the expected skip. An unreachable Babel server now skips rather than erroring out of the fixture.

Testing

220 unit tests pass (261 collected; the remainder are integration). Integration tests run against whatever BABEL_URL points at and skip when that release does not publish the Parquet files.

Verified against the live servers: the public default produces the missing-Parquet error; the internal latest/ resolves to 2026jul22 via VERSION.txt; a 2025nov19 → 2026jul22 flip expires the .meta sidecars while leaving Concord.parquet in place; the NodeNorm skew check fires before any download; and a full xrefs query returns cross-references end to end.

[Copilot]

Pull request overview

This pull request implements a basic version of babel-explorer in Python using the uv package manager. It's a tool for querying Babel intermediate files to understand why biological/chemical identifiers are considered equivalent. The implementation includes a downloader for large Parquet files with MD5 validation and resume support, NodeNorm API integration for label enrichment, DuckDB-based cross-reference querying, and a Click-based CLI.

Changes:

• Initial project structure with uv-based package management (pyproject.toml, Python 3.11+)
• Core functionality: BabelDownloader with streaming downloads and MD5 validation, NodeNorm API client with LRU caching, BabelXRefs for DuckDB-based Parquet queries
• CLI with three commands: xrefs, ids, and test-concord
• Comprehensive test suite with 80 tests split between unit tests (mocked) and integration tests (real network calls)

Reviewed changes

Copilot reviewed 15 out of 19 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
pyproject.toml	Project configuration with dependencies (click, duckdb, requests, tqdm) and pytest markers
.python-version	Specifies Python 3.11 requirement
.gitignore	Excludes /data directory for downloaded files
README.md	User documentation with setup, usage examples, and testing instructions
CLAUDE.md	AI assistant guidance documentation (contains outdated wget reference)
src/babel_explorer/cli.py	Click-based CLI with xrefs, ids, and test-concord commands
src/babel_explorer/core/downloader.py	Streaming file downloader with MD5 validation and resume capability
src/babel_explorer/core/nodenorm.py	NodeNorm API client for identifier normalization
src/babel_explorer/core/babel_xrefs.py	DuckDB-based cross-reference query engine (has frozen dataclass bug)
tests/conftest.py	Session-scoped pytest fixtures for shared test resources
tests/constants.py	Shared test constants and CURIE loader utility
tests/data/valid_curies.txt	Parametrized test data (one CURIE)
tests/test_downloader.py	26 tests for BabelDownloader (22 unit, 3 integration, 1 slow)
tests/test_nodenorm.py	23 tests for NodeNorm (18 unit, 5 integration)
tests/test_babel_xrefs.py	31 tests for BabelXRefs (22 unit, 8 integration, 1 slow)

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 20 out of 25 changed files in this pull request and generated 4 comments.

Suppressed comments (7)

tests/test_babel_xrefs.py:400

• get_curie_xref is now a regular method backed by BabelXRefs._xref_cache; it has no cache_clear attribute. Every integration test containing this call will raise AttributeError as soon as the Parquet fixture stops skipping. Replace all such calls with a supported cache-reset mechanism (or add a public reset helper).

    babel_xrefs.get_curie_xref.cache_clear()

src/babel_explorer/core/downloader.py:347

• HTTP 416 means only that the requested range is unsatisfiable; it does not prove the local .tmp is complete. If a stale partial is larger than the current remote object, this branch returns success and get_downloaded_file promotes corrupted bytes to the final Parquet file. Compare the temporary file size with the HEAD Content-Length; delete and restart when they differ.

                    if response.status_code == 416:
                        self.logger.info(f"File already complete: {local_path}")
                        # The 416 headers describe the error body, not the file; saving
                        # them as this file's metadata would record a bogus
                        # content_length and force a full re-download on the next check.
                        head = requests.head(url, timeout=self.timeout)
                        head.raise_for_status()
                        return head.headers

.github/workflows/ci.yml:39

• Repository secrets are not exposed as environment variables automatically. Even after adding the planned BABEL_URL secret, this step will keep using the public default and skip the Parquet-dependent integration suite. Pass the secret into the test process, retaining the public URL as a fallback when it is unset.

      - run: uv run pytest -v -m "integration and not slow"

src/babel_explorer/formatting.py:109

• This renderer does not follow the repository’s console-label convention: it emits curie='A:1', nodenorm_label="label" instead of placing "label" immediately after the CURIE. Route the CURIE and label through curie_with_label() (formatting.py:87-97), then render the remaining Parquet fields separately.

    parts = [f"curie={record.curie!r}"]
    if record.nodenorm_label:
        parts.append(f'nodenorm_label="{escape_label(record.nodenorm_label)}"')
    parts.extend(f"{name}={value!r}" for name, value in record.extra_fields)

FUTURE.md:6

• Batch NodeNorm lookup is implemented by NodeNorm.normalize_curies() and already used by all enrichment paths, so issue #12 is no longer future work. Remove this completed item and close/update the issue to keep the roadmap accurate.

- [#12](https://github.com/TranslatorSRI/babel-explorer/issues/12) — Batch NodeNorm lookups to reduce N round-trips when `--labels` is set

README.md:42

• The version check does not run for plain xrefs; it runs only when NodeNorm is consulted by xrefs --labels or ids --labels. The current wording incorrectly tells users that every xrefs query can be rejected and omits the equivalent ids behavior.

`xrefs` refuses to run when NodeNorm was built from a different Babel release than the one being
queried, since the labels and cliques would not match the cross-references. Pass
`--allow-version-mismatch` to override.

src/babel_explorer/cli.py:307

• For a single CURIE, --paths cannot produce a pair, but this is checked only inside _print_paths after the recursive DuckDB query has scanned the large Concord file and expanded the component. Reject fewer than two CURIEs here, before creating the downloader, to avoid an expensive query that can only print a warning.

    if paths:
        # Checked before anything is downloaded. Only the console renderer knows how to
        # lay out paths; the other formats would silently emit the full recursive xref
        # list instead, which looks like a successful --paths run but is not one.
        if fmt != "console":
            raise click.UsageError(
                f"--paths is only supported with --format console, not --format {fmt}. "
                f"Drop --paths to emit the full cross-reference list as {fmt}."
            )
        recurse = True