Skip to content

[OSI][Omni] Add Omni semantic model converter#175

Open
jdavidson wants to merge 3 commits into
apache:mainfrom
jdavidson:add-omni-converter
Open

[OSI][Omni] Add Omni semantic model converter#175
jdavidson wants to merge 3 commits into
apache:mainfrom
jdavidson:add-omni-converter

Conversation

@jdavidson

Copy link
Copy Markdown

Add Omni bidirectional converter

Adds a bidirectional, offline converter between OSI semantic models and
Omni semantic model files, filling the Omni
spoke in the hub-and-spoke architecture. Follows the conventions of the
Databricks converter (#171): same package shape, custom_extensions stash
protocol, and fail-loudly philosophy.

What's included

  • Export (osi-omni export): OSI semantic model -> Omni model files.
    Because an Omni model is a directory rather than one document, export
    produces the Omni local-editor/git layout: one views/<name>.view.yaml per
    dataset, a relationships.yaml join list, and one generated
    topics/<name>.topic.yaml rooted at the fact dataset (FK-sink heuristic,
    --base-view override).
  • Import (osi-omni import): Omni model directory -> OSI. Omni-only
    features (formats, timeframes, hidden/tags/group labels, join types, topic
    curation, the model file itself, query views) are preserved in
    custom_extensions[OMNI], so Omni -> OSI -> Omni is lossless.
  • A unified osi-omni CLI and a Python API (OSI YAML string <-> {filename: YAML} dict).
  • A README with a mapping table and limitations; an OMNI row in converters/index.md.

Design highlights

  • Round-trip fidelity via the custom_extensions[OMNI] stash -- the
    approach the converter guide recommends. Import stashes only what the OSI
    expression cannot reconstruct; export restores stashed originals verbatim.
  • Native mapping is broad: dataset <-> view (schema/table_name/catalog,
    or a SQL view for subquery sources), field <-> dimension (labels,
    descriptions, synonyms, ai_context, is_time <-> timeframes,
    primary keys incl. composite via custom_compound_primary_key_sql),
    relationship <-> relationships.yaml entry (from/to columns <->
    on_sql equi-join, OSI's many-side-first orientation <-> relationship_type),
    metric <-> measure (AGG(view.field) <-> aggregate_type + sql: ${field},
    COUNT(*) <-> count, anything else as a raw-SQL measure), and
    model <-> topic (description/AI context).
  • Dialects: Omni SQL is the connection's dialect and the OSI dialect enum
    has no OMNI entry, so import emits ANSI_SQL and export prefers ANSI_SQL
    with a --dialect flag to prefer a warehouse dialect (e.g. SNOWFLAKE).
  • Fails loudly, never silently. A non-equi on_sql, a schema-less source,
    an ambiguous base view, and name-sanitization collisions are rejected with a
    clear ConversionError; anything dropped on export (unique_keys,
    relationship ai_context, foreign-vendor extensions, unusable dialects)
    emits a warning naming the object.

Testing

cd converters/omni
pip install -e . && pip install pytest hypothesis
python -m pytest tests/

Example-based unit tests, fixture round-trips in both directions (including a
TPC-DS baseline derived from examples/tpcds_semantic_model.yaml), and
Hypothesis property-based round-trip tests (with a seeded-random fallback when
hypothesis is not installed). Imported OSI validates against the core-spec
JSON schema via validation/validate.py, including sqlglot SQL validation of
the emitted expressions.

Notes

  • Omni model YAML verified against docs.omni.co (views, dimensions, measures,
    relationships, topics, model parameters) as of July 2026.
  • One generated topic per OSI model; query views, composite topics, access
    grants/filters, and templated filters round-trip via the stash without OSI
    semantics (documented in the README).
  • On duplicate metric names (the same measure name on several views -- e.g.
    Omni's idiomatic per-view count), the import name-qualifies as
    <view>__<measure> and restores the original name on export -- the
    resolution Add Snowflake → OSI import converter (snowflake_to_osi) #163 floats as an open question for the Snowflake importer.

🤖 Generated with Claude Code

jdavidson and others added 3 commits July 5, 2026 18:53
Adds converters/omni: an offline OSI <-> Omni semantic model converter,
filling the Omni spoke in the hub-and-spoke architecture.

- Export (osi-omni export): OSI -> Omni model files (views/*.view.yaml,
  relationships.yaml, a generated topic, model.yaml when stashed).
- Import (osi-omni import): Omni -> OSI. Omni-only features are preserved
  in custom_extensions[OMNI], so Omni -> OSI -> Omni is lossless.
- Unified osi-omni CLI plus a string-in / files-dict-out Python API.
- Example-based unit tests, fixture round-trips (incl. TPC-DS), and
  Hypothesis property-based round-trip tests with a seeded-random fallback.
- Adds the OMNI vendor row to converters/index.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Testing against production Omni instances surfaced model shapes the
fixtures did not cover; importing all of them now round-trips losslessly
(verified on a 3,498-file and a 427-file production model plus the
Ecommerce demo model, all validating against the OSI spec):

- Resolve schema-qualified view names (schema__table) from the
  "# Reference this view as" header, accept per-schema folder layouts,
  and preserve original file paths through a round trip.
- Stash-and-restore instead of erroring: extends-only views, non-equi/
  range joins, joins touching query/extends views, and fields or
  measures using Omni template ({{...}}) syntax.
- Preserve explicit Omni defaults verbatim (join_type, relationship_type,
  redundant table_name, self-named sql), non-canonical on_sql (aliases,
  casing, spacing), ${view.field} compound keys, empty-string metadata,
  and timeframes: [].
- Accept real Omni identifiers (_fivetran_id, trailing underscores,
  camelCase); collision checks stay case-insensitive.
- Quote source parts that need it ("Omni Views".upload), tag date values
  in the JSON stash, and suffix duplicate generated relationship names.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
An expression that cannot be tokenized (e.g. templated SQL) crashed the
validator instead of being reported as a finding.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@jbonofre jbonofre self-requested a review July 14, 2026 18:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant