[OSI][Omni] Add Omni semantic model converter#175
Open
jdavidson wants to merge 3 commits into
Open
Conversation
Adds converters/omni: an offline OSI <-> Omni semantic model converter, filling the Omni spoke in the hub-and-spoke architecture. - Export (osi-omni export): OSI -> Omni model files (views/*.view.yaml, relationships.yaml, a generated topic, model.yaml when stashed). - Import (osi-omni import): Omni -> OSI. Omni-only features are preserved in custom_extensions[OMNI], so Omni -> OSI -> Omni is lossless. - Unified osi-omni CLI plus a string-in / files-dict-out Python API. - Example-based unit tests, fixture round-trips (incl. TPC-DS), and Hypothesis property-based round-trip tests with a seeded-random fallback. - Adds the OMNI vendor row to converters/index.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Testing against production Omni instances surfaced model shapes the
fixtures did not cover; importing all of them now round-trips losslessly
(verified on a 3,498-file and a 427-file production model plus the
Ecommerce demo model, all validating against the OSI spec):
- Resolve schema-qualified view names (schema__table) from the
"# Reference this view as" header, accept per-schema folder layouts,
and preserve original file paths through a round trip.
- Stash-and-restore instead of erroring: extends-only views, non-equi/
range joins, joins touching query/extends views, and fields or
measures using Omni template ({{...}}) syntax.
- Preserve explicit Omni defaults verbatim (join_type, relationship_type,
redundant table_name, self-named sql), non-canonical on_sql (aliases,
casing, spacing), ${view.field} compound keys, empty-string metadata,
and timeframes: [].
- Accept real Omni identifiers (_fivetran_id, trailing underscores,
camelCase); collision checks stay case-insensitive.
- Quote source parts that need it ("Omni Views".upload), tag date values
in the JSON stash, and suffix duplicate generated relationship names.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
An expression that cannot be tokenized (e.g. templated SQL) crashed the validator instead of being reported as a finding. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add Omni bidirectional converter
Adds a bidirectional, offline converter between OSI semantic models and
Omni semantic model files, filling the Omni
spoke in the hub-and-spoke architecture. Follows the conventions of the
Databricks converter (#171): same package shape,
custom_extensionsstashprotocol, and fail-loudly philosophy.
What's included
osi-omni export): OSI semantic model -> Omni model files.Because an Omni model is a directory rather than one document, export
produces the Omni local-editor/git layout: one
views/<name>.view.yamlperdataset, a
relationships.yamljoin list, and one generatedtopics/<name>.topic.yamlrooted at the fact dataset (FK-sink heuristic,--base-viewoverride).osi-omni import): Omni model directory -> OSI. Omni-onlyfeatures (formats, timeframes, hidden/tags/group labels, join types, topic
curation, the model file itself, query views) are preserved in
custom_extensions[OMNI], so Omni -> OSI -> Omni is lossless.osi-omniCLI and a Python API (OSI YAML string <-> {filename: YAML} dict).OMNIrow inconverters/index.md.Design highlights
custom_extensions[OMNI]stash -- theapproach the converter guide recommends. Import stashes only what the OSI
expression cannot reconstruct; export restores stashed originals verbatim.
schema/table_name/catalog,or a SQL view for subquery sources), field <-> dimension (labels,
descriptions, synonyms,
ai_context,is_time<->timeframes,primary keys incl. composite via
custom_compound_primary_key_sql),relationship <->
relationships.yamlentry (from/tocolumns <->on_sqlequi-join, OSI's many-side-first orientation <->relationship_type),metric <-> measure (
AGG(view.field)<->aggregate_type+sql: ${field},COUNT(*)<->count, anything else as a raw-SQL measure), andmodel <-> topic (description/AI context).
has no
OMNIentry, so import emitsANSI_SQLand export prefersANSI_SQLwith a
--dialectflag to prefer a warehouse dialect (e.g.SNOWFLAKE).on_sql, a schema-less source,an ambiguous base view, and name-sanitization collisions are rejected with a
clear
ConversionError; anything dropped on export (unique_keys,relationship
ai_context, foreign-vendor extensions, unusable dialects)emits a warning naming the object.
Testing
Example-based unit tests, fixture round-trips in both directions (including a
TPC-DS baseline derived from
examples/tpcds_semantic_model.yaml), andHypothesis property-based round-trip tests (with a seeded-random fallback when
hypothesis is not installed). Imported OSI validates against the core-spec
JSON schema via
validation/validate.py, including sqlglot SQL validation ofthe emitted expressions.
Notes
relationships, topics, model parameters) as of July 2026.
grants/filters, and templated filters round-trip via the stash without OSI
semantics (documented in the README).
Omni's idiomatic per-view
count), the import name-qualifies as<view>__<measure>and restores the original name on export -- theresolution Add Snowflake → OSI import converter (snowflake_to_osi) #163 floats as an open question for the Snowflake importer.
🤖 Generated with Claude Code