Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
374db33
RoPE: public rotate_half in primitives/macos/rope
stikves Oct 22, 2026
6184ee9
EmbeddingGemma 2: text and image encoders for GPU and Neural Engine
stikves Oct 20, 2026
03bb974
coreai.embedding.export: embedding bundle exporter
stikves Oct 20, 2026
85db172
EmbeddingGemma 2: model README and catalog entry
stikves Oct 20, 2026
90e08e1
Merge branch 'main' into sukru/embedding-gemma2-export
stikves Oct 27, 2026
3b15173
Keep ScaledRMSNorm finite in float16 for rows with max-abs below 1/256
stikves Oct 27, 2026
5b5e7f1
Clamp the squared norm in the Neural Engine L2 normalize
stikves Oct 28, 2026
0ea9996
Store the Neural Engine token table in float16
stikves Oct 28, 2026
18b2ce6
Say which encoder the family precision fields describe
stikves Oct 28, 2026
6b00b9c
Check the float16 Neural Engine text encoder against real weights
stikves Oct 28, 2026
ce3b018
Store the macOS text weights once when the bundle has an image tower
stikves Oct 28, 2026
561f019
Merge branch 'main' into sukru/embedding-gemma2-export
stikves Oct 28, 2026
0f054ec
Record the requested compute precision in the bundle metadata
stikves Oct 29, 2026
98966aa
Require the layer layout in the EmbeddingGemma 2 text config
stikves Oct 29, 2026
7f38ce4
Use the shared 2-D RoPE in the EmbeddingGemma 2 image encoder
stikves Oct 29, 2026
9b45f9f
Set the image-tower facts per family and fix the precision message
stikves Oct 29, 2026
4fbd256
Tidy the embedding export CLI and clean up after a failed export
stikves Oct 29, 2026
59dc4b9
Correct the EmbeddingGemma 2 docstrings and report missing weight files
stikves Oct 29, 2026
d1c0f03
Share the converter setup and per-function tracing with diffusion.gpu
stikves Oct 29, 2026
44dd5a5
Use one image reference-input helper for the GPU and Neural Engine tr…
stikves Oct 29, 2026
3747ec4
Split the embedding exporter into shared, GPU and Neural Engine modules
stikves Oct 29, 2026
d2c7485
Note the host-gathered token table as a follow-up for the Neural Engi…
stikves Oct 29, 2026
a8364dc
Revert the Muse Glimmer rotate_half move
stikves Oct 30, 2026
188d4bc
Share the RoPE frequency and 1x1 conv helpers publicly
stikves Oct 30, 2026
531775f
Read the EmbeddingGemma 2 checkpoint in one place and load strictly
stikves Oct 30, 2026
a57aacd
Call add_function directly in the embedding exporters
stikves Oct 30, 2026
55ae6ad
Look up the RoPE head dims through the text config
stikves Oct 30, 2026
b1f93de
Promote the Gemma 4 max-abs RMSNorm to primitives/ios
stikves Oct 30, 2026
ad81f5e
Reshape each iOS text projection to per-head layout once
stikves Oct 30, 2026
2f2e086
Share the macOS attention and sandwich layer between text and image
stikves Oct 30, 2026
377f503
Drop a rationale from a section comment in the macOS image encoder
stikves Oct 30, 2026
d7a56f2
Fold overlapping embedding schema tests and reuse the shared tiny con…
stikves Oct 30, 2026
3559d62
Trim the EmbeddingGemma 2 model tests to one path per check
stikves Oct 30, 2026
5b3db09
Clarify embedding helper names and derive bundle assets from the pipe…
stikves Oct 30, 2026
ac7fe2a
Make the base.py safetensors helpers public
stikves Oct 30, 2026
d979627
Read EmbeddingGemma 2 checkpoints through the base.py safetensors hel…
stikves Oct 30, 2026
6f29e93
Restore the EmbeddingGemma 2 test guards dropped by the trims
stikves Oct 30, 2026
49f7336
Test the Neural Engine embedding conversion on tiny configs
stikves Oct 30, 2026
1915334
Record each embedding asset once and the component output dtypes
stikves Oct 30, 2026
41b294b
Tighten the EmbeddingGemma 2 comments and docstrings
stikves Oct 30, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 16 additions & 2 deletions models/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,7 @@ Pass `--include-debug-info` to switch the converter to `DEBUG` mode, which embed
uv run coreai.llm.export Qwen/Qwen3-0.6B --include-debug-info
```

The flag is available on `coreai.llm.export`, `coreai.vlm.export`, `coreai.diffusion.export`, `coreai.segmentation.export`, and every standalone `models/<name>/export.py` recipe, so all export paths produce assets carrying the same debug information by default.
The flag is available on `coreai.llm.export`, `coreai.vlm.export`, `coreai.diffusion.export`, `coreai.segmentation.export`, `coreai.embedding.export`, and every standalone `models/<name>/export.py` recipe, so all export paths produce assets carrying the same debug information by default.

**Note:** `--include-debug-info` is independent of `--verbose`/`-v`. `--verbose` only raises the console log level; it does not change what goes into the asset.

Expand Down Expand Up @@ -189,6 +189,16 @@ decoder (`main`), token-embedding lookup (`embedding`), vision encoder
(`vision`), tokenizer, and `metadata.json`. Pass `--skip-vision` to export the
text portion only.

### Embedding Models

```bash
uv run coreai.embedding.export --list-models
uv run coreai.embedding.export embeddinggemma-2 # text + image (macOS)
uv run coreai.embedding.export embeddinggemma-2 --platform iOS # Neural Engine
```

See the [EmbeddingGemma 2](embedding_gemma2) model card.

### Standalone Export Scripts

Models with a standalone `export.py` are run directly:
Expand Down Expand Up @@ -245,9 +255,13 @@ uv run models/<name>/export.py --include-debug-info # embed debug information
- [RoBERTa](roberta)
- [T5](t5)

### Embedding Models

- [EmbeddingGemma 2](embedding_gemma2)

## Adding a Model

To make a new model exportable via short-name, add a `ModelPreset(...)` entry to `LLM_PRESETS` or `DIFFUSION_PRESETS` in [`python/src/coreai_models/model_registry.py`](../python/src/coreai_models/model_registry.py). Set the short name, HuggingFace ID, family, variant, and the export defaults (compression, compute precision, max context length).
To make a new model exportable via short-name, add a `ModelPreset(...)` entry to `LLM_PRESETS` or `DIFFUSION_PRESETS` in [`python/src/coreai_models/model_registry.py`](../python/src/coreai_models/model_registry.py). Set the short name, HuggingFace ID, family, variant, and the export defaults (compression, compute precision, max context length). Models exported by `coreai.embedding.export` need an `EmbeddingFamily` in `EMBEDDING_FAMILIES` ([`python/src/coreai_models/embedding/contract.py`](../python/src/coreai_models/embedding/contract.py)) and a `UtilityModel` entry with task `embedding` in `UTILITY_PRESETS`.

For models with bespoke export logic that doesn't fit the standard `coreai.llm.export` / `coreai.diffusion.export` flow, write a standalone recipe under `models/<name>/export.py` — see existing recipes for the [PEP 723](https://peps.python.org/pep-0723/) pattern and `models/README.md` for the contribution checklist.

Expand Down
57 changes: 57 additions & 0 deletions models/embedding_gemma2/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# EmbeddingGemma 2

Multimodal retrieval embedding model that maps text and images into one shared space: each input becomes a single 768-dimensional, L2-normalized vector, and its leading 512, 256 or 128 dimensions, re-normalized, are valid smaller embeddings.[^1]

## Setup

If you haven't installed `uv`, install it by

```bash
brew install uv
```

## Export

```sh
uv run coreai.embedding.export embeddinggemma-2
```

Saves to `<repo-root>/exports/embeddinggemma-2/` (text and image towers, GPU). `--platform iOS` writes `embeddinggemma-2-ios/` and `--skip-vision` appends `-text`. Pass `--output-dir <path>` to change the parent directory.

```sh
uv run coreai.embedding.export --help
```

**Options:**

| Flag | Description | Default |
| ---------------------- | ---------------------------------------------------- | ------------------------------------ |
| `--platform` | `macOS` (GPU bundle) or `iOS` (Neural Engine bundle) | `macOS` |
| `--skip-vision` | Export the text tower only | — |
| `--compute-precision` | `float32`, `bfloat16` on macOS; `float16` on iOS | `float32` on macOS, `float16` on iOS |
| `--output-dir` | Parent directory for the bundle | `<repo-root>/exports/` |
| `--overwrite` | Overwrite an existing bundle | — |
| `--include-debug-info` | Embed conversion debug info in the `.aimodel` files | — |
| `--dry-run` | Print the export plan without writing | — |
| `--list-models` | List the supported models and exit | — |

**Supported models:**

| Model | HuggingFace ID | Parameters |
| ---------------- | ----------------------- | ---------------------------- |
| embeddinggemma-2 | google/embeddinggemma-2 | 439M (text 271M, image 168M) |

## Bundle

`metadata.json` lists each `.aimodel` file once under `assets`. Each entry of `embedding.components` names the asset it lives in (`asset`), its functions and its `output_dtype`:

| Component | macOS asset (GPU) | iOS asset | Functions | Output dtype |
| --------------- | ---------------------------- | ------------------------------- | ------------------ | ------------------------------ |
| `text_encoder` | `text_encoder` | — | `len{L}_b{B}` | `float32` |
| `token_table` | `text_encoder` (images only) | `token_table` (GPU) | `main` | `float32` macOS, `float16` iOS |
| `encoder` | `text_encoder` (images only) | `encoder` (Neural Engine) | `embeds_len{L}_b1` | `float32` macOS, `float16` iOS |
| `image_encoder` | `image_encoder` | `image_encoder` (Neural Engine) | `main` | `float32` macOS, `float16` iOS |

The lengths `{L}` differ by platform; read the function names from `embedding.components.<component>.functions`.

[^1]: [Documentation](https://ai.google.dev/gemma/docs/embeddinggemma) · [HuggingFace](https://huggingface.co/google/embeddinggemma-2)
1 change: 1 addition & 0 deletions python/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ dependencies = [
"coreai.llm.export" = "coreai_models.llm.export:main"
"coreai.llm.eval" = "coreai_models.llm.eval:main"
"coreai.vlm.export" = "coreai_models.vlm.export:main"
"coreai.embedding.export" = "coreai_models.embedding.export:main"
"coreai.diffusion.export" = "coreai_models.diffusion.export:main"
"coreai.model.registry" = "coreai_models.model_registry:main"

Expand Down
71 changes: 40 additions & 31 deletions python/src/coreai_models/diffusion/gpu.py
Original file line number Diff line number Diff line change
Expand Up @@ -82,13 +82,7 @@ def export_fn(module: torch.nn.Module) -> torch.export.ExportedProgram:
decomposed: torch.export.ExportedProgram = exported.run_decompositions(coreai_decomp_table)
return decomposed

converter = coreai_torch.TorchConverter(
mode=(
coreai_torch.TorchConverter.Mode.DEBUG
if include_debug_info
else coreai_torch.TorchConverter.Mode.RELEASE
)
)
converter = make_converter(include_debug_info)
converter.add_pytorch_module(
wrapper,
export_fn=export_fn,
Expand All @@ -99,6 +93,42 @@ def export_fn(module: torch.nn.Module) -> torch.export.ExportedProgram:
return program


def make_converter(
include_debug_info: bool = DEFAULT_INCLUDE_DEBUG_INFO,
) -> coreai_torch.TorchConverter:
"""Return a ``TorchConverter``; ``include_debug_info`` keeps debug info in the asset."""
mode = coreai_torch.TorchConverter.Mode
return coreai_torch.TorchConverter(mode=mode.DEBUG if include_debug_info else mode.RELEASE)


def add_function(
converter: coreai_torch.TorchConverter,
name: str,
module: torch.nn.Module,
args: tuple[torch.Tensor, ...],
input_names: tuple[str, ...] | list[str],
output_names: tuple[str, ...] | list[str],
*,
dynamic_shapes: tuple[dict[int, torch.export.Dim] | None, ...] | None = None,
decompositions: dict | None = None,
) -> None:
"""Trace ``module`` on ``args`` and add it to ``converter`` as function ``name``.

Functions added to one converter share their weights. ``decompositions`` defaults to Core AI's
table.
"""
module.eval()
with torch.no_grad():
exported = torch.export.export(module, args=args, dynamic_shapes=dynamic_shapes)
table = coreai_torch.get_decomp_table() if decompositions is None else decompositions
converter.add_exported_program(
exported.run_decompositions(table),
input_names=input_names,
output_names=output_names,
entrypoint_name=name,
)


def export_multifunction(
functions: list[tuple[str, torch.nn.Module, tuple[torch.Tensor, ...]]],
input_names: tuple[str, ...],
Expand All @@ -122,29 +152,8 @@ def export_multifunction(
Returns:
An optimized AIProgram with multiple named functions.
"""
converter = coreai_torch.TorchConverter(
mode=(
coreai_torch.TorchConverter.Mode.DEBUG
if include_debug_info
else coreai_torch.TorchConverter.Mode.RELEASE
)
)
coreai_decomp_table = coreai_torch.get_decomp_table()

converter = make_converter(include_debug_info)
for name, wrapper, dummy_inputs in functions:
wrapper.eval()
logger.info(f"Tracing function '{name}' (seq dims: {[t.shape for t in dummy_inputs[:2]]})")

with torch.no_grad():
exported = torch.export.export(wrapper, args=dummy_inputs)
decomposed = exported.run_decompositions(coreai_decomp_table)

converter.add_exported_program(
decomposed,
input_names=input_names,
output_names=output_names,
entrypoint_name=name,
)

program = converter.to_coreai()
return program
add_function(converter, name, wrapper, dummy_inputs, input_names, output_names)
return converter.to_coreai()
6 changes: 6 additions & 0 deletions python/src/coreai_models/embedding/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# Copyright 2026 Apple Inc.
#
# Use of this source code is governed by a BSD-3-clause license that can
# be found in the LICENSE file or at https://opensource.org/licenses/BSD-3-Clause

"""Retrieval-embedding bundle export; CLI: `uv run coreai.embedding.export`."""
Loading
Loading