Skip to content

Add initial T5Gemma S/S support#947

Open
maasir554 wants to merge 7 commits into
google:devfrom
maasir554:t5gemma-support
Open

Add initial T5Gemma S/S support#947
maasir554 wants to merge 7 commits into
google:devfrom
maasir554:t5gemma-support

Conversation

@maasir554

@maasir554 maasir554 commented Jul 8, 2026

Copy link
Copy Markdown

Summary

Adds initial T5Gemma S/S support to gemma.cpp.

This includes:

  • T5Gemma S/S model config and tensor metadata
  • HF safetensors -> SBS conversion through python/convert_from_safetensors.py
  • Encoder-decoder runtime path for fresh seq2seq generation
  • BF16 default conversion path for correctness/parity
  • Basic docs and focused config/tensor/cache tests

Scope

This PR targets T5Gemma v1 S/S only.

Out of scope:

  • T5Gemma 2
  • Other T5Gemma sizes
  • Multiturn/cache reuse with changing encoder inputs
  • Fully optimized T5-specific attention/logits kernels

Local Validation

  • bazel test -c opt //:configs_test //:tensor_info_test //:kv_cache_test
  • Local optimized smoke test with t5gemma-s-s-it
    • prompt: Hello
    • output begins: I am a new user, ...
    • local speed: roughly 30-35 generated tok/s on MacBook M1, 8GB RAM

Notes

Opened as draft to get early maintainer feedback on scope, naming, and whether this should be split further.

Additional Note: //:gemma_lib_test currently fails locally for me in an existing GemmaAttentionFlash/EMU128 golden mismatch on Apple Silicon; this reproduced outside the T5Gemma changes.

Related to #661

@google-cla

google-cla Bot commented Jul 8, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@maasir554 maasir554 marked this pull request as draft July 8, 2026 18:31

@jan-wassenberg jan-wassenberg left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remarkably nice work, very clean code, congrats :) A few suggestions:

Comment thread gemma/activations.h Outdated
Comment thread gemma/configs.h Outdated
Comment thread gemma/gemma.cc Outdated
Comment thread gemma/gemma.cc Outdated
Comment thread gemma/gemma.cc Outdated
Comment thread gemma/gemma.cc Outdated
@maasir554

maasir554 commented Jul 11, 2026

Copy link
Copy Markdown
Author

Remarkably nice work, very clean code, congrats :) A few suggestions:

Thank you, really appreciate the review and kind words :)
I’ll address your suggestions in a follow-up update.

@maasir554 maasir554 marked this pull request as ready for review July 15, 2026 18:35
@maasir554

Copy link
Copy Markdown
Author

Hi, I addressed the suggestions in the latest push:

  • added a shared Activations::num_layers
  • made encoder/decoder config fields conditional on is_encoder_decoder
  • moved/reused T5 token IDs via tokenizer.h / BOS_ID
  • reused MaybeLogitsSoftCap
  • hoisted T5Gemma encoder scratch buffers into Activations
  • switched internal encoder-cache checks to HWY_DASSERT

Thanks again for the review!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants