Add a required-prefix constraint for word completion

[Mason363]

Summary

CotabbyInferenceEngine::setRequiredPrefix(sequence, utf8, length) makes the next generation on a sequence begin with the given bytes. While any remain, every token whose text is inconsistent with them is masked before sampling (the seed in decodePrompt and each sampleNext), each sampled token consumes the bytes it covers, and the constraint clears once they have all been produced.

Why

Completing a word the user has half typed ("…the draft yest") works poorly with the partial word in the prompt: the tokenizer sees a word cut at an arbitrary byte, and the model tends to continue another word or produce a misspelled tail. With this constraint the caller prompts up to the word boundary, so the tokenizer sees whole words, and requires the completion to begin with the boundary whitespace plus the typed letters (" yest"), so the model finishes the user's word (" yesterday"). Cotabby uses it for every mid-word request: FuJacob/cotabby#827.

Details

• Token texts are cached once at model load (token_pieces), so the mask costs one pass over the vocabulary per constrained token, and nothing when no prefix is set.
• A token is consistent when its text is a prefix of the remaining bytes, or the remaining bytes are a prefix of its text. Tokens that render to nothing, EOG among them, are never consistent while bytes remain.
• If no token is consistent, the constraint is dropped for that row instead of sampling from an all-masked distribution.
• An empty prefix clears a pending constraint, and an unknown sequence is ignored. The API is additive, so existing callers are unaffected.

Tests

• testRequiredPrefixConstrainsTheStartOfTheCompletion (model-backed): with " yest" required after "Thanks for sending over the draft", the completion begins " yesterday".
• testRequiredPrefixIsClearedByAnEmptyPrefixAndIgnoredWithoutASequence.

COTABBY_TEST_MODEL_PATH=/path/to/gemma-4-E2B.i1-Q6_K.gguf swift test
# Executed 16 tests, with 0 failures

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: cf22edd1-62aa-4600-b09a-a992dd8da0bf

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}