Repository navigation
Conversation
InputLayout uses the output named "logits", else the first output.
processTokenBatch takes a LogitsRequest. Decode and prefill copy the last row; all-position paths copy every row.
Logits are read back as Float16, so a bfloat16 logits output now fails at load.
Embeds tokens and scatter-merges image embeddings on prefill. Not wired into an engine yet.
Verbatim move of the VLM engine's image and video encoding; VLMModelConfig gets its own file.
SequentialDecodeLoop owns the token sequence, stop handling and sampling for both iterators; GenerationSequence keeps its public name.
Optional VisionFrontEnd and EmbeddingsInputHandler, image prefill on the session, prefix reuse cut at the first image, 2-4 states (apple#276).
makeVLMEngine builds the sequential engine and rejects variant overrides other than coreai-sequential.
A deprecated typealias and init keep existing callers compiling. is CoreAISequentialVLMEngine now matches every sequential engine; use acceptsMultimodalInput.
| let (commonPrefix, _) = sessionState.history.resolve(input: input) | ||
| let plan = Self.prefixResetPlan( | ||
| commonPrefix: commonPrefix, | ||
| commonPrefix: Self.reusablePrefix( |
There was a problem hiding this comment.
Currently a text-only request whose tokens contain image placeholders silently loses the image
This is important since this whole refactor is to make multimodal reuse the text path
| } | ||
|
|
||
| extension MultimodalInferenceEngine { | ||
| public var acceptsMultimodalInput: Bool { true } |
There was a problem hiding this comment.
The flag being true makes a text only static shape engine claim image support, the risk is that for such a flag (capability flag), a default true is risky. This means that conformer who never thought about it now claims that. Let's not have a default, or a conservative false
| languageModelURL: URL, | ||
| options: EngineOptions | ||
| ) async throws -> any InferenceEngine { | ||
| _ = try resolveVLMVariant(override: options.variant) |
There was a problem hiding this comment.
The variant checks one token_merge branch but not the other
So --inference-engine-variant coreai-sequential on a Gemma 4 VLM quietly builds a StaticShapeEngine, while the wrong variant on Qwen throws. Let's keep the switch visionConfig.tokenMerge check the variant for both branches
| let strategy = engine.selectPrefillStrategy(newTokenCount: allTokens.count) | ||
| switch strategy { | ||
| case .chunked(let chunkSize): | ||
| allLogits = try await engine.processChunkedPromptAllLogits( |
There was a problem hiding this comment.
Batched forced continuation ignored prefix cache?
CoreAISequentialVLMEngineduplicated the sequential engine's decode loop, session state and input handling. Vision-language bundles now run onCoreAISequentialEngine, which gains an optionalVisionFrontEnd(image and video encoding) and anEmbeddingsInputHandler(token embedding plus image-embedding merge). The VLM class is deleted; a deprecated typealias keeps existing callers compiling.Changes
SequentialDecodeLoop: a shared decode loop for both sequential iterators.Behavior changes
coreai-sequential(the variant was ignored before).is CoreAISequentialVLMEnginematches every sequential engine; useacceptsMultimodalInput.configisModelConfig; vision settings are invisionConfig.reset(to:)on the VLM path cancels in-flight generation on partial resets too, and applies the sliding-window and hybrid-state checks.Verification
mainon smollm2, qwen3-1.7b and phi-4-mini (sequential; pipelined text identical).VLMSessionParityTests5/5.main(M2 Max, 3 interleaved pairs): qwen3-1.7b decode −0.8%, prefill +3.3% at 512 tokens and −2.8% at 4,096, within run-to-run noise. Muse Glimmer 30B VLM image decode 12.42 vs 12.30 tok/s (5 interleaved pairs, 256 tokens), also within noise.