[pull] main from danny-avila:main - #162
Merged
Merged
Conversation
* feat: shared adaptive stream smoother engine Bounded producer/consumer queue with backlog-proportional piece sizing: piece = max(4, ceil(buffered * tick / 250ms)), sliced at dequeue time so render lag stays pinned near the target latency regardless of provider chunk size or reply length. Three-way item classification (splittable / atomic / passthrough), first-token zero delay, cadenced sleeps, canonical abort semantics, strict FIFO for unsmoothed metadata chunks. * refactor: migrate OpenAI chat-model family onto the shared smoother delayStreamChunks keeps its signature but delegates to smoothStream; plain text splits adaptively, logprobs/finish_reason and reasoning-delta chunks pace whole (atomic), tool-call/usage-only chunks pass through undelayed in FIFO order. Defaults change from undefined to 25ms via resolveStreamDelay; _lc_stream_delay: 0 disables. * refactor: migrate Anthropic client onto the shared smoother The bespoke bounded queue (which the engine generalizes) is replaced by a producer generator yielding classified SmoothItems; extractToken/cloneChunk stay as the provider adapter. Empty text-block starts are still skipped, tool-input and usage/id-only chunks still bypass pacing in FIFO order, and the early-break producer close is preserved via abortUpstream. Adaptive sizing now bounds render lag on fast streams; default stays 25ms. * refactor: migrate Bedrock Converse client onto the shared smoother Delta pieces are built lazily via the exact per-piece pipeline (sliced delta -> chunk -> enrichment -> stream-limit link) against an arrival-time snapshot of seen block indices, so output is byte-identical to the legacy splitter. toolUse seals, metadata and misc events pass through unpaced in FIFO order (seal timing test added). Default changes 0 -> 25ms; explicit 0 disables (parameter-passthrough tests updated accordingly). * feat: adaptive stream smoothing for Google and Vertex clients Both clients previously streamed raw provider chunks with no pacing at all. A generic ChatGenerationChunk adapter (shared with Mistral) splits plain-text deltas — string content or google-common's single text-part array shape — while thought/media/function parts pace whole and metadata passes through unpaced. Raw generators no longer dispatch runManager callbacks; dispatch moves per emitted piece so callback-echo consumers observe smoothed deltas. _lc_stream_delay hoisted onto the shared client-options surface (StreamSmoothingOptions) for every provider. * feat: Mistral subclass with adaptive stream smoothing CustomChatMistralAI wraps the parent stream with the generic smoothing adapter, registered for both MISTRALAI and MISTRAL provider keys. The Jest stub for @langchain/mistralai (ESM-only) becomes a minimal functional double so the subclass is constructible under tests; honest failure moves from the constructor to the network layer. * feat: linear tail drain + end-to-end cadence benchmark Once the producer finishes, remaining backlog drains linearly across the target latency window instead of decaying geometrically — the measured tail drops from ~3.2x to under 3x target. Benchmark drives the real ChatOpenAI against a local SSE server emitting 110-char chunks at 20ms, asserting even cadence, sub-10ms jitter, bounded lag, and that lag stays flat as reply length quadruples. * v3.4.0 * test: load-tolerant benchmark bounds; document logprobs predicate split Absolute cadence/jitter/lag bounds loosen for parallel-worker and CI scheduling noise; the discriminating flat-lag-with-length assertion stays tight. Cross-reference comments pin why the OpenAI-family and generic adapters intentionally differ on logprobs (DeepSeek contract vs google-common's always-present empty logprobs). * style: satisfy eslint — emit return types, dead guard removal, formatting Explicit return types on SmoothItem emit arrows (anthropic/bedrock), drop a provably-dead null guard on additional_kwargs, widen a runtime type probe to unknown, and auto-fixed indentation/blank lines. * fix: keep reasoning-bearing chunks atomic; scope split-piece metadata; sync lockfile Addresses Codex review: chunks pairing visible text with reasoning payloads in additional_kwargs now pace whole in both the generic and OpenAI-family adapters — split pieces would each clone the kwargs and the aggregator's dict merge concatenates string fields once per piece. Generic clone functions now carry additional_kwargs/response_metadata/ usage_metadata on the first piece only (pinned by new adapter tests). package-lock.json version synced to 3.4.0. * fix: cover reasoning_details and camelCase finishReason; abort-aware idle wait; bounded teardown Second Codex round: OpenRouter reasoning_details join the unified reasoning-kwargs predicate (shared by both adapters); google-common's camelCase finishReason now blocks splitting like finish_reason; a consumer parked on an empty queue wakes on signal abort instead of waiting for a provider that may ignore it; and generator teardown force-closes the source and bounds the producer wait with a 1s grace so a stalled stream can no longer block abort propagation indefinitely (previously it could — proven by the new parked-abort test). * style: drop unnecessary optional chain on Node timeout * fix: third Codex round — lazy disabled path, admission-side text cap, tool/usage split guards - delayMs <= 0 now takes a direct pass-through path: no background producer, no read-ahead — disabled smoothing is byte- and laziness-identical to the pre-engine behavior. - Oversized splittable items are segmented at admission (4096-char word boundary segments), so a single giant provider chunk parks the producer at the text cap like the legacy split-before-enqueue queues; wrapped emit maps segment pieces back to chunk-global isFirst/isLast. - Mixed text/tool-call deltas pace whole in both adapters (hasToolCallChunks) — split pieces would duplicate tool arguments. - Generic clone functions scope generationInfo to the first piece (Vertex reports authoritative usage there; replication inflated per-piece trace observations). OpenAI-family clones keep per-piece generationInfo deliberately: token indices live there and scalar metadata dedup already handles repetition. * fix: hard-cut boundary-free runs at a 64-char lookahead past target size Fourth Codex round: findStreamChunkBoundary previously scanned to end of string when no boundary char existed, letting base64/minified deltas defeat both the admission segment cap and per-tick piece sizing. The word-boundary search now extends at most 64 chars past the target before hard-cutting — natural language is unaffected (boundaries land within a few chars); boundary-free runs stay within budget. Pinned by unit and engine tests. * fix: fifth Codex round — cross-item tick batching, reasoning-only pacing, robustness guards - P1: one cadence tick now drains up to the adaptive budget ACROSS queued items, so token-sized provider deltas coalesce instead of costing a full tick each (400 one-char deltas: ~25s -> under target window). Passthrough items flush free mid-batch in FIFO order; atomic items take their own tick. - Reasoning-only deltas (Gemini thoughts, OpenRouter reasoning_details, reasoning summaries) now pace atomically via a shared getReasoningKwargsText extractor wired into both adapters, instead of passing through unsmoothed. - resolveStreamDelay normalizes non-finite values to the default, and the engine's disabled-path guard inverts to catch NaN defensively. - Producer failures are tracked with an explicit failed flag so nullish rejection values still propagate instead of truncating silently. * fix: extend StreamSmoothingOptions to Azure and OpenRouter option types Codex: typed factory usage for AZURE/OPENROUTER rejected _lc_stream_delay even though both runtime classes read it with the new 25ms default. AzureClientOptions and the OPENROUTER ProviderOptionsMap entry intersect StreamSmoothingOptions; ChatOpenRouterInput carries the field inline (avoids a types<->openrouter import cycle). Compile-time pin added to the smoke suite. * fix: close the split-safety class — lossless-reassembly property test Instead of another denylist entry, pin the invariant itself: for an adversarial matrix of chunk shapes (unknown kwargs, scalar response metadata, usage, reasoning variants, tool calls, google array shapes), aggregating the emitted pieces must reproduce the original chunk's observable payload. The property immediately caught three live gaps, all now fixed: - OpenAI-family clone replicated additional_kwargs/response_metadata on every split piece (unknown string fields concatenated downstream); now first-piece-only, matching the generic adapter. generationInfo deliberately stays per-piece (token indices + scalar dedup own it). - Generic array-shape split required no merge key, so index-less parts aggregated as N separate content parts; splitting now requires a numeric part index, index-less parts pace whole. - (Third failure was harness noise: concat() adds empty token-detail objects; usage asserts on token counts.) * style: drop unused type import in reassembly property test
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )