Repository navigation
0.4.0: cache hit rates from Langfuse - #1
Merged
Merged
Conversation
…without Bedrock invocation logging can check their cache from the traces they already have New `cachecanary langfuse` command (also in the GitHub Action). It reads only token counts, model names and timestamps for the project's generations through the Langfuse public API (v2 on Langfuse v4, v1 on older servers), and reports hit rate and input cost per model like `logs` does. Langfuse stores the counts the way each integration sends them, and some don't add up: Strands before 1.54 makes the input look like 0 (so the cache looks perfect), Strands 1.53 sends its cache writes under a name Langfuse doesn't recognise, and LiteLLM's langfuse callback counts cache writes twice. When a call carries a total that was really sent, the uncached input is rebuilt from it and the output says how many calls were fixed. LiteLLM routing prefixes are stripped so one model stays one group and keeps its global endpoint price. Tested against records that Strands 1.61 and 1.53, LiteLLM (both Langfuse integrations) and LangChain wrote into a real Langfuse 4.64 server, and against a production Langfuse Cloud project, where the numbers matched a separate recount for every model.
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
cachecanary langfuse: hit rate and dollars lost per model from the generations a Langfuse project already records, so teams without Bedrock invocation logging can check their cache. Also available in the GitHub Action.