Skip to content

Improve langfuse tracing of lesson insights - #74314

Open
lfryemason wants to merge 3 commits into
stagingfrom
lc/snap-intermediary
Open

Improve langfuse tracing of lesson insights#74314
lfryemason wants to merge 3 commits into
stagingfrom
lc/snap-intermediary

Conversation

@lfryemason

Copy link
Copy Markdown
Contributor

Improvements:

  • Fix folder that langfuse prompts are retrieved from
  • Add prompt name and version to trace to associate with the langfuse prompt
  • Change from sending in the entire system prompt to adding the specific variables - this will allow us to create datasets and test different prompts on the trace
  • Add lesson feedback traces
  • Filter out everything except the expected end output from the langfuse response so we can also add intermediate steps to the AI response and log them to langfuse
  • Use the development tagged prompts in development

Now working - see correctly associating prompt version
image

Adds correct tracing variables
image

Also adds input variables
image

Links

@lfryemason
lfryemason requested a review from etaderhold August 3, 2026 20:46

def self.notify_and_raise(message, context)
exception = StandardError.new(message)
Honeybadger.notify(exception, context: context)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there's some ideal where Honeybadger entries are supposed to represent actionable bugs that need to be fixed in our code. There's of course a lot of noise there right now, and adding more entries for things like external API timeouts outside of our control might make it even noisier. Most of the usages of notify_and_raise do seem like encountering them would represent an actionable bug fix opportunity though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants