Repository navigation
fix(prompt): Say that replies are not top-level channel posts - #2350
Merged
Merged
Conversation
#2315 merged the two Slack delivery rules into one line. The merged line tied the explanation to tool calls, so a plain reply that repeats the requested text looked allowed. The delivery integration eval for a top-level post request then failed in both live samples. State the fact that a reply is not a channel post, and say that the agent must not reply with only the requested text. Co-Authored-By: David Cramer <david@sentry.io>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
These tests only checked that the prompt builder output contains static strings. Evals own prompt behavior. Co-Authored-By: David Cramer <david@sentry.io>
dcramer
marked this pull request as ready for review
October 29, 2026 14:22
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit d9ecc3a. Configure here.
The rule now covers only requests to post right now. Scheduled automations can still post to the active channel later.
dcramer
approved these changes
Oct 29, 2026
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The integration eval
conversation/delivery.eval.ts> "when asked for a top-level channel post, explain the limitation instead" fails onmainsince #2315. This change fixes the Slack delivery rule inprompt.tsthat the eval checks.Why it fails
#2315 merged the two Slack delivery rules into one line. The merged line ties the explanation to tool calls ("explain the limitation instead of calling a tool"). It also says that assistant text goes to the current conversation. For
post this to the channel: deploy is unblocked, a plain reply with the text calls no tool and lands in the current conversation, so it looked allowed. Both live samples on #2317 replied onlydeploy is unblocked.Before #2315, every recorded live sample passed, and each reply used the old wording: "I can't create a new top-level channel post…".
Is the eval correct?
Yes. The mention is a top-level channel message. Junior can only reply in the thread under it, so a reply with only the requested text is wrong. The rubric follows the product rule, and the eval file has not changed since Oct 8.
Change
The delivery rule stays one line, so the #2315 cleanup holds. It now:
This PR also removes
tests/unit/prompt.test.ts. It only checked that the prompt output contains static strings. Evals own prompt behavior.Verification
tsc --noEmitpasses for@sentry/junior.packages/junior/src/**, so this PR has thetrigger-evals-integrationlabel. The prompt change makes every model request new, so the whole integration suite runs live. That gives a real comparison against the failures on feat(images): Generate images with GPT Image through the AI SDK #2317.Open question
Integration and behavioral evals do not run for
packages/junior/src/**changes, and no eval suite runs onmain. That gap let #2315 merge without this eval. It is out of scope here.via David Cramer.
--
View Junior Session [Sentry]