Repository navigation
fix(evals): Record automation fixture channels as public - #2449
Merged
Merged
Conversation
Since #2440 (15ee5c1), an automation run no longer writes the visibility saved on its Automation to the Destination row. Only a live Slack signal does. A run into a channel with no row stores `unknown`, and readers treat it as private. In production, the Slack thread where a person created the Automation records the channel. The eval fixture inserts Automations without that thread, so their channels stayed `unknown`. The fixture reads results as a viewer who is not a participant, so the reporting API redacted the run Conversations. Each tool call came back with no input and no output. Evals that read `finishAutomationRun` results or tool arguments of scheduled or event automation runs then failed. `insertScheduledAutomation` and `insertEventAutomation` now store the creation thread through the Conversation store with public visibility, as the live Slack event would.
The Conversation store defaults the activity time, so the helper does not need a time argument. Describe the shared setup timestamp for both of its uses.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Sentry Evalsjunior-behavioral
Base commit junior-integration
Base commit |
dcramer
approved these changes
Oct 30, 2026
dcramer
added a commit
that referenced
this pull request
Oct 30, 2026
…r mention (#2435) An Automation runs later without the conversation that created it. Words such as "this channel" or "here" mean nothing then. Before this change, the tools told the agent to rewrite "me" as the creator's Slack mention in the saved instruction. Now the agent calls the creator "the Automation creator". The run-time `Created by:` line says who that is and where to write the mention. Refs #2362 **Run-time creator line** (`chat/task-input.ts`) ```text Created by: <@u123>. This person is the Automation creator. "The Automation creator", "me", and "my" in the instructions mean this person. Where the instructions refer to this person, write this mention in the message. If they do not, do not mention this person. ``` - "Me" and "my" stay a normal run-time meaning, because people can save "remind me" in the dashboard editor. - The line says where to write the mention. A line that only says who "me" is (#2052), or "write this mention for them", made the model drop the mention in live evals. - The #2052 rule stays: if the instructions do not refer to the creator, the reply does not mention them. - The mention still comes from `createdBy.slackUserId`. The create tools require that it is the creator identity's Slack user in the same workspace. **Tool description** (`chat/automations/edit-schema.ts`) The shared instruction description for the scheduled and event create and update tools now says: - The run does not see this conversation, so words like "this channel" or "here" will mean nothing then. Write the exact target instead, such as a channel mention or a link. - Call the person who created the Automation "the Automation creator", not "me". Each run knows who that is. - Name every other person with their Slack mention, including a requester who did not create the Automation. The full term "the Automation creator" keeps "issue creator" or "PR creator" from matching. A requester who did not create the Automation is covered by this guidance and an eval, not by a fixed keyword check. **Evals** (integration) - Scheduler creation: "remind me in this channel to submit my timesheet" saves "Automation creator", without "me", "my", or "this channel". - Scheduler delivery: "Remind me to do healthchecks." and "Remind the Automation creator to do healthchecks." both deliver `<@u0test>`. - Scheduler management: a requester who did not create the reminder asks to be reminded too. The update saves `<@u0test>`, without "me", "my", or "Automation creator". **Live results on c570f89** (integration run 41101189024, merged with main 15ee5c1) - All of the cases above pass, and all 6 OAuth cases pass. - 4 other cases fail: the two event-automation "post nothing" cases, "block it and tell only its creator", and scheduler "no message outcome". Each one reads the `finishAutomationRun` result, which was null in the eval session for dispatch runs. They failed the same way on other PRs against the same main, and #2449 fixes the fixture. These failures do not come from this PR. **Not in this PR** - `renderCurrentInstruction` (`chat/current-instruction.ts`) XML-escapes the task text, so the model sees `<@u0test>`. This can also make the model drop a mention. It applies to every current instruction, so it needs its own change. - Moving the creator consumers (credentials, DMs, creator checks, display) from the `createdBy` copy to `creatorIdentityId` is a separate hard cutover. <!-- junior-request-attribution:start --> via **David Cramer**. <!-- junior-request-attribution:end --> <!-- junior-session-footer:start --> <!-- junior-conversation-id:agent%3A13810b1c5e3e841f9cae2bd2d5d58933 --> -- [View Session](https://junior-prod.sentry.dev/conversations/agent%3A13810b1c5e3e841f9cae2bd2d5d58933) [[Sentry]](https://sentry.sentry.io/explore/conversations/agent%3A13810b1c5e3e841f9cae2bd2d5d58933/?project=4510944073809921) <!-- junior-session-footer:end --> --------- Co-authored-by: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com> Co-authored-by: David Cramer <david@sentry.io>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Since #2440 (
15ee5c130), an automation run no longer writes its saved visibility to the Destination row. Only a live Slack signal does. A run into a channel with no row storesunknown, and readers treat it as private.In production, the Slack thread where a person created the Automation records the channel. The eval fixture inserted Automations without that thread, so their channels stayed
unknown. The fixture reads results asWEB_VIEWER_EMAIL, who is not a participant. SoreadConversationAccessFromSqlreturnedcanViewPrivateContent: false, and the reporting API hid tool payloads for every scheduled and event automation run. Tool calls came back with no input and no output (result null [4B]on every tool).finishAutomationRunresults and tool arguments were empty. This broke these cases on unrelated branches (#2435, #2442, fix/flaky-behavioral-evals, chore/remove-slash-command):Event automation delivery(no-match silence ×2, blocked automation) andScheduled Delivery(no message outcome)Scheduled Delivery(post in this channel, provider that Junior does not have) andSentry Skill Workflows(creator-bound scheduled Sentry work)There is no product change. Fail-closed
unknownis intended (#2440). Automations are created from a live Slack turn in their channel, so production channels have a known visibility.Fix
packages/junior-evals/src/fixture/insert.ts:insertScheduledAutomationandinsertEventAutomationnow store the creation thread through the Conversation store withvisibility: "public", as the live Slack event would. No assertions, cases, or timeouts change.Checks
pnpm lint,pnpm typecheck,pnpm --filter @sentry/junior-evals testpass.slackChannel()was redacted (unknown). Channels from either insert function were readable (public).cloudflaredfor eval egress. CI must confirm them.via David Cramer.
--
View Session [Sentry]