Add exec-brief-scenario command for the Entity Analytics AI brief by hop-dev · Pull Request #493 · elastic/security-documents-generator

hop-dev · 2026-10-19T13:22:07Z

Summary

Adds exec-brief-scenario, which seeds one space with the data the Entity Analytics AI brief (elastic/kibana#297677) needs to say something useful, and writes a ground truth to check the brief against.

yarn start exec-brief-scenario

What it seeds

Five days of data, anchored to now (or --anchor):

S1, needs action: a privileged user with stolen credentials moves from their laptop to a jump box and escapes a production container. Alerts follow the attack stages day by day, the user is resolved to their IdP identity, and the hosts have asset criticality. A brief should lead with a.rodriguez, LAPTOP-FIN03, jump-box-01 and docker-host-prod-01.
S2, being handled: MFA bombing and a session hijack on a marketing user, with an open case, so a brief can tell it apart from S1.
S3, no attack discovery: an SSH password spray and curl exfiltration on a build server.
Noise: ordinary users and hosts, so the priorities have something to beat.

Around those it runs the rules, risk scoring, entity relationships and resolution, the privileged watchlist, vulnerabilities, attack discoveries (--ad-mode), a lead and ML anomalies (skipped with --no-ml).

Output

exec-brief-output/ground_truth.json describes each storyline: entities, alerts, attack-stage order and expected priorities. verification.txt checks the seeded data against it, so a failed seed shows up before you generate a brief.

Other changes

The entity history index is now .entities.v2.history.<space>.*, matching Entity Store v2. This fixes entity_maintainers and the space clean-up in risk_score_v2.
kibanaFetch takes extra headers, and runEntityMaintainer can run synchronously.
createVulnerabilities takes an optional hostId, so vulnerabilities attach to the right host entity.

Testing

Against a Kibana running elastic/kibana#297677:

yarn start exec-brief-scenario --clean

Check verification.txt passes, then open Security > Entity Analytics, pick "Last 7d" and click "Generate AI brief".

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ream targets work

…s shape, with legacy fallback

… page in Agent Builder (elastic#297677) ## Summary Adds a "Generate AI brief" button to the header of the new Entity Analytics page, shown when Agent Builder is available to the user. It opens Agent Builder with the view the user is looking at, and a new tool turns that view into a short brief: what needs attention, the period's counts, a recommended next step and up to three entities to look at first. The brief is posted to the conversation as an inline HTML card, and the agent adds one sentence. Follow-up questions use the existing Entity Analytics tools. This is V0 of elastic/security-team#20150. | The page | The brief | | --- | --- | | <img width="1043" height="693" alt="Screenshot 2026-10-22 at 14 33 43" src="https://github.com/user-attachments/assets/a234baf4-23f5-4d57-8364-0a517d50e132" /> | <img width="424" height="954" alt="Screenshot 2026-10-22 at 13 33 12" src="https://github.com/user-attachments/assets/89d16019-efb5-4c6b-9983-1125f2d90833" /> | ## What the brief contains In the order it is read: - **Title and scope:** a headline the model writes for the period's pattern (at most eight words, no counts or entity names), then the period, every filter the page applied as the page labels them, and when the brief was captured. Notes follow only when they apply: counts that couldn't be read, flagged entities missing from the Entity Store, alert details that couldn't be read, or that the brief covers a sample. - **What needs attention:** one or two sentences on the pattern of the period, not the counts. - **This period at a glance:** one row with the page's counts and their change, copied from the page and never summed. A count that is zero and didn't change is left out. - **Recommended next step:** one step across the whole view, above the priorities so it is never below the card's fold. - **Priorities:** up to three entities. Each has its title row with pills for risk (a resolution group's carries the icon the entity table uses), asset criticality with the page's colour dot, and every watchlist the entity is on, named as the flyout names them, three at most then "+N more". Then the model's assessment, why the entity ranks first, a next step and, for the first priority a risk count flags, its risk history. - The **assessment** is one sentence on what the evidence looks like taken together, and **why first** is one sentence on why the entity ranks ahead of the others. The card states the facts and the model interprets them, so neither repeats the risk score, criticality, privilege or first seen beside them, and both name what to confirm rather than what happened. - The **alert rules and anomaly jobs** are named inside those sentences with their counts. The model is given each entity's two most important rules (the highest alert risk score first, then the most alerts), the number of other rules, and the Security ML job with the highest anomaly score when that job has a display name, plus the number of jobs. It cites them only as `{{ rule <index> }}` and `{{ job <index> }}`, and the view model resolves the tokens to names, so neither can be invented. A token that names nothing of that entity is removed. - The **risk history** states the first and last score as text and draws a chart from three or more points. The existing `security.entity` table is added under the card, because it is the way to open an entity from the brief. The agent's reply is one sentence, the brief, the line "To go further, open a priority from the table below, or ask me about any of them." and the table. ## How it's generated The button captures the view into a `security.entity_analytics_ai_brief` attachment: the period, entity filters, the selected count, and each count's title, description, value, change and up to 200 entity ids. Each captured view brings its own brief tool, bound to its attachment: the tool exists only in a conversation that holds a view, so it isn't one of Agent Builder's built-in tools and can't be used from a general chat. The tool checks the user's Entity Analytics privileges in the space, then runs the steps listed in `server/agent_builder/tools/entity_analytics/brief/index.ts`: 1. **Read entities:** one Entity Store search for the flagged ids, as the user, in the request's space. The brief follows the counts, so it doesn't apply the page's search or filter pills, which the counts don't apply either. Candidates rank by the selected count, then how many counts flag them, then risk, and the top 25 go on. A second search reads their resolution group members (`fetchGroupMembers`, shared with case counts), so a group has the values the page shows. Risk is resolution first. 2. **Read alert rules and anomalies:** one query on the space's alerts index, as the user, for the open alerts of the period, and one over the ML results index restricted to the installed Security jobs. An alert is matched on the `kibana.alert.entity.id` it is stamped with. An alert without a stamp is matched, as the tiles derive it, on the identity fields of the candidate's and its members' records with the Entity Store's own record filter (`euid.dsl.getEuidFilterBasedOnEntityRecord`). A group's alerts are its members' alerts, counted once, with at most 20 members matched by record. For each candidate it returns the alert count, the two rules ranked by their most severe alert and then by count, and how many other rules raised alerts. Anomalies are matched the same way and summed per candidate, naming only a job the page itself names. If the user can't read alerts or anomalies, the brief says so in its notes or carries on without them. 3. **Build facts:** one JSON object per candidate, with why it's flagged, its risk, criticality, alerts and rules, watchlists and when it was first seen. 4. **Draft:** one `inferenceClient.output()` call through the conversation's model, retried once if the answer doesn't match the schema. A priority's text is two fields, `assessment` and `whyFirst`. Everything sent to a model, including what the agent is told, is in `brief/prompts.ts`. 5. **Verify:** each priority must be about a different flagged entity, at most three, mentions must be flagged entities, and the title keeps none. 6. **View model:** `brief/view_model.ts` turns the verified draft, the facts and the page's counts into one versioned zod schema, and resolves entity and rule mentions to names. It is the seam: the renderer reads nothing else, so a new signal is an optional field there and a function in the renderer. 7. **Render and post:** `brief/render_brief_html.ts` is a pure function, `renderBriefHtml(viewModel) => { html, height }`. The markup is `brief.mustache.html`, the styles `brief.css`, and the risk history chart a partial, `risk_history_chart.mustache.html`, with its geometry in `risk_history_chart.ts` (d3-scale). They are read once and rendered with Mustache, whose escape is ours, so every `{{ }}` is HTML-escaped with `{` included. The tool adds the HTML with `attachments.add` as a `visualization` attachment with `renderer: custom_content`, and returns the render tags with a short result, so the agent writes one sentence, the brief tag, the hand-off line and the entity table tag. Everything the template prints goes through one escape function passed to Mustache (HTML entities, and `{` replaced), and entity names come from the Entity Store record, never from model text. The HTML uses only the `--cc-*` theme tokens the card injects, so it follows light and dark. The card runs the HTML through its own sanitiser, and a test checks that the brief passes through it unchanged. The card is at most 1,200 px tall. The height is estimated from the sections, because the sandboxed card can't measure its content, and a brief with three priorities and a chart runs to about 1,300 px, so its third priority scrolls inside the card. The recommended step and the first two priorities are always above the fold. Long watchlist names truncate and the pills wrap under the title rather than squeezing it. ## Outside the brief's own folder - `buildRenderAttachmentTag` also accepts the hyphenated ids `attachments.add` returns. - `security_solution`'s `tsconfig.json` and `moon.yml` gain references to `@kbn/custom-content-common` for the card's height limits and `@kbn/custom-content-renderer` for the sanitiser used by one test. Mustache and d3-scale were already dependencies of the plugin or the repo. ## Tests - **Jest, in `brief/`:** the view model (notes, counts, priorities, sample line, quiet views, watchlist names, and entity, rule and job tokens: resolved, an unknown index, another entity's, a bare "rule 1"), the renderer (a snapshot of the HTML, the counts rule, escaping of model text and names, the empty brief, the chart from three points and the caption from two, and the pass through the card's sanitiser under jsdom), the alert rules and anomalies (the aggregation shapes, the risk-score ranking, the counts of other rules and of jobs, the Security-job and display-name rules, the filter for stamped and unstamped alerts, the cap of 20 members matched by record, and a failed read), and the existing steps. - **Jest, tool:** the privilege check. - **Scout:** the main brief (the tool result carries the tags, the conversation holds the `visualization` attachment with the priority name, and the agent's reply holds both tags unchanged), the quiet brief without a model call, and a user restricted to another space. Measured on the demo seed with Claude Sonnet 5: tags copied intact in 10 of 10 runs, the agent's own text 22 to 38 words, median 11.2 s to the last token against 15.1 s for the earlier markdown brief, and 880 output tokens against 1,537. ## Testing 1. Check out this branch and bootstrap. Add this to `config/kibana.dev.yml`: ```yaml feature_flags.overrides: securitySolution.useNewEntityAnalyticsPage: true ``` 2. Start Elasticsearch and Kibana, and make sure Agent Builder has a default model. 3. Seed the demo scenario with the documents generator's `exec-brief-scenario` command, from [elastic/security-documents-generator#493](elastic/security-documents-generator#493): ```bash git clone -b exec-brief-scenario https://github.com/hop-dev/security-documents-generator.git cd security-documents-generator && yarn && yarn start exec-brief-scenario ``` It asks for your Elasticsearch and Kibana details on first run, and writes `exec-brief-output/ground_truth.json` with the entities a brief should prioritise. Add `--no-ml` if ML jobs don't start in your environment. 4. Open Security > Entity Analytics, pick "Last 7d" and click "Generate AI brief". Expect priorities from the main storyline: a.rodriguez, jump-box-01 and docker-host-prod-01, their alert rules named in each assessment, and a risk history under a.rodriguez. 5. Try "Last 24h" with Risk level Critical (a quiet brief with no model call), and filtering the table by a count before generating (its entities rank first). 6. Switch Kibana to dark mode and generate again. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

Copilot

🟡 Changes recommended

Cleanup and risk-copy paths can lose unrelated data, while replay, ML lifecycle, and verification issues undermine scenario reliability.

10 open issues

Scope ML cleanup to scenario-owned jobs · New
Scope risk rebadging to scenario entities · New
Verify bulk copies before deleting original risk documents · New
Reconcile alert references per discovery on retries · New
Cancel and await ML generation after timeout · New
Clear the timeout timer when ML settles · New
Reset scenario alerts before replaying the risk timeline · New
Validate tactic IDs and their expected order · New
Skip Attack Discovery verification when mode is none · New
Restrict case verification to seeded S2 alerts · New

What changed in this PR

Adds exec-brief-scenario to seed correlated security data for the Entity Analytics AI brief, with ground truth and verification output for evaluating the generated brief.

Changes:

Adds three attack storylines, background activity, and supporting entity, risk, discovery, case, vulnerability, lead, and ML data.
Adds command options, cleanup, chronological seeding, and verification reports.
Updates Entity Store v2 history paths and extends shared helpers for headers, synchronous maintainers, and vulnerability host IDs.

File	Description
`src/utils/kibana_api.ts`	Supports extra headers and synchronous maintainer runs.
`src/index.ts`	Registers the scenario command module.
`src/generators/create_vulnerability.ts`	Adds optional host IDs to vulnerabilities.
`src/commands/exec_brief/verify.ts`	Checks seeded data and scenario expectations.
`src/commands/exec_brief/utils.ts`	Provides deterministic IDs, timestamps, and bulk utilities.
`src/commands/exec_brief/source_events.ts`	Builds and indexes phased source events.
`src/commands/exec_brief/scenario.ts`	Defines entities, alerts, rules, and storylines.
`src/commands/exec_brief/run.ts`	Orchestrates seeding and writes output reports.
`src/commands/exec_brief/rules.ts`	Installs rules with custom fallbacks.
`src/commands/exec_brief/risk.ts`	Runs risk scoring and backdates history.
`src/commands/exec_brief/ml.ts`	Wraps optional ML generation with a timeout.
`src/commands/exec_brief/index.ts`	Defines CLI options and validation.
`src/commands/exec_brief/ground_truth.ts`	Builds the expected scenario record.
`src/commands/exec_brief/extras.ts`	Seeds discoveries, cases, vulnerabilities, and leads.
`src/commands/exec_brief/entities.ts`	Extracts entities and applies relationships and modifiers.
`src/commands/exec_brief/clean.ts`	Implements scenario and ML cleanup.
`src/commands/exec_brief/alerts.ts`	Builds correlated detection alert documents.
`src/commands/entity_store/risk_score_v2.ts`	Corrects the history index cleanup pattern.
`src/commands/entity_store/entity_maintainers.ts`	Corrects history snapshot index names.

_{Review effort: Balanced}

+    const jobs = await client.ml.getJobs({ job_id: '_all', allow_no_match: true });
+    const ids = jobs.jobs.filter((j) => j.groups?.includes('security')).map((j) => j.job_id);


+  if (!stamp) return { rebadged: 0 };
+
+  await client.indices.refresh({ index: riskIndex(space), ignore_unavailable: true });
+  const query = { range: { '@timestamp': { gte: runStart } } };


+  await client.bulk({ operations, refresh: true });
+  await client.deleteByQuery({


+  result.attackIdsBackReferenceNative = withBackRef.count > 0;
+  if (!result.attackIdsBackReferenceNative) {


+    await Promise.race([
+      generateAnomalousBehaviorDataWithMlJobs(space, 6, false, true, true),


+        setTimeout(
+          () => reject(new Error(`ML step timed out after ${ML_TIMEOUT_MS / 1000}s`)),
+          ML_TIMEOUT_MS,
+        );


+    // Idempotent on retries: the day loop rebuilds the whole risk history.
+    await cleanRiskDocs(space);
+    const alerts = buildAlerts();


+      const pass =
+        s1Tactics.length >= 5 && missing === 1 && ecs.includes('TA0008') && prodTactics.length >= 2;


+      return check(
+        '9',
+        'Attack Discovery (S1 + S2) via _find',
+        (res.total ?? titles.length) >= 2 && direct.count >= 2,


+      const res = await client.search({
+        index: getAlertIndex(space),
+        size: 0,
+        query: { exists: { field: 'kibana.alert.case_ids' } },


hop-dev and others added 7 commits October 8, 2026 22:28

Add exec-brief-scenario command and helper fixes

cffd2fa

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

exec-brief-scenario: tune risk, idempotent retries, verification

c0d0e25

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

exec-brief-scenario: checker hints

e76ef62

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

exec-brief-scenario: idempotent retries, ML cleanup, lead index

99fd937

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

exec-brief-scenario: write vulnerabilities with create ops so data-st…

bf899d1

…ream targets work

exec-brief-scenario: case alert attachments in the unified attachment…

8dd8499

…s shape, with legacy fallback

Merge elastic/main into exec-brief-scenario

aba1058

hop-dev mentioned this pull request Oct 19, 2026

[Entity Analytics] AI brief: generate a brief of the Entity Analytics page in Agent Builder elastic/kibana#297677

Merged

hop-dev marked this pull request as ready for review October 22, 2026 21:41

hop-dev requested a review from a team as a code owner October 22, 2026 21:41

hop-dev requested review from ymao1 and a balanced review from Copilot October 22, 2026 21:41

Copilot started reviewing on behalf of hop-dev October 22, 2026 21:42

View session

hop-dev added 2 commits October 22, 2026 22:43

prettier

0a39c58

Merge branch 'main' into exec-brief-scenario

97e3d3b

hop-dev merged commit 0946fc4 into elastic:main Oct 22, 2026
2 checks passed

hop-dev deleted the exec-brief-scenario branch October 22, 2026 21:45

Copilot AI reviewed Oct 22, 2026

View reviewed changes

Navigation Menu

Repository navigation

Add exec-brief-scenario command for the Entity Analytics AI brief - #493

Add exec-brief-scenario command for the Entity Analytics AI brief#493
hop-dev merged 9 commits into
elastic:mainfrom
hop-dev:exec-brief-scenario

hop-dev commented Oct 19, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		const jobs = await client.ml.getJobs({ job_id: '_all', allow_no_match: true });
		const ids = jobs.jobs.filter((j) => j.groups?.includes('security')).map((j) => j.job_id);

		await client.bulk({ operations, refresh: true });
		await client.deleteByQuery({

		result.attackIdsBackReferenceNative = withBackRef.count > 0;
		if (!result.attackIdsBackReferenceNative) {

		await Promise.race([
		generateAnomalousBehaviorDataWithMlJobs(space, 6, false, true, true),

		const pass =
		s1Tactics.length >= 5 && missing === 1 && ecs.includes('TA0008') && prodTactics.length >= 2;

Conversation

hop-dev commented Oct 19, 2026

Summary

What it seeds

Output

Other changes

Testing

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants