[Security][CPS] Document detection rule behavior and correct app availability - #8064
[Security][CPS] Document detection rule behavior and correct app availability#8064nastasha-solomon wants to merge 20 commits into
Conversation
Elastic Docs AI PR menuCheck the box to run an AI review for this pull request.
Powered by GitHub Agentic Workflows and docs-actions. For more information, reach out to the docs team. |
✅ Elastic Docs Style Checker (Vale)No issues found on modified lines! The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale. |
Co-authored-by: Cursor <cursoragent@cursor.com>
shainaraskas
left a comment
There was a problem hiding this comment.
hello thank you!
overall thoughts:
- I think it would be good if we could be clearer about when things use the space scope
- We can probably cut back/unify some of the info about API keys on rules - it reads as pretty cohesive between the two surfaces now and we should probably just document what is different about detection rules in this spot
| ::: | ||
|
|
||
| - **{{ml-cap}}:** {{anomaly-detect-cap}} job {{dfeeds}} and transforms can read data from linked projects. Jobs and results live on the origin project. {{ml-cap}} rules alert on those origin results, including anomalies produced from linked-project data. | ||
| - **Timeline:** Tables display documents from linked projects. Actions that don't apply to remote documents are disabled. |
There was a problem hiding this comment.
is this using the space scope?
There was a problem hiding this comment.
@PhilippeOberti @michaelolo24 does Timeline always use the space’s default CPS scope, or can it follow the CPS scope selector selection in the project's header? For reference, here's the latest draft.
| - **Dashboards:** The Detection & Response and Data Quality dashboards support {{cps-init}}. | ||
| - **Intelligence:** Threat intelligence indicator searches support {{cps-init}}. |
There was a problem hiding this comment.
@chuddy-elastic who can clarify whether the threat intel features always use the space’s default CPS scope, or can follow the CPS scope selector selection in the project's header? For reference, here's the latest draft.
Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com>
…y-security-apps.md Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com>
…y-security-apps.md Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com>
…y-security-apps.md Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com>
shainaraskas
left a comment
There was a problem hiding this comment.
looking nice! a couple of considerations for you about the table columns and some questions about rule scope.
the cutback on the API keys stuff for rules is looking especially nice 🎉
| **Max alerts per run** (optional) | ||
| : Specify the maximum number of alerts the rule can create each time it executes. Default is 100. | ||
|
|
||
| {applies_to}`serverless: preview` When {{cps}} is enabled, this limit covers the combined results from the origin project and all linked projects in the space-level scope rather than each project separately. Refer to [{{cps-cap}} and detection rules](/solutions/security/detect-and-alert/cross-project-search-detection-rules.md#cps-rules-max-alerts). |
There was a problem hiding this comment.
🤔 if we can limit the scope with project routing, is this exactly true? or is it the scope of the query, which is by default the same as the space-level scope unless there's a query-level override?
There was a problem hiding this comment.
Hm, good callout. I'm unsure what the relationship is and if the setting supersedes the space-level scope and or query-level overrides. @chuddy-elastic can you answer this?
Claude edited this description as follows:
{applies_to}
serverless: previewWhen {{cps}} is enabled, this limit is shared across every project the rule searched, not applied separately to each project. A query-level override (SET project_routingor a qualified index expression) changes which projects that execution searches; the limit still applies once. Refer to {{cps-cap}} and detection rules.
|
|
||
| If your data spans ECH, ECE, ECK, or self-managed clusters rather than linked {{serverless-short}} projects, refer to [{{ccs-cap}} and detection rules](/solutions/security/detect-and-alert/cross-cluster-search-detection-rules.md) instead. | ||
|
|
||
| ## Which projects a rule searches and where alerts are stored [cps-rules-scope-and-alerts] |
There was a problem hiding this comment.
is this stuff covered in the core alerting/rules docs?
There was a problem hiding this comment.
At the moment, it's not. @chuddy-elastic @tiamliu who would be able to provide this information for Kibana alerting v1 and Observability incident management?
shainaraskas
left a comment
There was a problem hiding this comment.
thank you for these edits! just a couple more comments, mostly around precision re: rule scope, and some stuff about making the snippet fit for purpose across its two homes :) ✅ anyway
|
|
||
| - **{{esql}} rules:** Add [`SET project_routing`](/explore-analyze/cross-project-search/cross-project-search-project-routing.md) at the start of the rule query to override the space-level scope. | ||
| - **Rules that use index patterns:** Use [qualified index expressions](/explore-analyze/cross-project-search/cross-project-search-search.md#search-expressions) in the index pattern to target specific projects. | ||
| - **{{ml-cap}} rules:** These rules alert on {{anomaly-detect}} results stored on the origin project. {{anomaly-jobs-cap}} can read linked-project data; jobs and results stay on the origin. |
There was a problem hiding this comment.
I think that this doesn't get to the heart of the takeaway, but is more like implementation detail. consider
| - **{{ml-cap}} rules:** These rules alert on {{anomaly-detect}} results stored on the origin project. {{anomaly-jobs-cap}} can read linked-project data; jobs and results stay on the origin. | |
| - **{{ml-cap}} rules:** These rules alert on {{anomaly-detect}} job results, so the job's scope determines what the rule can detect. To control which projects the job searches, set [`project_routing`](/explore-analyze/machine-learning/anomaly-detection/ml-ad-run-jobs.md) on the job's {{dfeed}}. |
| **Max alerts per run** (optional) | ||
| : Specify the maximum number of alerts the rule can create each time it executes. Default is 100. | ||
|
|
||
| {applies_to}`serverless: preview` When {{cps}} is enabled, this limit is shared across every project the rule searched, not applied separately to each project. A query-level override (`SET project_routing` or a qualified index expression) changes which projects that execution searches. The limit still applies once. Refer to [{{cps-cap}} and detection rules](/solutions/security/detect-and-alert/cross-project-search-detection-rules.md#cps-rules-max-alerts). |
There was a problem hiding this comment.
this is not exactly true because not all rules allow query-level overrides. we prob want to lean on https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/8064/explore-analyze/alerting/alerts/create-manage-rules#cps-scope-for-rules
|
|
||
| The **Max alerts per run** [advanced setting](/solutions/security/detect-and-alert/common-rule-settings.md#rule-ui-advanced-params) limits the number of alerts a rule creates in a single execution. Under {{cps}}, that limit covers the combined results from every project the rule queries in that run, rather than each project separately. By default, the rule queries the origin project and all linked projects in the space-level scope. The default limit is 100. | ||
|
|
||
| A rule that stayed under the limit on a single project can reach the limit after you link projects, which leaves matching events without alerts. Review the limit for rules that run across a broad scope. To search fewer projects, add [`SET project_routing`](/explore-analyze/cross-project-search/cross-project-search-project-routing.md) at the start of the **{{esql}} query**, or enter a [qualified index expression](/explore-analyze/cross-project-search/cross-project-search-search.md#search-expressions) in **Index patterns**, instead of raising the limit. |
There was a problem hiding this comment.
these are not the only ways
a) space scope
b) these two
c) ml job
consider linking up/out to wherever solutions/_snippets/cps-sec-obs-rules.md goes
There was a problem hiding this comment.
sorry I keep coming back to this single snippet.
The problem we have now is that it's being used to do two things:
... but it's not doing double duty as well because more things work now
I suggest that we do the following:
- kill the snippet
- put the content directly on Managing cross-project search scope in your project apps and rebrand as "support" rather than "limitations"
- Create a mini list reduced to limitations, and put it on Configure cross-project search, and link across
shortlist of limitations might look like this
- **Alerts:** The Alerts page doesn't show alerts that a linked project generated independently. Only alerts created by origin project rules appear.
- **Attack Discovery:** Discoveries are based on origin project alerts only. Alerts from linked projects aren't included.
- **Cases:** You can't attach an alert or event from a linked project to a case.
- **Entity store:** A host that appears in more than one project isn't combined into a single entity at the origin. Risk scoring runs on the origin project only.
- **Flyouts:** Session View isn't available for documents from linked projects. Some actions are hidden or disabled.
- **{{elastic-defend}} and Osquery:** Policies, artifacts, response actions, and Osquery saved queries and packs can't be shared or managed across linked projects.
- **{{ml-cap}} rules:** {{ml-cap}} rules don't use the space-level {{cps}} scope. They use the scope of the underlying {{anomaly-detect}} job's {{dfeed}}, which might differ from the space default.
- **SIEM Readiness and Value report:** These features don't include data from linked projects.
- **Timeline:** Some actions are disabled for documents from linked projects.
| - **Detection rules:** {{esql}} rules support `SET project_routing`. For non-{{esql}} rules that use index patterns, you can use [qualified index expressions](/explore-analyze/cross-project-search/cross-project-search-search.md#search-expressions). Origin rules write alerts to the origin project. The **Max alerts per run** limit applies across the projects the rule queries. A rule searches only the linked projects the user who last saved it can access. For details, refer to [{{cps-cap}} and detection rules](/solutions/security/detect-and-alert/cross-project-search-detection-rules.md). | ||
| - **{{elastic-defend}} and Osquery:** The **Endpoints** page, host details, **Response actions history**, and Osquery query results include data from linked projects. Policies, artifacts, response action dispatch, and Osquery saved queries and packs stay per project because they're managed through Fleet. | ||
| - **Entity store:** Origin profiles include entities from every linked project. Each project still builds its own store, and risk scoring stays on the origin project. A host that appears in more than one project isn't combined into a single entity at the origin. | ||
| - **{{ml-cap}}:** {{anomaly-detect-cap}} job {{dfeeds}} can read data from linked projects. Jobs and results are stored on the origin project. {{ml-cap}} rules alert on those stored results, including anomalies produced from linked-project data. |
There was a problem hiding this comment.
siiide note, I was doing a little poking around for another ml thing and I suspect that when i try to start a prebuilt job using this flow, it searches every linked project (doesn't use the space scope): https://www.elastic.co/docs/solutions/security/advanced-entity-analytics/anomaly-detection#manage-jobs
might be worth adding as a limitation and/or tweaking that page directly (or maybe it's a bug? :/)
Summary
Contributes to #8050.
Corrects the cross-project search (CPS) availability information for Elastic Security apps and documents rule-level CPS behavior ahead of GA.
Previews
CPS availability in Elastic Security apps - Corrected and expanded the availability list. (The same snippet also renders on Configure CPS.) Made the following changes to the section and moved the info into a table to align the format used for Kibana and Observability.
CPS and detection rules — Expands the detection-rules row. The intro snippet also renders on Observability Create and manage rules, and states space default, read-only selector, and query-level overrides inline:
Common rule settings - Max alerts per run notes that, with CPS enabled, the limit covers combined results from the origin and linked projects in the space-level scope, with a link to the full explanation.
Deliberately not included
The "Using cross-project search with Elastic Security" guidance page (item 2 of #8050) is not included as it's not a blocker to CPS GA'ing. It will be added in a separate PR.
Lifecycle note
CPS pages remain at
serverless: preview. Flipping them to GA is intentionally left to a separate PR. The one inline{applies_to}tag added here, on Max alerts per run, will need to flip with them.Generative AI disclosure