Add guidance on splitting an event into multiple documents - #8110
Open
bmorelli25 wants to merge 8 commits into
Open
Add guidance on splitting an event into multiple documents#8110bmorelli25 wants to merge 8 commits into
bmorelli25 wants to merge 8 commits into
Conversation
Adds a how-to page under Transform and enrich data clarifying that Elastic Agent processors and Elasticsearch ingest pipelines cannot split one event into multiple documents, and documenting the two supported options: splitting at collection time via input settings, or a Logstash pipeline with the elastic_integration and split filters. Also adds short pointers from the pages where users hit this limitation: the Agent processors reference (limitations list and options table), the ingest pipelines page, the transform-enrich landing page, and the Agent-to-Logstash reference architecture. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
Elastic Docs AI PR menuCheck the box to run an AI review for this pull request.
Powered by GitHub Agentic Workflows and docs-actions. For more information, reach out to the docs team. |
Contributor
Contributor
✅ Elastic Docs Style Checker (Vale)No issues found on modified lines! The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale. |
Fixes from a source-code validation pass against product HEAD: - "one event in, one document out" -> processors and pipelines can also drop an event, so state the model as transform-or-drop (verified against the Processor contracts in elasticsearch and beats, which are strictly one-in, at-most-one-out) - "CEL Custom API integration" -> "CEL Custom API input integration" and "Custom TCP" -> "Custom TCP Logs", matching the actual package titles in elastic/integrations - integrations "require" data streams/ECS output -> "are designed to work with", matching the logstash-output-elasticsearch guidance (the hard error is only data_stream with ecs_compatibility disabled) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Docs convention: don't call out explicit subscription levels; link to elastic.co/subscriptions instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The splitting caveat was too prominent as a note in the page intro. Move it into a new Limitations section at the end of the page, alongside two related consequences of the one-document-at-a-time model, both verified against elasticsearch source at HEAD: - no cross-document reads except the enrich processor (the only shipped processor besides inference that receives a client, and the only one that queries indexed documents) - pipelines run at index time only; reindex (dest.pipeline) or update by query (pipeline param) apply a pipeline to existing documents, and update by query works on data streams Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Reword the agent-ls ingest model line as a full sentence - Remove the splitting signpost from the transform-enrich landing page - Trim the agent-processors limitation bullet to the bare statement (the options table below it keeps the link to the how-to page) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Move the split-processor disambiguation out of the intro paragraph into a note after the scoping paragraph - Replace "simplest" with "easiest" phrasing - Trim the collection-time section's closing sentence Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
shainaraskas
approved these changes
Aug 27, 2026
shainaraskas
left a comment
shainaraskas
left a comment
Member
There was a problem hiding this comment.
approving the toc change because we own it. lgtm? 🤷
nastasha-solomon
approved these changes
Aug 28, 2026
nastasha-solomon
left a comment
nastasha-solomon
left a comment
Member
There was a problem hiding this comment.
new page looks v nice, v good
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Users frequently assume that Elastic Integrations or Elasticsearch ingest pipelines can split one incoming event into multiple documents. They (for the most part) can't (see elastic/elasticsearch#56769). These docs attempt to reduce confusion around this topic.
Closes #6491.
Changes
Adds a new page: "Split an event into multiple documents" (
manage-data/ingest/transform-enrich/), structured to avoid Logstash unless actually required:splitprocessor splits a field value into an array, but doesn't create documentsrerouteprocessor and@custompipelinesAdds a few signposts linking to the new page
Attribution
🤖 Generated with Claude Code