[FEATURE] Add support for ELK - #41
Open
Grand-Duc wants to merge 2 commits into
Open
Conversation
There was a problem hiding this comment.
Pull request overview
This PR introduces Elastic (ELK) stack support alongside the existing Splunk integration, updating setup/config flows and docker orchestration so users can choose splunk, elasticsearch, or both as the SIEM backend while keeping module parsing/normalization consistent.
Changes:
- Add Elasticsearch/Kibana services and related Dockerfiles/templates/pipelines to the master docker-compose stack.
- Extend master/agent setup + sample configs to capture and persist Elastic connection details and SIEM selection (including port persistence into
.envfor compose). - Add an internal Elastic indexer module (
indexer_elastic) and wire UI links (sidebar) to point to configured Splunk/Kibana endpoints.
Reviewed changes
Copilot reviewed 22 out of 25 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| setup/setup_scripts/setup_docker.sh | Persists selected ports into .env before running docker compose and ensures Elastic data dir exists/owned correctly. |
| setup/setup_scripts/save_setup_config.sh | Writes master/agent YAML config files with new SIEM + Elastic placeholders. |
| setup/setup_scripts/clean_elastic.sh | New helper to erase Elastic data directory for re-install scenarios. |
| setup/setup_scripts/check_elastic.sh | New helper to detect existing Elastic data before (re)install. |
| setup/master/master_setup.sh | Adds SIEM selection (splunk/elastic/both), elastic install paths, and order-independent compose profile accumulation. |
| setup/master/dockerfile/sources/elastic/template.json | Adds an index template tuned for OSIR ingestion into Elasticsearch. |
| setup/master/dockerfile/sources/elastic/pipeline.json | Adds an ingest pipeline to promote timestamp to @timestamp and set event.ingested. |
| setup/master/dockerfile/MasterFile | Extends PATH to include /OSIR/OSIR/bin in master image. |
| setup/master/dockerfile/KibanaFile | New Kibana image definition wired to local Elasticsearch. |
| setup/master/dockerfile/ElasticFile | New Elasticsearch image definition (single-node) with performance/settings tweaks. |
| setup/master/docker-compose.yml | Adds Elasticsearch/Kibana services (online/offline profiles) and makes Splunk ports configurable via env. |
| setup/master/.env.example | Documents Elastic/Kibana env vars in the example env file. |
| setup/conf/master_sample.yml | Adds SIEM selection + Elastic settings to the master sample YAML. |
| setup/conf/agent_sample.yml | Adds Elastic settings to the agent sample YAML. |
| setup/agent/dockerfile/sources/requirements.txt | Adds the Python elasticsearch client to agent image dependencies. |
| setup/agent/dockerfile/AgentFile | Updates PATH and installs Elastic Python client (currently redundantly). |
| setup/agent/agent_setup.sh | Extends agent setup to collect/export Elastic settings and SIEM choice. |
| requirements.txt | Adds Python elasticsearch dependency at repo level. |
| README.md | Updates TOC and documents Elastic support + new elastic indexer module. |
| OSIR/src/osir_web/osir_web/utils/OsirWebSidebar.py | Adds Kibana link and uses configured ports/hosts for sidebar links. |
| OSIR/src/osir_lib/osir_lib/modules/indexer/indexer_elastic.py | New internal module to dispatch json2elastic-rs over parsed module outputs. |
| OSIR/src/osir_lib/osir_lib/core/OsirAgentConfig.py | Adds ElasticSearch configuration model + accessors to agent config. |
| OSIR/configs/modules/elastic/indexer_elastic.yml | Adds the Elastic indexer module configuration invoking json2elastic-rs. |
| .gitignore | Ignores Elastic data directory and .opencode file. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+52
to
+53
| sed -i "s/{local_elastic_previous_data}/$LOCAL_ELASTIC_PREVIOUS_DATA/g" $conf | ||
| sed -i "s/{local_splunk_previous_data}/$LOCAL_SPLUNK_PREVIOUS_DATA/g" $conf |
| (echo >&2 "${INFO} $elastic_path contains files, ElasticSearch was previously installed.") | ||
| exit 1 | ||
| else | ||
| (echo >&2 "${INFO} $elastic_path does not contain file, ElasticSearch can be installed.") |
Comment on lines
+142
to
+146
| elasticsearch: | ||
| build: | ||
| context: ./dockerfile | ||
| dockerfile: ElasticFile | ||
| container_name: master-elasticsearch |
Comment on lines
+194
to
+198
| kibana: | ||
| build: | ||
| context: ./dockerfile | ||
| dockerfile: KibanaFile | ||
| container_name: master-kibana |
Comment on lines
+10
to
+12
| # Enable CORS so Kibana / external dashboards can query from the browser if needed. | ||
| RUN echo 'http.cors.enabled: true' >> /usr/share/elasticsearch/config/elasticsearch.yml && \ | ||
| echo 'http.cors.allow-origin: "*"' >> /usr/share/elasticsearch/config/elasticsearch.yml |
Comment on lines
+4
to
+8
| ENV discovery.type=single-node \ | ||
| xpack.security.enabled=false \ | ||
| xpack.security.http.ssl.enabled=false \ | ||
| ES_JAVA_OPTS="-Xms8g -Xmx8g" \ | ||
| bootstrap.memory_lock=true |
Comment on lines
67
to
+80
| try: | ||
| agent_config = OsirAgentConfig() | ||
| host = agent_config.master_host | ||
| splunk_host = agent_config.splunk_host if agent_config.splunk_host not in ["host.docker.internal", "127.0.0.1"] else host | ||
| elastic_host = agent_config.elastic_host if agent_config.elastic_host not in ["host.docker.internal", "127.0.0.1"] else host | ||
| splunk_port = agent_config.splunk_port | ||
| kibana_port = agent_config.elastic_kibana_port | ||
| except FileNotFoundError: | ||
| # agent.yml missing -> happens if master launched before agent is installed | ||
| host = "localhost" | ||
| splunk_host = host | ||
|
|
||
| elastic_host = host | ||
| splunk_port = 8000 | ||
| kibana_port = 5601 |
Comment on lines
+96
to
115
| host: str | ||
| user: str | ||
| password: str | ||
| port: int | ||
| ssl: bool | ||
| kibana_port: int = 5601 | ||
|
|
||
|
|
||
| class FullAgentConfig(BaseModel): | ||
| """ | ||
| The root validation model for the 'agent.yml' configuration file. | ||
|
|
||
| It acts as a single point of truth, validating that the Master, Windows Box, | ||
| and Splunk sections are correctly formatted and present before the agent starts. | ||
| Splunk and ElasticSearch sections are correctly formatted and present before the agent starts. | ||
| """ | ||
| master: MasterConfig | ||
| windows_box: WindowsBoxConfig | ||
| splunk: SplunkConfig | ||
| elasticsearch: ElasticConfig | ||
|
|
Comment on lines
60
to
+62
| RUN set -eux; \ | ||
| python -m pip install --no-cache-dir --break-system-packages --ignore-installed -r /tmp/requirements.txt | ||
| python -m pip install --no-cache-dir --break-system-packages --ignore-installed -r /tmp/requirements.txt; \ | ||
| python3.12 -m pip install --no-cache-dir --break-system-packages elasticsearch |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Introduction of ELK stack support to OSIR, alongside the existing Splunk support.
This PR lets users choose the SIEM backend at setup time (
splunk,elasticsearch, orboth) with feature parity: same parsing pipeline, same module datamodels, same VRL normalization rules. Only the indexing destination changes.New files
docker-compose.yml(online/offline/all profiles, mirroring the Splunk setup), with newElasticFile/KibanaFileDockerfiles tuned.json2elastic-rs: a Rust binary (companion tojson2splunk-rs) dispatched by its internal Python wrapper. Reuses the existing per-modulesplunk:datamodel, so none of the parsing modules needed changes. Acceptelasticorsiemkeywords. Auto-creates the ES index/template/pipeline and embeds the VRL engine directly. (seehttps://github.com/Grand-Duc/json2elastic-rsfor details)master_setup.sh/agent_setup.shnow ask which SIEM to use and configure ES/Kibana (host, port etc.) the same way Splunk already is, for both interactive and config-file installs..envand used by docker compose, instead of being silently discarded in favor of hardcoded defaults. Web sidebar links (Splunk/Kibana) now use the real configured ports.master_setup.shto remove an order-dependency bug.Performance
Benchmarked ingesting the same DFIR-ORC collection:
json2splunk-rs)json2elastic-rs)json2splunk-rs)json2elastic-rs)Performance seems correct on the order of magnitude expected. Note that it has not been tested with large collections.
ES pipeline is still a single-node setup with no replication or sharding.
Follow-ups
json2elastic-rs(bulk size, refresh interval).