Repository navigation
Fork vs last release - #11
Open
rdettai-sk wants to merge 118 commits into
Open
rdettai-sk wants to merge 118 commits into
rdettai-sk wants to merge 118 commits into
Conversation
The multilang feature pulls heavy dependencies (lindera, whichlang) for little apparent usage. Remove it along with associated code, benchmarks, and tests to reduce build times and binary size.
Bumps [oneshot](https://github.com/faern/oneshot) from 0.1.11 to 0.1.12. - [Changelog](https://github.com/faern/oneshot/blob/main/CHANGELOG.md) - [Commits](faern/oneshot@v0.1.11...v0.1.12) --- updated-dependencies: - dependency-name: oneshot dependency-version: 0.1.12 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [time](https://github.com/time-rs/time) from 0.3.44 to 0.3.47. - [Release notes](https://github.com/time-rs/time/releases) - [Changelog](https://github.com/time-rs/time/blob/main/CHANGELOG.md) - [Commits](time-rs/time@v0.3.44...v0.3.47) --- updated-dependencies: - dependency-name: time dependency-version: 0.3.47 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [bytes](https://github.com/tokio-rs/bytes) from 1.11.0 to 1.11.1. - [Release notes](https://github.com/tokio-rs/bytes/releases) - [Changelog](https://github.com/tokio-rs/bytes/blob/master/CHANGELOG.md) - [Commits](tokio-rs/bytes@v1.11.0...v1.11.1) --- updated-dependencies: - dependency-name: bytes dependency-version: 1.11.1 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [keccak](https://github.com/RustCrypto/sponges) from 0.1.5 to 0.1.6. - [Commits](RustCrypto/sponges@keccak-v0.1.5...keccak-v0.1.6) --- updated-dependencies: - dependency-name: keccak dependency-version: 0.1.6 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…d binary size (quickwit-oss#6125) Optimizes hyper-util, tokio-util, prometheus, dialoguer, zstd, and env_logger by disabling default features that aren't used by the codebase. Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
This is fixed upstream in memory cache improvement PR.
Rustfmt 1.9.0 (shipped in nightly 2026-03-01) changed behavior for `imports_granularity = "Module"`: top-level imports are no longer merged. Expand grouped `pub use` re-exports and fix sort order. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This is a squash of @Darkheir's pr 6107
Signed-off-by: Darkheir <raphael.cohen@sekoia.io>
Signed-off-by: Darkheir <raphael.cohen@sekoia.io>
Signed-off-by: Darkheir <raphael.cohen@sekoia.io>
Signed-off-by: Darkheir <raphael.cohen@sekoia.io>
Signed-off-by: Darkheir <raphael.cohen@sekoia.io>
Signed-off-by: Darkheir <raphael.cohen@sekoia.io>
Signed-off-by: Darkheir <raphael.cohen@sekoia.io>
Signed-off-by: Darkheir <raphael.cohen@sekoia.io>
Signed-off-by: Darkheir <raphael.cohen@sekoia.io>
Signed-off-by: Darkheir <raphael.cohen@sekoia.io>
Signed-off-by: Darkheir <raphael.cohen@sekoia.io>
Signed-off-by: Darkheir <raphael.cohen@sekoia.io>
Also add caller to searcher thread pool metrics.
* feat: On disk cache for split footers Signed-off-by: Darkheir <raphael.cohen@sekoia.io> * feat: Async disk cache access Signed-off-by: Darkheir <raphael.cohen@sekoia.io> * Apply suggestions from code review Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * feat: Async disk cache open Signed-off-by: Darkheir <raphael.cohen@sekoia.io> * feat: Sub-folder sharding Signed-off-by: Darkheir <raphael.cohen@sekoia.io> * feat: Apply PR review suggestion Signed-off-by: Darkheir <raphael.cohen@sekoia.io> --------- Signed-off-by: Darkheir <raphael.cohen@sekoia.io> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Run multiple day spans if None configured * Add cron based execution * Improve logging
* feat: Add new search metrics Signed-off-by: Darkheir <raphael.cohen@sekoia.io> * feat: Add cost classification to queries Signed-off-by: Darkheir <raphael.cohen@sekoia.io> * feat: Apply review suggestion Signed-off-by: Darkheir <raphael.cohen@sekoia.io> * feat: Apply review suggestion (part 2) Signed-off-by: Darkheir <raphael.cohen@sekoia.io> --------- Signed-off-by: Darkheir <raphael.cohen@sekoia.io>
* Fix metadata responses considered as cache miss * Port the rest of the changes * Apply copi suggestions
* Fix fetch docs metrics * Fix thread pool task cost classes * Use tag for tantivy dependency
* Fix cache eviction metric * Still increase recency and frequency on concurrent metric put * Fix file description eviction metric * Fix fd eviction metric in case of duplicate eviction
…LRU across restarts (#85) * feat: Improve DiskSizedCache Signed-off-by: Darkheir <raphael.cohen@sekoia.io> * feat: Apply review suggestion Signed-off-by: Darkheir <raphael.cohen@sekoia.io> --------- Signed-off-by: Darkheir <raphael.cohen@sekoia.io>
* fix: Only drop a vanished disk cache entry if it is the one that was read Signed-off-by: Darkheir <raphael.cohen@sekoia.io> * feat: scan the disk cache shards in parallel when opening Signed-off-by: Darkheir <raphael.cohen@sekoia.io> * Some extra comments, in places not currently changed in this PR --------- Signed-off-by: Darkheir <raphael.cohen@sekoia.io> Co-authored-by: Remi Dettai <remi.dettai@sekoia.io>
* Add CPU scheduler * Add warmup queue state to split * Avoid panic on uninitialized query * Update running count even when panicking * Fix split outcome accumulation * Fix pump yielding * Add missing tests * Trying to fix the pump but keep running into reentering issues * Use actor instead of pump loops * Fix actor shutdown * Fix clippy * Avoid double boxing on tasks * Tiny variable name impr
* get_slice zero copy for single-segment bodies * Bump bytes and add more arguments about safety * Fix azure download counting
* Add timeout on root search and avoid leaf retry on leaf timeout * Apply copi PR review
* Cache DNS results using upstream quickwit-oss#6584 and quickwit-oss#6586 Important: this also activates TCP_NODELAY as a side effect. * Add DNS histogram metric * Avoid waiting DNS_REFRESH_COOLDOWN if lookup fails
* feat: Avoid a second stat per entry when scanning the disk cache Signed-off-by: Darkheir <raphael.cohen@sekoia.io> * feat: Open the split footer cache concurrently with the node startup Signed-off-by: Darkheir <raphael.cohen@sekoia.io> --------- Signed-off-by: Darkheir <raphael.cohen@sekoia.io>
Each regex or wildcard leaf compiles to one automaton per targeted field when the request is validated against each index schema. A request with hundreds of them, fanned out over hundreds of indexes, can spend minutes of CPU in planning alone (2026-09-28 wildcard planning incident). Count them on the resolved query AST and return a 400 above QW_MAX_AUTOMATON_QUERIES_PER_REQUEST (default 200). Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Add metric with AWS SDK connection attempts * Add DNS result shuffling * Refine documentation for aws_connection_attempts_total Removed redundant explanation from the metric documentation.
…ion (#105) * Report tokio worker busy time in microseconds `tokio_worker_busy_duration_milliseconds_total` is renamed to `tokio_worker_busy_duration_microsecs_total`. Millisecond truncation on every 1s scrape interval is too coarse to compare busy time with the poll durations recorded by the slow poll detector. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Instrument CPU contention of runtime * Add a slow poll detector to search and split cache futures `detect_slow_poll(name)` records the duration of every poll of the wrapped future in `quickwit_runtime_task_poll_duration_seconds{name}` and logs a rate-limited warning above `QW_SLOW_POLL_LOG_THRESHOLD_MS` (default 50ms). A poll blocks its tokio worker for its whole duration, so this pinpoints the futures doing CPU or blocking work on the runtime. Instrumented: root search and its phases, leaf multi-index / single-index / single-split searches, each warm-up step (terms, term ranges, fast fields, field norms, automatons), fetch docs, and the split cache download and get. Nested points are named `outer:inner` and report inclusive times. Buckets are kept to 1ms, 16ms, 256ms and 4s to bound the number of series. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Attribute the S3 client's background task polls to `s3` The AWS SDK's hyper client runs each connection (socket reads, TLS, HTTP parsing) in a task spawned by the first request that needed it and shared through the pool, so this work can't be wrapped with `detect_slow_poll`. With `QW_TOKIO_TASK_POLL_ATTRIBUTION=true` (requires `tokio_unstable`), task hooks on the main runtime record the polls of the tasks spawned while an S3 request is polled, and of their descendants, under `quickwit_runtime_task_poll_duration_seconds{name="s3"}`. The S3 client's HTTP client is wrapped to mark its requests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Always record the S3 client's task polls Drop the `QW_TOKIO_TASK_POLL_ATTRIBUTION` opt-in: the task hooks are now installed on the main runtime unconditionally (they still require `tokio_unstable`, which the build always sets). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Revert "Always record the S3 client's task polls" S3 task poll attribution must stay opt-in: `main.rs` configures the runtime of every node type, and the per-poll hook cost is only wanted on searchers. Set `QW_TOKIO_TASK_POLL_ATTRIBUTION=true` on searcher pods only. This reverts commit 333ef2c. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…r CPU-heavy polls on tokio (#106) * Port upstream priority thread pool for search Adapted from quickwit-oss#6512. The search thread pool becomes a `ThreadPoolWithPriority`: `Priority::High` tasks are served before the `Normal` ones. Merging leaf results (`finalize`) and root results (`root_merge`) are on the response's critical path, so they run with high priority instead of queueing behind split searches. The thread pool module is split into `simple` (the existing pool, plus a small tasks pool) and `with_priority`. Doc fetching moves to the small tasks pool so it doesn't queue behind split searches either. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Build warm-up regexes on the search thread pool Compiling a regex automaton (`tantivy_fst::Regex::from_patterns`) during the automaton warm-up ran inline on the tokio worker. For complex patterns this takes long enough to block the runtime. It now runs on the search thread pool, under the `automaton_warmup_build` caller. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Run automaton warm-up block selection on the search thread pool Bump tantivy to SekoiaLab/tantivy@573de9bf, where `warm_postings_automaton` takes two executors: one for selecting the term dictionary blocks the automaton can match, which used to run inline on the tokio worker and iterates over the whole sstable index, and one for scanning the terms of the selected blocks. Both now run on the search thread pool, under the `automaton_warmup_select_blocks` and `automaton_warmup_scan_terms` callers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Keep small result merges off the search thread pool Every leaf `finalize` and every root merge went through the search thread pool, paying the hand-off and the queue wait even when there was nothing expensive to merge. They now run inline unless there are intermediate aggregation results of at least `QW_AGGREGATION_MERGE_OFFLOAD_THRESHOLD_NUM_BYTES` (default 128 KiB) to merge, or several trace id aggregations. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Assemble large S3 bodies on the search thread pool on searcher-only nodes A GetObject body that arrives in several segments is concatenated into one buffer. For multi-megabyte bodies, the copy and the page faults of the fresh buffer take milliseconds on the tokio worker polling the download. On nodes running only the searcher service, the storage resolver now hands the search thread pool to the S3 storages, and bodies of at least `QW_S3_ASSEMBLY_OFFLOAD_THRESHOLD_NUM_BYTES` (default 5 MiB) are assembled there with high priority, since a request is waiting on them. Other nodes keep assembling inline. Single-segment bodies are still handed over without a copy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Prioritize search work by the remaining cost of its query Adapted from quickwit-oss#6798, with its query cost model: `cost::compute_query_complexity_factor` derives a factor from the query AST shape and the aggregations, and the cost of searching a split is its number of docs scaled by that factor. - Search permits: leaf requests are served by lowest remaining cost instead of in arrival order. The remaining cost is shared by the permit requests of all the indexes targeted by a leaf request, and decreases as its splits get permits. - Search thread pool: `Normal` priority tasks are served by lowest remaining cost awaiting a permit, then by lowest remaining cost in progress, then in arrival order. Queries closest to completion finish first instead of every query slowing down together under load. This replaces the `QueryCostClass` heuristic, which was only used to label metrics. The `cost_class` label is removed from the thread pool metrics (`ongoing_tasks`, `pending_tasks`, `queue_wait_time_secs`, `run_time_secs`), from `leaf_search_single_split_tasks`, and from the leaf search request metrics. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Pin tantivy to the v0.26.1_20261008 tag The tag contains the previously pinned commit plus a fix restoring the `will_always_match` fast path in automaton block selection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Share the merge offload rule between leaf finalize and root merge The root merge only looked at the size of the aggregation results, so several small trace id aggregation results were merged on the tokio worker, while the leaf `finalize` offloads them. Both now use `is_aggregation_merge_cpu_intensive`: several trace id results, or tantivy aggregation results of at least `QW_AGGREGATION_MERGE_OFFLOAD_THRESHOLD_NUM_BYTES`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Check that large S3 bodies are assembled on the thread pool The test only checked the assembled bytes, which an inline assembly produces too. It now occupies the only thread of the pool and checks that the assembly waits for it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Decide aggregation merge offloading with an AggregationMerge summary Replace the `is_aggregation_merge_cpu_intensive(bool, usize, usize)` helper with an `AggregationMerge` struct summarizing the intermediate results about to be merged. It is built from the leaf's `QuickwitIncrementalAggregations` or from the root's `QuickwitAggregations` and leaf responses, and `is_cpu_intensive` gives the verdict. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Turn AggregationMerge into an AggregationMergeSummary enum Each variant only carries what the decision needs: the number of trace id results, or the number of bytes of tantivy aggregation results. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…CY (#102) * Properly limit parallel s3 get requests * Remove QW_S3_MAX_CONCURRENCY, release GetObject permits early - Remove the S3 request semaphore (QW_S3_MAX_CONCURRENCY): its permit was held across retries and backoff, so it bounded neither requests nor connections. QW_S3_MAX_GET_OBJECT_CONNECTIONS is now the only limit. - Validate QW_S3_MAX_GET_OBJECT_CONNECTIONS: an unset variable means unlimited, any value outside 1..=Semaphore::MAX_PERMITS fails storage resolution with an InvalidConfig error instead of deadlocking, being ignored or panicking. - get_slice_stream releases the permit once the requested range has been read, at EOF or on a read error, rather than when the reader is dropped. - copy_to releases the permit before flushing the output. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Hold the get_slice_stream permit until the reader is dropped Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
`QW_TOKIO_TASK_POLL_ATTRIBUTION` becomes `QW_TOKIO_S3_TASK_POLL_ATTRIBUTION` and `configure_task_poll_attribution` becomes `configure_s3_task_poll_attribution`: the hooks only attribute the polls of the tasks spawned by S3 requests. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Fix the metadata of backfilled secondary timestamps * Cleanup test index
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR shows the diff between the
sekoiabranch and the last deployed release.