Skip to content

feat(topics): cap age of target streams with an XADD MINID safety trim - #3951

Open
ryanmelt wants to merge 1 commit into
mainfrom
feat/stream-safety-trim
Open

ryanmelt wants to merge 1 commit into
mainfrom
feat/stream-safety-trim

Conversation

@ryanmelt

Copy link
Copy Markdown
Member

What

Every write to the high-rate target streams (TELEMETRY__, COMMAND__, DECOM__, DECOMCMD__) now passes an approximate XADD ... MINID ~ <cutoff>, so Redis/Valkey never keeps entries older than a configurable max age.

  • New env var OPENC3_STREAM_MAX_AGE_SECONDS: default 600. Set it to 0 or empty to disable. Documented in configuration/environment.md.
  • For TELEMETRY__/COMMAND__, the max age is never less than 2 × (log cycle time + LogWriter::CLEANUP_DELAY), which is 1320 s with the default 600 s cycle. The reason: historical streaming reads data that isn't in the bucket yet (the current log file) from the stream. The per-target tlm_log_cycle_time/cmd_log_cycle_time is read from TargetModel once and cached per process.
  • DECOM__/DECOMCMD__ use the plain max age. Historical DECOM data comes from TSDB, and TSDB normally keeps 60 s of these streams.
  • The cutoff is measured back from the id being written, or from now when the id is *. Decom reuses the raw packet's id, so an entry written by a decom that's running behind is never trimmed by its own add.
  • write_topic in Ruby and Python (store + Topic) takes an optional minid: keyword. Existing callers are unaffected.
  • The trimming contract is now written down in comments in Topic, LogWriter, and TsdbMicroservice.

Why

Until now these streams were bounded only by their consumers trimming them: the log microservice after it uploads a file, and the TSDB microservice. If logging is disabled (PacketLogWriter#write returns early, so no trim offsets are ever queued), TSDB isn't deployed, or a trimmer falls behind, the streams grow without bound until Redis runs out of memory. This cap bounds memory even when no trimmer is running, and it sits well above normal retention so it doesn't change behavior on a healthy system.

Tradeoff: a consumer that falls further behind than the cap loses the trimmed entries. That's the intended failure mode, better than taking down Redis. Warning the user when this happens belongs in a separate PR about lag and data-loss alerting.

Testing

  • New spec/topics/topic_spec.rb and test/topics/test_topic.py: minid math (now-relative, id-relative, min age, disabled), log cycle lookup and caching, write_topic passing MINID through to XADD (fakeredis confirms old entries are actually trimmed), and TelemetryTopic queued and unqueued writes.
  • Updated the existing topic specs and tests for the new minid argument.
  • bundle exec rspec spec/topics spec/microservices spec/logs spec/api spec/utilities/store*: 643 examples, 1 failure. The failure is decom_microservice_spec.rb:58, an order-dependent limits-set test that passes on its own and also fails on main.
  • uv run pytest: 3040 passed. uv run ruff check openc3: clean.

🤖 Generated with Claude Code

TELEMETRY/COMMAND/DECOM/DECOMCMD streams were bounded only by the log and
TSDB microservices trimming them, so disabled logging, a missing TSDB, or a
lagging trimmer let Redis grow without limit. Writes now pass an approximate
MINID (OPENC3_STREAM_MAX_AGE_SECONDS, default 600s, never less than two log
cycles for raw streams) so the streams stay bounded without a trimmer.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@sonarqubecloud

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
Bugs with severity Major found (required < Minor)
Code smells with severity Major found (required < Major)

See analysis details on SonarQube Cloud

@codecov

codecov Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.87234% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 80.09%. Comparing base (688f928) to head (3625c81).

Files with missing lines Patch % Lines
openc3/lib/openc3/topics/topic.rb 97.22% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main    #3951   +/-   ##
=======================================
  Coverage   80.08%   80.09%           
=======================================
  Files         901      901           
  Lines       68356    68450   +94     
  Branches     2645     2698   +53     
=======================================
+ Hits        54743    54823   +80     
- Misses      12946    12969   +23     
+ Partials      667      658    -9     
Flag Coverage Δ
frontend 66.68% <ø> (-0.27%) ⬇️
python 80.16% <ø> (+0.02%) ⬆️
ruby-api 82.51% <ø> (+0.50%) ⬆️
ruby-backend 85.67% <97.87%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant