Conversation
The shard lookup regex matched the stream type (TELEMETRY) instead of the
{TARGET} hashtag and ignored scope, so on multi-shard deployments target
streams outside shard 0 were never trimmed by the log writer. Python also
iterated offset dicts without .items() and used datetime.min for minutes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ryanmelt
force-pushed
the
fix/log-writer-trim-shard
branch
from
September 28, 2026 00:42
6453ef4 to
f44bd4d
Compare
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #3953 +/- ##
==========================================
+ Coverage 80.08% 80.14% +0.06%
==========================================
Files 901 901
Lines 68356 68368 +12
Branches 2645 2645
==========================================
+ Hits 54743 54796 +53
+ Misses 12946 12910 -36
+ Partials 667 662 -5
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| # Returns the Redis db_shard holding a target stream such as | ||
| # SCOPE__TELEMETRY__{TARGET}__PACKET, or 0 if no target can be determined | ||
| def self.db_shard_for_topic(redis_topic) | ||
| target_match = redis_topic.match(/\{([^}]+)\}/) |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
LogWriter: the shard lookup used/__\{?([^}_]+)\}?__/, which onSCOPE__TELEMETRY__{TARGET}__PACKETmatchesTELEMETRY, not the target. It also always looked up theDEFAULTscope. AddedLogWriter.db_shard_for_topic, which reads the{TARGET}hashtag and the scope prefix.LogWriter, same fix plus two latent bugs in the same code:cleanup_offsets/last_offsetsdicts were iterated without.items(), so any queued cleanup would raiseutc_now.min(thedatetime.minclass constant) instead ofutc_now.minute, so it could never triggertrim_due_cleanups()so it can be tested.Why
On deployments with more than one Redis db_shard,
TELEMETRY__streams for targets on non-zero shards were never trimmed by the log writer and grew without bound.XTRIMagainst the wrong shard is a silent no-op. Single-shard deployments are unaffected. The PythonLogWriteris currently only used by interface stream logs, which don't record Redis offsets, so the Python dict-iteration bug hasn't been hit in practice.Testing
openc3/spec/logs/log_writer_spec.rb;bundle exec rspec spec/logs: 60 examples, 0 failuresopenc3/python/test/logs/test_log_writer.py(shard lookup, plus close → trim on the correct shard);uv run pytest test/logs: 56 passed; ruff clean🤖 Generated with Claude Code