Skip to content

Fix/operator process metrics - #3959

Open
jmthomas wants to merge 4 commits into
mainfrom
fix/operator-process-metrics
Open

jmthomas wants to merge 4 commits into
mainfrom
fix/operator-process-metrics

Conversation

@jmthomas

@jmthomas jmthomas commented Sep 29, 2026 •

Copy link
Copy Markdown
Member

What changed

- Add OpenC3::ProcessStats to sample cpu/memory for an arbitrary pid,
  using Etc.sysconf rather than shelling out to getconf and splitting
  /proc/<pid>/stat from the last paren so a comm containing spaces or
  parens can't shift the field offsets
- Sample every child pid from Operator#publish_process_metrics each
  cycle, keeping one ProcessStats per microservice so cpu deltas
  accumulate across cycles and resetting it when a respawn changes pid
- Store operator reported values under MetricModel::PROCESS_PRIMARY_KEY
  and merge them on read, so the operator and the microservice are
  never two writers of one field
- Skip Metric.add_update_generator when OPENC3_OPERATOR_PROCESS_METRICS
  is set, which microservice_operator.rb sets on every child it spawns
- Drop the bootstrap metrics row in PluginMicroservice#run before exec
  so the stale sample is never published

Why it changed

plugin_microservice.rb exec()s the configured cmd, which replaces the
process image and kills the Metric update thread it just started. A cmd
that is itself an OpenC3::Microservice builds a new Metric and keeps
reporting, but a plugin running a Rails app, a python script or any
other binary never loads the OpenC3 libraries, so its cpu and memory
froze at whatever the bootstrap sampled and never changed again.

Testing strategy

Built enterprise and installed the CFDP plugin. Watched the System Health Microservice Metrics on the CFDP processes to verify the Sample Cpu Utilization was changing during scripts. Here is a screenshot of them at rest:

image

Here's an example of running the cfdp_test_suite.py. Note the Sample Cpu Utilization ... previously that would have just been 0 for the DEFAULT__USER__CFDP and DEFAULT__USER__CFDP2.

image

jmthomas and others added 3 commits September 29, 2026 07:31
plugin_microservice.rb exec()s the configured cmd, which replaces the
process image and kills the Metric update thread it just started. A cmd
that is itself an OpenC3::Microservice builds a new Metric and keeps
reporting, but a plugin running a Rails app, a python script or any
other binary never loads the OpenC3 libraries, so its cpu and memory
froze at whatever the bootstrap sampled and never changed again.

- Add OpenC3::ProcessStats to sample cpu/memory for an arbitrary pid,
  using Etc.sysconf rather than shelling out to getconf and splitting
  /proc/<pid>/stat from the last paren so a comm containing spaces or
  parens can't shift the field offsets
- Sample every child pid from Operator#publish_process_metrics each
  cycle, keeping one ProcessStats per microservice so cpu deltas
  accumulate across cycles and resetting it when a respawn changes pid
- Store operator reported values under MetricModel::PROCESS_PRIMARY_KEY
  and merge them on read, so the operator and the microservice are
  never two writers of one field
- Skip Metric.add_update_generator when OPENC3_OPERATOR_PROCESS_METRICS
  is set, which microservice_operator.rb sets on every child it spawns
- Drop the bootstrap metrics row in PluginMicroservice#run before exec
  so the stale sample is never published

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- Describe the processed/ and error/ split under decom_logs and how to
  retry a file by moving it back once the cause is fixed
- Add error/ to the directories to remove after verifying the data
- Note the migration microservice idles when done, so the plugin should
  be uninstalled from the Admin Console

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
_log_info always printed to stdout, so info messages bypassed the
microservice logger that _log_warn and _log_error already use.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings September 29, 2026 14:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@codecov

codecov Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.93939% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.16%. Comparing base (2ad8e7b) to head (29a9071).
⚠️ Report is 4 commits behind head on main.

Files with missing lines Patch % Lines
openc3/lib/openc3/utilities/process_stats.rb 92.95% 5 Missing ⚠️
openc3/lib/openc3/operators/operator.rb 92.30% 2 Missing ⚠️
...c3/lib/openc3/microservices/plugin_microservice.rb 80.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3959      +/-   ##
==========================================
+ Coverage   80.11%   80.16%   +0.05%     
==========================================
  Files         901      903       +2     
  Lines       68370    68516     +146     
  Branches     2699     2699              
==========================================
+ Hits        54773    54929     +156     
+ Misses      12929    12918      -11     
- Partials      668      669       +1     
Flag Coverage Δ
frontend 67.01% <ø> (-0.02%) ⬇️
python 80.12% <ø> (-0.01%) ⬇️
ruby-api 82.67% <ø> (+0.44%) ⬆️
ruby-backend 85.70% <93.93%> (+0.04%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Microservice.run publishes INITIALIZED, then sets @State to RUNNING in
memory only and calls run. Plugins never start the periodic status
thread, and PluginMicroservice#run exec()s the configured cmd, so the
RUNNING state was never written. A plugin running a Rails app, a python
script or any other binary showed INITIALIZED in the Microservices tab
forever even though it was running. A cmd that is itself an
OpenC3::Microservice publishes its own status and overwrites this.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sonarqubecloud

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
4.9% Duplication on New Code (required ≤ 3%)
Code smells with severity Critical found (required < Major)

See analysis details on SonarQube Cloud

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants