Skip to content

Fix sign dependent flat line detection in get_railed_percentage - #848

Open
adityasingh2400 wants to merge 3 commits into
brainflow-dev:masterfrom
adityasingh2400:fix-railed-percentage
Open

Fix sign dependent flat line detection in get_railed_percentage#848
adityasingh2400 wants to merge 3 commits into
brainflow-dev:masterfrom
adityasingh2400:fix-railed-percentage

Conversation

@adityasingh2400

Copy link
Copy Markdown

DataFilter.get_railed_percentage decides whether a channel is flat before it converts the peak amplitude into a percentage of the ADC range. The flat line test in get_railed_percentage is written as (abs (raw_data[i - 1]) - raw_data[i]) > 0.00001, so the closing parenthesis sits after the abs call rather than around the difference. It compares the absolute value of the previous sample against the signed value of the current sample instead of measuring how far apart two consecutive samples are.

That makes the check depend on the sign of the data. For a positive signal the condition only becomes true when the signal is decreasing, so a monotonically rising channel never clears the flag and is reported as 100 percent railed. For a negative signal abs (prev) - cur is -prev - cur, which is positive for essentially any pair of negative samples, so the flag is cleared immediately and a channel stuck at a negative value is never detected as railed. Detecting a stuck channel is the main thing this function exists to do, and a negative rail is just as common as a positive one. Separately, the running maximum is declared int cur_max = abs (raw_data[0]), so the fractional part of every peak is discarded and peaks below 1 uV collapse to exactly 0.

The fix moves the parenthesis so the difference is taken before abs, which matches the pattern abs (data[i] - avg_filter[i - 1]) already used by detect_peaks_z_score a hundred lines further down in the same file, and stores the running maximum as a double. The behavior for signals that were already handled correctly is unchanged.

I built the project on macOS arm64 with cmake .. && make -j8 and ran the reproduction through the Python binding against the freshly built libDataHandler. Before the change a flat line at +100 uV returned 100.0 while a flat line at -100 uV returned 0.0533, a rising ramp from 100 uV to 5000 uV returned 100.0, and uniform noise bounded by 0.9 uV returned exactly 0.0. After the change both flat lines return 100.0, the ramp returns 2.6667 in both signs, and the noise case returns 0.00047889, which matches max (abs (data)) / max_val * 100 computed independently in NumPy. Signals at the actual rail and a flat line at zero still return 100.0, and uniform noise bounded by 90000 uV returns 47.989 which again matches the closed form.

There is no existing test that touches get_railed_percentage, so I could not extend one. As a regression check I ran the synthetic board signal processing examples that CI already runs, denoising.py, signal_filtering.py, transforms.py, downsampling.py, band_power.py and windowing.py, and all six pass. clang-format with the repository .clang-format reports no change in the edited region.

The straight line check compared abs (raw_data[i - 1]) against the signed
raw_data[i] because the closing parenthesis was placed after the abs call
instead of around the difference. That made the test sign dependent, so a
channel stuck at a negative value was never reported as railed and a
monotonically rising positive signal was always reported as 100 percent
railed.

The running maximum was also stored in an int, which discarded the
fractional part of every peak and collapsed sub microvolt peaks to zero.

Store the maximum as a double and take the absolute value of the
difference between consecutive samples, matching the pattern already used
by detect_peaks_z_score in the same file.

Copy link
Copy Markdown
Member

The consecutive-difference correction looks right, and preserving the fractional maximum also fixes a real precision bug. However, this PR contains two observable behavior changes, so the statement that behavior for previously correct signals is unchanged should be narrowed.

Please add focused regression coverage for:

  • positive and negative flat signals;
  • a monotonic ramp;
  • fractional amplitudes below 1;
  • a signal crossing or ending at zero.

The zero case matters because the existing abs(raw_data[i]) > 0.00001 condition still suppresses a detected difference when the current sample is near zero. It would be useful to capture the intended zero-transition semantics explicitly.

@Andrey1994

Copy link
Copy Markdown
Member

Sorry, that was codex and autoreview, change looks fine, thanks

Covers positive and negative flat signals, ascending and descending
ramps, fractional amplitudes below 1, and signals crossing or ending at
zero. The zero cases pin the existing semantics rather than changing
them: a step into zero is suppressed, a step back out of zero is not.
@adityasingh2400

Copy link
Copy Markdown
Author

Added python_package/examples/tests/railed_percentage.py and wired it into run_unix.yml. You were right that the description understated the change, so here is the measured before and after at gain 24:

case before after
positive flat 100.0 100.0
negative flat 0.002667 100.0
monotonic ramp 100.0 0.003733
fractional below 1 0.0 0.00048
crosses zero 0.001600 unchanged
ends at zero 0.002667 unchanged

So three observable changes, not two. The ramp one is the worst: with the old comparison an increasing ramp always produced a negative difference, so every ascending ramp was reported as fully railed.

On the zero semantics, the test pins the current behavior rather than changing it. abs(current) > 0.00001 means a step into zero is suppressed while the step back out is counted, so a flat line ending at zero still reads as 100 and a zero in the middle of a flat line does not. Both are asserted explicitly with that asymmetry spelled out in a comment, so if you want different semantics the test is where to change it.

@adityasingh2400

Copy link
Copy Markdown
Author

I missed your follow-up before posting that, sorry. I had started on the tests when the first comment came in and did not re-check the thread before pushing.

The test and the CI line are a separate commit, so if you would rather merge this as the one-line fix you already reviewed, say so and I will drop it. If you want to keep it, nothing further is needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants