Repository navigation
Use an unsigned shift when reading 32-bit function samples - #22380
Huang-404-Q wants to merge 1 commit into
Conversation
The bit reservoir can have the high bit set for 32 bits-per-sample data, in which case the signed right shift sign-extends and yields a negative sample value.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #22380 +/- ##
==========================================
- Coverage 90.58% 90.48% -0.10%
==========================================
Files 278 278
Lines 68558 68558
==========================================
- Hits 62100 62034 -66
- Misses 6458 6524 +66
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Viewer previewCommit 6b5f138 (build logs).
🔒 A user with write access must approve this commit with the |
Merging this PR will improve performance by 11.13%
Performance Changes
Tip Curious why performance improved? Comment Comparing Footnotes
|
A type 0 (sampled) function with
BitsPerSample 32decodes samples with the high bit set as negative, and they are then clamped to the bottom of the range.PDFFunction.getSampleArrayextracts each sample with an arithmetic shift:codeBufis a 32-bit accumulator, so oncebpsis above 24 the sample itself can occupy bit 31. Withbps = 32the shift amount is 0,0x80000000 >> 0is-2147483648, and0xFFFFFFFF >> 0is-1; both multiply to a negative value, which is then clamped to 0. So80 00 00 00decodes to 0 instead of 0.5, andFF FF FF FFdecodes to 0 instead of 1.Using the unsigned shift fixes it:
For
bps <= 24nothing changes: the accumulator cannot reach bit 31, so>>and>>>agree.Test
Added to
test/unit/function_spec.js, extending the sampled-function helper with abitsPerSampleoption (it was hard-coded to 8). The case feeds80 00 00 00 FF FF FF FFatBitsPerSample: 32and expects0x80000000 / (2**32 - 1)and1.Without the change:
With it:
eslinton both files is clean.