Skip to content

Use an unsigned shift when reading 32-bit function samples - #22380

Open
Huang-404-Q wants to merge 1 commit into
mozilla:masterfrom
Huang-404-Q:fix/sampled-function-bps32
Open

Huang-404-Q wants to merge 1 commit into
mozilla:masterfrom
Huang-404-Q:fix/sampled-function-bps32

Conversation

@Huang-404-Q

Copy link
Copy Markdown
Contributor

A type 0 (sampled) function with BitsPerSample 32 decodes samples with the high bit set as negative, and they are then clamped to the bottom of the range.

PDFFunction.getSampleArray extracts each sample with an arithmetic shift:

array[i] = (codeBuf >> codeSize) * sampleMul;

codeBuf is a 32-bit accumulator, so once bps is above 24 the sample itself can occupy bit 31. With bps = 32 the shift amount is 0, 0x80000000 >> 0 is -2147483648, and 0xFFFFFFFF >> 0 is -1; both multiply to a negative value, which is then clamped to 0. So 80 00 00 00 decodes to 0 instead of 0.5, and FF FF FF FF decodes to 0 instead of 1.

Using the unsigned shift fixes it:

array[i] = (codeBuf >>> codeSize) * sampleMul;

For bps <= 24 nothing changes: the accumulator cannot reach bit 31, so >> and >>> agree.

Test

Added to test/unit/function_spec.js, extending the sampled-function helper with a bitsPerSample option (it was hard-coded to 8). The case feeds 80 00 00 00 FF FF FF FF at BitsPerSample: 32 and expects 0x80000000 / (2**32 - 1) and 1.

Without the change:

12 specs, 1 failure
Expected $[0] = 0 to equal 0.5000000001164153

With it:

12 specs, 0 failures

eslint on both files is clean.

The bit reservoir can have the high bit set for 32 bits-per-sample data,
in which case the signed right shift sign-extends and yields a negative
sample value.
@codecov-commenter

codecov-commenter commented Oct 29, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.48%. Comparing base (87b369c) to head (6b5f138).

Additional details and impacted files
@@            Coverage Diff             @@
##           master   #22380      +/-   ##
==========================================
- Coverage   90.58%   90.48%   -0.10%     
==========================================
  Files         278      278              
  Lines       68558    68558              
==========================================
- Hits        62100    62034      -66     
- Misses       6458     6524      +66     
Flag Coverage Δ
browsertest 66.27% <100.00%> (+0.01%) ⬆️
fonttest 8.99% <ø> (ø)
integrationtest 69.38% <100.00%> (-0.16%) ⬇️
unittest 61.26% <100.00%> (ø)
unittestcli 59.00% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

Copy link
Copy Markdown
Contributor

Viewer preview

Commit 6b5f138 (build logs).

Viewer Build Preview
Modern (gulp generic) ✅ not published
Legacy (gulp generic-legacy) ✅ not published

🔒 A user with write access must approve this commit with the viewer-preview label to publish the viewers.

@codspeed

codspeed Bot commented Oct 29, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 11.13%

⚡ 1 improved benchmark
✅ 87 untouched benchmarks
⏩ 174 skipped benchmarks1

Performance Changes

Benchmark BASE HEAD Efficiency
⚡ PDFPageProxy.render: luminosity soft mask (issue18956.pdf) 22.5 ms 20.3 ms +11.13%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing Huang-404-Q:fix/sampled-function-bps32 (6b5f138) with master (87b369c)

Open in CodSpeed

Footnotes

  1. 174 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants