Fix MSI DigitalSignature verification with MsiDigitalSignatureEx

[olszomal]

Pull Request Type

• Bug fix
• New feature
• Code style / formatting / renaming
• Refactoring (no functional or API changes)
• Build / CI related changes
• Documentation
• Other (please describe):

Related Issue

Fixes #507

Current Behavior

MSI verification prepends the recalculated MsiDigitalSignatureEx value to the content digest and skips the stored MsiDigitalSignatureEx stream.

This can produce a different DigitalSignature digest than Windows signtool for valid MSI files.

New Behavior

During verification, the recalculated MsiDigitalSignatureEx value is checked against the stored value.

The stored MsiDigitalSignatureEx stream is then included in the MSI content digest at its sorted position.

Signing behavior remains unchanged.

Scope of Changes

• Verify the recalculated MsiDigitalSignatureEx value.
• Allow msi_hash_dir() to include the MsiDigitalSignatureEx stream during verification.
• Keep the existing signing behavior unchanged.

Testing

• Existing tests
• New tests added
• Manual testing

Verified an affected MSI file successfully with both osslsigncode and Windows signtool.

Additional Notes

License Declaration

• I hereby agree to license my contribution under the project's license.

[mtrojnar]

Signing and verification now disagree on MSI digest ordering

msi.c:480 — Verification now hashes MsiDigitalSignatureEx at its sorted position, but signing still prepends it in msi_calc_MsiDigitalSignatureEx(). When another stream sorts before it, this branch successfully signs a file that it cannot subsequently verify. Previously generated signatures also stop
verifying.

Reproducer using the existing fixture, renaming its embedded-cabinet stream so it sorts before DSE:

python3 - <<'PY'
from pathlib import Path
b = bytearray(Path("tests/files/unsigned.msi").read_bytes())
b[7168] = 0
Path("early-stream.msi").write_bytes(b)
PY

openssl req -x509 -newkey rsa:2048 -nodes \
  -keyout key.pem -out cert.pem -subj /CN=Review \
  -days 1 -addext extendedKeyUsage=codeSigning

build/osslsigncode sign -certs cert.pem -key key.pem \
  -add-msi-dse -in early-stream.msi -out signed.msi
build/osslsigncode verify -CAfile cert.pem -in signed.msi

Result: base passes; this branch fails with Calculated DigitalSignature ... MISMATCH!!!, although the stored and calculated DSE values match. Both versions produce signatures accepted by base and rejected by this branch.

The unmodified fixture with DSE, and the modified fixture without DSE, pass on both versions.

Suggestion: make signing and verification use the same digest ordering, including the detached-signature digest path. Explicitly address compatibility with previously generated signatures and add an early-sorting-stream regression test.