Repository navigation
Make memory service canonical JSON locale-independent and lossless - #3221
Open
Kevin Turcios (KRRT7) wants to merge 1 commit into
Open
Kevin Turcios (KRRT7) wants to merge 1 commit into
Kevin Turcios (KRRT7) wants to merge 1 commit into
Conversation
Replace four copies of the sort-keys-then-serialize helper with one
shared canonicalJson module.
- Procedure and view hashes sorted keys with the host's default locale,
so the same document hashed differently on, e.g., Estonian or Danish
hosts. Pin an "en" collator, which matches hashes already stored by
en-locale hosts.
- Agent session request hashes depended on key order, so an identical
retry with reordered keys was rejected as different content. Hash a
canonical form and still accept hashes stored in the old format.
- Session event redaction silently dropped a value when two keys
redacted to the same text. Keep both ("[REDACTED] (2)") and accept
the old lossy digest for events stored before this change.
Session event text keeps code-point key order, since stored event
digests depend on it.
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
canonicalizeProcedure,viewHash) sorted object keys withlocaleCompareusing the host's default locale. The same document could hash differently across machines: Estonian and Danish hosts put keys in a different order than en-US. Keys now sort with a pinnedencollator. That matches the hashes already stored by en-locale hosts, so stored review content hashes and fingerprints stay valid.sha256(JSON.stringify(request)), so an identical retry with keys in a different order failed with "idempotency key has different content". Requests now hash a canonical form, and hashes stored in the old format are still accepted (marked with a TODO to remove).sk-...keys both becoming[REDACTED]). Both values are now kept ([REDACTED],[REDACTED] (2)). An event stored before this change may not match its saved digest when re-sent. In that case only, the chunk is re-parsed the old way so the stored digest is still recognized instead of being rejected as conflicting content.canonicalJson.ts. Session event text keeps code-point key order, because stored event digests depend on it.Scope
Seven files, all in
memory/service: five source files, plus a new spec and additions to an existing spec. No changes to stored formats, dependencies, or other packages.Testing
memory/service: all 73 suites pass (853 tests), including the newcanonicalJson.spec.tsand collision tests inagentSessionParsing.spec.ts.mainon colliding, nested-colliding, and non-colliding events: all match, and non-colliding events hash the same as before.origin/main.