Repository navigation
[lexical-y][lexical-website] Bug Fix: Write text holding a lone surrogate like any other text, and never throw on it - #9714
Conversation
…gate like any other text, and never throw on it ## Description A selection an app sets between the two halves of a surrogate pair (`anchor.set(key, 3, 'text')` in `ab😀cd`; Lexical's own input never does this) makes Lexical split the pair into two text nodes or keep one half. Reported in #9701, `@lexical/y` then behaved three ways: - A range that **starts** inside the pair threw `@lexical/y: a write splits a surrogate pair` (a development check on the way to the range's start): nothing was written, the author kept Lexical's split and the other clients kept the emoji. - A range that **ends** inside it was written: Yjs split the stored item there and replaced both halves with U+FFFD, on every client; the author kept the lone halves and its self-check warned. - Typing, deleting or replacing next to a lone half wrote the lone halves: the author's editor and its in-memory document kept them, while Yjs's UTF-8 encoding sent U+FFFD to every other client (and a reload of the author's own document read U+FFFD). D57 accepts lone surrogates reaching other clients as U+FFFD (malformed text is the author's input), so the binding should write such text like any other text and never throw. This change: - The desired content of a line (`$pushInline`) holds each text node's text with every surrogate half that has no pair in that node as U+FFFD, which is what Yjs's UTF-8 encoding sends and a reload reads, so the author's document holds the same text as every other client. A pair whose halves Lexical put in two text nodes is then new text (two U+FFFD), so no write splits a stored pair: the dev check `a write splits a surrogate pair` cannot fire on such input, and undoing the write brings the character back. - Such a line is read again after the write (`DesiredContext.unpaired`, through the same `refused` re-read that two text nodes stored as one run already take), so the author's editor shows U+FFFD too. - The keystroke fast path leaves a text node holding a lone half to the line's write, and the record of text edits does not keep an old character whose stored value changed (the half written as U+FFFD). - The self-check builds what it expects with the same desired content, so it compares the U+FFFD written, and still reports a write that splits a pair the editor keeps whole. - D57 and y.md's Limits say what the binding does: never throws, writes the text with U+FFFD for each half without its pair in its text node, the author reads the line again, an undo brings the character back. Fixes the throw reported in #9701 (the issue is closed under D57). ## Test plan New `packages/lexical-y/src/__tests__/unit/regressions/SurrogateSelection.test.ts`: 24 cases x Yjs gc off and on x both client id orders (96 cells). Bold a range that ends inside, starts inside, covers one half, or runs backwards from inside; type inside the pair; type over a half; delete either half; delete from inside to the end; type a lone half before the emoji and at the end of the line; split the text node inside the pair and bold one side; a pair split across two text nodes; a lone half typed at the end and at the start of a text node; two text nodes Lexical keeps apart that store the same attributes (one run) holding the halves; a split inside the emoji while it is being composed; a collaborator typing in the line at the same time; a collaborator composing at the end of the line while the author bolds from inside the pair; the author undoing the write; controls (a collapsed selection inside, bolding the whole emoji). Each cell expects no warnings or errors and every client, the author and a fresh read of the author's document to show Lexical's text with each half lacking its pair in its text node as U+FFFD (the undo cell: the emoji back). ### Before On main (gc off, client ids [1, 2] shown; the other three settings fail the same cells the same way): ``` $ ./node_modules/.bin/vitest --run --project unit packages/lexical-y/src/__tests__/unit/regressions/SurrogateSelection.test.ts FAIL ... (gc false, ids [ 1, 2 ]) > bold a range that ends inside the emoji AssertionError: expected [ Array(1) ] to deeply equal [] FAIL ... (gc false, ids [ 1, 2 ]) > bold a range that starts inside the emoji FAIL ... (gc false, ids [ 1, 2 ]) > bold the low half FAIL ... (gc false, ids [ 1, 2 ]) > bold backwards from inside the emoji FAIL ... (gc false, ids [ 1, 2 ]) > split the text node inside the emoji and bold its second half FAIL ... (gc false, ids [ 1, 2 ]) > a collaborator types in the line at the same time FAIL ... (gc false, ids [ 1, 2 ]) > the author undoes the write Error: @lexical/y: a write splits a surrogate pair FAIL ... (gc false, ids [ 1, 2 ]) > bold the high half AssertionError: expected [ Array(1) ] to deeply equal [] FAIL ... (gc false, ids [ 1, 2 ]) > type inside the emoji AssertionError: expected 'paragraph("ab\ud83dX\ude00cd")' to be 'paragraph("ab\ufffdX\ufffdcd")' FAIL ... (gc false, ids [ 1, 2 ]) > type over the low half AssertionError: expected 'paragraph("ab\ud83dXcd")' to be 'paragraph("ab\ufffdXcd")' FAIL ... (gc false, ids [ 1, 2 ]) > delete the high half AssertionError: expected 'paragraph("ab\ude00cd")' to be 'paragraph("ab\ufffdcd")' FAIL ... (gc false, ids [ 1, 2 ]) > delete the low half AssertionError: expected 'paragraph("ab\ud83dcd")' to be 'paragraph("ab\ufffdcd")' FAIL ... (gc false, ids [ 1, 2 ]) > delete from inside the emoji to the end AssertionError: expected 'paragraph("ab\ud83d")' to be 'paragraph("ab\ufffd")' FAIL ... (gc false, ids [ 1, 2 ]) > type a lone high half before the emoji AssertionError: expected 'paragraph("ab\ud83d😀cd")' to be 'paragraph("ab\ufffd😀cd")' FAIL ... (gc false, ids [ 1, 2 ]) > type a lone low half at the end of the line AssertionError: expected 'paragraph("ab😀cd\ude00")' to be 'paragraph("ab😀cd\ufffd")' FAIL ... (gc false, ids [ 1, 2 ]) > a pair split across two text nodes AssertionError: expected 'paragraph("ab\ud83d"{1} "\ude00cd")' to be 'paragraph("ab\ufffd"{1} "\ufffdcd")' FAIL ... (gc false, ids [ 1, 2 ]) > a lone high half typed at the end of a text node AssertionError: expected 'paragraph("ab\ud83d" "cd"{1})' to be 'paragraph("ab\ufffd" "cd"{1})' FAIL ... (gc false, ids [ 1, 2 ]) > a lone low half typed at the start of a text node AssertionError: expected 'paragraph("\ude00ab" "cd"{1})' to be 'paragraph("\ufffdab" "cd"{1})' FAIL ... compositions (gc false, ids [ 1, 2 ]) > a collaborator composes at the end of the line while the author bolds [3, 6) Error: @lexical/y: a write splits a surrogate pair FAIL ... compositions (gc false, ids [ 1, 2 ]) > two text nodes that store the same attributes hold the halves AssertionError: expected 'ab😀cd' to be 'ab\ufffd\ufffdcd' FAIL ... compositions (gc false, ids [ 1, 2 ]) > a split inside the emoji while it is composed AssertionError: expected 'ab😀cd' to be 'ab\ufffd\ufffdcd' FAIL ... compositions (gc false, ids [ 1, 2 ]) > a collaborator composes at the end of the line while the author bolds [0, 3) AssertionError: expected [ …(2) ] to deeply equal [] Tests 88 failed | 8 passed (96) ``` (The 8 passing cells are the two controls in each setting.) ### After ``` $ ./node_modules/.bin/vitest --run --project unit packages/lexical-y/src/__tests__/unit/regressions/SurrogateSelection.test.ts packages/lexical-y/src/__tests__/unit/Surrogates.test.ts Tests 111 passed (111) ``` `packages/lexical-y` unit tests: ``` LEXICAL_Y_GC=0 main: 19322 passed | 351 expected fail | 0 failed LEXICAL_Y_GC=0 this change: 19418 passed | 351 expected fail | 0 failed LEXICAL_Y_GC=1 main: 19319 passed | 354 expected fail | 0 failed LEXICAL_Y_GC=1 this change: 19415 passed | 354 expected fail | 0 failed ``` `tsc`, `tsc -p tsconfig.test.json` and `eslint` on the changed files pass.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
AI review status for this pull request.
|
lexical
@lexical/a11y
@lexical/clipboard
@lexical/code
@lexical/code-core
@lexical/code-prism
@lexical/code-shiki
@lexical/compiler
@lexical/devtools-core
@lexical/dragon
@lexical/eslint-plugin
@lexical/extension
@lexical/fast-check
@lexical/file
@lexical/hashtag
@lexical/headless
@lexical/history
@lexical/html
@lexical/internal
@lexical/link
@lexical/list
@lexical/mark
@lexical/markdown
@lexical/mdast
@lexical/offset
@lexical/overflow
@lexical/plain-text
@lexical/react
@lexical/rich-text
@lexical/selection
@lexical/table
@lexical/tailwind
@lexical/text
@lexical/utils
@lexical/y
@lexical/yjs
commit: |
There was a problem hiding this comment.
Thanks, this looks good. Writing lone-surrogate text like any other text, and having the author read the container again so it shows the same U+FFFD as every peer, gives the single consistent outcome #9701 asks for and matches D57 as updated here. The new matrix in SurrogateSelection.test.ts fails on main and passes with this change across gc on/off and both client-id orders, and the changeset follows .changeset/README.md, including the Verify: paragraph AGENTS.md asks for on behaviour-changing fixes.
Two optional, non-blocking wording suggestions:
packages/lexical-website/docs/collaboration/y.md, Limits → Lone surrogates (line 1278). "Yjs 13 does the same." now follows the sentence about the author's editor reading the line again, so it reads as if the Yjs 13 binding (@lexical/yjs) also re-reads, which it does not. Moving it next to the UTF-8 clause (for example "…as Yjs encodes text as UTF-8; Yjs 13 does the same.") keeps the comparison to the storage behaviour, as D57's Why column already phrases it.packages/lexical-y/src/__tests__/unit/regressions/SurrogateSelection.test.ts, lines 16-17. The header says each cell checks "a fresh read of the author's document", butexpectConvergedbuilds its fresh reader frompeers[0](clienta), while every edit runs ont.b. The author side is still covered, becauset.b's editor has to match after reading its own document again, so only the wording needs adjusting (for example "a fresh read of the document").
| selection that starts or ends between the two halves of one) is stored | ||
| with U+FFFD in place of each lone half, as Yjs encodes text as UTF-8. | ||
| The author's editor reads the line again after such a write, so it shows | ||
| U+FFFD there too, like every other client. Yjs 13 does the same. |
There was a problem hiding this comment.
Optional: "Yjs 13 does the same." now follows the re-read sentence, so it reads as if the Yjs 13 binding (@lexical/yjs) also reads the line again, which it does not. Moving it next to the UTF-8 clause ("…as Yjs encodes text as UTF-8; Yjs 13 does the same.") keeps the comparison to the storage behaviour, as D57's Why column phrases it.
There was a problem hiding this comment.
Agreed it reads better next to the UTF-8 clause. Deferred to a follow-up to keep this head stable for landing (the reworked head 6400d02 keeps the sentence order).
| // no pair in its text node as U+FFFD, on every client, in a fresh read of the | ||
| // author's document, and in the author's own editor (the binding reads the |
There was a problem hiding this comment.
Optional wording: expectConverged builds its fresh reader from peers[0] (client a), while the edits run on t.b, so this is a fresh read of the collaborator's document. The author side is still covered by t.b's editor matching after its re-read; "a fresh read of the document" would be accurate.
There was a problem hiding this comment.
Agreed: expectConverged freshly reads the first client's document. The author side is covered by its editor matching after the re-read, and the fresh read of the author's own document is asserted separately in the cells that check it (the equal-attribute and composition cells). Making check freshly read every client is deferred to a follow-up to keep this head stable for landing.
6fe2f8f to
6400d02
Compare
There was a problem hiding this comment.
This looks good, thanks! Writing unpaired surrogate halves as U+FFFD in $pushInline and re-reading the line makes the author match every other client and a reload, and removes the throw reported in #9701. The new SurrogateSelection.test.ts fails 88 of its 96 cells on main and passes here, and the whole @lexical/y unit project passes at this head.
A few suggestions, all optional and non-blocking:
- Selection after the re-read (
packages/lexical-y/src/internal/Write.ts:2982). Afterbold(3, 6)inab😀cd, the restored selection starts at the end of the plainab\ufffdnode, not on the bolded\ufffdcdnode. Typing over it then givesparagraph("ab\ufffdZ")withZnot bold, where Lexical without collaboration keepsZbold;[6, 3)and[2, 3)behave the same.toWellFormedkeeps every offset, so restoring the selection by offset in the re-read line would keep it on the node it was on. Every cell clears the selection (SurrogateSelection.test.ts:188,:209), so a cell that keeps it and types over it would pin this down. - Re-read cost (
packages/lexical-y/src/internal/Binding.ts:5301). A refused container is read in full, and for a top-level paragraph that is the whole document. With 2,000 paragraphs a lone-half write took about 32 ms (about 71 ms at 5,000), against about 3 ms for an ordinary keystroke. It happens once per such write, so it can wait; re-reading only the affected line would keep it proportional to the line. The D57 row and y.md describe it as reading "the line". - Fast-path check (
packages/lexical-y/src/internal/Write.ts:3103).hasLoneSurrogaterebuilds the node's text whenever it holds any emoji: about 18 µs per keystroke for a 1,000-character node, about 165 µs for 10,000. A feature-detectedString.prototype.isWellFormed, with the current loop as the fallback, is about 20 times cheaper and keeps the fast path proportional to the edit, as its JSDoc says. - Docs (
packages/lexical-website/docs/collaboration/y.md:1279). "Yjs 13 does the same." now follows the sentences about re-reading the line and undo, but@lexical/yjshas no lone-surrogate handling, so there the author keeps the lone half until it reloads. Suggested wording: "Yjs 13 (@lexical/yjs) also delivers U+FFFD to other clients, but there the author keeps the lone half until it reloads." - Test inputs.
packages/lexical-y/AGENTS.mdasks for test inputs drawn fromsrc/__tests__/catalog/, and every cell uses a top-level paragraph. The same edits in a list item, quote, heading, table cell, link and code block converge, so this is only about coverage.
| const ctx = this.$ctx(true); | ||
| $pushInline(atoms, $lineChildren(d), ctx); | ||
| if (ctx.flattened === true) { | ||
| if (ctx.flattened === true || ctx.unpaired === true) { |
There was a problem hiding this comment.
Optional: after this re-read, a range that started on the bolded node comes back at the end of the plain node before it. After bold(3, 6) in ab😀cd the selection is ["ab\ufffd", 3] to ["\ufffdcd", 3], so typing over it inserts unbolded text (ab\ufffdZ), where Lexical without collaboration keeps Z bold. toWellFormed keeps every offset, so restoring the selection by offset in the re-read line would keep it on its node. Separately, the refused container is read in full, and for a top-level paragraph that is the whole document: about 32 ms at 2,000 paragraphs against about 3 ms per keystroke, once per such write.
There was a problem hiding this comment.
Confirmed: the re-read rebuilds the line and the selection lands at the end of the previous node at the same offset. Deferred to a follow-up (restore the selection by its offset in the re-read line, plus a cell that keeps the selection and types over it) to keep this head stable for landing.
| changes.some(([changed]) => $runsWithNeighbour(changed)) | ||
| changes.some( | ||
| ([changed]) => | ||
| $runsWithNeighbour(changed) || hasLoneSurrogate(changed.__text), |
There was a problem hiding this comment.
Optional: whenever the node holds any emoji, this rebuilds its whole text with toWellFormed: about 18 µs per keystroke at 1,000 characters and about 165 µs at 10,000. A feature-detected String.prototype.isWellFormed, with this as the fallback, is about 20 times cheaper and keeps the fast path proportional to the edit.
There was a problem hiding this comment.
Agreed; deferred to the same follow-up: feature-detected String.prototype.isWellFormed with the current loop as fallback, so the keystroke path stays proportional to the edit.
| with U+FFFD in place of each half that has no pair in its text node, as | ||
| Yjs encodes text as UTF-8. The author's editor reads the line again after | ||
| such a write, so it shows U+FFFD there too, like every other client, and | ||
| undoing the write brings the character back. Yjs 13 does the same. |
There was a problem hiding this comment.
Optional wording: @lexical/yjs has no lone-surrogate handling, so its author keeps the lone half until it reloads. After the new sentences about re-reading and undo, "Yjs 13 does the same" reads as if it did those too. Suggested: "Yjs 13 (@lexical/yjs) also delivers U+FFFD to other clients, but there the author keeps the lone half until it reloads."
There was a problem hiding this comment.
Agreed; deferred to the same follow-up, with your wording (@lexical/yjs delivers U+FFFD to other clients but its author keeps the lone half until it reloads).
Description
A selection an app sets between the two halves of a surrogate pair (
anchor.set(key, 3, 'text')inab😀cd; Lexical's own input never does this) makes Lexical split the pair into two text nodes or keep one half. Reported in #9701,@lexical/ythen behaved three ways:@lexical/y: a write splits a surrogate pair(a development check on the way to the range's start): nothing was written, the author kept Lexical's split and the other clients kept the emoji.D57 accepts lone surrogates reaching other clients as U+FFFD (malformed text is the author's input), so the binding should write such text like any other text and never throw. This change:
$pushInline) holds each text node's text with every surrogate half that has no pair in that node as U+FFFD, which is what Yjs's UTF-8 encoding sends and a reload reads, so the author's document holds the same text as every other client. A pair whose halves Lexical put in two text nodes is then new text (two U+FFFD), so no write splits a stored pair: the dev checka write splits a surrogate paircannot fire on such input, and undoing the write brings the character back.DesiredContext.unpaired, through the samerefusedre-read that two text nodes stored as one run already take), so the author's editor shows U+FFFD too.Fixes the throw reported in #9701 (the issue is closed under D57).
Test plan
New
packages/lexical-y/src/__tests__/unit/regressions/SurrogateSelection.test.ts: 24 cases x Yjs gc off and on x both client id orders (96 cells). Bold a range that ends inside, starts inside, covers one half, or runs backwards from inside; type inside the pair; type over a half; delete either half; delete from inside to the end; type a lone half before the emoji and at the end of the line; split the text node inside the pair and bold one side; a pair split across two text nodes; a lone half typed at the end and at the start of a text node; two text nodes Lexical keeps apart that store the same attributes (one run) holding the halves; a split inside the emoji while it is being composed; a collaborator typing in the line at the same time; a collaborator composing at the end of the line while the author bolds from inside the pair; the author undoing the write; controls (a collapsed selection inside, bolding the whole emoji). Each cell expects no warnings or errors and every client, the author and a fresh read of the author's document to show Lexical's text with each half lacking its pair in its text node as U+FFFD (the undo cell: the emoji back).Before
On main (gc off, client ids [1, 2] shown; the other three settings fail the same cells the same way):
(The 8 passing cells are the two controls in each setting.)
After
packages/lexical-yunit tests:tsc,tsc -p tsconfig.test.jsonandeslinton the changed files pass.