fix: replacing Unicode chars with NCRs unexpectedly changes formatting

[grnch]

When using xmlWhitespaceSensitivity: "ignore", normal Unicode text inside XML elements gets formatted as you'd expect, but if one of the chars is replaced with a Numeric Character Reference (NCR) or an XML entity, the formatting changes in unexpected ways.

For example, if you have a simple element with Unicode text:

<p>piñata</p>

it gets formatted like this (i.e. no change):

<p>piñata</p>

However, if you just replace the non-ASCII char with an NCR:

<p>pi&#241;ata</p>

it suddenly gets reformatted like this:

<p>
  pi &#241; ata
</p>

Not only does it add an extra indent, it also breaks apart the piñata.

An NCR is just an alternate encoding for the same char, so it shouldn't change how the entire word is formatted (unless of course it causes it to exceed the printWidth, which is not the case here).

Things get worse if you include mixed content (text side-by-side with elements).

If we add an extra element to the original Unicode text:

<p>piñata<br/></p>

it gets formatted like this (pretty sensible):

<p>
  piñata
  <br />
</p>

However, if we replace the non-ASCII char with an NCR again:

<p>pi&#241;ata<br/></p>

it gets formatted like this instead:

<p>
  pi
  &#241;
  ata
  <br />
</p>

The piñata is not only broken apart, but split across separate lines.

Same behavior can be observed with XML entities too, not just NCRs.

For example, if we replace the apostrophe in "Let's go" with an entity:

<p>Let&apos;s go<br/></p>

it gets formatted like this, similarly broken up across separate lines:

<p>
  Let
  &apos;
  s go
  <br />
</p>

The root cause is in how the original code handled some special cases. If an XML element had only text, it would get printed on a single line, but if it had mixed content it would fall back to general code that just inserted hardlines between every token.

This is what caused the word breaking apart across multiple lines. In case of references it would change the delimiter to a space instead, which explains the case where the word would be broken apart by spaces instead of split across multiple lines.

From what I can infer, the intent of this code was to print XML elements on separate lines, but it looks like NCRs/entities were unintentionally caught by the logic intended for elements.

I removed those special cases and (hopefully) simplified the code a bit.

The mixed content loop now carefully inserts hardlines only before/after elements, while leaving the other tokens undisturbed. This accomplishes the goal of printing each XML element on a separate line, while allowing the surrounding text to flow freely around the elements.

There is one special case remaining, to handle elements that only have a single child element and nothing else:

<p><br/></p>

Without this special case, the element above would be printed unchanged, on one line.

If I understood the intent of this formatter correctly, each XML element is supposed to start on a separate line, so the element above should be formatted like this:

<p>
  <br />
</p>

This is what that special case does.

I added some test cases based on the examples above, and confirmed that they got formatted properly before updating the snapshots. None of the existing test cases were affected by these code changes.