Python: Bug: split_plaintext_paragraph / split_markdown_paragraph can return a chunk larger than max_tokens #14566

Describe the bug

split_plaintext_paragraph and split_markdown_paragraph can return a paragraph whose token count is above max_tokens.

At the end of _split_text_paragraph, a short last paragraph is merged into the previous one. The merge check counts words (len(paragraph.split(" "))) instead of calling token_counter:

A word is usually more than one token, so the merged paragraph can go over the limit even though the word count is under it. This matters when max_tokens is the embedding model's input limit.

To Reproduce

With the default token counter, using the input of the existing test_split_text_paragraph_evenly test:

from semantic_kernel.text import split_plaintext_paragraph

text = [
    "This is a test of the emergency broadcast system. This is only a test.",
    "We repeat, this is only a test. A unit test.",
    "A small note. And another. And once again. Seriously, this is the end. We're finished. All set. Bye.",
    "Done.",
]
chunks = split_plaintext_paragraph(text, 15)
print([len(c) // 4 for c in chunks])  # [12, 5, 11, 10, 16] -> last chunk is 16 tokens, limit is 15

With a real tokenizer (tiktoken, cl100k_base) passed as token_counter, on generated multi-line English text and limits from 64 to 512, 955 of 8000 runs (about 12%) returned an over-limit chunk. The worst one was 314 tokens for max_tokens=256.

Expected behavior

No returned paragraph is above max_tokens. The last paragraph should only be merged into the previous one if the merged text fits, measured with token_counter.

Platform

• Language: Python
• Source: main at cc8a15f
• OS: Windows 11, Python 3.12

Additional context

Eight existing unit tests in tests/unit/text/test_text_chunker.py expect an over-limit last chunk (for example 16 tokens with max_token_per_line = 15). A fix would change their expected output, so I wanted to raise it here first. I have a small fix with tests ready and I'm happy to open the PR.