Conversation
…nd clean the reply
replacePlaceholders() substituted the page HTML into the template with a
plain String.replace. Jsoup writes script and style contents and comments
out verbatim, so a page could carry the template's own marker lines, such
as <|HTML_CONTENT_END|>, and open its own instruction section. The marker
tokens of the form <|NAME|> found in the configured template are now
removed from the HTML before it is substituted, which also covers custom
templates. {REQUEST} is substituted before {HTML}, so a page containing
{REQUEST} no longer pulls in the configured request.
text() returned the reply as it came. It now takes the content of the
<content> envelope the default prompt asks for, or the whole reply if
there is none, keeps only its text nodes so that no markup is returned,
and truncates it to textextractor.llm.text.maxlength (default -1, no
limit).
The default prompt no longer tells the model to ignore its guidelines
when a user instruction is present. An operator who wants that can say so
in textextractor.llm.user_request or a custom template.
dpol1
reviewed
Sep 25, 2026
dpol1
left a comment
dpol1
left a comment
Member
There was a problem hiding this comment.
One on the config key and two on the reply cleaning, inline.
| public static final String USER_PROMPT = "textextractor.llm.prompt"; | ||
| public static final String USER_REQUEST = "textextractor.llm.user_request"; | ||
| public static final String LISTENER_CLASS = "textextractor.llm.listener.clazz"; | ||
| public static final String TEXT_MAX_LENGTH = "textextractor.llm.text.maxlength"; |
Member
There was a problem hiding this comment.
we already have textextractor.skip.after on TextExtractor for this, can we reuse it? the README line saying it's unsupported here can then go
| } | ||
|
|
||
| /** keeps the text nodes of the input and their line breaks, dropping elements and comments */ | ||
| private static String stripMarkup(String text) { |
Member
There was a problem hiding this comment.
this eats the code blocks the prompt asks for: List<String> comes back as List, x<y and y>z as xz. strip only outside the ``` fences, or keep the envelope and skip the stripping?
| return sb.toString(); | ||
| } | ||
|
|
||
| private String removeMarkers(String html) { |
Member
There was a problem hiding this comment.
wouldn't html.replace("<|", "< |") do the same in one pass, whatever the template? no marker set, no loop, and nested markers stay linear
Contributor
Author
|
@dpol1 Feel free to directly push to this branch ;-) |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2093.
Marker tokens of the form
<|NAME|>found in the prompt template are now removed from the page HTML before it is substituted, and{REQUEST}is filled in before{HTML}. The reply is reduced to the content of its<content>envelope (or the whole reply if there is none), markup is removed, and the newtextextractor.llm.text.maxlength(default -1) caps the length. The default prompt no longer tells the model to ignore its guidelines when a user instruction is present.