Fix PPL search command dropping wildcards on values with whitespace (#5682)

[penghuo]

Description

On the Calcite path, search source=idx name="foo bar*" against a keyword field returned 0 hits instead of matching the whole-value pattern foo bar*. The parser marked whitespace-containing literals as phrases, which emitted name:"foo bar*" — inside a Lucene phrase, * is a literal character, so the query searched for docs containing * in the stored value and found none.

Emission strategy

PPL emits a single query_string filter for the entire search predicate (never a specific Lucene query type — the query_string parser inside OpenSearch decides that at execution time based on the parsed operators and the target field's analyzer). The emitter's job is to produce the right string.

The decision is driven by three orthogonal properties of the PPL literal and its enclosing field:

1. whitespace in the value — set at parse time as SearchLiteral.isPhrase.
2. an unescaped wildcard (* / ?) — scanned by hasUnescapedWildcard.
3. field is text-like — true when the mapping is text or match_only_text. Read from the OpenSearch table's field-type map directly (AbstractOpenSearchTable.getFieldTypes()), because the Calcite RelDataType round trip collapses text to plain VARCHAR and loses the distinction.

value contains whitespace? (isPhrase)
├── no  → [A] unquoted; escape Lucene specials (wildcards preserved)
└── yes → contains unescaped * or ??
         ├── no  → [B] quoted phrase; Lucene specials escaped inside
         └── yes → field is text-like? (text | match_only_text)
                  ├── yes → [C] quoted phrase; specials escaped inside
                  └── no  → [D] unquoted; escape specials AND whitespace

Decision notes

[A] Whitespace-free value → unquoted, escape specials, keep wildcards. Field-type-agnostic. The query_string parser sees a single term; the field's own analyzer (or lack of one, for keyword) decides matching at execution time.

• name=foo → name:foo
• name="foo-bar" → name:foo\-bar
• name="foo*" → name:foo* (parser sees an unescaped *, builds a prefix query)
• name="foo/*" → name:foo\/* (/ escaped so it isn't parsed as a regex delimiter, * preserved)

[B] Whitespace value without wildcards → quoted phrase. Preserves phrase semantics without needing to think about the field type: on keyword it matches the whole term; on text it becomes a PhraseQuery over analyzed tokens.

• name="foo bar" → name:"foo bar"
• name="hello world" → name:"hello world"

[C] Whitespace + wildcard on a text-like field → quoted phrase. Inside a phrase, the text analyzer strips * / ? as punctuation and matches the remaining tokens. That's the natural bag-of-words behavior users expect on text fields, and it matches pre-fix behavior (no regression).

• name="foo bar*" on text → name:"foo bar*" (analyzer emits tokens [foo, bar], matches phrases containing them)
• name="*foo bar*" on text → name:"*foo bar*" (same, wildcards stripped inside the phrase)

[D] Whitespace + wildcard on a non-text field (keyword, constant_keyword, wildcard, numeric, etc.) → unquoted, escape everything including the space. This is the reported bug (#5682) fix. Emitting a quoted phrase here on keyword makes the literal * / ? search for those characters in the stored value (they aren't operators inside a phrase), which returns zero. Emitting unquoted with the space un-escaped is worse: the query_string parser splits at the space into two clauses (name:foo + unfielded bar*), dropping the field binding on the right half. Escaping the space keeps the value a single term with active wildcards; the parser builds a whole-value pattern match against the keyword's stored term.

• name="foo bar*" on keyword → name:foo\ bar* (matches foo bar, foo barbaz)
• name="*foo bar*" on keyword → name:*foo\ bar* (matches foo bar, foo barbaz)
• name="foo b?r" on keyword → name:foo\ b?r (matches foo bar)

About "text-like". The predicate returns true only for text and match_only_text. Every other mapping — keyword, constant_keyword, wildcard, numeric types, date, boolean — routes to [D]. When the field type can't be resolved (scan doesn't unwrap to AbstractOpenSearchTable, field missing from the type map), the predicate returns true as a regression-safe fallback so we take the [C] phrase branch. This case is dead code for well-formed queries against real indices.

Measured contract

Fixture. Two indices with identical documents. test_5682_keyword maps name to keyword; test_5682_text maps name to text (standard analyzer). 11 documents each, values: foo, foobar, food, FOO, foo bar, foo barbaz, foo-bar, foo_bar, foo.bar, foo/bar, foo@bar.

Group 1 — no special chars, no wildcards

Row	PPL query	Emitted on text	Text hits	Emitted on keyword	Keyword hits
1.1	name=foo	name:foo	7	name:foo	1
1.2	name="foo"	name:foo	7	name:foo	1

Group 2 — special chars in value, no wildcards

Row	PPL query	Emitted on text	Text hits	Emitted on keyword	Keyword hits
2.1	name="foo_bar"	name:foo_bar	1	name:foo_bar	1
2.2	name="foo.bar"	name:foo.bar	1	name:foo.bar	1
2.3	name="foo-bar"	name:foo\-bar	7	name:foo\-bar	1
2.4	name="foo/bar"	name:foo\/bar	7	name:foo\/bar	1
2.5	name="foo@bar"	name:foo@bar	7	name:foo@bar	1
2.6	name="foo bar"	name:"foo bar"	4	name:"foo bar"	1

Group 3 — trailing wildcard (postfix)

Row	PPL query	Emitted on text	Text hits	Emitted on keyword	Keyword hits
3.1	name=foo*	name:foo*	11	name:foo*	10
3.2	name="foo*"	name:foo*	11	name:foo*	10
3.3	name="foo_*"	name:foo_*	1	name:foo_*	1
3.4	name="foo.*"	name:foo.*	1	name:foo.*	1
3.5	name="foo-*"	name:foo\-*	0	name:foo\-*	1
3.6	name="foo/*"	name:foo\/*	0	name:foo\/*	1
3.7	name="foo bar*"	name:"foo bar*"	4	name:foo\ bar*	2 (the reported bug #5682, fixed)

Group 4 — leading wildcard (prefix)

Row	PPL query	Emitted on text	Text hits	Emitted on keyword	Keyword hits
4.1	name="*foo"	name:*foo	7	name:*foo	1
4.2	name="*bar"	name:*bar	7	name:*bar	7
4.3	name="*foo bar"	name:"*foo bar"	4	name:*foo\ bar	1

Group 5 — interior wildcard (* in-between)

Row	PPL query	Emitted on text	Text hits	Emitted on keyword	Keyword hits
5.1	name="f*r"	name:f*r	3	name:f*r	7
5.2	name="foo*bar"	name:foo*bar	3	name:foo*bar	7
5.3	name="foo *baz"	name:"foo *baz"	0	name:foo\ *baz	1
5.4	name="*foo bar*"	name:"*foo bar*"	4	name:*foo\ bar*	2

Note on 5.3 text: the analyzer strips * inside the phrase and produces tokens [foo, baz]. On this corpus no doc has adjacent analyzed tokens foo → baz (the closest is foo barbaz, which tokenizes to [foo, barbaz] — barbaz is one token, not [bar, baz]). Zero on text is a corpus/analyzer interaction, not an emission failure. Same phrase shape with a present token — e.g. name:"foo *bar" — returns 4 hits on text against this corpus (bar is an indexed token).

Group 6 — ? wildcard (exactly one character)

Row	PPL query	Emitted on text	Text hits	Emitted on keyword	Keyword hits
6.1	name="foo?"	name:foo?	1	name:foo?	1
6.2	name="?oo"	name:?oo	7	name:?oo	1
6.3	name="f?o"	name:f?o	7	name:f?o	1
6.4	name="foo?bar"	name:foo?bar	2	name:foo?bar	6
6.5	name="foo b?r"	name:"foo b?r"	0	name:foo\ b?r	1

Note on 6.5 text: analyzer produces [foo, br] (drops ? as punctuation, splits at whitespace). No corpus doc has adjacent tokens foo → br. Mathematically correct given the analyzer's behavior.

Tests

• 54 new Group1–Group6 tests in CalciteSearchCommandIT covering the full text × keyword × wildcard-placement matrix on the fixture above.

Related Issues

Resolves #5682

Check List

• New functionality includes testing.
• New functionality has been documented.
• New functionality has javadoc added.
• New functionality has a user manual doc added.
• New PPL command checklist all confirmed.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff or -s.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Impact
Possible issue	Handle consecutive backslashes correctly The method does not handle consecutive backslashes correctly. A sequence like \* (escaped backslash followed by wildcard) will incorrectly skip the wildcard, treating it as escaped when it should be considered unescaped. Track whether the previous character was an unescaped backslash. core/src/main/java/org/opensearch/sql/ast/expression/SearchLiteral.java [91-103] private static boolean hasUnescapedWildcard(String s) { + boolean escaped = false; for (int i = 0; i < s.length(); i++) { char c = s.charAt(i); - if (c == '\\' && i + 1 < s.length()) { - i++; + if (c == '\\' && !escaped) { + escaped = true; continue; } - if (c == '*' || c == '?') { + if ((c == '*' || c == '?') && !escaped) { return true; } + escaped = false; } return false; } Suggestion importance[1-10]: 7 __ Why: The suggestion identifies a legitimate edge case where consecutive backslashes (\\*) could be mishandled. The improved logic with the escaped flag properly tracks escape state across iterations, fixing a potential bug in wildcard detection.	Medium
General	Add fallback for expression resolution If getOriginalExpression() returns non-null but toQueryString() throws an exception or returns null/empty, the fallback to getQueryString() is never attempted. Add error handling to ensure the pre-computed query string is used when expression-based resolution fails. core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java [307-324] if (node.getOriginalExpression() != null) { ... - queryString = node.getOriginalExpression().toQueryString(typesByName::get); + try { + queryString = node.getOriginalExpression().toQueryString(typesByName::get); + } catch (Exception e) { + queryString = node.getQueryString(); + } } else { queryString = node.getQueryString(); } Suggestion importance[1-10]: 6 __ Why: Adding error handling to fall back to getQueryString() when toQueryString() fails is a reasonable defensive programming practice. However, the suggestion assumes exceptions might occur without evidence from the PR context, making it a moderate improvement rather than a critical fix.	Low
	Verify space escape ordering The space replacement logic may fail if escapeLuceneSpecialCharacters introduces backslashes before spaces. This could result in \ becoming \ (double-escaped). Verify that the escape function does not already escape spaces, or apply space replacement before escaping special characters. core/src/main/java/org/opensearch/sql/ast/expression/SearchLiteral.java [62-64] if (isPhrase && !isTextLike(indexType) && hasUnescapedWildcard(str)) { - return QueryStringUtils.escapeLuceneSpecialCharacters(str).replace(" ", "\\ "); + String escaped = QueryStringUtils.escapeLuceneSpecialCharacters(str); + return escaped.replace(" ", "\\ "); } Suggestion importance[1-10]: 3 __ Why: The suggestion raises a valid concern about escape ordering, but the improved_code is identical to the existing_code, making the practical impact minimal. The concern is worth verifying but doesn't constitute a critical fix.	Low

[penghuo]

@vamsimanohar Please help review.