[core] Support FM indexes for primary-key tables

docs/docs/multimodal-table/global-index.mdx

@@ -35,6 +35,7 @@ without full-table scans. Paimon supports multiple global index types:
 - **[BTree Index](./global-index/btree)**: A B-tree based index for scalar column lookups. Supports equality, IN, range predicates, and can be combined across multiple columns with AND/OR logic.
 - **[Bitmap Index](./global-index/bitmap)**: A bitmap based index for enum-like scalar dimensions and tag columns. Supports equality, IN, prefix match on string columns, complement predicates, and null checks with compressed row-id bitmaps.
 - **[Multivalue Index](./global-index/multivalue)**: A bitmap-backed index for element-membership predicates on `ARRAY` columns.
+- **[FM Index](./global-index/fm)**: An exact partitioned substring index for `CONTAINS` predicates on character columns.
 - **[Vector Index](./global-index/vector)**: An approximate nearest neighbor (ANN) index powered by Paimon's vector index library for vector similarity search.
 - **[Full-Text Index](./global-index/full-text)**: A full-text search index backed by the native full-text engine for text retrieval. Supports term matching and relevance scoring.
 - **[Hybrid Search](./global-index/hybrid-search)**: A multi-route search API that combines results from multiple vector routes, multiple full-text routes, or both before reading table rows.
@@ -44,6 +45,7 @@ without full-table scans. Paimon supports multiple global index types:
 | BTree | Scalar filters on numeric, string, date, and timestamp columns | Best when predicates are selective, such as equality, IN, range, and null checks. |
 | Bitmap | Enum-like dimensions and tag columns | Best for equality, IN, string prefix match, complement predicates, and null checks over compressed row-id bitmaps. |
 | Multivalue | Membership tests on arrays of supported scalar elements | Best for `ARRAY_CONTAINS`, `ARRAYS_OVERLAP`, and `ARRAY_CONTAINS_ALL`. |
+| FM | Exact substring filters on character columns | Supports needles of any byte length and partitions the indexed text for bounded-memory construction and demand-loaded reads. |
 | Vector | Top-K similarity search on embeddings | Uses ANN algorithms. Tune build-time and search-time options to balance recall, latency, and index size. |
 | Full-Text | Keyword search over text columns | Uses full-text scoring and tokenizer configuration stored with each index file. |
 | Hybrid Search | Combining multiple vector routes, multiple full-text routes, or vector and full-text retrieval together | Runs multiple scored routes and merges them with a ranker before reading rows. |
@@ -396,7 +398,7 @@ These table options affect global index build and read behavior:
 |---|---|---|
 | `global-index.enabled` | `true` | Whether scans can use global indexes. |
 | `global-index.search-mode` | Not set | Legacy fallback search mode for global-index queries. Family-specific options take precedence. |
-| `scalar-index.search-mode` | `fast` | Search mode for BTree, Bitmap, and Multivalue queries. |
+| `scalar-index.search-mode` | `fast` | Search mode for BTree, Bitmap, Multivalue, and FM queries. |
 | `vector-index.search-mode` | `fast` | Search mode for vector queries. |
 | `full-text-index.search-mode` | `fast` | Search mode for full-text queries. |
 | `global-index.external-path` | Not set | Root directory for global index files. If not set, files are stored under the table index directory. |
@@ -427,6 +429,12 @@ Use Multivalue indexes for element-membership predicates on `ARRAY` columns. See
 [Multivalue Index](./global-index/multivalue) for Data Evolution build examples, Core query usage,
 options, and null/empty membership semantics.
 
+## FM Index
+
+Use FM indexes for exact substring predicates on character columns. See
+[FM Index](./global-index/fm) for build examples, exactness and coverage semantics, tuning options,
+and primary-key table configuration.
+
 ## Vector Index
 
 Use Vector indexes for approximate nearest neighbor (ANN) search. See [Vector Index](./global-index/vector)

docs/docs/multimodal-table/global-index/fm.mdx

@@ -0,0 +1,111 @@
+---
+title: "FM Index"
+sidebar_position: 4
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# FM Index
+
+The FM index is an exact byte-oriented substring index for `CHAR`, `VARCHAR`, and `STRING`
+columns. It supports `CONTAINS` needles of any byte length without a configured gram size. Null
+values do not match; empty needles follow the normal Paimon predicate semantics.
+
+The writer divides source rows into independent partitions and appends them to one checksummed
+container file. Each partition stores a compressed wavelet matrix, sampled suffix-array values,
+row boundaries, null rows, and exact verification pages. Reads demand-load bounded blocks instead
+of downloading the complete container. If locating matches would cost more than exact
+verification, the reader scans the relevant verification pages and still returns an exact result.
+
+## Create a Global FM Index
+
+Create the index on a Data Evolution table with row tracking enabled:
+
+```sql
+CALL sys.create_global_index(
+    table => 'db.documents',
+    index_column => 'content',
+    index_type => 'fmindex',
+    options => 'fm-index.partition-row-count=100000'
+);
+```
+
+Drop it with the same index type:
+
+```sql
+CALL sys.drop_global_index(
+    table => 'db.documents',
+    index_column => 'content',
+    index_type => 'fmindex'
+);
+```
+
+See [Global Index](../global-index) for row-tracking requirements, lifecycle, and partial-coverage
+semantics.
+
+## Query
+
+After a connector converts a literal substring expression to Paimon's `CONTAINS` predicate, the
+normal batch scan uses the FM index automatically. For example, Spark pushes down:
+
+```sql
+SELECT id, content
+FROM documents
+WHERE content LIKE '%needle%';
+```
+
+Results are exact inside every indexed row range. A table-wide global-index query can still omit
+matches in row ranges that have never been indexed; rebuild the index for newly appended ranges or
+use the visibility callback described on the Global Index page.
+
+For a primary-key table, configure `pk-fm.index.columns` instead. Primary-key FM indexes follow
+data compaction and scan uncovered files through the ordinary data path, so partial coverage does
+not make `CONTAINS` results incomplete. See [Primary-Key Indexes](../../primary-key-table/global-index).
+
+## Options
+
+Global options can be passed to `create_global_index`. For a primary-key index, place the same
+keys in `fields.<column>.pk-fm.index.options`; the `fm-index.` prefix may be omitted inside that
+JSON object.
+
+| Option | Default | Description |
+|---|---|---|
+| `fm-index.partition-size` | `16 mb` | Maximum encoded text buffered by one independently readable partition. |
+| `fm-index.partition-row-count` | `100000` | Maximum source rows in one partition. |
+| `fm-index.sa-sample-rate` | `32` | Suffix-array sample rate. A smaller power of two speeds locate at the cost of a larger index. |
+| `fm-index.compression` | `lz4` | Compression codec for independently checksummed FM blocks. |
+| `fm-index.compression-level` | `1` | Compression level for codecs which support levels. |
+| `fm-index.read-cache-size` | `64 mb` | Maximum decoded rank and sample block cache per indexer. |
+| `fm-index.demand-page-size` | `512 kb` | Target contiguous range size when demand-loading blocks. |
+| `fm-index.locate-cost-ratio` | `0.001` | Maximum estimated suffix-array locate work relative to exact stored-value scan bytes before verification fallback. |
+
+`fm-index.partition-size` and `fm-index.partition-row-count` bound build memory and the unit of
+independent reads. Smaller partitions reduce peak construction memory but increase the number of
+partitions searched per query. All partitions produced by one writer are stored in one physical
+index file and represented by one index manifest entry. A lower `fm-index.sa-sample-rate`
+accelerates locating matched rows but stores more suffix-array samples.
+
+## Limitations
+
+- The index is single-column and accepts only character string types.
+- Matching is byte-oriented and case-sensitive; it does not apply a tokenizer, collation, or
+  Unicode normalization.
+- Global FM indexes inherit the partial-coverage behavior of table-wide global indexes.
+- Primary-key FM indexes are built from eligible compact output, not directly from Level-0 appends.

docs/docs/multimodal-table/index.mdx

@@ -38,7 +38,7 @@ Key capabilities:
 - **[Variant Storage](./variant)**: Store and query schema-flexible semi-structured data with optional typed sub-column shredding.
 - **[Blob Storage](./blob)**: Store large binary objects (images, videos, audio) in dedicated `.blob` files with efficient column projection.
 - **[Vector Storage](./vector)**: Store and manage vector embeddings in dedicated Vortex-format files optimized for vector workloads.
-- **[Global Index](./global-index)**: Build BTree, Bitmap, Multivalue, vector, and full-text indexes for efficient lookups and similarity search.
+- **[Global Index](./global-index)**: Build BTree, Bitmap, Multivalue, FM, vector, and full-text indexes for efficient lookups, exact substring filtering, and similarity search.
 
 Data Evolution, Blob Storage, Vector Storage, and Global Index require the following table properties.
 Variant Storage can also be used in a regular Paimon table without these properties:

docs/docs/primary-key-table/global-index.mdx

@@ -27,7 +27,7 @@ under the License.
 
 # Primary-Key Indexes
 
-Primary-key tables can maintain Vector, Full Text, BTree, Bitmap, and Multivalue indexes together
+Primary-key tables can maintain Vector, Full Text, FM, BTree, Bitmap, and Multivalue indexes together
 with compact data files. These indexes are bucket-local and source-backed: every index group records
 its source data files and maps matches back to physical row positions. Deletion vectors are applied
 when indexed rows are read, so updates and deletes remain exact.
@@ -100,11 +100,23 @@ For an append-only or Data Evolution table whose full-text index is built indepe
 
 </TabItem>
 
+<TabItem value="fm" label="FM">
+
+Use FM for exact substring predicates on a `CHAR`, `VARCHAR`, or `STRING` column. Normal batch
+scans apply it automatically after a connector converts a literal substring expression to
+Paimon's `CONTAINS` predicate. Needles of any byte length are supported without choosing a gram
+size.
+
+For an append-only or Data Evolution table whose FM index is built independently, see
+[FM Index](../multimodal-table/global-index/fm).
+
+</TabItem>
+
 </Tabs>
 
 Different columns in one table can use different index families. One column can occur in at most
 one of `pk-vector.index.columns`, `pk-full-text.index.columns`, `pk-btree.index.columns`,
-`pk-bitmap.index.columns`, and `pk-multivalue.index.columns`.
+`pk-bitmap.index.columns`, `pk-multivalue.index.columns`, and `pk-fm.index.columns`.
 
 ## Requirements
 
@@ -120,14 +132,14 @@ supported by this configuration.
 
 <Tabs groupId="primary-key-index-requirements">
 
-<TabItem value="sorted" label="BTree, Bitmap, and Multivalue">
+<TabItem value="sorted" label="BTree, Bitmap, Multivalue, and FM">
 
-BTree, Bitmap, and Multivalue additionally require:
+BTree, Bitmap, Multivalue, and FM additionally require:
 
 - `deletion-vectors.enabled = true`.
 - `deletion-vectors.merge-on-read = false`.
 - A supported scalar column type for BTree and Bitmap, or an `ARRAY` of supported scalar elements
-  for Multivalue.
+  for Multivalue. FM requires a `CHAR`, `VARCHAR`, or `STRING` column.
 
 Each entry creates an independent single-column index. Multiple columns are supported as long as a
 column is not listed by another primary-key index family.
@@ -166,8 +178,9 @@ Exactly one full-text column is currently supported per table.
 
 ## Create a Table
 
-The following table uses all five families on different columns: Vector for `embedding`, Full Text
-for `content`, BTree for `amount`, Bitmap for `status`, and Multivalue for `tags`.
+The following table uses all six families on different columns: Vector for `embedding`, Full Text
+for `content`, FM for `raw_text`, BTree for `amount`, Bitmap for `status`, and Multivalue for
+`tags`.
 
 <Tabs groupId="primary-key-index-create-table">
 
@@ -180,6 +193,7 @@ CREATE TABLE items (
     amount DECIMAL(12, 2),
     tags ARRAY<STRING>,
     content STRING,
+    raw_text STRING,
     embedding ARRAY<FLOAT> COMMENT '__VECTOR_FIELD;3',
     PRIMARY KEY (id) NOT ENFORCED
 ) WITH (
@@ -191,6 +205,8 @@ CREATE TABLE items (
     'fields.embedding.pk-vector.index.options' = '{"nlist":"256"}',
     'pk-full-text.index.columns' = 'content',
     'fields.content.pk-full-text.index.options' = '{"tokenizer":"jieba"}',
+    'pk-fm.index.columns' = 'raw_text',
+    'fields.raw_text.pk-fm.index.options' = '{"partition-row-count":"100000"}',
     'pk-btree.index.columns' = 'amount',
     'fields.amount.pk-btree.index.options' = '{"block-size":"64 kb"}',
     'pk-bitmap.index.columns' = 'status',
@@ -215,6 +231,7 @@ CREATE TABLE items (
     amount DECIMAL(12, 2),
     tags ARRAY<STRING>,
     content STRING,
+    raw_text STRING,
     embedding ARRAY<FLOAT> COMMENT '__VECTOR_FIELD;3'
 ) USING paimon
 TBLPROPERTIES (
@@ -227,6 +244,8 @@ TBLPROPERTIES (
     'fields.embedding.pk-vector.index.options' = '{"nlist":"256"}',
     'pk-full-text.index.columns' = 'content',
     'fields.content.pk-full-text.index.options' = '{"tokenizer":"jieba"}',
+    'pk-fm.index.columns' = 'raw_text',
+    'fields.raw_text.pk-fm.index.options' = '{"partition-row-count":"100000"}',
     'pk-btree.index.columns' = 'amount',
     'fields.amount.pk-btree.index.options' = '{"block-size":"64 kb"}',
     'pk-bitmap.index.columns' = 'status',
@@ -253,6 +272,8 @@ schema validation.
 | `fields.<column>.pk-vector.index.options` | Not set | JSON object containing build options for the selected ANN implementation. |
 | `pk-full-text.index.columns` | Not set | Character column to index. Exactly one full-text column is currently supported. |
 | `fields.<column>.pk-full-text.index.options` | Not set | JSON object containing native analyzer options. Unqualified keys are scoped to `full-text`. |
+| `pk-fm.index.columns` | Not set | Comma-separated character columns which own independent exact FM indexes. |
+| `fields.<column>.pk-fm.index.options` | Not set | JSON object containing FM build and read options. Unqualified keys are scoped to `fm-index`. |
 | `pk-btree.index.columns` | Not set | Comma-separated columns which own independent BTree indexes. |
 | `fields.<column>.pk-btree.index.options` | Not set | JSON object containing BTree build options. Unqualified keys are scoped to `btree-index`. |
 | `pk-bitmap.index.columns` | Not set | Comma-separated columns which own independent Bitmap indexes. |
@@ -268,7 +289,9 @@ schema validation.
 
 For algorithm-specific options, see the corresponding
 [BTree](../multimodal-table/global-index/btree),
-[Bitmap](../multimodal-table/global-index/bitmap), or
+[Bitmap](../multimodal-table/global-index/bitmap),
+[Multivalue](../multimodal-table/global-index/multivalue),
+[FM](../multimodal-table/global-index/fm), or
 [Vector](../multimodal-table/global-index/vector) index page. Full-text analyzer options are listed
 in the [`paimon-full-text` README](https://github.com/apache/paimon/blob/master/paimon-full-text/README.md).
 
@@ -289,13 +312,14 @@ output; simply assigning or upgrading a pending file does not make it an index s
 
 ### Data-Level Maintenance
 
-Each indexed column maintains one immutable index payload for the complete eligible source-file
-set in every non-zero data level. When data compaction changes a level, Paimon rebuilds that whole
-level payload, including files in the target level which were not direct compaction inputs. A
-level payload is used only when its ordered source names and row counts exactly match the current
-data level; partial, duplicate, stale, and cross-level payloads are rejected.
+Each indexed column maintains one immutable index group for the complete eligible source-file set
+in every non-zero data level. Each group writes one payload file; an FM payload can contain several
+independently readable internal partitions. When data compaction changes a level, Paimon rebuilds
+that whole level group, including files in the target level which were not direct compaction
+inputs. A level group is used only when its ordered source names and row counts exactly match the
+current data level; stale payloads and cross-level payloads are rejected.
 
-A rebuild atomically replaces the old payload after the complete new payload is ready. Unrelated
+A rebuild atomically replaces the old group after the complete new group is ready. Unrelated
 data levels retain their existing payloads.
 
 Index construction can execute asynchronously inside the writer. A writer which waits for
@@ -307,7 +331,7 @@ Coverage behavior depends on the index family and search mode. `vector-index.sea
 queries search only data covered by an active index group. Uncovered files are not searched, so
 partial coverage can make results incomplete.
 
-- BTree, Bitmap, and Multivalue scans always read uncovered files through the ordinary data path.
+- BTree, Bitmap, Multivalue, and FM scans always read uncovered files through the ordinary data path.
   Partial coverage affects their acceleration, not result completeness. The original predicate and
   deletion vectors are applied after index pruning.
 - Vector search in `full` or `detail` mode evaluates files without an active ANN group exactly and
@@ -318,12 +342,24 @@ persistent archives and ignores uncovered files; `full` and `detail` are rejecte
 merge-aware logical-row fallback is not implemented. Consequently, newly appended Level-0 rows
 become full-text searchable only after compaction publishes an eligible data file and archive.
 
-## BTree, Bitmap, and Multivalue Queries
+## BTree, Bitmap, Multivalue, and FM Queries
 
-BTree, Bitmap, and Multivalue indexes are applied automatically to snapshot-scoped Core batch
+BTree, Bitmap, Multivalue, and FM indexes are applied automatically to snapshot-scoped Core batch
 scans after a supported predicate reaches Paimon. BTree and Bitmap can accelerate equality, `IN`,
 null checks, comparisons and ranges, complement predicates, and supported string predicates.
 Multivalue accelerates `ARRAY_CONTAINS`, `ARRAYS_OVERLAP`, and `ARRAY_CONTAINS_ALL` predicates.
+FM accelerates exact `CONTAINS` predicates. For example, Spark converts the following literal
+substring filter to Paimon's `CONTAINS` predicate:
+
+```sql
+SELECT id, raw_text
+FROM items
+WHERE raw_text LIKE '%needle%';
+```
+
+The FM result is exact for indexed files. Paimon reads uncovered files normally and reapplies the
+original predicate and deletion vectors, so partial index coverage affects performance rather than
+result completeness.
 
 The Java/Core predicate can be constructed directly:
 
@@ -558,7 +594,7 @@ create a table with the desired definition and migrate the data.
 
 ## Limitations
 
-- BTree, Bitmap, and Multivalue definitions are single-column indexes.
+- BTree, Bitmap, Multivalue, and FM definitions are single-column indexes.
 - Multivalue supports arrays of scalar element types accepted by the global-index key serializer.
   Null elements are not indexed. Higher array and element cardinality increases the expanded-sort
   I/O and index-file size.

docs/sidebars.js

@@ -130,6 +130,7 @@ const sidebars = {
           "multimodal-table/global-index/btree",
           "multimodal-table/global-index/bitmap",
           "multimodal-table/global-index/multivalue",
+          "multimodal-table/global-index/fm",
           "multimodal-table/global-index/vector",
           "multimodal-table/global-index/full-text",
           "multimodal-table/global-index/hybrid-search"