[Bug] [Hive] IndexOutOfBoundsException when converting an unavailable dynamic BETWEEN predicate

[kane0409]

Search before asking

• I searched in the issues and found nothing similar.

Paimon version

Paimon 1.4, using the Hive 3.1 connector.

Compute Engine

• Hive 3.1.2
• Tez
• Java 8

Minimal reproduce step

A Hive SQL query joining multiple Paimon tables occasionally fails while Tez is initializing input splits.

The failure happens before any data is read:

ROOT_INPUT_INIT_FAILURE
Vertex Input: nws_fleet_van_proof_parcel_detail_p initializer failed

java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
    at java.util.ArrayList.get(ArrayList.java:435)
    at org.apache.paimon.hive.SearchArgumentToPredicateConverter.convertLeaf(
        SearchArgumentToPredicateConverter.java:163)
    at org.apache.paimon.hive.SearchArgumentToPredicateConverter.convertTree(
        SearchArgumentToPredicateConverter.java:124)
    at org.apache.paimon.hive.SearchArgumentToPredicateConverter.convert(
        SearchArgumentToPredicateConverter.java:91)
    at org.apache.paimon.hive.utils.HiveUtils.createPredicate(HiveUtils.java:96)
    at org.apache.paimon.hive.utils.HiveSplitGenerator.generateSplits(
        HiveSplitGenerator.java:68)
    at org.apache.paimon.hive.mapred.PaimonInputFormat.getSplits(
        PaimonInputFormat.java:45)

The failing code is in the BETWEEN branch:

List<Object> literalList = leaf.getLiteralList();
return builder.between(
        idx,
        toLiteral(columnType, literalList.get(0)),
        toLiteral(columnType, literalList.get(1)));

Hive Tez dynamic semijoin reduction can inject a runtime predicate similar to:

proof_id BETWEEN DynamicValue(min) AND DynamicValue(max)

When Paimon generates splits, these dynamic values may not be available yet. In this case, PredicateLeaf.getLiteralList() returns an empty list, but Paimon accesses elements 0 and 1 without checking the list size.
The issue is intermittent because whether Hive generates the runtime filter depends on the execution plan, table statistics, data size, and runtime timing.

Minimal reproduce steps

1. Enable the Tez dynamic semijoin optimization:

SET hive.execution.engine=tez;
SET hive.tez.dynamic.semijoin.reduction=true;

2. Run a join where Hive pushes a dynamic min/max predicate into a Paimon table scan:

SELECT COUNT(*)
FROM paimon_db.proof_table p
JOIN paimon_db.proof_detail_table d
  ON p.id = d.proof_id
WHERE p.p_date >= date_sub(current_date, 7)
  AND d.p_date >= date_sub(current_date, 8)
  AND d.relation_category IN (1, 3);

3. When the dynamic values are unavailable during split initialization, the Paimon input initializer fails with IndexOutOfBoundsException.
   Disabling dynamic semijoin reduction prevents the failure:

SET hive.tez.dynamic.semijoin.reduction=false;

What doesn't meet your expectations?

Expected behavior

An unavailable or incomplete dynamic predicate should not cause the query to fail.
Because predicate pushdown is a best-effort optimization, Paimon should skip this predicate and allow Hive to apply the normal join/filter logic.
For example:

List<Object> literalList = leaf.getLiteralList();
if (literalList == null || literalList.size() < 2) {
    throw new UnsupportedOperationException(
            "BETWEEN literals are not available");
}

convert() already catches UnsupportedOperationException, so the unavailable predicate can be ignored safely.

Actual behavior

Paimon directly calls literalList.get(0) and literalList.get(1), causing the entire Tez DAG to fail during root input initialization.

Anything else?

The Hive project documents that dynamic BETWEEN values may be unavailable while predicate pushdown is being evaluated:

• https://issues.apache.org/jira/browse/HIVE-18827
• https://issues.apache.org/jira/browse/HIVE-23295
  Paimon source:
• https://github.com/apache/paimon/blob/release-1.4/paimon-hive/paimon-hive-connector-common/src/main/java/org/apache/paimon/hive/SearchArgumentToPredicateConverter.java#L159-L164
  The current workaround is:

SET hive.tez.dynamic.semijoin.reduction=false;

However, this disables a useful Hive runtime optimization and may increase the amount of data read and shuffled.

Are you willing to submit a PR?

• I'm willing to submit a PR!