Skip to content

DefaultDataBag.iterator: IndexOutOfBoundsException when the bag holds exactly the spill threshold #4409

Open

Description

@MatthiasVanDE

Split off from #4407 (found with the help of an AI coding assistant).

Summary: With ARQ.spillToDiskThreshold set to 100, an INSERT ... WHERE that has exactly 100 solutions throws IndexOutOfBoundsException: Index 0 out of bounds for length 0 from DefaultDataBag.iterator. With 99 or 101 solutions it works. Only users who enabled spilling are affected, since the default policy never spills.

Cause: DefaultDataBag.add only spills when an item arrives after the threshold is reached. iterator(), however, uses policy.isThresholdExceeded() to choose between memory and the spill file, and that is already true after the 100th item. So nothing was spilled, but iterator() reads getSpillFiles().get(0) from an empty list.

Fix: use the existing spilled flag in iterator() instead of policy.isThresholdExceeded(). With this change on 6.2.0, the test below passes (99 / 100 / 101).

Seen on 5.6.0, 6.2.0 and 6.3.0-SNAPSHOT.

package org.apache.jena.sparql.modify;

import static org.junit.jupiter.api.Assertions.assertEquals;

import org.junit.jupiter.api.Test;

import org.apache.jena.graph.Node;
import org.apache.jena.graph.NodeFactory;
import org.apache.jena.query.ARQ;
import org.apache.jena.sparql.core.DatasetGraph;
import org.apache.jena.sparql.core.DatasetGraphFactory;
import org.apache.jena.system.Txn;
import org.apache.jena.update.UpdateAction;

/** INSERT..WHERE with exactly ARQ.spillToDiskThreshold solutions. */
public class TestUpdateSpillThreshold {
    private static final Node g = NodeFactory.createURI("http://example/g");
    private static final Node p = NodeFactory.createURI("http://example/p");

    private static void test(int n) {
        DatasetGraph dsg = DatasetGraphFactory.createTxnMem();
        dsg.getContext().set(ARQ.spillToDiskThreshold, 100L);
        Txn.executeWrite(dsg, () -> {
            for ( int i = 0 ; i < n ; i++ )
                dsg.add(g, NodeFactory.createURI("http://example/s/" + i), p, NodeFactory.createURI("http://example/o/" + i));
        });
        // In 6.2.0: IndexOutOfBoundsException from DefaultDataBag.iterator when n == threshold.
        Txn.executeWrite(dsg, () -> UpdateAction.parseExecute("INSERT { ?s ?p ?o } WHERE { GRAPH <http://example/g> { ?s ?p ?o } }", dsg));
        assertEquals(n, Txn.calculateRead(dsg, () -> dsg.getDefaultGraph().size()));
    }

    @Test public void spill_below_threshold()   { test(99); }
    @Test public void spill_at_threshold()      { test(100); }
    @Test public void spill_above_threshold()   { test(101); }
}

Activity

  1. Zhou-Kan commented on Nov 15, 2026

    @Zhou-Kan

    Could I still work on this issue as my first ticket here? @MatthiasVanDE

  2. MatthiasVanDE commented on Nov 15, 2026

    @MatthiasVanDE
    Author

    Sure, go ahead! The cause and a one-line fix are in the issue (use the spilled flag in DefaultDataBag.iterator()), and the test above should pass with it. The maintainers decide on review and merge, of course.

  3. Zhou-Kan commented on Nov 16, 2026

    @Zhou-Kan

    Hi @MatthiasVanDE I added a PR for it #4420

  4. MatthiasVanDE commented on Nov 16, 2026

    @MatthiasVanDE
    Author

    Thanks @Zhou-Kan ! The fix matches what I tested on 6.2.0, and the test covers the exact-threshold case. LGTM from my side.

  5. deepikavalluru commented on Nov 17, 2026

    @deepikavalluru

    Hi!
    I'm interested in contributing to Apache Jena and would like to work on this issue. I've been working with Java and am currently developing my backend engineering skills. This issue looks like a good opportunity to understand Jena's DefaultDataBag implementation, investigate the threshold-handling behaviour, and add appropriate regression tests. Is this issue still open for contributions? I'd be happy to investigate and submit a PR.

    Thanks!

  6. MatthiasVanDE commented on Nov 17, 2026

    @MatthiasVanDE
    Author

    Thanks for your interest! This one is already taken: @Zhou-Kan opened #4420 with a fix. But #4408 is still open and similar in size: the cause, a one-line fix and a test are in the issue. Feel free to pick that one up; the maintainers decide on review and merge.

  7. Zhou-Kan commented on Nov 17, 2026

    @Zhou-Kan

    @afs I think I've fixed this issue, but I'm not sure if it's ready to be merged. PR is #4420

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions