Skip to content

Cassandra: use one consistency policy (LOCAL_QUORUM / LOCAL_SERIAL) for every statement #770

Open

Description

@jcshepherd

Why

The Cassandra backend has no consistency policy. Statements default to the
driver's ONE, and individual call sites have been upgraded to
LOCAL_QUORUM one at a time, as bugs turned up. With RF 3, a ONE read can
miss an acknowledged write, and the code often acts on "not found" as a fact
(see the comments at cassandra_util.rs:116, put_get_item.rs:606,
data/index.rs:50, and PR #508). Upgrading reads alone does not help: a
quorum read overlaps only writes that were also made at quorum. A single
rule — LOCAL_QUORUM for every statement, LOCAL_SERIAL for every
conditional one — is correct for DynamoDB's strong reads, conditional writes
and transactions. It costs nothing on single-node (RF 1) deployments and
removes the per-call-site decision that keeps going wrong. Separately,
init creates every keyspace, including the catalog, with RF 1, so a
multi-node cluster set up by the docs keeps all control-plane state on one
node.

Changes

  1. In crates/storage-cassandra/src/cassandra_util.rs, make every helper set
    its level explicitly:
    • query_rows, query_optional, execute: LOCAL_QUORUM
      (query_rows_quorum / execute_quorum become redundant);
    • query_lwt: unchanged (LOCAL_SERIAL, commit at LOCAL_QUORUM);
    • add a helper that returns a BatchQueryBuilder at LOCAL_QUORUM.
  2. Move every direct session.query_with_values, query_with_params,
    .query( and BatchQueryBuilder::new() call in non-test source onto
    those helpers. Every IF ... statement goes through query_lwt.
  3. Add a test that fails if non-test source in the crate makes those calls
    outside cassandra_util.rs.
  4. Do not add a consistency setting. Serve ConsistentRead=false at
    LOCAL_QUORUM too; a strong read is a valid eventually-consistent read.
  5. init: create keyspaces with the configured replication_factor and
    datacenter, not 1. serve: log a warning when a keyspace's RF differs
    from the config. Docs: raising RF needs nodetool repair before serving.
  6. CI: add one job that runs the Cassandra integration tests on a 3-node,
    RF 3 cluster. On RF 1, every consistency level behaves the same, so
    today's CI cannot catch these bugs.

Items 1-3 are one PR. Items 5 and 6 can be separate.

Activity

  1. self-assigned this
    on Nov 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

backend:cassandraRelated to the Cassandra backend specifically

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions