Skip to content

qmark queries reuse cached results that ran with different parameters #941

Description

@laughingman7743

Problem

With the qmark paramstyle, client-side result caching (cache_size / cache_expiration_time) can return the result of an earlier query that ran with different parameters.

With qmark, _prepare_query() keeps the SQL text unchanged and sends the parameters separately as ExecutionParameters (pyathena/common.py:1188-1191).
_find_previous_query_id() then matches earlier executions only by query text, database, and catalog (pyathena/common.py:1118-1128).
It never compares the earlier execution's ExecutionParameters, even though AthenaQueryExecution.execution_parameters exposes them (pyathena/model.py:213).
So SELECT * FROM t WHERE id = ? with ["2"] can reuse the execution that ran with ["1"].

With pyformat, the parameters are substituted into the query text before the lookup, so this does not happen.

Expected: a cached execution is reused only if its ExecutionParameters equal the current ones.

Reproduction

Reproduced against Athena on master 9b74767 (the tag makes the query text unique to the run):

import os, uuid
from pyathena import connect

cursor = connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2").cursor()
sql = f"SELECT ? AS v, '{uuid.uuid4().hex[:8]}' AS run"
cursor.execute(sql, ["'1'"], paramstyle="qmark")
print(cursor.query_id, cursor.fetchall())
cursor.execute(sql, ["'2'"], paramstyle="qmark", cache_size=50)
print(cursor.query_id, cursor.fetchall())

Output:

1e139c07-24ff-45cc-bf71-661d46918876 [('1', 'b8778e57')]
1e139c07-24ff-45cc-bf71-661d46918876 [('1', 'b8778e57')]

The second call, with parameter '2', returns the first execution and its '1' result.
Found during the review of #940 (#927).

Environment

  • PyAthena master 9b74767 (qmark support since v3.12.0), any cursor that uses cache_size / cache_expiration_time with paramstyle="qmark".

Proposed fix (optional)

Pass the execution parameters to _find_previous_query_id() and require execution.execution_parameters == (execution_parameters or []) in the match.
An offline unit test can feed _find_previous_query_id executions with different parameters; one qmark integration case with cache_size covers the real path.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions