Problem
With the qmark paramstyle, client-side result caching (cache_size / cache_expiration_time) can return the result of an earlier query that ran with different parameters.
With qmark, _prepare_query() keeps the SQL text unchanged and sends the parameters separately as ExecutionParameters (pyathena/common.py:1188-1191).
_find_previous_query_id() then matches earlier executions only by query text, database, and catalog (pyathena/common.py:1118-1128).
It never compares the earlier execution's ExecutionParameters, even though AthenaQueryExecution.execution_parameters exposes them (pyathena/model.py:213).
So SELECT * FROM t WHERE id = ? with ["2"] can reuse the execution that ran with ["1"].
With pyformat, the parameters are substituted into the query text before the lookup, so this does not happen.
Expected: a cached execution is reused only if its ExecutionParameters equal the current ones.
Reproduction
Reproduced against Athena on master 9b74767 (the tag makes the query text unique to the run):
import os, uuid
from pyathena import connect
cursor = connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/", region_name="us-west-2").cursor()
sql = f"SELECT ? AS v, '{uuid.uuid4().hex[:8]}' AS run"
cursor.execute(sql, ["'1'"], paramstyle="qmark")
print(cursor.query_id, cursor.fetchall())
cursor.execute(sql, ["'2'"], paramstyle="qmark", cache_size=50)
print(cursor.query_id, cursor.fetchall())
Output:
1e139c07-24ff-45cc-bf71-661d46918876 [('1', 'b8778e57')]
1e139c07-24ff-45cc-bf71-661d46918876 [('1', 'b8778e57')]
The second call, with parameter '2', returns the first execution and its '1' result.
Found during the review of #940 (#927).
Environment
- PyAthena master 9b74767 (qmark support since v3.12.0), any cursor that uses
cache_size / cache_expiration_time with paramstyle="qmark".
Proposed fix (optional)
Pass the execution parameters to _find_previous_query_id() and require execution.execution_parameters == (execution_parameters or []) in the match.
An offline unit test can feed _find_previous_query_id executions with different parameters; one qmark integration case with cache_size covers the real path.
Problem
With the
qmarkparamstyle, client-side result caching (cache_size/cache_expiration_time) can return the result of an earlier query that ran with different parameters.With
qmark,_prepare_query()keeps the SQL text unchanged and sends the parameters separately asExecutionParameters(pyathena/common.py:1188-1191)._find_previous_query_id()then matches earlier executions only by query text, database, and catalog (pyathena/common.py:1118-1128).It never compares the earlier execution's
ExecutionParameters, even thoughAthenaQueryExecution.execution_parametersexposes them (pyathena/model.py:213).So
SELECT * FROM t WHERE id = ?with["2"]can reuse the execution that ran with["1"].With
pyformat, the parameters are substituted into the query text before the lookup, so this does not happen.Expected: a cached execution is reused only if its
ExecutionParametersequal the current ones.Reproduction
Reproduced against Athena on master 9b74767 (the tag makes the query text unique to the run):
Output:
The second call, with parameter
'2', returns the first execution and its'1'result.Found during the review of #940 (#927).
Environment
cache_size/cache_expiration_timewithparamstyle="qmark".Proposed fix (optional)
Pass the execution parameters to
_find_previous_query_id()and requireexecution.execution_parameters == (execution_parameters or [])in the match.An offline unit test can feed
_find_previous_query_idexecutions with different parameters; one qmark integration case withcache_sizecovers the real path.