Invalidate object cache after raw SQL delete in idx_clean_transients() - #560
Open
pmallek wants to merge 1 commit into
Open
Invalidate object cache after raw SQL delete in idx_clean_transients()#560pmallek wants to merge 1 commit into
pmallek wants to merge 1 commit into
Conversation
idx_clean_transients() removes idx_*_cache options with a direct DELETE query, bypassing delete_option(). On installs backed by a persistent object cache (Memcached, Redis) the cached copies survive the delete, so get_option() keeps returning a value for a row that no longer exists. update_option() only falls back to add_option() when get_option() returns false. Otherwise it issues an UPDATE that matches zero rows, hits "if ( ! $result ) return false;" and never reaches wp_cache_set(). The option is then permanently unwritable and stays expired forever. In practice this traps idx_clients_accounttype_cache, which engage_account_type() consults on every bootstrap, so each WordPress request re-queries /clients/accounttype until the account exhausts its hourly API limit and every IDX call returns 412. Collect the matching option names before the DELETE and drop their object-cache entries afterwards, along with the notoptions and alloptions buckets. Refs idxbroker#57 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Idx_Api::idx_clean_transients()deletesidx_*_cacheoptions with a raw SQLDELETE, bypassingdelete_option(). On installs backed by a persistent objectcache (Memcached, Redis) the cached copies survive the delete. The affected
options then become permanently unwritable, which on at least one production
site turned into a sustained 412 loop that took all IDX content down.
idx_clean_transients()runs on plugin update (Initiate_Plugin::update_triggered()),so any site on such a host can enter this state after upgrading.
Root cause
idx/idx-api.php:687Once the row is gone but the object-cache entry is not:
get_option()returns the stale cached value, so it is notfalse;update_option()only falls back toadd_option()whenget_option()returns
false(wp-includes/option.php:928);UPDATEthat matches zero rows, hitsif ( ! $result ) return false;and never reacheswp_cache_set().The value can never be refreshed or repaired. It stays expired forever, and
every request re-queries the API.
Impact
The trapped entry that matters is
idx_clients_accounttype_cache.engage_account_type()has 14 call sites, 8 of them on the normal bootstrappath (
initiate-plugin.phpx2,register-impress-shortcodes.phpx2,register-blocks.php,create-impress-widgets.php,register-shortcode-for-ui.php,idx-pages.php), and it has no staticmemoization. With the cache trapped, one WordPress request can mean several
/clients/accounttypecalls.On a production site on Pressable (IDX Broker Engage, 500 calls/hour) we
measured
hourly-access-key-usage: 968roughly ten minutes into the hour —about 16 requests per minute, every one of them returning 412.
Observed timeline on that site, from its own options:
idx_api_cache_repaired_at = 1787946960idx_clients_accounttype_cachewritten,expiration = 1788033361Why this looks intermittent
We found four options present in Memcached but absent from
wp_options, all ofthem small. Every large cache entry had recovered on its own —
featured(410 KB),
leads(149 KB),postalcodes(48 KB),cities(38 KB). Memcachedevicted those under LRU pressure, so
get_option()returnedfalseandadd_option()rebuilt the row correctly.Only small, cache-resident entries stay broken. That makes the bug
host-dependent and load-dependent, which may be why it has been hard to pin down.
Reproduction
idx_clients_accounttype_cache.idx_clean_transients()(a plugin update does it).wp_options,get_option()still returns theold value from the object cache, and
update_option()on that key returnsfalseforever.The fix
Collect the matching option names before the
DELETE, thenwp_cache_delete()each one plus the
notoptionsandalloptionsbuckets. Both branches of thefunction are covered. Passes
php -l.Recovering an already-affected site
The fix prevents new occurrences but does not heal existing ones, since the
trapped entries are already orphaned. Dropping the orphaned keys from the object
cache lets
add_option()recreate the rows. We did this on the affected site andthe call rate dropped to zero immediately, with no downtime.
Related
"Stop http requests once hourly API limit is reached". The 412 cooldown in
idx_api()is real, but it cannot help while its own state option is trapped.idx_create_idx_pages/idx_delete_idx_pagesrun everythree minutes by default (
idx-pages.php:19), and each run bootstraps theplugin, adding ~40 more
accounttypechecks per hour on top of organic traffic.Two related observations, not in this PR
Happy to open separate PRs for either:
engage_account_type()would benefit from static memoization — even with ahealthy cache it does 8+
get_option()calls per request.'%idx_%_cache'pattern has a leading wildcard and an unescapedunderscore (a single-character wildcard in
LIKE), so it can match anddelete options belonging to other plugins.
$wpdb->esc_like()with ananchored prefix would tighten it.