Repository navigation
Failed event publications that are skipped or keep failing block resubmission of all others #1906
Description
FailedEventPublications.resubmit(…) reads at most min(batchSize, maxInFlight - resubmitted) failed publications, ordered by publication date, and only then applies ResubmissionOptions.getFilter(). So the read window always holds the oldest failed publications:
- a publication that fails on every attempt is read again on every run and stays at the head of the queue for good,
- a publication the filter skips still takes a slot in the window. We use the filter for a retry backoff (last attempt plus a pause growing with
getCompletionAttempts()), so recently retried publications are skipped until their pause is over.
Once batchSize of these pile up, no other failed publication gets resubmitted: every run reads the same rows, skips or fails them again, and stops. A larger batch size only moves the threshold. We ran into this while adding the backoff filter on top of a scheduled resubmit(…).
Reading the least recently attempted publications first fixes it: order by LAST_RESUBMISSION_DATE, falling back to the publication date for publications that were never resubmitted. A publication that has just been resubmitted and failed again goes to the end of the queue, so every failed publication gets its turn, and a skipped one only holds its slot until it's due. The JDBC repository already initializes LAST_RESUBMISSION_DATE with the publication date, JPA, MongoDB and Neo4j leave it null until the first resubmission, hence the fallback.
Seen on 2.1.1, main has the same queries. PR follows.
Activity
odrotbohm commentedon Oct 1, 2026
I am not quite sure that we can reorder the resubmissions as you suggest. Let's say the publications target the same entity and the events can only be reacted to in order (e.g., an “order cancelled” can only be reacted to after an “order placed” has successfully been processed. In this case, it doesn't make sense to proceed with subsequent events if the processing of the previous events has failed.
For 2.2, we're introducing an additional “abandoned” state, which event publications will enter after a configurable amount of resubmissions or programmatic selection. Would that help mitigate the problem you face? Furthermore, couldn't you already skip offending publications by setting up a
ResubmissionOptionsinstance accordingly? I wonder if we could add a flag to opt-in to the lookup algorithm you prefer onResubmissionOptionsto cover scenarios that can be more relaxed on the overall ordering.char-yb commentedon Oct 6, 2026
I hit the same problem on 2.1.1 from a slightly different direction, so here's a data point.
On 2.1.1 we tried to cap retries with
withFilter(it -> it.getCompletionAttempts() < 60). It doesn't work, for the reason described above: the filter runs afterfindFailedPublications(…)has already applied the limit (min(batchSize, maxInFlight - resubmitted), oldest first). Once 500 publications (our batch size) have hit the cap, they fill the window on every run and nothing behind them gets resubmitted, even after the broker is back. With a one-minute schedule that's a broker outage of about an hour with more than 500 failed publications. So skipping throughResubmissionOptionsmade things worse for us, and we ended up dropping the cap and relying on idempotent consumers.I had a look at the abandonment support on
main, and for that case it does solve the problem: the policy is applied when the publication fails, andfindFailedPublications(…)only matchesFAILED, so abandoned publications never take a slot in the window. It doesn't help with the backoff filter from the issue description, though. A publication that is skipped but not abandoned still occupies its slot, so that part would still need the opt-in ordering.One thing I'm not sure is covered: failures that never reach a listener invocation don't seem to go through the
AbandonPolicy.TransactionalEventListeners.invoke(…)evaluatespublication.getApplicationEvent()before callingprocessEvent(…), so a payload that can't be deserialized throws into the catch block ofprocessPublications(…)(the same spot as #1904) and the publication staysRESUBMITTED. If the staleness monitor flips it back later, that goes throughEventPublicationRepository.markFailed(…)directly, so the policy isn't consulted there either. The reference docs list "a payload that can never be deserialized" as a typical case for abandoning, so unless I'm missing something, that case can't actually end up abandoned right now. I can open a separate issue for it if you prefer.On ordering, I agree it matters for events targeting the same entity. Two things I noticed, though. The blocking is global today, so a publication for one aggregate holds back retries for unrelated ones. And the reference docs already point out that later events might overtake earlier ones during resubmission unless
serialize-externalizationis enabled, so for externalized events strict ordering isn't guaranteed anyway. An opt-in flag onResubmissionOptionssounds like a reasonable compromise to me.AdzerKI commentedon Oct 9, 2026
Sorry for the late reply. The abandoned state covers publications that keep failing: they leave
FAILEDand stop taking slots in the window. It does not cover the backoff filter: a publication the filter skips is stillFAILEDand is still read into the window. Skipping offending publications throughResubmissionOptionshits the same limit, because the filter only sees the rows that were already read (char-yb's cap above is exactly that case).An opt-in on
ResubmissionOptionsworks for us. I'll rework #1907 that way: the current order stays the default, and an option along the lines ofwithLeastRecentlyAttemptedFirst()makes the repositories order by the last resubmission date, falling back to the publication date.spring-projectsGH-1906 - Allow resubmitting least recently attempted …