Competency Mastery Concurrency ADR#657
Conversation
|
Thanks for the pull request, @jesperhodge! This repository is currently maintained by Once you've gone through the following steps feel free to tag them in a comment and let them know that your changes are ready for engineering review. 🔘 Get product approvalIf you haven't already, check this list to see if your contribution needs to go through the product review process.
🔘 Provide contextTo help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:
🔘 Get a green buildIf one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green. 🔘 Update the status of your PRYour PR is currently marked as a draft. After completing the steps above, update its status by clicking "Ready for Review", or removing "WIP" from the title, as appropriate. Where can I find more information?If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources: When can I expect my changes to be merged?Our goal is to get community contributions seen and reviewed as efficiently as possible. However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:
💡 As a result it may take up to several weeks or months to complete a review and merge your PR. |
openedx-webhooks
added
the
open-source-contribution
Rejected-alternative items 1 and 2 nested Pros/Cons bullet lists directly under a continuation paragraph at a different indent with no blank line separator, and item 3's closing paragraph was indented to neither the list body nor its sub-bullets. docutils flagged these as errors under -W, failing the readthedocs build.
jesperhodge
changed the title
jesperhodge
changed the title
mgwozdz-unicon
left a comment
mgwozdz-unicon
left a comment
There was a problem hiding this comment.
Following up on feedback I'd shared earlier outside this PR favoring the batch-lock approach over keyed partitioning, given the Kafka/Redis dependency and the accuracy-over-latency tradeoff for this use case: this revision reflects that direction, and the Kafka/Redis coupling concern is resolved.
I'm wondering if we should also consider a per-learner lock instead of the single deployment-wide lock. A DB advisory lock keyed on a hash of user_id would give the same same-learner serialization the chosen approach relies on, but would let different learners' batches run in parallel, which the current design gives up entirely. It also sidesteps everything Alternative 1 was rejected for: no event-transport dependency, no partition-key contract with openedx-platform, no reconciliation backstop.
Rejected Alternative 2's argument doesn't quite cover this: it treats "per-learner isolation" as equivalent to "keyed partitioning" ("Making parallel evaluation correct requires... per-learner isolation, i.e. the keyed-partitioning alternative above"), but a per-learner DB lock gets you per-learner isolation without touching the transport at all. If there's a reason this doesn't hold up here (lock-management overhead across many concurrent per-learner locks, contention patterns, something else), that reasoning is worth adding to the doc.
Requesting changes to get this addressed, either by adding the analysis to Rejected Alternatives or by folding it into the Decision.
| - **Same-learner correctness.** Grade-change events arrive asynchronously and can be delivered out | ||
| of order and processed on more than one worker. Because writes are append-only, two evaluations | ||
| for the same learner that overlap can each read a stale snapshot of the sibling leaf statuses and | ||
| each append a derived roll-up computed from an incomplete picture (a write-skew). Leaf rows are | ||
| always correct, since each leaf is a pure function of its own grade; only the derived | ||
| group/competency rows can be left wrong. Nothing crashes and no constraint is violated, but a | ||
| learner's stored competency status can be silently incorrect. |
There was a problem hiding this comment.
Maybe this is a silly question, but can't we mitigate this by making sure to commit the transaction of the leaf statuses, and then re-read the latest commited state of the leaves before doing the roll-up?
There was a problem hiding this comment.
Another thing we can do (if it's necessary) is to arrange it so that the roll-ups are a log that have an incrementing version number, and create a composite unique index that would cause a conflict that would reject concurrent writes. The losing write would have to catch the error, increment the version number again, and re-read the database state.
5 participants
Claude / AI assisted.
Summary
ADR for
openedx_learning: how learner competency mastery is recordedcorrectly under concurrent, out-of-order grade-change events without
paying a per-event serialization cost at scale.
Decision: consume grade-change events in windowed micro-batches and
serialize each batch's read-evaluate-write behind a single
deployment-wide lock, so no two workers can evaluate the same learner
concurrently.