Competency Mastery Concurrency ADR#657
Conversation
|
Thanks for the pull request, @jesperhodge! This repository is currently maintained by Once you've gone through the following steps feel free to tag them in a comment and let them know that your changes are ready for engineering review. 🔘 Get product approvalIf you haven't already, check this list to see if your contribution needs to go through the product review process.
🔘 Provide contextTo help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:
🔘 Get a green buildIf one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green. DetailsWhere can I find more information?If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources: When can I expect my changes to be merged?Our goal is to get community contributions seen and reviewed as efficiently as possible. However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:
💡 As a result it may take up to several weeks or months to complete a review and merge your PR. |
openedx-webhooks
added
the
open-source-contribution
Rejected-alternative items 1 and 2 nested Pros/Cons bullet lists directly under a continuation paragraph at a different indent with no blank line separator, and item 3's closing paragraph was indented to neither the list body nor its sub-bullets. docutils flagged these as errors under -W, failing the readthedocs build.
jesperhodge
changed the title
jesperhodge
changed the title
mgwozdz-unicon
left a comment
mgwozdz-unicon
left a comment
There was a problem hiding this comment.
Following up on feedback I'd shared earlier outside this PR favoring the batch-lock approach over keyed partitioning, given the Kafka/Redis dependency and the accuracy-over-latency tradeoff for this use case: this revision reflects that direction, and the Kafka/Redis coupling concern is resolved.
I'm wondering if we should also consider a per-learner lock instead of the single deployment-wide lock. A DB advisory lock keyed on a hash of user_id would give the same same-learner serialization the chosen approach relies on, but would let different learners' batches run in parallel, which the current design gives up entirely. It also sidesteps everything Alternative 1 was rejected for: no event-transport dependency, no partition-key contract with openedx-platform, no reconciliation backstop.
Rejected Alternative 2's argument doesn't quite cover this: it treats "per-learner isolation" as equivalent to "keyed partitioning" ("Making parallel evaluation correct requires... per-learner isolation, i.e. the keyed-partitioning alternative above"), but a per-learner DB lock gets you per-learner isolation without touching the transport at all. If there's a reason this doesn't hold up here (lock-management overhead across many concurrent per-learner locks, contention patterns, something else), that reasoning is worth adding to the doc.
Requesting changes to get this addressed, either by adding the analysis to Rejected Alternatives or by folding it into the Decision.
| - **Same-learner correctness.** Grade-change events arrive asynchronously and can be delivered out | ||
| of order and processed on more than one worker. Because writes are append-only, two evaluations | ||
| for the same learner that overlap can each read a stale snapshot of the sibling leaf statuses and | ||
| each append a derived roll-up computed from an incomplete picture (a write-skew). Leaf rows are | ||
| always correct, since each leaf is a pure function of its own grade; only the derived | ||
| group/competency rows can be left wrong. Nothing crashes and no constraint is violated, but a | ||
| learner's stored competency status can be silently incorrect. |
There was a problem hiding this comment.
Maybe this is a silly question, but can't we mitigate this by making sure to commit the transaction of the leaf statuses, and then re-read the latest commited state of the leaves before doing the roll-up?
There was a problem hiding this comment.
Another thing we can do (if it's necessary) is to arrange it so that the roll-ups are a log that have an incrementing version number, and create a composite unique index that would cause a conflict that would reject concurrent writes. The losing write would have to catch the error, increment the version number again, and re-read the database state.
There was a problem hiding this comment.
I think this question is not relevant anymore with the current version, right? Since now we only process on one worker at a time.
@mgwozdz-unicon I'll improve the description in the ADR. Here's why this solution is not good: It fights the batching that the performance depends on. The chosen design's throughput comes from batching across learners: one bulk read of current statuses, evaluate the whole batch in memory, one bulk write — a few round-trips regardless of how many learners/events are in the batch. A per-learner lock is naturally per-learner (in the earlier design, per-event): acquire lock → read that learner → evaluate → write → release, one learner at a time. That reintroduces per-learner (or per-event) transaction/commit overhead plus lock acquire/release churn, which is exactly the overhead batching was collapsing. Under bursty grading across many learners, that per-unit overhead dominates. Lock-lifecycle machinery would be multiplied across millions of keys. One deployment-wide lock has exactly one lifecycle to run: acquisition, timeout, stale-lock recovery on a crashed worker. A per-learner lock needs a per-learner lock table (or keyed advisory locks) and that same lifecycle replicated across potentially millions of learner keys, held and waited on concurrently, plus a connection+worker tied up per contended lock. |
| 2. ``competency_criteria_id``: logical foreign key to ``CompetencyCriterion.id`` (no database-level constraint) | ||
| 3. ``user_id``: logical foreign key to the learner's user id (no database-level constraint) | ||
| 4. ``status_id``: foreign key to ``CompetencyMasteryStatuses.id`` | ||
| 5. ``effective_source_timestamp``: the timestamp of the grade change this status was computed from, used for the out-of-order defense in :ref:`openedx-learning-adr-0004` |
There was a problem hiding this comment.
How is effective_source_timestamp different from created?
ormsbee
left a comment
ormsbee
left a comment
There was a problem hiding this comment.
It seems like this entire ADR's premise is that per-event processing is prohibitively expensive, and we must therefore batch event processing under various locking mechanisms. It's not clear to me why that is. The subsection grade update is already a somewhat expensive, async task. We could plausibly wrap any updates we're doing to competency mastery data in the same transaction that writes the subsection grade update, so that those things are always kept in sync.
Approximately much work are we expecting to happen in order to do the mastery recalculation?
5 participants
UPDATE: ADRs in this PR have been completely rewritten as of July 17, 2026.
Strong involvement from Claude (AI).
I haven't fully reviewed changes to diagrams and ADRs 2 and 3 yet, made with Claude, need my own review still.
Summary
These are some significant changes to the data model and approach. They will warrant a good amount of discussion.
A main concern is that the leaf node table and leaf node history table could possibly contain billions of rows, at the scale of the
PersistentSubsectionGradetable of edx-platform. The current solution accepts this, but is this acceptable for OpenEdx? If not, there is also a rejected alternative to not store leaf nodes and just compute them at read time, but that would make it harder to keep subsection statuses frozen so that later competency criteria changes don't change the status.ADR 4: How learner competency mastery is recorded
correctly under concurrent, out-of-order grade-change events without
paying a per-event serialization cost at scale.
ADR 5: How competency criteria statuses for learners are stored at scale, given the tables
can be massive (billions of rows).