Paper 1: official corrected-2009 audit + exploratory refit + fresh 2015 replication (author review)

.github/workflows/arxiv-verification.yml

@@ -1,4 +1,4 @@
-name: arXiv verification: real-data robustness
+name: "arXiv verification: real-data robustness"
 on:
   push:
     branches:

.github/workflows/confirmatory-2015-stage2.yml

@@ -0,0 +1,60 @@
+name: External cohort confirmation — second untouched ASSISTments 2015 cohort
+on:
+  push:
+    branches:
+      - research/kt-fresh-cohort-20261022
+    paths:
+      - CONFIRMATORY_STAGE2_PROTOCOL_2015.md
+      - scripts/confirmatory_2015_stage2.py
+      - .github/workflows/confirmatory-2015-stage2.yml
+  workflow_dispatch:
+permissions:
+  contents: write
+jobs:
+  second-cohort:
+    runs-on: ubuntu-latest
+    timeout-minutes: 120
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.11'
+      - name: Install frozen computational packages
+        run: |
+          python -m pip install -r requirements-repro.txt
+          python -m pip install 'gdown>=5.2,<6'
+      - name: Compile and inspect frozen code
+        run: python -m py_compile scripts/confirmatory_2015.py scripts/confirmatory_2015_stage2.py
+      - name: Run untouched population study
+        run: PYTHONPATH=src python scripts/confirmatory_2015_stage2.py
+      - name: Check full independent source and test outputs
+        run: |
+          python - <<'PY'
+          import json
+          from pathlib import Path
+          a=json.loads(Path('results/confirmatory_assist2015_stage2.json').read_text())
+          assert a['data']['sha256']=='75e1f46131d5897a388c82411bc3032279899cb1f1e66a9043354d41234407a0'
+          assert a['data']['stage2_intersection_stage1']==0
+          assert a['data']['stage1_excluded_learners']==5000
+          assert a['data']['stage2_sampled_learners']==5000
+          assert sum(a['protocol'][x] for x in ('train_learners','validation_learners','test_learners'))==5000
+          assert a['paired_brier']['n_learners']==750
+          assert a['paired_brier']['n_bootstrap']==2000
+          print('Recorded status:',a['status'],'directional replication:',a['directional_replication_confirmed'])
+          PY
+      - uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: assistments2015-stage2-validation
+          path: results/confirmatory_assist2015_stage2.json
+          if-no-files-found: warn
+      - name: Commit frozen results (not raw student records)
+        if: success()
+        run: |
+          git config user.name 'github-actions[bot]'
+          git config user.email '41898282+github-actions[bot]@users.noreply.github.com'
+          git add results/confirmatory_assist2015_stage2.json
+          if ! git diff --cached --quiet; then
+            git commit -m 'research: record untouched cohort confirmatory findings'
+            git push origin HEAD:research/kt-fresh-cohort-20261022
+          fi

.github/workflows/confirmatory-2015.yml

@@ -0,0 +1,62 @@
+name: External cohort confirmation — ASSISTments 2015
+on:
+  push:
+    branches:
+      - research/kt-fresh-cohort-20261022
+    paths:
+      - scripts/confirmatory_2015.py
+      - .github/workflows/confirmatory-2015.yml
+      - CONFIRMATORY_PROTOCOL_2015.md
+      - scripts/robustness_extension.py
+      - src/run_experiment.py
+  workflow_dispatch:
+permissions:
+  contents: write
+jobs:
+  external-cohort:
+    runs-on: ubuntu-latest
+    timeout-minutes: 120
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+      - name: Install verified environment and Google Drive downloader
+        run: |
+          python -m pip install -r requirements-repro.txt
+          python -m pip install 'gdown>=5.2,<6' pytest
+      - name: Compile frozen confirmatory analysis
+        run: |
+          python -m py_compile scripts/confirmatory_2015.py
+      - name: Run independent-cohort analysis
+        run: PYTHONPATH=src python scripts/confirmatory_2015.py
+      - name: Validate output schema and data isolation
+        run: |
+          python - <<'PY'
+          import json
+          from pathlib import Path
+          a = json.loads(Path('results/confirmatory_assist2015.json').read_text())
+          assert a['protocol']['sample_learners'] == 5000
+          assert sum(a['protocol'][k] for k in ('train_learners','validation_learners','test_learners')) == 5000
+          assert a['brier_comparison']['n_learners'] == a['protocol']['test_learners']
+          assert a['brier_comparison']['n_bootstrap'] == 2000
+          assert len(a['data']['source_sha256']) == 64
+          print('External-cohort result recorded. Confirmation:', a['directional_replication_confirmed'])
+          PY
+      - name: Upload results and logs even if run fails
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: knowledge-tracing-2015-confirmation
+          path: results/confirmatory_assist2015.json
+          if-no-files-found: warn
+      - name: Commit metrics to branch without changing previous paper results
+        if: success()
+        run: |
+          git config user.name "github-actions[bot]"
+          git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
+          git add results/confirmatory_assist2015.json
+          if ! git diff --cached --quiet; then
+            git commit -m "research: freeze ASSISTments 2015 external-cohort test outcome"
+            git push origin HEAD:research/kt-fresh-cohort-20261022
+          fi

.github/workflows/corrected-2009-reanalysis.yml

@@ -0,0 +1,55 @@
+name: "Exploratory corrected-source 2009 GRU versus enriched PFA"
+on:
+  push:
+    branches:
+      - research/kt-fresh-cohort-20261022
+    paths:
+      - scripts/corrected_2009_reanalysis.py
+      - .github/workflows/corrected-2009-reanalysis.yml
+      - CORRECTED_2009_REANALYSIS_PROTOCOL.md
+  workflow_dispatch:
+permissions:
+  contents: write
+jobs:
+  reanalysis:
+    runs-on: ubuntu-latest
+    timeout-minutes: 100
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.11'
+      - name: Install frozen numeric and raw-source packages
+        run: |
+          python -m pip install -r requirements-repro.txt
+          python -m pip install 'gdown>=5.2,<6'
+      - name: Inspect Python code
+        run: python -m py_compile scripts/audit_corrected_2009.py scripts/corrected_2009_reanalysis.py
+      - name: Analyze corrected official-source data without redistributing student records
+        run: PYTHONPATH=src python scripts/corrected_2009_reanalysis.py
+      - name: Check required source-audit provenance and output schema
+        run: |
+          python - <<'PY'
+          import json
+          r=json.load(open('results/corrected_2009_sensitivity.json'))
+          assert r['status'] in ('completed_exploratory_corrected_source','inconclusive_diagnostics','blocked_or_failed')
+          if r['status']=='completed_exploratory_corrected_source':
+            assert r['source']['corrected_sha256']=='162ef8d2d28bcbfea6591a282994062bd8d5eaa00636544292a0d268dca6e5da'
+            assert r['paired_brier']['n_bootstrap']==2000
+          print('Corrected-source research status:',r['status'])
+          PY
+      - uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: corrected-2009-aggregate-sensitivity
+          path: results/corrected_2009_sensitivity.json
+      - name: Archive aggregate outcome in research branch
+        if: success()
+        run: |
+          git config user.name 'github-actions[bot]'
+          git config user.email '41898282+github-actions[bot]@users.noreply.github.com'
+          git add results/corrected_2009_sensitivity.json
+          if ! git diff --cached --quiet; then
+            git commit -m 'research: record exploratory corrected-2009 source sensitivity'
+            git push origin HEAD:research/kt-fresh-cohort-20261022
+          fi

.github/workflows/official-2009-parity.yml

@@ -0,0 +1,52 @@
+name: "Audit corrected official ASSISTments 2009 against pinned third-party mirror"
+on:
+  push:
+    branches:
+      - research/kt-fresh-cohort-20261022
+    paths:
+      - scripts/audit_corrected_2009.py
+      - .github/workflows/official-2009-parity.yml
+  workflow_dispatch:
+permissions:
+  contents: write
+jobs:
+  source-parity:
+    runs-on: ubuntu-latest
+    timeout-minutes: 50
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.11'
+      - name: Install source-audit dependencies
+        run: |
+          python -m pip install 'pandas==2.3.3' 'pyarrow==23.0.1' 'huggingface-hub==1.33.0' 'gdown>=5.2,<6'
+      - name: Verify audit script syntax
+        run: python -m py_compile scripts/audit_corrected_2009.py
+      - name: Compare official corrected raw records against pinned mirror
+        run: python scripts/audit_corrected_2009.py
+      - name: Verify audit records uncertainty explicitly
+        run: |
+          python - <<'PY'
+          import json
+          from pathlib import Path
+          r=json.loads(Path('results/official_2009_mirror_parity_audit.json').read_text())
+          assert r['parity_status'] in ('blocked_not_established','exact_full_dataset_parity_disproved_by_necessary_conditions','aggregate_necessary_conditions_pass_but_event_parity_not_proven')
+          assert 'student_records' not in r
+          print('SOURCE AUDIT STATUS:',r['parity_status'])
+          PY
+      - uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: official-2009-mirror-parity-metadata-only
+          path: results/official_2009_mirror_parity_audit.json
+      - name: Preserve report in research branch
+        if: success()
+        run: |
+          git config user.name 'github-actions[bot]'
+          git config user.email '41898282+github-actions[bot]@users.noreply.github.com'
+          git add results/official_2009_mirror_parity_audit.json
+          if ! git diff --cached --quiet; then
+            git commit -m 'research: archive corrected 2009 versus mirror provenance audit'
+            git push origin HEAD:research/kt-fresh-cohort-20261022
+          fi

CONFIRMATORY_PROTOCOL_2015.md

@@ -0,0 +1,13 @@
+# Frozen external-cohort replication protocol (2026-10-22)
+
+**Research question.** On a previously unanalysed, different-year learner sample from ASSISTments 2015, does the pre-existing extended GRU pipeline achieve lower held-out pooled Brier loss than the pre-existing enriched PFA pipeline? The models are retrained on 2015 training learners. This tests cross-cohort *replication of the comparison*, not direct transport of the 2009 fitted model or causal effects of architecture.
+
+- Source: official ASSISTments 2015 Skill Builder Data, `2015_100_skill_builders_main_problems.csv`, Google Drive file ID `0B_hO8cnpcIMgUGZzRnh3bHJrSjQ` linked from https://sites.google.com/site/assistmentsdata/datasets/2015-assistments-skill-builder-data.
+- Require `user_id`, `log_id` (event order), `sequence_id` (100 skill-builder concepts), and binary `correct`. Require non-null, unambiguous records and unique learner-event pairs. Sort each learner chronologically using `log_id`. Never use later responses as model inputs for earlier outcomes.
+- Learner sampling: uniformly sample without replacement **5,000 distinct learners**, deterministically with seed `20261022` after sorting IDs lexicographically. No filtering based on outcomes or model performance, except invalid records verified before learner sampling; log exclusions.
+- Split: shuffle selected learners with seed `20261022`, then 70/15/15, disjoint between training/validation/test. A new dataset, independent of the 2009 train/validation/test learners. All model selection on training/validation; test inspected once.
+- Fixed models: pre-existing stronger PFA `stronger_pfa()` (skill success/failure and cross-skill prior counts, logistic regression C=1, SAGA max_iter=1000) and `GRUKnowledgeTracer` model (32-token,16-query,64-hidden, Adam lr=0.002, validation early stopping, max 25 epochs, patience 4, random seed 42). No test-driven tuning.
+- Primary estimand: interaction-weighted paired Brier-score difference `GRU - enriched PFA`. Resample **test learners** with replacement (2,000 replicates, seed 20261022), retaining all responses per selected learner. 95% percentile confidence interval. Claim directional external-cohort confirmation **only if** upper CI bound < 0 and computation/data gates pass; if the upper bound >= 0, record as not confirmed. Sign criterion is not proof of architecture causation.
+- Secondary descriptive quantities: ROC-AUC, log loss, calibration/ECE, learner and interaction counts, outcome prevalence, checksums and provenance. No choosing between datasets/seeds after seeing test outcomes.
+- Scope caveats: This is replication within ASSISTments in another year, on a random 5,000-learner subsample, not independent educational-system validity; 2015 skill-builder `sequence_id` concepts may not be the same units as 2009 mirror skill IDs, and enriched PFA and GRU have unequal information/capacity.
+- No outcome claims before executed logs and machine-readable artifacts have been verified. Inability to retrieve/verify the official source means **blocked**, not passed.

CONFIRMATORY_STAGE2_PROTOCOL_2015.md

@@ -0,0 +1,18 @@
+# Frozen stage-2, disjoint-cohort confirmatory replication
+
+**Reason for independent rerun.** The predeclared first external-cohort execution (GitHub run 39754046492; source SHA 75e1f46131d5897a388c82411bc3032279899cb1f1e66a9043354d41234407a0) generated estimates, but PFA `saga` emitted a nonconvergence warning and the planned diagnostic gate marked the test `completed_with_unresolved_diagnostics`. That result is retained without a confirmatory label. The correction addresses only numerical solver convergence; it must not select hyperparameters based on the observed direction or magnitude of the first test result.
+
+## New learner cohort and data provenance
+
+Use the **same official ASSISTments 2015 raw CSV bytes**, SHA-256 above, linked from the official ASSISTments portal. Apply exactly the same predeclared validity filters and chronological order (`log_id`) as in stage 1. Lexicographically sort eligible learner IDs, reconstruct the 5,000 participants originally sampled with `random.Random(20261022).sample(ids, 5000)`, exclude ALL 5,000 of those previously analysed learners, and sample 5,000 learners from the remaining eligible IDs with `random.Random(20261023).sample(remaining, 5000)`. Use a learner-disjoint 70/15/15 train/validation/test split with seed `20261023`. Assert zero overlap with stage 1 participants. No stage-2 outcomes have been inspected when this protocol was specified.
+
+## Fixed models and criterion
+
+- **PFA**: the same enriched information features (`skill`, within-skill previous correct/incorrect counts, across-skill previous correct/incorrect counts), StandardScaler for counts, OneHotEncoder for skill, logistic regression regularization `C=1.0`. Replace the nonconverged `saga` numerical solver by fixed `lbfgs` with `max_iter=2000`, `tol=1e-8`, to estimate the *same regularized logistic objective*. Record and reject nonconvergence warnings; **no alternative solvers, seeds, or hyperparameters after this check**.
+- **GRU**: same implementation as first execution; train only on stage-2 training learners, select epoch only on stage-2 validation learners. Seed 42, up to 25 epochs, patience 4, Adam LR .002, token dim 32, query dim 16, hidden 64.
+- Primary estimand: paired pooled interaction-weighted Brier loss GRU minus PFA across stage-2 test learners. Learner-cluster percentile bootstrap 2,000 replicates, seed 20261023. Directional result counts as *confirmed on this external 2015 subset* only if upper 95% percentile endpoint < 0 and all data/provenance/convergence guards pass. Otherwise it is a failed or indeterminate test; publish unchanged. Do not compare different PFA solvers on these stage-2 test outcomes to pick the favorable one.
+- Secondary descriptive measures: ROC-AUC, log loss, ECE, sample sizes and observed correctness prevalence; not selection criteria.
+
+## Scope
+
+A successful stage-2 test would independently replicate the direction of a predictive difference on a different-year ASSISTments cohort, with a numerically converged baseline. It does not prove that GRU architecture alone causes the difference, demonstrate source parity between the original 2009 mirror and corrected official data, or establish generalizability outside the analyzed contexts. This is not a registered external trial or blind third-party peer review.

CORRECTED_2009_REANALYSIS_PROTOCOL.md

@@ -0,0 +1,9 @@
+# Exploratory official-corrected-2009 source sensitivity protocol (22 Oct 2026)
+
+**Rationale:** Direct independent source audit proves that the frozen 2009 Atomi learner-sequence mirror and the official corrected 2009--2010 skill-builder release are not event-equivalent. The previously published model differences are conditional on the mirror. Execute one source-corrected sensitivity comparison on the source bytes whose provenance was just verified.
+
+**Source:** The exact corrected file linked from the ASSISTments maintainers' 2009--2010 Skill Builder page: Google Drive ID `1NNXHFRxcArrU0ZJSb9BIL56vmUt5FhlE`, SHA-256 `162ef8d2d28bcbfea6591a282994062bd8d5eaa00636544292a0d268dca6e5da`. Preserve original file; never redistribute its raw student records. Exclude only rows with absent/invalid binary outcome or unlabeled skill; retain one record per corrected student--problem event; source has 346,860 rows and 283,105 labeled rows spanning 4,163 learners. Normalize simple numeric skill IDs, preserve `_` composite skill keys. Order events within learner by increasing numeric `order_id`. Assert unique (student, order_id) and deterministic tie treatment. Event chronology depends on the source documentation's `order_id` semantics; treat this as an assumption, not independently reconstructed timestamps.
+
+**Preselected experiment:** Train/test split by whole learner with deterministic fixed seed 20261024 and `kt.split_rows` 70/15/15. Fit *enriched PFA*, using skill identity, prior same-skill successes/failures and prior across-skill successes/failures, `StandardScaler` for counts, `OneHotEncoder(handle_unknown='ignore')`, regularized logistic regression with `solver=lbfgs`, `C=1.0`, `max_iter=2000`, `tol=1e-8`. Reject nonconvergent fits. Fit the existing compact GRU, seed 42, token_dim=32, query_dim=16, hidden=64, max 25 epochs, patience 4, Adam 0.002; select stop epoch with validation learners only. First test event uses training skill/global prior. Compare test Brier scores (primary), AUC, log loss and calibration (secondary). Use 2,000 paired learner-cluster bootstrap draws seed 20261024 on pooled interaction-weighted GRU-minus-PFA Brier. Do not choose a different seed/hyperparameter to rescue a failed result. Record null/failure status honestly if invalid or interrupted.
+
+**Interpretation:** This analysis tests a corrected-source *sensitivity*, not an independent untouched 2009 hypothesis confirmation, because the original 2009 population and comparison direction were already analyzed. It does **not** establish a recurrence-only architecture effect, direct parity of the mirror with official data, or generalizability beyond the cohort. Do not edit the original mirror outcome or supplant the separate fresh 2015 replication.