Paper 1 (2026): matched-history budget audit reverses GRU-over-classical ranking (draft)

.github/workflows/matched-history-budget-2026.yml

@@ -0,0 +1,71 @@
+name: "Paper 1 - 2026 matched-history-budget experiment"
+on:
+  push:
+    branches:
+      - research/kt-methodological-2026-history-budget
+    paths:
+      - METHODOLOGICAL_2026_PROTOCOL.md
+      - scripts/matched_history_budget_2026.py
+      - .github/workflows/matched-history-budget-2026.yml
+  workflow_dispatch:
+permissions:
+  contents: write
+jobs:
+  matched-budget:
+    if: github.actor != 'github-actions[bot]'
+    runs-on: ubuntu-latest
+    timeout-minutes: 120
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.11'
+      - name: Install pinned scientific stack
+        run: |
+          python -m pip install -r requirements-repro.txt
+          python -m pip install 'gdown>=5.2,<6'
+      - name: Code compilation and causal prefix smoke checks
+        run: |
+          python -m py_compile scripts/matched_history_budget_2026.py
+          PYTHONPATH=src python scripts/matched_history_budget_2026.py --smoke
+      - name: Execute same-target independent-cohort budget comparison
+        run: PYTHONPATH=src python -u scripts/matched_history_budget_2026.py
+      - name: Validate learner and calculation invariants
+        run: |
+          python - <<'PY'
+          import json
+          a=json.load(open("results/matched_history_budget_2026.json"))
+          assert a["status"]=="passed_all_computational_gates",a.get("diagnostic_failures")
+          assert a["source"]["source_sha256"]=="75e1f46131d5897a388c82411bc3032279899cb1f1e66a9043354d41234407a0"
+          assert a["source"]["cohort_overlap_previous"]==0
+          assert a["source"]["stage3_learners"]==4000
+          assert a["design"]["test_learners"]==600
+          assert set(a["comparisons"])=={"1","20"}
+          for b in a["comparisons"].values():
+            assert set(b["models"])=={"window_count","ordered_logistic","window_gru"}
+            for x in b["models"].values():
+              assert 0<x["brier"]<1 and x["n"]>10000
+            for x in b["paired_GRU_minus"].values():
+              assert 400 <= x["n_learners"] <= 600 and x["n_bootstrap"]==2000
+              assert x["n_interactions"]==b["models"]["window_gru"]["n"]
+          print("Verified fresh learner cohort and history-parity output gates.")
+          PY
+      - name: Upload aggregate evidence even if a research gate fails
+        uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: matched-history-budget-2026-aggregate
+          path: results/matched_history_budget_2026.json
+          if-no-files-found: warn
+      - name: Archive de-identified aggregate results
+        if: success()
+        run: |
+          git config user.name "github-actions[bot]"
+          git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
+          git add results/matched_history_budget_2026.json
+          if ! git diff --cached --quiet; then
+            git commit -m "research: archive matched-history budget empirical results"
+            git push origin HEAD:research/kt-methodological-2026-history-budget
+          fi

METHODOLOGICAL_2026_PROTOCOL.md

@@ -0,0 +1,32 @@
+# Frozen 2026 methodological extension: matched history budgets (26 October 2026)
+
+This document is committed **before any stage-3 outcome is evaluated**. Previous ASSISTments 2009 mirror, corrected-source 2009, and 2015 stage-1/2 results have been examined, so this is a **new, prospectively specified 2015 holdout analysis conditional on earlier model development**, not a public registry preregistration and not a new educational intervention.
+
+## Research question and defensible novelty
+How do the *number of available prior student responses* and the *representation of their order* interact with the predictive differences between count-based generalized PFA and a small recurrent knowledge tracer, on the **same response targets**? This study does not claim to invent information-parity evaluation (see Padi et al., 2026) or cross-cohort benchmarks (Polydoros, 2026). The contribution tested here is a frozen **history-budget response curve** on a fresh learner cohort, linked to independent corrected-source/mirror provenance audits.
+
+## Official source and untouched sample
+Use exact original bytes of **ASSISTments 2015 Skill Builder Data**, SHA-256 `75e1f46131d5897a388c82411bc3032279899cb1f1e66a9043354d41234407a0`, downloaded only through the official portal link (as in existing scripts). The source contains 19,840 learners eligible under fixed existing stage-1/2 filters, ordered within learner by numeric `log_id`; use `sequence_id` as the 2015 skill-builder unit.
+
+Recreate stage-1 sampled learners with `random.Random(20261022).sample(sorted_eligible_ids, 5000)`; remove them. Recreate stage-2 sampled learners with `random.Random(20261023).sample(sorted_remaining, 5000)`; remove them. Independently sample **4,000** learners with `random.Random(20261026).sample(sorted_remaining_after_both,4000)`. Assert absolutely zero overlap with all previously evaluated 10,000 learners. Learner-disjoint **70/15/15** train/validation/test, split seed **20261026** (2,800/600/600). Test uses only responses at index t>=1 (the first response has no past to compare), regardless of history budget.
+
+## Models and information budget
+History budgets `K=1` and `K=20`, determined by number of *prior response events*, never time. At target t, each model may access only current skill ID and the preceding `min(K,t)` pairs (skill ID and correctness); **never current correctness**. Training and inference both use exactly this restriction.
+
+1. **Window-count PFA+**: logistic regression with current skill one-hot, within-current-skill correct/incorrect counts and all-skill correct/incorrect counts derived *only* from the last K prior events. Regularized logistic regression `C=1.0`, `lbfgs`, `tol=1e-7`, `max_iter=1600`, fitted on training interactions only. Standardize numerical features on training only.
+2. **Ordered-feature logistic**: same as window-count PFA+, plus correctness of latest prior interaction, latest occurrence of current skill (0 absent; +/-1 for correct/incorrect) and distance to most recent same-skill observation capped at K+1. The selected features are all derivable from the identical allowed history. This tests a transparent nonlinear-history proxy without upgrading PFA to an unrelated data stream.
+3. **Window-GRU**: train a compact recurrent model on only the last K (skill,response) tokens with the current skill query embedded separately; variable-length sequences padded only to batch maximum; final valid hidden state is read per target. Embeddings 24/12, hidden 48; Adam LR0.002, batch 256, 12 epochs max, early stopping patience3 on *validation only*, seed42. Same model capacity across K.
+
+Each K uses the exact same train/validation/test labels and learners, no within-test tuning. Models retrain for each K. This is **information-source parity**, not parameter/complexity parity, and does not identify architectural causal effects.
+
+## Estimands and numerical criteria
+Primary descriptive outputs: for each model and K report held-out ROC-AUC, Brier, log loss, ECE10, held-out interactions, and learner count; report GRU minus ordered-logistic and GRU minus count-logistic pooled interaction-weighted Brier differences with paired **2000 learner-cluster bootstrap percentile CIs** (seed 20261026). Report change in each model's Brier moving K=1 to K=20 and GRU-vs-ordered gap change. These comparisons are **exploratory multiple comparisons**; intervals are descriptive and not a controlled family-wise hypothesis rejection or causal effect. Do not label the result a state-of-the-art benchmark or universal replication.
+
+Sanity gates: checksum, exclusion counts, zero prior-cohort overlap, exact test IDs and target index agreement across all models/K, full train-only preprocessing, history prefix tests verifying the target answer is absent and only previous K events used, numerical convergence no warnings, finite outputs, numeric probabilities in [0,1], 2000 valid learner bootstrap draws. If any fail, output `inconclusive` and preserve diagnostics; never fabricate or quietly substitute results.
+
+## Non-overclaiming and next steps
+Only merge observed results into manuscript after actual GitHub Actions real-data execution and critical result inspection. Earlier corrected-2009 and 2015 evidence remains separately reported. A 2026 survey/literature novelty statement must explicitly recognize recent matched-input and leakage papers. Additional modern datasets and AKT are useful but NOT claimed executed in this extension. No student-level data exported; only cohort hashes and aggregate predictions/statistics retained.
+
+## Post-execution correction to a software assertion (26 October 2026; no model/test retuning)
+
+The predeclared test split comprises **600 distinct learners**, but 18 test learners have no response at index t>=1 and therefore generate no history-eligible targets. **582 learners / 20,592 interactions** are evaluable on the common response target set. The first Actions run 40310088864 successfully executed both K budgets and wrote aggregate metrics (`status: passed_all_computational_gates`) but the downstream release assertion incorrectly demanded 600 **evaluated** learners. The workflow assertion was changed to verify **up to 600** evaluation learners, requiring more than 400, and exact event counts across paired models. There was no change to sampling, models, features, targets, seeds, or inferential analysis. The initial diagnostic artifact is retained. This correction and the initially failed pipeline must be disclosed as technical QA history; the new clean workflow run is the final computational release gate.

paper/METHODOLOGICAL_2026_EXECUTED_RESULTS.md

@@ -0,0 +1,41 @@
+# Executed 2026 matched-history-budget results — 26 October 2026
+
+**Scientific outcome:** An interpretable order-aware logistic regression scored better Brier loss than the compact GRU on the same next-response targets under both fixed observable history budgets. The earlier GRU-over-PFA result does not generalize to every classical baseline under matched input stream availability.
+
+The official 2015 source SHA-256 was verified; stage 3 excluded the full 5,000-learner stage-1 sample and full 5,000-learner stage-2 sample. Of 19,840 source-eligible students, 9,840 were eligible for further sampling and 4,000 were selected for stage 3 (138,129 source events). Learner splits: 2,800 train / 600 validation / 600 test. **582** test learners produced **20,592** eligible targets at t>=1. The other 18 test learners had no post-first response and were properly excluded from next-response metrics.
+
+## Actual same-target results
+
+| Previous history K | Model | Brier (lower) | ROC-AUC (higher) | Log loss | ECE-10 |
+|---|---|---:|---:|---:|---:|
+| 1 | Count-based PFA+ | 0.177882 | 0.684440 | 0.534589 | 0.009320 |
+| 1 | Order-aware logistic | **0.177882** | 0.684436 | 0.534589 | 0.009321 |
+| 1 | GRU | 0.180003 | 0.670909 | 0.540658 | 0.005781 |
+| 20 | Count-based PFA+ | 0.176674 | 0.691058 | 0.531590 | 0.006126 |
+| 20 | Order-aware logistic | **0.173885** | 0.705044 | 0.524095 | 0.007251 |
+| 20 | GRU | 0.175140 | 0.699361 | 0.527153 | 0.011658 |
+
+Paired learner-cluster bootstrap, 2,000 draws: **GRU minus order-aware Brier K=1 +0.002121** (95% interval [0.001198,0.003013]), **K=20 +0.001255** ([0.000075,0.002426]). GRU-minus-count at K=20 is **−0.001533**, interval [−0.002761, −0.000279], demonstrating the difference that comparator choice makes.
+
+Improvements in Brier from K=1 to K=20: count-based 0.001208, order-aware 0.003997, GRU 0.004863. Interpret intervals descriptively: fixed data split/model fits, one GRU seed, no multiple-comparison correction.
+
+## Audited execution status
+
+- New protocol committed **before evaluating stage-3 cohort**: [frozen protocol](../METHODOLOGICAL_2026_PROTOCOL.md)
+- [Python implementation](../scripts/matched_history_budget_2026.py)
+- [Real-data, checksum-verified and learner-disjoint aggregate JSON](../results/matched_history_budget_2026.json)
+- [Successful GitHub Actions run 40310733288](https://github.com/devissaputra/knowledge_tracing_benchmark/actions/runs/40310733288).
+- [Final 13-page author-review manuscript](main_methodological_2026.tex).
+- [41 internal manuscript and numerical QA checks](METHODOLOGICAL_2026_FINAL_QA.json).
+- [Figure generator](generate_figures.py).
+- User-facing final PDF, minimal arXiv ZIP, and complete evidence package are preserved in ChatGPT Library folder `/Paper1_Methodological_2026`.
+
+### Transparency on prior release assertion
+The initial run 40310088864 completed all model training, both history horizons and aggregate output. Its release gate nevertheless failed on a mistaken assertion requiring 600 **evaluated** learners, confusing split members with learners having t>=1 targets. The corrected check permits 582 observed evaluable test learners and verifies exactly equal event counts across models, without changing any data, model, tuning, or observed result. The final run succeeds.
+
+### Contribution boundaries
+The study is **not** the first knowledge-tracing information-parity comparison, not the first source correction study and not a model architecture innovation. Polydoros (2026, DOI 10.5281/zenodo.22131530), Padi et al. (2026, arXiv:2609.28502), and Badran & Preisach (2026, DOI 10.1007/s42979-026-05344-3) are material prior work. The incremental contribution is a frozen, newly nonoverlapping 2015 cohort with explicitly constrained K=1 versus K=20 history budgets and order-feature comparator, interpreted alongside independently verified 2009 source mismatch.
+
+There is no basis for causal effect on educational outcomes, current schooling deployment, universal superiority of logistic models, or cross-year frozen-weight transfer. Next strengthening steps would be independent contemporary datasets, multiple splits/seeds and attention baselines; **none are claimed completed**.
+
+**Status:** Audited working preprint for named human author's scientific approval, not externally peer reviewed and not submitted to arXiv.

paper/METHODOLOGICAL_2026_FINAL_QA.json

@@ -0,0 +1,181 @@
+{
+  "date": "2026-10-26",
+  "scope": "Internal computation/transcription/document QA, not outside peer review",
+  "checks_total": 41,
+  "passed": 41,
+  "failed": 0,
+  "checks": [
+    {
+      "name": "Official 2015 CSV SHA-256 confirmed",
+      "passed": true
+    },
+    {
+      "name": "Fresh cohort has no overlap with first/second sample",
+      "passed": true
+    },
+    {
+      "name": "Correct stage3 learner splits",
+      "passed": true
+    },
+    {
+      "name": "Result scientifically completed",
+      "passed": true
+    },
+    {
+      "name": "Both history budgets present",
+      "passed": true
+    },
+    {
+      "name": "Same 20592 targets across both budgets",
+      "passed": true
+    },
+    {
+      "name": "582 evaluable history-eligible test learners",
+      "passed": true
+    },
+    {
+      "name": "2k learner cluster resamples",
+      "passed": true
+    },
+    {
+      "name": "No logistic convergence warnings",
+      "passed": true
+    },
+    {
+      "name": "All six models report bounded metrics",
+      "passed": true
+    },
+    {
+      "name": "Manuscript Brier 1 window_count",
+      "passed": true
+    },
+    {
+      "name": "Manuscript AUC 1 window_count",
+      "passed": true
+    },
+    {
+      "name": "Manuscript Brier 1 ordered_logistic",
+      "passed": true
+    },
+    {
+      "name": "Manuscript AUC 1 ordered_logistic",
+      "passed": true
+    },
+    {
+      "name": "Manuscript Brier 1 window_gru",
+      "passed": true
+    },
+    {
+      "name": "Manuscript AUC 1 window_gru",
+      "passed": true
+    },
+    {
+      "name": "Positive GRU-minus-ordered contrast for K=1",
+      "passed": true
+    },
+    {
+      "name": "Paired interval excludes zero positive at K=1",
+      "passed": true
+    },
+    {
+      "name": "Manuscript Brier 20 window_count",
+      "passed": true
+    },
+    {
+      "name": "Manuscript AUC 20 window_count",
+      "passed": true
+    },
+    {
+      "name": "Manuscript Brier 20 ordered_logistic",
+      "passed": true
+    },
+    {
+      "name": "Manuscript AUC 20 ordered_logistic",
+      "passed": true
+    },
+    {
+      "name": "Manuscript Brier 20 window_gru",
+      "passed": true
+    },
+    {
+      "name": "Manuscript AUC 20 window_gru",
+      "passed": true
+    },
+    {
+      "name": "Positive GRU-minus-ordered contrast for K=20",
+      "passed": true
+    },
+    {
+      "name": "Paired interval excludes zero positive at K=20",
+      "passed": true
+    },
+    {
+      "name": "New figure exists",
+      "passed": true
+    },
+    {
+      "name": "Multiple comparison caveat",
+      "passed": true
+    },
+    {
+      "name": "Late arxiv file includes AI-assistance caveat",
+      "passed": true
+    },
+    {
+      "name": "No novel architect invention claim",
+      "passed": true
+    },
+    {
+      "name": "Inherited figure kt_auc.png present",
+      "passed": true
+    },
+    {
+      "name": "Inherited figure kt_brier.png present",
+      "passed": true
+    },
+    {
+      "name": "Inherited figure robust_comparison.png present",
+      "passed": true
+    },
+    {
+      "name": "arXiv archive ZIP integrity",
+      "passed": true
+    },
+    {
+      "name": "Clean-room pdflatex pass 1",
+      "passed": true
+    },
+    {
+      "name": "Clean-room pdflatex pass 2",
+      "passed": true
+    },
+    {
+      "name": "Final PDF contains 20,592 evaluation count",
+      "passed": true
+    },
+    {
+      "name": "Final PDF contains new method section",
+      "passed": true
+    },
+    {
+      "name": "No LaTeX overfull warning",
+      "passed": true
+    },
+    {
+      "name": "No unresolved citations",
+      "passed": true
+    },
+    {
+      "name": "Final manuscript PDF hash matches clean-room compiled PDF",
+      "passed": true
+    }
+  ],
+  "source_sha256": "75e1f46131d5897a388c82411bc3032279899cb1f1e66a9043354d41234407a0",
+  "stage3_results_sha256": "e9a8cba68c2cb5b945d3a286aa27c0bfdc24b8841cd5052badc5ea4ac2e7381d",
+  "arxiv_source_sha256": "6d15e720f8bb613e40d9f4c1e6e4f9118d162092125dc749122f92cc2edcb6ff",
+  "pdf_sha256": "b0783067f84ac3c5b1f6b478f673e59465d6ecc2655833a0443831af4546609a",
+  "author_approval": false,
+  "external_peer_review": false,
+  "modern_independent_source_validation": false,
+  "scientific_caveat": "Single stage-3 split and GRU seed; matched input-stream availability does not match parameter count or optimization; descriptive bootstrap comparisons and 2009 source audit only"
+}