Skip to content

Fix CCR test setup issues and add opt-in CCR study tooling - #106

Open
Frank Zalkow (fzalkow) wants to merge 46 commits into
microsoft:masterfrom
fzalkow:frankzalkow/ccr-test-fixes
Open

Frank Zalkow (fzalkow) wants to merge 46 commits into
microsoft:masterfrom
fzalkow:frankzalkow/ccr-test-fixes

Conversation

@fzalkow

@fzalkow Frank Zalkow (fzalkow) commented Nov 4, 2026 •

Copy link
Copy Markdown

While setting up my first CCR test with this repository, I ran into a number of issues. This PR addresses them.

It is admittedly a diverse set of changes, split into focused commits. If you would prefer to drop or roll back any specific change, please let me know.

Issues and fixes

CCR template

  • allowed_max_hit_in_project: 0 did not block workers in a fresh browser, because the limit was only checked when a HIT cookie existed. The check now always runs.
  • Practice answers were appended to a non-existent #av-tabContent (typo of #nav-tabContent).
  • The loudspeaker image in the device question returned 404 (loudspeaker.png). It now uses louadspeaker.png, like the other templates.
  • The clip highlight switched when play() was called rather than when audio started, and the elapsed-time display updated in coarse steps. The highlight now follows actual media playback, and the counter is repainted from DoubleClip.currentTime on each animation frame. Inter-clip gap timing is unchanged from the original template: the displayed time holds during the one-second gap and accounts for it when the second clip advances; resuming from a pause inside the gap still starts the second clip immediately.
  • Next after setup or after the last practice item scrolled to the middle of the next section. It now scrolls to the start of that section.
  • Workers could not see how many practice and rating comparisons remained. Added progress bars.
  • Qualification form: the birth year accepted any text; the mother-tongue checkboxes were hard to see and select and had no option for unlisted languages; the prior-participation questions had no answer for "more than a month ago". Birth year is now limited to four digits, the languages are toggle buttons with an Other option (rejected unless configured as accepted), and More than a month ago was added.

All templates

  • The self-reported device check rejected workers who own both headphones and loudspeakers. It now passes if any selected device matches required_playback_device. The acoustic checks still verify the device that is actually in use.
  • The device-check retry text told workers to run the check again with Start, which stays disabled until they submit an answer. It now tells them to select I heard no sound, submit, and then retry.

CCR/DCR setup and result parsing

  • CCR/DCR sessions never received bandwidth-check sets, because create_input assigned them only for ACR. The assignment moved into a shared assign_bw_sets.
  • The generated parser config ignored allowed_max_hit_in_project from [hit_app_html], because it was looked up at the wrong level.
  • Trailing commas in the parser config template turned bw_min/bw_max into values like SWB,, so non-default bandwidth limits were silently not enforced.
  • The parser raised a KeyError for configs without a gold_question section, which is the case for CCR.
  • CCR "about the same" (0) votes were discarded as "cannot rate". That rule now applies only to P.804 dimensions.
  • Sessions without trap questions still got a ${TP} question, and the parser required one correct trap. The question is now added only when traps are configured, and the threshold is 0 otherwise.

Opt-in features (no behavior change unless configured)

  • There was no way to supply per-session comparison assignments, e.g. balanced repeated bundles. Added clip_packing_strategy: prepacked, and src/pack_sessions.py to build bundles with exact coverage, balanced A/B order and optional group/stratum quotas.
  • There was no way to prescribe or record each question's A/B order, comparison ID and role (main or control). Added comparison_metadata with Qn_ID/ROLE/ORDER/EXPOSURE columns. Training rows may set pair_id/order, and append_training_trap: false skips the appended training trap.
  • Instructions and panels could only be customized by editing the template. Added an optional presentation_file (trusted JSON).
  • Results were aggregated by file URL, which merges comparisons that share audio and mixes in controls. aggregate_by_pair_id aggregates by comparison ID, excludes controls, and exports raw trials and control-candidate audits. expected_votes_per_file is now configurable.

The new options are documented in docs/prep_dcr_ccr.md. Tests cover most fixes and features in tests/test_create_input.py, tests/test_result_parser.py, tests/test_master_script.py and tests/test_pack_sessions.py.

Frank Zalkow and others added 13 commits November 4, 2026 17:34
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 25a0ca09-56e2-41d3-9692-fe2b939d8071
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 25a0ca09-56e2-41d3-9692-fe2b939d8071
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 25a0ca09-56e2-41d3-9692-fe2b939d8071
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 25a0ca09-56e2-41d3-9692-fe2b939d8071
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 25a0ca09-56e2-41d3-9692-fe2b939d8071
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 25a0ca09-56e2-41d3-9692-fe2b939d8071
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 25a0ca09-56e2-41d3-9692-fe2b939d8071
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 25a0ca09-56e2-41d3-9692-fe2b939d8071
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 25a0ca09-56e2-41d3-9692-fe2b939d8071
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 25a0ca09-56e2-41d3-9692-fe2b939d8071
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 25a0ca09-56e2-41d3-9692-fe2b939d8071
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 25a0ca09-56e2-41d3-9692-fe2b939d8071
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 25a0ca09-56e2-41d3-9692-fe2b939d8071
@fzalkow

Copy link
Copy Markdown
Author

Frank Zalkow (Frank Zalkow (@fzalkow)) please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.

@microsoft-github-policy-service agree [company="{your company}"]

Options:

  • (default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.
@microsoft-github-policy-service agree
  • (when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.
@microsoft-github-policy-service agree company="Microsoft"

Contributor License Agreement

@microsoft-github-policy-service agree company="Microsoft"

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved defects discard valid annotated votes and controls, corrupt comparison IDs during Prolific CSV processing, and reject feasible session schedules.

4 open issues
What changed in this PR

This PR fixes CCR study setup and result-processing issues in the P.808 toolkit, improves participant-facing templates, and adds opt-in tools for controlled comparison studies.

Changes:

  • Improve CCR playback feedback, navigation, qualification forms, and progress indicators; update device eligibility and retry guidance across six templates.
  • Correct CCR/DCR bandwidth assignment and parser configuration, and preserve valid zero-valued CCR ratings.
  • Add prepacked sessions, balanced comparison packing, configurable presentation, and pair-based aggregation with audit exports.
File Description
tests/​test_result_parser.py Cover CCR scores, pair identity, controls, and audit exports.
tests/​test_pack_sessions.py Test packing constraints, reproducibility, and CLI outputs.
tests/​test_master_script.py Cover template behavior and generated configurations.
tests/​test_create_input.py Test prepacked assignments and bandwidth-set preservation.
src/​result_parser.py Add pair-based CCR aggregation and control audits; fix score handling.
src/​pack_sessions.py Add balanced session packing and reproducibility audits.
src/​P808Template/​P835_template.html Update device eligibility and retry instructions.
src/​P808Template/​P835_personalized_template3.html Update device eligibility and retry instructions.
src/​P808Template/​P808_multi.html Update device eligibility and retry instructions.
src/​P808Template/​DCR_template.html Update device eligibility and retry instructions.
src/​P808Template/​CCR_template.html Improve participant flow and support optional metadata and presentation.
src/​P808Template/​ACR_template.html Update device eligibility and retry instructions.
src/​master_script.py Wire optional study settings and correct parser configuration generation.
src/​create_input.py Support prepacked sessions and shared bandwidth-set assignment.
src/​assets_master_script/​dcr_ccr_result_parser_template.cfg Correct bandwidth values and expose aggregation and trap settings.
docs/​prep_dcr_ccr.md Document packing, metadata, presentation, and aggregation options.

Review effort: Balanced

Comment thread src/result_parser.py Resolved
Comment thread src/result_parser.py Resolved
Comment thread src/result_parser.py Resolved
Comment thread src/pack_sessions.py Resolved
Frank Zalkow and others added 4 commits November 16, 2026 16:04
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 25a0ca09-56e2-41d3-9692-fe2b939d8071
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 25a0ca09-56e2-41d3-9692-fe2b939d8071
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 25a0ca09-56e2-41d3-9692-fe2b939d8071
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 25a0ca09-56e2-41d3-9692-fe2b939d8071

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved configuration and metadata defects can generate studies whose results fail to parse, while URL-based trap screening can allow failed quality checks to pass.

3 open issues
1 previously missed issue

In code that hasn't changed since last review

Medium severity Read training identities as raw strings to preserve IDs

src/​master_script.py:269

The new training identities have already undergone pandas type inference before this conversion. For example, training IDs 001 and 002 become 1 and 2, while NA is omitted entirely, so the recorded practice identities no longer match the corresponding main comparisons. Read these fields as raw strings, while continuing to treat empty optional order cells as unset.

4 resolved since last review

Review effort: Balanced

Comment thread src/master_script.py Resolved
Comment thread src/master_script.py Resolved
Comment thread src/result_parser.py Resolved
Frank Zalkow and others added 4 commits November 17, 2026 09:42
Reject aggregate_by_pair_id without comparison_metadata, reject empty prepacked IDs and unknown roles, and read training identities as raw strings.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 25a0ca09-56e2-41d3-9692-fe2b939d8071
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 25a0ca09-56e2-41d3-9692-fe2b939d8071
In pair-ID mode, check_tps now skips questions with a pair ID, matching transform and the audit, so a control reusing the trap audio cannot pass a failed trap.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 25a0ca09-56e2-41d3-9692-fe2b939d8071
An invalid training order made the HIT template throw during page setup, leaving no questions. A prepacked ID naming different audio or roles silently pooled votes in pair-ID mode. Both are now rejected at setup.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 25a0ca09-56e2-41d3-9692-fe2b939d8071

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The omitted result_parser.py diff leaves the full parser change surface and regression scope unclear, requiring human review of the whitespace-ignored diff.

1 open issue
1 resolved since last review

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The overwrite check compares path strings rather than file identity, allowing symlink aliases to bypass it and overwrite the original assignments CSV.

1 open issue
1 resolved since last review

Review effort: Balanced

Comment thread src/create_input.py Resolved
Use file identity to prevent symlinked output paths from overwriting the prepacked assignments CSV.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 25a0ca09-56e2-41d3-9692-fe2b939d8071

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

CCR and worker-screening flows need end-to-end validation with real browser audio beyond mocks and static checks, while pair-ID aggregation still needs verification against a representative collected CCR export.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

@babaknaderi

Copy link
Copy Markdown
Contributor

What is the usage of pack_sessions and what problem does it solve?

Explain comparisons, sessions and bundles, why repeated comparisons need exact repeats, balanced playback order and source-disjoint bundles, and how the assignment plan becomes a prepacked sessions file. List the prepacked strategy in the master configuration reference and add help text to the packer arguments.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 25a0ca09-56e2-41d3-9692-fe2b939d8071
@fzalkow

Copy link
Copy Markdown
Author

What is the usage of pack_sessions and what problem does it solve?

pack_sessions.py plans, before master_script.py runs, which comparisons each listener gets. In a CCR test, a comparison is two clips of the same scene that the listener hears one after the other. A session is one listener's single sitting, that is, one HIT.

In my recent listenign test, each session holds 8 test comparisons plus 2 control comparisons. We wanted every comparison rated 10 times by different listeners (800 comparisons × 10 = 8,000 ratings in 1,000 sessions). The script guarantees:

  • Exact repeats: every comparison is rated exactly 10 times.
  • Balanced play order: each comparison is played clip A then B in 5 of its ratings, and B then A in the other 5. Listeners may favour whatever they hear first or second. With equal counts that bias cancels out for each comparison instead of leaning on a few of them.
  • No repeated material within a session: no comparison twice, and no two comparisons from the same scene, because they share the same speech.
  • Fixed mix and varied company: every session has the same mix of comparison types, all sessions differ, and the same comparisons are not always heard together.

The existing random strategy doesn't control this. When I shuffled the same design randomly, with a coin flip for play order, 170 of 1,000 sessions had two comparisons from one scene. 25 sessions repeated a comparison. 591 of 800 comparisons ended up with an uneven order split, for example 7 A-then-B against 3 B-then-A. With the packer all three counts are 0. Its output feeds clip_packing_strategy: prepacked.

I have extended documenation to explain this. (55f6d28)

Frank Zalkow and others added 10 commits November 19, 2026 08:34
Prepacked CCR/DCR sessions with Qn_ORDER columns were accepted when comparison_metadata was false, but the HIT app then randomized playback order and silently discarded the planned A/B balance. Fail at generation time instead.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 25a0ca09-56e2-41d3-9692-fe2b939d8071
Pass optional practice_example labels through to generated practice pairs so custom feedback can distinguish examples. Document the field and cover the passthrough in the presentation regression test.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 25a0ca09-56e2-41d3-9692-fe2b939d8071
Replace dynamic evaluation with direct array indexing so practice completion persists when HITAppServer forbids eval. Cover incomplete, repeated, and out-of-order playback with dynamic code generation disabled.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 25a0ca09-56e2-41d3-9692-fe2b939d8071
DCR always presents the reference and processed clips in fixed positions, so prescribed Qn_ORDER and comparison_metadata cannot be honored. Reject them with method-specific errors instead of silently ignoring the planned order, and clarify the CCR-only documentation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 25a0ca09-56e2-41d3-9692-fe2b939d8071
Add cmp_pass_threshold (1-4, default 2) to the master configuration, validate it, and pass it to the HIT app configuration and the result-parser templates so the number of correct setup comparisons required to pass is tunable again. The option was added on master after this branch diverged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 25a0ca09-56e2-41d3-9692-fe2b939d8071
The in-browser setup check accepted a clip if it was correct for any of the four pairs, and the offline answer keys used one global set of correct clips. Resolve the correct clip per displayed pair (explicit pair_ans, validated against the pair, or the legacy SNR filename prefix), write per-question ans_cmpK and cmp_correct_hashK columns, and make every active template compare against the hash of its own question.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 25a0ca09-56e2-41d3-9692-fe2b939d8071
Pack random and balanced_block sessions so the same clip, or the same two clips in either order, are not repeated within a session when feasible. Repeated input occurrences are retained and padded. When repeat-free packing is impossible at the requested session count, print a clear warning and continue with repetitions instead of failing. The packer uses scipy.optimize.milp, so require scipy>=1.9.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 25a0ca09-56e2-41d3-9692-fe2b939d8071
The CCR page showed its thank-you message unconditionally. Reveal a submit reminder only after practice and every comparison are answered and fully played, hide the Next button on the last comparison, and keep the progress label and bar optional. Say explicitly that the responses have not been submitted yet.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 25a0ca09-56e2-41d3-9692-fe2b939d8071
Add convert_assignments.py, which turns a pack_sessions.py plan into the wide-format prepacked sessions CSV consumed by master_script.py, and a Prepacked sessions guide that explains when to use the pipeline, how it differs from random and balanced_block packing, and the full planning, conversion and generation walkthrough. Slim the README, conf_master and DCR/CCR preparation docs down to short pointers to the guide.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 25a0ca09-56e2-41d3-9692-fe2b939d8071
Master added the comparison pass threshold in microsoft#105, which this branch had also restored. Keep master's result-parser templates, including their line-ending normalization, and the identical threshold tests and code from both sides.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 25a0ca09-56e2-41d3-9692-fe2b939d8071

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Legacy CMP pairs without pair_ans and with equal SNR prefixes are forced to an a/b answer, causing valid o responses to be marked incorrect.

1 open issue

Review effort: Balanced

Comment thread src/create_input.py Resolved
Emit o for equal-SNR legacy setup comparisons instead of forcing a winning clip. Keep explicit answer overrides and cover swapped order and offline grading.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 25a0ca09-56e2-41d3-9692-fe2b939d8071

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human judgment is needed to determine whether the unpublished, low-severity line-ending convention concern should affect approval.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

Convert the P831 DCR template to CRLF and remove whitespace warnings without changing behavior.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 25a0ca09-56e2-41d3-9692-fe2b939d8071
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants