Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
0c0ccdb
Fix CCR template defects
Nov 4, 2026
caf553e
Sync CCR clip highlight with playback
Nov 4, 2026
ab98eb4
Scroll CCR navigation to section starts
Nov 4, 2026
84ccc8e
Accept any eligible listening device
Nov 4, 2026
8f0b68f
Clarify device-check retry instructions
Nov 4, 2026
4a250e8
Fix CCR result parsing
Nov 4, 2026
d39189a
Allow CCR/DCR sessions without trap questions
Nov 4, 2026
17addd2
Add session bundle packer
Nov 4, 2026
6f094dd
Support prepacked CCR/DCR sessions
Nov 4, 2026
c5e63a0
Add optional CCR comparison metadata and presentation
Nov 4, 2026
928658e
Aggregate CCR results by comparison ID
Nov 4, 2026
da5a4b1
Improve CCR qualification form usability
Nov 4, 2026
461866e
Show CCR practice and comparison progress
Nov 4, 2026
7cc2d4b
Keep annotated CCR trials that reuse trap audio
Nov 16, 2026
01861f0
Preserve CCR identities in Prolific merge
Nov 16, 2026
7b34a64
Keep NA CCR identities with pandas 3.1
Nov 16, 2026
3b8e87e
Pass comparison threshold in CCR analyzer test
Nov 16, 2026
95919aa
Validate CCR pair metadata before setup
Nov 17, 2026
8dcaadb
Match CCR conditions against audio URLs in pair mode
Nov 17, 2026
2714682
Ignore annotated trials in CCR trap acceptance
Nov 17, 2026
e805cfd
Reject invalid CCR training orders and conflicting pair IDs
Nov 17, 2026
67ba4f3
Keep annotated CCR trials in variance check
Nov 17, 2026
01968b4
Validate CCR metadata values and bandwidth limits
Nov 17, 2026
e8c79e6
Skip CCR/DCR bandwidth limit check without online qualification
Nov 17, 2026
55ea634
Remove unused CCR options and redundant template checks
Nov 17, 2026
13a6d1b
Simplify CCR playback counter to follow clip time
Nov 17, 2026
5cdb8bd
Check out result_parser.py with CRLF line endings
Nov 17, 2026
8bf2b89
Use four-space indentation and JSDoc in changed template scripts
Nov 17, 2026
cd38b06
Store result_parser.py with CRLF line endings
Nov 17, 2026
c7679e3
Export optional CCR training pair IDs
Nov 17, 2026
3e39a80
Keep result_parser.py line endings as on master
Nov 18, 2026
80a357b
Preserve training IDs through Prolific merge
Nov 18, 2026
1e5957b
Reject prepacked output aliases
Nov 18, 2026
55f6d28
Document session packer purpose and CLI options
Nov 18, 2026
32be1b1
Reject prepacked Qn_ORDER without comparison metadata
Nov 19, 2026
86fb71c
Restore practice-example metadata for custom presentations
Nov 19, 2026
2ce71f3
Remember CCR practice completion under strict CSP
Nov 19, 2026
45a7e78
Reject unsupported DCR order metadata
Nov 19, 2026
2d22cac
Restore configurable setup-comparison pass threshold
Nov 20, 2026
d870150
Grade setup comparisons against each pair's own correct clip
Nov 20, 2026
79b6a03
Avoid repeating items within random and balanced sessions
Nov 20, 2026
5ebf0d0
Show the CCR submit reminder only when all ratings are complete
Nov 20, 2026
59498f4
Add prepacked session conversion and guide
Nov 20, 2026
bb2fe0c
Merge origin/master into frankzalkow/ccr-test-fixes
Nov 20, 2026
4891628
Preserve legacy CMP tie answers
Nov 20, 2026
fada56d
Clean up line endings and trailing whitespace
Nov 20, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,10 @@ If you use this tool in your research please cite it with the following referenc
* [Using Prolific](docs/running_test_prolific.md)
* [Analyzing Data](docs/results.md)

For repeated pairwise comparisons, such as choosing between microphones with no clean reference,
see [Prepacked sessions](docs/prepacked_sessions.md). The guide explains the use case, what offline
planning adds to random/condition-balanced packing, and how to generate upload-ready files.

## Using an AI Agent

You can use an AI coding agent (e.g. GitHub Copilot, Claude) to create and run studies
Expand Down
17 changes: 13 additions & 4 deletions docs/conf_master.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,10 +30,19 @@ This describes the configuration for the `master_script.py`. A sample configurat
Assuming the URL is `http://test.com/D501_C03_M2_S02.wav` is the clip URL, and "03" is the condition name.
The pattern will be `.*_c(?P<condition_num>\d{1,2})_.*.wav`, you should also use condition_keys with `condition_num`.
* (optional) `condition_keys:` comma separated list of keys appearing in the `condition_pattern`:
* (optional) `clip_packing_strategy:random`: Either `random` or `balanced_block`. It specifies how to select clips
which will be assessed in a same HIT. For the `balanced_block` design, `condition_pattern`, `condition_pattern`, and
`condition_pattern` should be specified. `number_of_clips_per_session` should be a multiple of the unique values of the
key specified in the `block_keys`.
* (optional) `clip_packing_strategy:random`: `random`, `balanced_block` or, for CCR/DCR, `prepacked`. It specifies how to select clips
which will be assessed in a same HIT. For `balanced_block`, specify `condition_pattern` and `condition_keys`;
specify `block_keys` as well when there is more than one condition key. `number_of_clips_per_session`
must be a multiple of the number of unique values of the first block key. `random` and `balanced_block`
avoid repeating the same clip, or the same two clips in either order, within a session when feasible.
Both accept repeated input occurrences and retain them, padding to fill the final session when necessary.
The number of sessions is the input row count divided by the session size, rounded up.
If repeat-free packing is impossible at that session count (while retaining condition balance for
`balanced_block`), generation prints a clear warning and continues with repetitions within sessions.
Invalid input/configuration still raises an error. These checks cover rating items, not separately
appended trapping/gold clips. For `prepacked`, `prepacked_sessions` is the path to a CSV with one row per session,
planned with `src/pack_sessions.py` and converted using `src/convert_assignments.py`;
see [Prepacked sessions](prepacked_sessions.md).
* (optional) `block_keys:`: The key(s) to be used for creating the blocks should be specified here. Up to two keys.
A comma separated list. For multiple keys, all values of the first key should appear in one block.

Expand Down
44 changes: 41 additions & 3 deletions docs/prep_dcr_ccr.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,10 @@ For all the resource files (steps 1-4) an example is provided in `src/test_input

**Note**: make sure to first perform steps listed in the [general preparation process](preparation.md).

**Note**: Within DCR and CCR method, the quality of a speech clip is compared to a `reference` clip e.g. a same clip
without being artificially processed.
**Note**: DCR compares a speech clip to a reference, for example the same clip without artificial
processing. CCR can also compare two alternatives with no clean reference, such as microphones
recording the same scene. The CSV column `references` then labels canonical A, not a known better
recording; see [Prepacked sessions](prepacked_sessions.md).

1. Upload your **speech clips** and the references in a cloud server and create the `rating_clips.csv` file which
contains all URLs to speech clips in a column named `rating_clips` and URLs to the corresponding reference clips in
Expand Down Expand Up @@ -57,4 +59,40 @@ column named `training_clips` and URLs to corresponding reference clips in colum
* `YOUR_PROJECT_NAME_publish_batch.csv`: List of dynamic content to be used during publishing batch in AMT.
* `YOUR_PROJECT_NAME_ccr_result_parser.cfg`: Customized configuration file to be used by `result_parser.py` script

Now, you are ready for running the test on [Prolific](running_test_prolific.md) or [Amazon Mechanical Turk](running_test_mturk.md).
Now, you are ready for running the test on [Prolific](running_test_prolific.md) or [Amazon Mechanical Turk](running_test_mturk.md).

## CCR listening-device availability

Eligibility passes when any selected device meets `required_playback_device` (`headset`,
`loudspeaker` or `any`). Acoustic checks verify the active device and audibility.

## Optional: vary session bundles

Use offline session planning when comparisons need repeated ratings with varied companions,
source-scene exclusion, stratum quotas or exact CCR playback-order balance. The
[Prepacked sessions guide](prepacked_sessions.md) explains the microphone-comparison use case and
provides a complete `pack_sessions.py` -> `convert_assignments.py` -> `master_script.py` walkthrough.
The usual `random` and `balanced_block` strategies remain available for simpler designs.

### Prepacked assignments

Set `clip_packing_strategy: prepacked` and `prepacked_sessions` in `[create_input]` to load a
wide-format session CSV instead of generating memberships. Supply the normal clip and training
CSVs as well. See the guide for [generation commands](prepacked_sessions.md#worked-ccr-pipeline),
[metadata and controls](prepacked_sessions.md#controls-and-format-details), and
[DCR compatibility](prepacked_sessions.md#dcr-compatibility).

## CCR practice and presentation

When online qualification is enabled, CCR/DCR requires all five bandwidth-check answers to match
their supplied hashes; `bw_min` does not relax that client-side check, and `bw_max` must be `FB`.
Training rows may specify `order` (`ab`, `ba` or empty) and an optional `pair_id`, exported
as `Tn_ID` with each shuffled practice comparison. Use only letters, digits, `_`, `.`, `:`
or `-` in the ID. `append_training_trap: false` omits the appended training comparison.
An optional `practice_example` column is passed through to custom presentation scripts for
practice-specific feedback; it has no effect in the stock template.

An optional `presentation_file` loads trusted JSON fields `title`, `style`, `script`,
`instructions`, `training` and `rating`. Custom replacements must retain stock IDs, containers,
prerequisites and native qualification/device/setup/form behavior. Omitting `training` or `rating`
keeps the native panels; training order is randomized unless `randomize_training_questions: false`.
Loading
Loading