FEAT: Add Local File Dataset Configuration for Dataset Tinkering by ValbuenaVC · Pull Request #2285 · microsoft/PyRIT

ValbuenaVC · 2026-07-28T23:40:04Z

Description

RapidResponse operators need a short local iteration loop: ask Copilot to create or edit YAML seeds, reinitialize the scenario, and run against the latest file contents without synchronizing those seeds to a database.

This change adds DatasetAttackConfiguration.from_yaml_file(...), a file-backed inline source that:

Rereads local YAML during each dataset resolution, normally once per scenario initialization.
Preserves YAML dataset names and prompt grouping for scenario identity and result display.
Reuses existing validation and sampling without querying or writing PyRIT seed memory.
Performs blocking YAML I/O off the async event loop.
Rejects subclass calls that would silently discard subclass grouping behavior.
Converts read, parse, and grouping failures into actionable DatasetConstraintError messages.

Because disk remains authoritative, successful local loads now warn users at every local provider boundary. These warnings explain that in-process seed mutations are not written back automatically and will be lost unless saved to disk. Failed loads do not emit false success warnings.

The standard RapidResponse example remains unchanged. A separate recipe demonstrates the Copilot edit/rerun workflow using ~/.pyrit/rapid_response_local.prompt, a fresh scenario instance per iteration, and visible persistence warnings.

This intentionally does not add file watchers, dataset identifiers, revisioning, synchronization commands, or database schema changes.

Tests and Documentation

Developed in red-green TDD cycles covering:

Rereading edited and removed seeds.
Dataset-name and prompt-group preservation.
No seed-memory access for file-backed scenarios.
Validation before sampling and malformed-group errors.
Missing, deleted, and invalid files.
Subclass factory misuse.
Named-inline validator semantics.
Persisted scenario identity and RapidResponse grouping.
Success-only warnings for all local providers and file-backed resolution.
Synchronized notebook documentation and visible warning admonitions.

Validation completed:

105 passed across local provider and dataset-configuration suites.
46 passed in the RapidResponse suite.
Ruff lint and formatting passed.
ty passed.
JupyText pairs match and existing outputs are preserved.
git diff --check passed.
Independent Copilot and Opus reviews found no remaining blockers.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 34c04c70-cd29-4216-9321-65a3beedc523

Copilot

Pull request overview

Adds file-backed YAML dataset configuration for rapid local scenario iteration without memory synchronization.

Changes:

Adds asynchronous file-backed dataset resolution with validation and sampling.
Adds unit coverage for reloads, grouping, validation, and memory isolation.
Updates the RapidResponse notebook workflow.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File	Description
`pyrit/scenario/core/dataset_configuration.py`	Implements local YAML-backed configuration.
`tests/unit/scenario/core/test_dataset_configuration.py`	Tests file-backed behavior.
`doc/scanner/airt.py`	Documents the local iteration workflow.
`doc/scanner/airt.ipynb`	Synchronizes notebook documentation.

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

tests/unit/scenario/airt/test_rapid_response.py:257

This new test omits explicit types for both mock fixtures and its return value. The repository requires every function parameter and return type to be annotated (.github/instructions/style-guide.instructions.md:81-84); please use MagicMock for these fixtures and None for the return.

        mock_objective_target,
        mock_objective_scorer,
    ):

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 34c04c70-cd29-4216-9321-65a3beedc523

romanlutz

I have a few worries here, particularly around the behavior when the file changes.

romanlutz · 2026-08-04T19:05:39Z

        if not seeds:
            raise ValueError(f"No jailbreak templates found in {self._templates_path}")
        logger.info(f"Loaded {len(seeds)} jailbreak templates from {self._templates_path}")
+        logger.warning(


Same, INFO. Nobody edits the shipped jailbreak templates, so there's nothing here to act on.

romanlutz · 2026-08-04T19:05:39Z

            dataset = SeedDataset.from_yaml_file(self.file_path)
            if not dataset.dataset_name:
                dataset.dataset_name = self.dataset_name
+            logger.warning(


This should be INFO. _register_local_datasets creates a loader per .prompt/.yaml under seed_datasets/local/ — 36 files here — so every AIRT run now prints 36 warnings about lost edits for read-only shipped datasets.

romanlutz · 2026-08-04T19:05:39Z

+    @property
+    def dataset_names(self) -> list[str]:
+        """The resolved YAML dataset name, or the file stem before first resolution."""
+        return [self._resolved_dataset_name or self._file_path.stem]


This is only correct after a resolution has happened, and it works today purely because initialize_async resolves seed groups (line 623) before building the identifier (line 630). Nothing documents or enforces that order — swap those two lines and persisted scenario identity silently changes from the YAML name to the file stem.

romanlutz · 2026-08-04T19:05:40Z

+        validators: Sequence[Callable[[ResolvedDataset], None]] | None = None,
+    ) -> None:
+        super().__init__(max_dataset_size=max_dataset_size, validators=validators)
+        self._file_path = Path(file_path)


Nothing reads the file here, so from_yaml_file(file_path="typo.prompt") constructs fine and only blows up deep inside initialize_async. Note SeedDataset.from_yaml_file has the opposite contract — same name, raises immediately.

Parsing once here would fail at the call site and let dataset_names be a real value up front (see below), while the reread in _build_groups_by_dataset_async still picks up edits.

romanlutz · 2026-08-04T19:05:40Z

+            TypeError: If called on a subclass, which would discard subclass-specific
+                grouping or validation behavior.
+        """
+        if cls is not DatasetAttackConfiguration:


A classmethod whose first act is rejecting subclasses is fighting itself. Concretely it means CompoundDatasetAttackConfiguration.from_yaml_file(...) is now an inherited public factory that always raises, sitting next to per_dataset. Make it a module-level function or a staticmethod and the guard goes away.

romanlutz · 2026-08-04T19:05:40Z

+        """
+        dataset = await self._load_dataset_async()
+        dataset_name = dataset.dataset_name or dataset.name or self._file_path.stem
+        self._resolved_dataset_name = dataset_name


Resume drift isn't caught here. _build_initial_scenario_metadata only snapshots objective_hashes when max_dataset_size is set, and scenario identity hashes dataset names, not content. So: file-backed config with no max_dataset_size → edit the YAML → resume with the same scenario_result_id → different objectives appended to the original result, no error.

That's fine for memory-backed datasets since they're stable, but editing between runs is the whole point of this one. Snapshotting hashes unconditionally for file-backed configs would reuse the existing loud "persisted objective hash(es) no longer present" path.

romanlutz · 2026-08-04T19:05:40Z

+        )
+        return {dataset_name: groups}, resolved
+
+    def _build_file_attack_groups(self, *, seeds: list[Seed]) -> list[AttackSeedGroup]:


_dataset_names is None for file-backed configs, so an empty resolution reports (datasets: <inline>) and require_nonempty() gives a bare "Resolved dataset is empty." Everywhere else in this PR you put the file path in the message — worth doing here too.

romanlutz · 2026-08-04T19:05:40Z

+# > scenario initialization, or it will be lost.
+
+# %%
+local_dataset_config = DatasetAttackConfiguration.from_yaml_file(


This cell builds a config for a file that doesn't exist and then never uses it — it only "passes" because construction is lazy. The prose says "create or copy the scratch file first" but doesn't show how, while doc/scanner/airt.py has the write_text bootstrap in a non-executed fence. Pick one treatment for both pages.

Also CONFIGURATION_DIRECTORY_PATH is ~/.pyrit, where .env lives. Probably not where we want to steer people's scratch data.

romanlutz · 2026-08-04T19:05:40Z

 from the registered ``SeedDatasetProvider`` into memory. If a configured dataset
 name still yields nothing, the resolver raises loudly rather than silently skipping it.
 Inline configs (``seeds=`` / ``seed_groups=``) never touch memory.
+File-backed configs created by ``DatasetAttackConfiguration.from_yaml_file`` also bypass


Worth calling out in the PR description: this is a second local-file path that deliberately never lands in memory, so these seeds are invisible to get_seeds and dataset discovery while still advertising a real dataset_name. Side effect is that any scenario composing forbid_inline_seeds() or restrict_dataset_names() rejects file-backed configs. Defensible, but scenario authors didn't opt into it.

FEAT add local file dataset configuration

8cd6b31

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 34c04c70-cd29-4216-9321-65a3beedc523

ValbuenaVC requested review from Copilot and fdubut July 28, 2026 23:41

Copilot started reviewing on behalf of ValbuenaVC July 28, 2026 23:41 View session

Copilot AI reviewed Jul 28, 2026

View reviewed changes

ValbuenaVC and others added 2 commits July 29, 2026 10:38

Merge branch 'main' into vvalbuena-microsoft-dataset-tinkering

4fe4dd2

FEAT: Added tests and expanded _LocalFileDatasetAttackConfiguration

0912d7a

ValbuenaVC marked this pull request as ready for review July 29, 2026 19:43

ValbuenaVC changed the title ~~[DRAFT] FEAT: Add Local File Dataset Configuration for Dataset Tinkering~~ FEAT: Add Local File Dataset Configuration for Dataset Tinkering Jul 29, 2026

ValbuenaVC requested a review from Copilot July 29, 2026 19:50

Copilot started reviewing on behalf of ValbuenaVC July 29, 2026 19:50 View session

Precommit

fcfef10

Copilot AI reviewed Jul 29, 2026

View reviewed changes

Victor Valbuena and others added 3 commits July 29, 2026 13:29

FEAT warn on local dataset mutations

3863701

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 34c04c70-cd29-4216-9321-65a3beedc523

Merge branch 'main' into vvalbuena-microsoft-dataset-tinkering

618a4fb

Merge branch 'main' into vvalbuena-microsoft-dataset-tinkering

541df2b

romanlutz reviewed Aug 4, 2026

View reviewed changes

Navigation Menu

Uh oh!

FEAT: Add Local File Dataset Configuration for Dataset Tinkering - #2285

FEAT: Add Local File Dataset Configuration for Dataset Tinkering#2285
ValbuenaVC wants to merge 7 commits into
microsoft:mainfrom
ValbuenaVC:vvalbuena-microsoft-dataset-tinkering

ValbuenaVC commented Jul 28, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

romanlutz left a comment

Uh oh!

romanlutz Aug 4, 2026

Uh oh!

romanlutz Aug 4, 2026

Uh oh!

romanlutz Aug 4, 2026

Uh oh!

romanlutz Aug 4, 2026

Uh oh!

romanlutz Aug 4, 2026

Uh oh!

romanlutz Aug 4, 2026

Uh oh!

romanlutz Aug 4, 2026

Uh oh!

romanlutz Aug 4, 2026

Uh oh!

romanlutz Aug 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

ValbuenaVC commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Tests and Documentation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

romanlutz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ValbuenaVC commented Jul 28, 2026 •

edited

Loading