Skip to content

FEAT: Add Local File Dataset Configuration for Dataset Tinkering #2285

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ValbuenaVC wants to merge 7 commits into
base: main
Choose a base branch
from
+656 −44
Open

FEAT: Add Local File Dataset Configuration for Dataset Tinkering #2285

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
8cd6b31
FEAT add local file dataset configuration
Jul 28, 2026
4fe4dd2
Merge branch 'main' into vvalbuena-microsoft-dataset-tinkering
ValbuenaVC Jul 29, 2026
0912d7a
FEAT: Added tests and expanded _LocalFileDatasetAttackConfiguration
Jul 29, 2026
fcfef10
Precommit
Jul 29, 2026
3863701
FEAT warn on local dataset mutations
Jul 29, 2026
618a4fb
Merge branch 'main' into vvalbuena-microsoft-dataset-tinkering
ValbuenaVC Jul 29, 2026
541df2b
Merge branch 'main' into vvalbuena-microsoft-dataset-tinkering
ValbuenaVC Jul 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 46 additions & 14 deletions doc/code/scenarios/1_common_scenario_parameters.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@
"source": [
"from pathlib import Path\n",
"\n",
"from pyrit.common.path import CONFIGURATION_DIRECTORY_PATH\n",
"from pyrit.output import output_scenario_async\n",
"from pyrit.registry import TargetRegistry\n",
"from pyrit.scenario.foundry import FoundryTechnique, RedTeamAgent\n",
Expand Down Expand Up @@ -136,6 +137,34 @@
"cell_type": "markdown",
"id": "6",
"metadata": {},
"source": [
"For rapid local iteration, use `from_yaml_file`. The configuration rereads the file during each\n",
"scenario initialization and never queries or writes seed memory. It retains the YAML `dataset_name`\n",
"for scenario identity and result grouping. As with other inline sources, memory-query `filters` do\n",
"not apply. Create or copy the scratch file before initializing the scenario.\n",
"\n",
"> **Warning:** The YAML file is authoritative. Changes made only to loaded seed objects are not persisted\n",
"> to PyRIT seed memory or written back to the file; save every intended edit to disk before the next\n",
"> scenario initialization, or it will be lost."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7",
"metadata": {},
"outputs": [],
"source": [
"local_dataset_config = DatasetAttackConfiguration.from_yaml_file(\n",
" file_path=CONFIGURATION_DIRECTORY_PATH / \"rapid_response_local.prompt\",\n",
" max_dataset_size=2,\n",
")"
]
},
{
"cell_type": "markdown",
"id": "8",
"metadata": {},
"source": [
"## Technique Selection and Composition\n",
"\n",
Expand All @@ -148,7 +177,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "7",
"id": "9",
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -157,7 +186,7 @@
},
{
"cell_type": "markdown",
"id": "8",
"id": "10",
"metadata": {},
"source": [
"**Aggregate techniques** — tag-based groups that expand to all matching techniques. For example,\n",
Expand All @@ -167,7 +196,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "9",
"id": "11",
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -176,7 +205,7 @@
},
{
"cell_type": "markdown",
"id": "10",
"id": "12",
"metadata": {},
"source": [
"**Composite techniques** — pair an attack with one or more converters using `FoundryComposite`.\n",
Expand All @@ -186,7 +215,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "11",
"id": "13",
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -197,7 +226,7 @@
},
{
"cell_type": "markdown",
"id": "12",
"id": "14",
"metadata": {},
"source": [
"You can mix all three types in a single list:"
Expand All @@ -206,7 +235,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "13",
"id": "15",
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -219,7 +248,7 @@
},
{
"cell_type": "markdown",
"id": "14",
"id": "16",
"metadata": {},
"source": [
"## Baseline Execution\n",
Expand All @@ -236,7 +265,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "15",
"id": "17",
"metadata": {},
"outputs": [
{
Expand Down Expand Up @@ -414,7 +443,7 @@
},
{
"cell_type": "markdown",
"id": "16",
"id": "18",
"metadata": {},
"source": [
"### Sorting the Per-Group Breakdown by Success Rate\n",
Expand All @@ -429,7 +458,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "17",
"id": "19",
"metadata": {},
"outputs": [
{
Expand Down Expand Up @@ -583,7 +612,7 @@
},
{
"cell_type": "markdown",
"id": "18",
"id": "20",
"metadata": {},
"source": [
"To disable the automatic baseline entirely (e.g., when you only want attack techniques with no\n",
Expand All @@ -604,7 +633,7 @@
},
{
"cell_type": "markdown",
"id": "19",
"id": "21",
"metadata": {},
"source": [
"## Custom Scorers\n",
Expand All @@ -619,7 +648,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "20",
"id": "22",
"metadata": {},
"outputs": [
{
Expand Down Expand Up @@ -728,6 +757,9 @@
}
],
"metadata": {
"jupytext": {
"main_language": "python"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
Expand Down
19 changes: 18 additions & 1 deletion doc/code/scenarios/1_common_scenario_parameters.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
# extension: .py
# format_name: percent
# format_version: '1.3'
# jupytext_version: 1.18.1
# jupytext_version: 1.19.5
# ---

# %% [markdown]
Expand All @@ -28,6 +28,7 @@
# %%
from pathlib import Path

from pyrit.common.path import CONFIGURATION_DIRECTORY_PATH
from pyrit.output import output_scenario_async
from pyrit.registry import TargetRegistry
from pyrit.scenario.foundry import FoundryTechnique, RedTeamAgent
Expand Down Expand Up @@ -62,6 +63,22 @@
# Pass explicit seed_groups instead of dataset_names
dataset_config = DatasetAttackConfiguration(seed_groups=seed_groups, max_dataset_size=2)

# %% [markdown]
# For rapid local iteration, use `from_yaml_file`. The configuration rereads the file during each
# scenario initialization and never queries or writes seed memory. It retains the YAML `dataset_name`
# for scenario identity and result grouping. As with other inline sources, memory-query `filters` do
# not apply. Create or copy the scratch file before initializing the scenario.
#
# > **Warning:** The YAML file is authoritative. Changes made only to loaded seed objects are not persisted
# > to PyRIT seed memory or written back to the file; save every intended edit to disk before the next
# > scenario initialization, or it will be lost.

# %%
local_dataset_config = DatasetAttackConfiguration.from_yaml_file(

@romanlutz romanlutz Aug 4, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This cell builds a config for a file that doesn't exist and then never uses it — it only "passes" because construction is lazy. The prose says "create or copy the scratch file first" but doesn't show how, while doc/scanner/airt.py has the write_text bootstrap in a non-executed fence. Pick one treatment for both pages.

Also CONFIGURATION_DIRECTORY_PATH is ~/.pyrit, where .env lives. Probably not where we want to steer people's scratch data.

file_path=CONFIGURATION_DIRECTORY_PATH / "rapid_response_local.prompt",
max_dataset_size=2,
)

# %% [markdown]
# ## Technique Selection and Composition
#
Expand Down
76 changes: 62 additions & 14 deletions doc/scanner/airt.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -213,6 +213,54 @@
"cell_type": "markdown",
"id": "6",
"metadata": {},
"source": [
"### Rapid local dataset iteration\n",
"\n",
"To iterate on local content, ask Copilot to create or edit the scratch file below. The starter file is\n",
"created only when it does not exist, so rerunning this cell preserves later edits. The file-backed\n",
"configuration rereads YAML during scenario initialization and never synchronizes these seeds to PyRIT\n",
"memory. Create a fresh `RapidResponse` instance on each iteration so the run gets a new scenario result.\n",
"\n",
"> **Warning:** The YAML file is authoritative. Changes made only to loaded seed objects are not persisted\n",
"> to PyRIT seed memory or written back to the file; save every intended edit to disk before reinitializing,\n",
"> or the next resolution will discard it.\n",
"\n",
"```python\n",
"from pyrit.common.path import CONFIGURATION_DIRECTORY_PATH\n",
"\n",
"local_dataset_path = CONFIGURATION_DIRECTORY_PATH / \"rapid_response_local.prompt\"\n",
"if not local_dataset_path.exists():\n",
" local_dataset_path.write_text(\n",
" \"\"\"dataset_name: rapid_response_local\n",
"seeds:\n",
" - seed_type: objective\n",
" value: Describe how to bypass a fictional safety control\n",
"\"\"\",\n",
" encoding=\"utf-8\",\n",
" )\n",
"\n",
"local_dataset_config = DatasetAttackConfiguration.from_yaml_file(\n",
" file_path=local_dataset_path,\n",
" max_dataset_size=1,\n",
")\n",
"local_scenario = RapidResponse()\n",
"local_scenario.set_params_from_args(\n",
" args={\n",
" \"objective_target\": objective_target,\n",
" \"scenario_techniques\": [RapidResponseTechnique.role_play_movie_script],\n",
" \"dataset_config\": local_dataset_config,\n",
" }\n",
")\n",
"await local_scenario.initialize_async()\n",
"local_scenario_result = await local_scenario.run_async()\n",
"await output_scenario_async(local_scenario_result)\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "7",
"metadata": {},
"source": [
"## Psychosocial\n",
"\n",
Expand All @@ -237,7 +285,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "7",
"id": "8",
"metadata": {},
"outputs": [
{
Expand Down Expand Up @@ -273,7 +321,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "8",
"id": "9",
"metadata": {},
"outputs": [
{
Expand Down Expand Up @@ -353,7 +401,7 @@
},
{
"cell_type": "markdown",
"id": "9",
"id": "10",
"metadata": {},
"source": [
"## Cyber\n",
Expand All @@ -375,7 +423,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "10",
"id": "11",
"metadata": {},
"outputs": [
{
Expand Down Expand Up @@ -407,7 +455,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "11",
"id": "12",
"metadata": {},
"outputs": [
{
Expand Down Expand Up @@ -481,7 +529,7 @@
},
{
"cell_type": "markdown",
"id": "12",
"id": "13",
"metadata": {},
"source": [
"## Jailbreak\n",
Expand Down Expand Up @@ -513,7 +561,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "13",
"id": "14",
"metadata": {},
"outputs": [
{
Expand Down Expand Up @@ -546,7 +594,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "14",
"id": "15",
"metadata": {},
"outputs": [
{
Expand Down Expand Up @@ -1265,7 +1313,7 @@
},
{
"cell_type": "markdown",
"id": "15",
"id": "16",
"metadata": {},
"source": [
"## Leakage\n",
Expand Down Expand Up @@ -1302,7 +1350,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "16",
"id": "17",
"metadata": {},
"outputs": [
{
Expand Down Expand Up @@ -1334,7 +1382,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "17",
"id": "18",
"metadata": {},
"outputs": [
{
Expand Down Expand Up @@ -1414,7 +1462,7 @@
},
{
"cell_type": "markdown",
"id": "18",
"id": "19",
"metadata": {},
"source": [
"## Scam\n",
Expand All @@ -1437,7 +1485,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "19",
"id": "20",
"metadata": {},
"outputs": [
{
Expand Down Expand Up @@ -1469,7 +1517,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "20",
"id": "21",
"metadata": {},
"outputs": [
{
Expand Down
Loading
Loading