[evals] Adding in some vally testing for some basic azd scenarios#9282
Draft
richardpark-msft wants to merge 13 commits into
Draft
[evals] Adding in some vally testing for some basic azd scenarios#9282richardpark-msft wants to merge 13 commits into
richardpark-msft wants to merge 13 commits into
Conversation
added 3 commits
July 22, 2026 16:36
Some evals/tests just didn't make the cut - either they were redundant with our normal unit tests (there were several that were checking help content), or they were evals that seemed to be eval'ing themselves (ie, the answer was encoded in the system prompt). I've reduced this down to two buckets: - eval-azd-troubleshooting.yaml, which is a series of prompts that just see what baseline copilot models can provide without any skills or extra materials. It's there mostly just to keep tabs on things and see if the advice changes in an interesting way, or if we start getting bad ad-hoc advice in chat sessions. - eval-azd-deploy.yaml - this is a suite I plan on expanding - right now I've got the node app test uncommented, and I plan on slowly introducing the others once we get this bootstrappable/trackable in some system (TBD, since there's a lot of moving parts/in-flight efforts here).
…s for some common scenarios - troubleshooting (ie, no azd around, but what does the LLM say?) and deploy, which will actually have some azd knowledge and is being asked - how do I deploy this bare-bones app, and other similar questions.
|
Azure Pipelines: Successfully started running 1 pipeline(s). 21 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Contributor
There was a problem hiding this comment.
Pull request overview
Replaces the legacy Waza evaluation framework with Vally-based azd scenarios, experiments, reporting, fixtures, and CI integration.
Changes:
- Adds Vally troubleshooting, deployment, and skill-baseline evaluations.
- Adds fixtures and a Markdown report generator.
- Removes legacy Waza tests, graders, and pipelines.
Show a summary per file
| File | Description |
|---|---|
eng/pipelines/eval-waza.yml |
Removes scheduled Waza evaluations. |
eng/pipelines/eval-unit.yml |
Removes legacy ADO unit-test pipeline. |
eng/pipelines/eval-report.yml |
Removes Waza report pipeline. |
eng/pipelines/eval-e2e.yml |
Removes Waza E2E pipeline. |
cli/azd/test/evals/skills/azd/SKILL.md |
Adds evaluation-only azd guidance. |
cli/azd/test/evals/README.md |
Documents Vally suites and commands. |
cli/azd/test/evals/package.json |
Adds suite and reporting scripts. |
cli/azd/test/evals/make-vally-report.go |
Generates Markdown evaluation reports. |
cli/azd/test/evals/fixtures/existing-node-api/package.json |
Defines the Node API fixture. |
cli/azd/test/evals/fixtures/existing-node-api/index.js |
Implements the fixture API. |
cli/azd/test/evals/fixtures/azd-project/staging.env |
Adds staging fixture values. |
cli/azd/test/evals/fixtures/azd-project/production.env |
Adds production fixture values. |
cli/azd/test/evals/fixtures/azd-project/dev.env |
Adds development fixture values. |
cli/azd/test/evals/fixtures/azd-project/config-production.json |
Selects the production fixture environment. |
cli/azd/test/evals/fixtures/azd-project/config-dev.json |
Selects the development fixture environment. |
cli/azd/test/evals/fixtures/azd-project/azure.yaml |
Defines the sample azd project. |
cli/azd/test/evals/eval-azd.experiments.yaml |
Compares skill and model variants. |
cli/azd/test/evals/eval-azd-troubleshooting.yaml |
Adds baseline troubleshooting evaluations. |
cli/azd/test/evals/eval-azd-deploy.yaml |
Adds deployment and lifecycle evaluations. |
cli/azd/test/evals/.gitignore |
Ignores Vally outputs and reports. |
cli/azd/test/eval/tsconfig.json |
Removes legacy TypeScript configuration. |
cli/azd/test/eval/tests/unit/help-text-quality.test.ts |
Removes Waza-era help tests. |
cli/azd/test/eval/tests/unit/flag-validation.test.ts |
Removes legacy flag tests. |
cli/azd/test/eval/tests/unit/command-sequencing.test.ts |
Removes legacy sequencing tests. |
cli/azd/test/eval/tests/unit/command-registry.test.ts |
Removes legacy registry tests. |
cli/azd/test/eval/tests/test-utils.ts |
Removes the old CLI test helper. |
cli/azd/test/eval/tests/human/error-recovery.test.ts |
Removes human error-recovery tests. |
cli/azd/test/eval/tests/human/command-discovery.test.ts |
Removes command-discovery tests. |
cli/azd/test/eval/tests/human/cli-workflow.test.ts |
Removes CLI workflow tests. |
cli/azd/test/eval/tasks/troubleshoot/quota-error.yaml |
Removes the Waza quota scenario. |
cli/azd/test/eval/tasks/troubleshoot/provision-role-conflict.yaml |
Removes the role-conflict scenario. |
cli/azd/test/eval/tasks/troubleshoot/config-error.yaml |
Removes the Waza config scenario. |
cli/azd/test/eval/tasks/troubleshoot/auth-error.yaml |
Removes the Waza authentication scenario. |
cli/azd/test/eval/tasks/negative/raw-azure-cli.yaml |
Removes the raw Azure CLI scenario. |
cli/azd/test/eval/tasks/negative/not-azure.yaml |
Removes the non-Azure scenario. |
cli/azd/test/eval/tasks/negative/general-coding.yaml |
Removes the general coding scenario. |
cli/azd/test/eval/tasks/lifecycle/teardown-only.yaml |
Removes the Waza teardown scenario. |
cli/azd/test/eval/tasks/lifecycle/full-lifecycle.yaml |
Removes the full lifecycle scenario. |
cli/azd/test/eval/tasks/environment/switch-env.yaml |
Removes the environment-switch scenario. |
cli/azd/test/eval/tasks/environment/delete-env.yaml |
Removes the environment-delete scenario. |
cli/azd/test/eval/tasks/environment/create-staging.yaml |
Removes the staging scenario. |
cli/azd/test/eval/tasks/deploy/deploy-python-webapp.yaml |
Removes the Python deployment scenario. |
cli/azd/test/eval/tasks/deploy/deploy-node-api.yaml |
Removes the Node deployment scenario. |
cli/azd/test/eval/tasks/deploy/deploy-existing-project.yaml |
Removes the existing-project scenario. |
cli/azd/test/eval/reports/.gitkeep |
Removes the legacy report placeholder. |
cli/azd/test/eval/README.md |
Removes Waza framework documentation. |
cli/azd/test/eval/package.json |
Removes Waza/Jest dependencies and scripts. |
cli/azd/test/eval/jest.config.ts |
Removes Jest configuration. |
cli/azd/test/eval/graders/test_graders.py |
Removes Python grader tests. |
cli/azd/test/eval/graders/infra_validator.py |
Removes infrastructure grader. |
cli/azd/test/eval/graders/cleanup_validator.py |
Removes cleanup grader. |
cli/azd/test/eval/graders/azure_auth.py |
Removes grader authentication helper. |
cli/azd/test/eval/graders/app_health.py |
Removes application-health grader. |
cli/azd/test/eval/eval.yaml |
Removes the Waza configuration. |
cli/azd/test/eval/.gitignore |
Removes legacy ignore rules. |
cli/azd/.vscode/cspell.yaml |
Removes obsolete Waza spelling overrides. |
.github/workflows/vally-ci.yml |
Splits Vally CI into two jobs. |
.github/workflows/eval-unit.yml |
Removes legacy GitHub unit workflow. |
Review details
Comments suppressed due to low confidence (1)
cli/azd/test/evals/make-vally-report.go:82
- azd-code-reviewer: This unexported field is initialized but never read. Staticcheck U1000 reports unused struct fields, so preflight will fail until both this field and its initializer are removed.
- Files reviewed: 57/59 changed files
- Comments generated: 5
- Review effort level: Medium
Co-authored-by: richardpark-msft <51494936+richardpark-msft@users.noreply.github.com>
added 4 commits
July 23, 2026 14:08
… closer to what it's actually doing.
…re-dev into azd-02-waza-port-evals
…t`, which actually can't create an environment - Use a prompt grader for fuzzy situations (these commands were listed in this order-ish) vs "hey, was the one piece of text mentioned?" which can be an output-matches. - Update the report so it reports _which_ checks failed in a failing test.
Contributor
There was a problem hiding this comment.
Review details
Files not reviewed (1)
- cli/azd/test/evals/package-lock.json: Generated file
Comments suppressed due to low confidence (1)
cli/azd/test/evals/eval-azd-deploy.yaml:48
- azd-code-reviewer: The PR description says only the Node app eval is enabled initially, but this second stimulus and all following stimuli are active. Because the workflow runs this file with
runs: 3, the deploy job executes 24 trials rather than the described 3. Either keep the planned gradual rollout by disabling the additional stimuli or update the PR description to reflect the expanded CI scope.
- Files reviewed: 57/60 changed files
- Comments generated: 4
- Review effort level: Medium
added 3 commits
July 23, 2026 15:22
…nicode characters on incorrect byte boundaries.
Contributor
There was a problem hiding this comment.
Review details
Files not reviewed (1)
- cli/azd/test/evals/package-lock.json: Generated file
Comments suppressed due to low confidence (1)
cli/azd/test/evals/eval-azd-deploy.yaml:2
- azd-code-reviewer: The PR description says only
deploy-node-apiis enabled and the other deploy scenarios will be introduced later, but this file activates all eight scenarios at three trials each, and the workflow runs the entire file. That materially expands CI cost and flakiness beyond the stated rollout. Either disable the remaining stimuli or update the PR description if enabling all of them is intentional.
- Files reviewed: 57/60 changed files
- Comments generated: 3
- Review effort level: Medium
added 2 commits
July 23, 2026 16:00
…t accidentally flag answers that might mention the 'bad" thing (azd init) in a "well, don't do that" type of answer.
… hopefully do a better job (has a limited rubric, so should be okay)
Contributor
Azure Dev CLI Install InstructionsInstall scriptsMacOS/Linux
bash: pwsh: WindowsPowerShell install MSI install Standalone Binary
MSI
Documentationlearn.microsoft.com documentationtitle: Azure Developer CLI reference
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Translating over some of the waza evals into vally evals.
Some evals/tests just didn't make the cut - either they were redundant with our normal unit tests (there were several that were checking help content), or they were evals that seemed to be eval'ing themselves (ie, the answer was encoded in the system prompt).
I've reduced this down to two buckets:
here).
I've tuned down the threshold for some of the flakier evals, just to give us a good reason to investigate if this job starts failing.
There's also an experiments file in there, which isn't yet running in CI, that can be useful for some manual testing as we refine things a bit.
Fixes #9199 (by adding in an experiments file)
Fixes #9186 (by porting waza evals)