diff --git a/README.md b/README.md index 7a5fde2..c47957b 100644 --- a/README.md +++ b/README.md @@ -56,7 +56,7 @@ claude mcp add --transport http testrigor https://api2.testrigor.com/api/v1/mcp --header "personal-access-token: " -s user ``` -Other MCP-capable agents (Cursor, etc.) point their own MCP config at `https://api2.testrigor.com/api/v1/mcp` with the same `personal-access-token` header. The server can list/retrieve suites & cases, run test cases or a green regression, list runs, and read run failures. Guide: https://testrigor.com/how-to-utilise-testrigors-mcp-server/ +Other MCP-capable agents (Cursor, etc.) point their own MCP config at `https://api2.testrigor.com/api/v1/mcp` with the same `personal-access-token` header. The server can list/retrieve suites & cases, validate step syntax without running, run test cases or a green regression, list runs, and read run failures. Guide: https://testrigor.com/how-to-utilise-testrigors-mcp-server/ ## Runnable examples diff --git a/testrigor-cli/SKILL.md b/testrigor-cli/SKILL.md index 456337d..dacbb4b 100644 --- a/testrigor-cli/SKILL.md +++ b/testrigor-cli/SKILL.md @@ -137,10 +137,11 @@ For the on-disk file schema (`.txt` vs `.yaml`, labels, datasets, `testCaseUuid` ## Command tree ``` -testrigor authenticate # INTERACTIVE prompt — human-once only; never in automation -testrigor test-suite config # --default | --show | --delete -testrigor test-suite run [ID] [flags] # push files + trigger a run -testrigor plugins # list installed plugins +testrigor authenticate # INTERACTIVE prompt — human-once only; never in automation +testrigor test-suite config # --default | --show | --delete +testrigor test-suite run [ID] [flags] # push files + trigger a run +testrigor test-suite validate-syntax [ID] [flags] # check step syntax, no run +testrigor plugins # list installed plugins testrigor help [command] ``` @@ -208,6 +209,22 @@ Reporting: |------|---------| | `--junit-report-save-path ` | Write a JUnit XML report (sync mode only) | +## `test-suite validate-syntax` flags + +Checks step syntax against the suite's reusable rules without running anything — no test execution, no mutation of the suite. Exactly one of `--steps`/`--test-cases-path` is required. + +| Flag | Purpose | +|------|---------| +| `--steps ` | Raw step text to validate directly (mutually exclusive with `--test-cases-path`) | +| `--test-cases-path ` | Test case files to validate — each is checked independently and reported by name; one invalid file doesn't stop the rest, and the command exits non-zero if any fail | +| `--rules-path ` | Ad hoc reusable rules to validate against, for rules not yet saved to this suite (same glob format as `test-suite run --rules-path`) | +| `--explicit-mutations` | Validate using only the supplied `--rules-path` rules instead of merging with the suite's saved rules | + +```bash +testrigor test-suite validate-syntax "$TEST_SUITE_ID" --steps 'click "Sign in"' +testrigor test-suite validate-syntax "$TEST_SUITE_ID" --test-cases-path "test-cases/**/*.{txt,yaml,yml}" --rules-path "rules/**/*.yaml" +``` + ## Localhost debugging (the tunnel) `--localhost --url http://localhost:3000` lets testRigor's cloud browsers reach an app running on your machine. Mechanics (so you know what to expect): @@ -253,7 +270,7 @@ claude mcp add --transport http testrigor https://api2.testrigor.com/api/v1/mcp --header "personal-access-token: " -s user ``` -Other MCP-capable agents (Cursor, etc.) point their own MCP config at the same endpoint (`https://api2.testrigor.com/api/v1/mcp`) with the `personal-access-token` header. The server exposes tools to list/retrieve test suites & cases, run test cases or a green regression, list runs, and read run failures. Use the MCP when you want programmatic run/inspect from the agent; use this CLI for terminal/CI pipelines and the file-push (mutation) workflow. +Other MCP-capable agents (Cursor, etc.) point their own MCP config at the same endpoint (`https://api2.testrigor.com/api/v1/mcp`) with the `personal-access-token` header. The server exposes tools to list/retrieve test suites & cases, validate step syntax without running, run test cases or a green regression, list runs, and read run failures. Use the MCP when you want programmatic run/inspect from the agent; use this CLI for terminal/CI pipelines and the file-push (mutation) workflow. > Guide: https://testrigor.com/how-to-utilise-testrigors-mcp-server/ @@ -299,6 +316,14 @@ testrigor test-suite run \ --explicit-mutations ``` +Gate a commit on syntax before the real run (seconds instead of minutes): + +```bash +testrigor test-suite validate-syntax "$TEST_SUITE_ID" \ + --test-cases-path "test-cases/**/*.txt" \ + --rules-path "rules/**/*.yaml" +``` + Upload and test a mobile build: ```bash diff --git a/testrigor-dev-loop/SKILL.md b/testrigor-dev-loop/SKILL.md index 3d40e56..abdb2f6 100644 --- a/testrigor-dev-loop/SKILL.md +++ b/testrigor-dev-loop/SKILL.md @@ -21,12 +21,12 @@ There is **no separate "deploy" step.** When you run with `--test-cases-path` / ## The loop ``` -edit files → run (localhost, --test-cases-path/--rules-path) → exit code + report.xml - ▲ │ - └──────────────────── fix failing steps ◀──────────────────────────────┘ +edit files → validate syntax → run (--test-cases-path/--rules-path) → exit code + report.xml + ▲ │ + └────────────────────────────── fix failing steps ◀──────────────────────────────┘ ``` -Stop when the exit code is `0` and `report.xml` shows no failures. +Validating first is optional but nearly free — it catches a mistyped command in a second instead of minutes into a run. See `testrigor-write-tests` → "Validate before you run". Stop when the exit code is `0` and `report.xml` shows no failures. ## 0. One-time setup @@ -91,7 +91,7 @@ grep -A2 ' **Cheap pre-check (runs take minutes).** After changing the app or a test, `grep` the page for the exact text and controls your tests assert before burning a full run — `grep -F "Thank you for your purchase!" end.html`. Catching a renamed label or a removed button statically is far faster than waiting on the cloud run. +> **Cheap pre-checks (runs take minutes).** Two are worth doing before burning a full run. **Syntax** — `validate_syntax`/`test-suite validate-syntax` tells you in a second whether every step parses (`testrigor-write-tests` → "Validate before you run"); if you just added a rule as a local file and haven't pushed it yet, pass it via `rules`/`--rules-path` or it'll be reported as unrecognized. **Content** — `grep` the page for the exact text and controls your tests assert, `grep -F "Thank you for your purchase!" end.html`. Catching a mistyped command or a renamed label statically is far faster than waiting on the cloud run. ## 4. "Update remote" is just re-running diff --git a/testrigor-write-tests/SKILL.md b/testrigor-write-tests/SKILL.md index 9a04873..063d9b5 100644 --- a/testrigor-write-tests/SKILL.md +++ b/testrigor-write-tests/SKILL.md @@ -1,6 +1,6 @@ --- name: testrigor-write-tests -description: Author testRigor automated test cases in plain-English commands — navigation, clicks, typing, assertions, reusable rules, variables, data-driven tests, API steps (`call api`/`mock api`), and login/email/SMS/2FA flows. Use when writing, editing, or converting manual test steps into testRigor test cases (as `test-cases/*.txt` files or steps pasted into the testRigor app), or when asked how to express a step in testRigor's language. Full command catalog in `reference.md`; worked authoring examples in `examples.md`; API steps in `api-testing.md`. To run suites use the `testrigor-cli` skill. +description: Author testRigor automated test cases in plain-English commands — navigation, clicks, typing, assertions, reusable rules, variables, data-driven tests, API steps (`call api`/`mock api`), and login/email/SMS/2FA flows. Use when writing, editing, or converting manual test steps into testRigor test cases (as `test-cases/*.txt` files or steps pasted into the testRigor app), or when asked how to express a step in testRigor's language, or to check that steps parse without spending a run (`validate_syntax`). Full command catalog in `reference.md`; worked authoring examples in `examples.md`; API steps in `api-testing.md`. To run suites use the `testrigor-cli` skill. --- # Writing testRigor Test Cases @@ -263,6 +263,43 @@ To **run** these files (a run also pushes them into the remote suite, updating i To call REST endpoints, extract JSON, assert on status codes, chain API + UI, or mock a dependency — all inside the same test case — see [api-testing.md](api-testing.md). +## Validate before you run + +A run takes minutes; a syntax check takes a second. After writing or editing steps, confirm they parse before running. + +**MCP** — call `validate_syntax` with the suite ID and the steps: + +``` +validate_syntax(testSuiteId: "", steps: "click \"Sign in\"\ncheck that page contains \"Welcome\"") +``` + +Valid steps return `{"summary": "No issues found.", "issues": []}`; otherwise `issues` lists each command that didn't parse, each as a `{command, error}` pair. + +**REST** — the same check without MCP. `auth-token` accepts a PAT or the suite's API token: + +```bash +curl -s -X POST "https://api2.testrigor.com/api/v1/apps/$SUITE/validate-syntax" \ + -H "Content-Type: application/json" -H "auth-token: $TESTRIGOR_API_KEY" \ + -d '{"steps":"click \"Sign in\"\ncheck that page contains \"Welcome\""}' +``` + +**CLI** — `testrigor test-suite validate-syntax`, either an inline string or a batch of files: + +```bash +testrigor test-suite validate-syntax "$SUITE" --token "$TESTRIGOR_API_KEY" --steps 'click "Sign in"' +testrigor test-suite validate-syntax "$SUITE" --token "$TESTRIGOR_API_KEY" --test-cases-path 'test_cases/**/*.{yaml,txt}' +``` + +With `--test-cases-path`, every matched file is validated independently and reported by name — one invalid file doesn't stop the rest from being checked, and the command exits non-zero if any file fails. + +Things to know about the result: + +- **The HTTP status is always `200`** once you're authenticated. The real verdict is the `status` field *inside* the body — `204` valid, `422` not valid. Don't gate on the HTTP code or a shell exit code. +- **The error message tells you how specific the failure is.** A recognized command that's malformed gets a precise message (e.g. `click` alone → "Missing reference to an element: click"). A command the parser can't recognize at all gets a generic fallback ("Unrecognizable instruction, can't detect action: ``") — treat that one as "re-check this line against [reference.md](reference.md)," not as a specific diagnosis. +- **A rule referenced but not yet saved to the suite is reported as unrecognized — unless you supply it ad hoc.** Built-ins (`login`, `fill out form`) always parse, and a rule already saved to the suite resolves normally. For a rule that only exists as a local file, pass it explicitly: MCP/REST take an optional `rules` list (`[{name, steps, labels}]`, labels optional), the CLI takes `--rules-path` (same glob format as `test-suite run --rules-path`). By default the supplied rules merge with the suite's saved ones — a rule with the same name as an already-saved one is used in place of it, no conflict. Set `explicitMutations`/`--explicit-mutations` to validate against *only* the supplied rules, ignoring the suite's saved ones entirely. + +Validation proves only that steps *parse*. It cannot tell you whether `"Sign in"` is actually on the page — that still takes a run. + ## When you're unsure of the exact keyword -Read [reference.md](reference.md) — it lists every command category with exact syntax. testRigor is forgiving about phrasing, but using the documented keywords (`click`, `enter ... into ...`, `check that page contains ...`, `grab ... and save it as ...`) is the most reliable. If a step is genuinely visual or ambiguous, add `using AI` (e.g. `check that page "shows a success banner" using ai`). +Read [reference.md](reference.md) — it lists every command category with exact syntax. testRigor is forgiving about phrasing, but using the documented keywords (`click`, `enter ... into ...`, `check that page contains ...`, `grab ... and save it as ...`) is the most reliable. If a step is genuinely visual or ambiguous, add `using AI` (e.g. `check that page "shows a success banner" using ai`). When you've guessed at phrasing, settle it with `validate_syntax` (above) rather than finding out mid-run.