Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ claude mcp add --transport http testrigor https://api2.testrigor.com/api/v1/mcp
--header "personal-access-token: <YOUR_PAT>" -s user
```

Other MCP-capable agents (Cursor, etc.) point their own MCP config at `https://api2.testrigor.com/api/v1/mcp` with the same `personal-access-token` header. The server can list/retrieve suites & cases, run test cases or a green regression, list runs, and read run failures. Guide: https://testrigor.com/how-to-utilise-testrigors-mcp-server/
Other MCP-capable agents (Cursor, etc.) point their own MCP config at `https://api2.testrigor.com/api/v1/mcp` with the same `personal-access-token` header. The server can list/retrieve suites & cases, validate step syntax without running, run test cases or a green regression, list runs, and read run failures. Guide: https://testrigor.com/how-to-utilise-testrigors-mcp-server/

## Runnable examples

Expand Down
35 changes: 30 additions & 5 deletions testrigor-cli/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -137,10 +137,11 @@ For the on-disk file schema (`.txt` vs `.yaml`, labels, datasets, `testCaseUuid`
## Command tree

```
testrigor authenticate # INTERACTIVE prompt — human-once only; never in automation
testrigor test-suite config # --default <id> | --show | --delete
testrigor test-suite run [ID] [flags] # push files + trigger a run
testrigor plugins # list installed plugins
testrigor authenticate # INTERACTIVE prompt — human-once only; never in automation
testrigor test-suite config # --default <id> | --show | --delete
testrigor test-suite run [ID] [flags] # push files + trigger a run
testrigor test-suite validate-syntax [ID] [flags] # check step syntax, no run
testrigor plugins # list installed plugins
testrigor help [command]
```

Expand Down Expand Up @@ -208,6 +209,22 @@ Reporting:
|------|---------|
| `--junit-report-save-path <file>` | Write a JUnit XML report (sync mode only) |

## `test-suite validate-syntax` flags

Checks step syntax against the suite's reusable rules without running anything — no test execution, no mutation of the suite. Exactly one of `--steps`/`--test-cases-path` is required.

| Flag | Purpose |
|------|---------|
| `--steps <text>` | Raw step text to validate directly (mutually exclusive with `--test-cases-path`) |
| `--test-cases-path <glob>` | Test case files to validate — each is checked independently and reported by name; one invalid file doesn't stop the rest, and the command exits non-zero if any fail |
| `--rules-path <glob>` | Ad hoc reusable rules to validate against, for rules not yet saved to this suite (same glob format as `test-suite run --rules-path`) |
| `--explicit-mutations` | Validate using only the supplied `--rules-path` rules instead of merging with the suite's saved rules |

```bash
testrigor test-suite validate-syntax "$TEST_SUITE_ID" --steps 'click "Sign in"'
testrigor test-suite validate-syntax "$TEST_SUITE_ID" --test-cases-path "test-cases/**/*.{txt,yaml,yml}" --rules-path "rules/**/*.yaml"
```

## Localhost debugging (the tunnel)

`--localhost --url http://localhost:3000` lets testRigor's cloud browsers reach an app running on your machine. Mechanics (so you know what to expect):
Expand Down Expand Up @@ -253,7 +270,7 @@ claude mcp add --transport http testrigor https://api2.testrigor.com/api/v1/mcp
--header "personal-access-token: <YOUR_PAT>" -s user
```

Other MCP-capable agents (Cursor, etc.) point their own MCP config at the same endpoint (`https://api2.testrigor.com/api/v1/mcp`) with the `personal-access-token` header. The server exposes tools to list/retrieve test suites & cases, run test cases or a green regression, list runs, and read run failures. Use the MCP when you want programmatic run/inspect from the agent; use this CLI for terminal/CI pipelines and the file-push (mutation) workflow.
Other MCP-capable agents (Cursor, etc.) point their own MCP config at the same endpoint (`https://api2.testrigor.com/api/v1/mcp`) with the `personal-access-token` header. The server exposes tools to list/retrieve test suites & cases, validate step syntax without running, run test cases or a green regression, list runs, and read run failures. Use the MCP when you want programmatic run/inspect from the agent; use this CLI for terminal/CI pipelines and the file-push (mutation) workflow.

> Guide: https://testrigor.com/how-to-utilise-testrigors-mcp-server/

Expand Down Expand Up @@ -299,6 +316,14 @@ testrigor test-suite run \
--explicit-mutations
```

Gate a commit on syntax before the real run (seconds instead of minutes):

```bash
testrigor test-suite validate-syntax "$TEST_SUITE_ID" \
--test-cases-path "test-cases/**/*.txt" \
--rules-path "rules/**/*.yaml"
```

Upload and test a mobile build:

```bash
Expand Down
10 changes: 5 additions & 5 deletions testrigor-dev-loop/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,12 +21,12 @@ There is **no separate "deploy" step.** When you run with `--test-cases-path` /
## The loop

```
edit files → run (localhost, --test-cases-path/--rules-path) → exit code + report.xml
▲ │
└──────────────────── fix failing steps ◀──────────────────────────────┘
edit files → validate syntax → run (--test-cases-path/--rules-path) → exit code + report.xml
└────────────────────────────── fix failing steps ◀──────────────────────────────┘
```

Stop when the exit code is `0` and `report.xml` shows no failures.
Validating first is optional but nearly free — it catches a mistyped command in a second instead of minutes into a run. See `testrigor-write-tests` → "Validate before you run". Stop when the exit code is `0` and `report.xml` shows no failures.

## 0. One-time setup

Expand Down Expand Up @@ -91,7 +91,7 @@ grep -A2 '<failure' testrigor/.run/report.xml # quick look at failures + re

Then fix the offending `.txt`/`.yaml` file and go back to step 2. Because re-running re-pushes the files, your fix updates the remote test automatically.

> **Cheap pre-check (runs take minutes).** After changing the app or a test, `grep` the page for the exact text and controls your tests assert before burning a full run — `grep -F "Thank you for your purchase!" end.html`. Catching a renamed label or a removed button statically is far faster than waiting on the cloud run.
> **Cheap pre-checks (runs take minutes).** Two are worth doing before burning a full run. **Syntax** — `validate_syntax`/`test-suite validate-syntax` tells you in a second whether every step parses (`testrigor-write-tests` → "Validate before you run"); if you just added a rule as a local file and haven't pushed it yet, pass it via `rules`/`--rules-path` or it'll be reported as unrecognized. **Content** — `grep` the page for the exact text and controls your tests assert, `grep -F "Thank you for your purchase!" end.html`. Catching a mistyped command or a renamed label statically is far faster than waiting on the cloud run.

## 4. "Update remote" is just re-running

Expand Down
41 changes: 39 additions & 2 deletions testrigor-write-tests/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
name: testrigor-write-tests
description: Author testRigor automated test cases in plain-English commands — navigation, clicks, typing, assertions, reusable rules, variables, data-driven tests, API steps (`call api`/`mock api`), and login/email/SMS/2FA flows. Use when writing, editing, or converting manual test steps into testRigor test cases (as `test-cases/*.txt` files or steps pasted into the testRigor app), or when asked how to express a step in testRigor's language. Full command catalog in `reference.md`; worked authoring examples in `examples.md`; API steps in `api-testing.md`. To run suites use the `testrigor-cli` skill.
description: Author testRigor automated test cases in plain-English commands — navigation, clicks, typing, assertions, reusable rules, variables, data-driven tests, API steps (`call api`/`mock api`), and login/email/SMS/2FA flows. Use when writing, editing, or converting manual test steps into testRigor test cases (as `test-cases/*.txt` files or steps pasted into the testRigor app), or when asked how to express a step in testRigor's language, or to check that steps parse without spending a run (`validate_syntax`). Full command catalog in `reference.md`; worked authoring examples in `examples.md`; API steps in `api-testing.md`. To run suites use the `testrigor-cli` skill.
---

# Writing testRigor Test Cases
Expand Down Expand Up @@ -263,6 +263,43 @@ To **run** these files (a run also pushes them into the remote suite, updating i

To call REST endpoints, extract JSON, assert on status codes, chain API + UI, or mock a dependency — all inside the same test case — see [api-testing.md](api-testing.md).

## Validate before you run

A run takes minutes; a syntax check takes a second. After writing or editing steps, confirm they parse before running.

**MCP** — call `validate_syntax` with the suite ID and the steps:

```
validate_syntax(testSuiteId: "<TEST_SUITE_ID>", steps: "click \"Sign in\"\ncheck that page contains \"Welcome\"")
```

Valid steps return `{"summary": "No issues found.", "issues": []}`; otherwise `issues` lists each command that didn't parse, each as a `{command, error}` pair.

**REST** — the same check without MCP. `auth-token` accepts a PAT or the suite's API token:

```bash
curl -s -X POST "https://api2.testrigor.com/api/v1/apps/$SUITE/validate-syntax" \
-H "Content-Type: application/json" -H "auth-token: $TESTRIGOR_API_KEY" \
-d '{"steps":"click \"Sign in\"\ncheck that page contains \"Welcome\""}'
```

**CLI** — `testrigor test-suite validate-syntax`, either an inline string or a batch of files:

```bash
testrigor test-suite validate-syntax "$SUITE" --token "$TESTRIGOR_API_KEY" --steps 'click "Sign in"'
testrigor test-suite validate-syntax "$SUITE" --token "$TESTRIGOR_API_KEY" --test-cases-path 'test_cases/**/*.{yaml,txt}'
```

With `--test-cases-path`, every matched file is validated independently and reported by name — one invalid file doesn't stop the rest from being checked, and the command exits non-zero if any file fails.

Things to know about the result:

- **The HTTP status is always `200`** once you're authenticated. The real verdict is the `status` field *inside* the body — `204` valid, `422` not valid. Don't gate on the HTTP code or a shell exit code.
- **The error message tells you how specific the failure is.** A recognized command that's malformed gets a precise message (e.g. `click` alone → "Missing reference to an element: click"). A command the parser can't recognize at all gets a generic fallback ("Unrecognizable instruction, can't detect action: `<command>`") — treat that one as "re-check this line against [reference.md](reference.md)," not as a specific diagnosis.
- **A rule referenced but not yet saved to the suite is reported as unrecognized — unless you supply it ad hoc.** Built-ins (`login`, `fill out form`) always parse, and a rule already saved to the suite resolves normally. For a rule that only exists as a local file, pass it explicitly: MCP/REST take an optional `rules` list (`[{name, steps, labels}]`, labels optional), the CLI takes `--rules-path` (same glob format as `test-suite run --rules-path`). By default the supplied rules merge with the suite's saved ones — a rule with the same name as an already-saved one is used in place of it, no conflict. Set `explicitMutations`/`--explicit-mutations` to validate against *only* the supplied rules, ignoring the suite's saved ones entirely.

Validation proves only that steps *parse*. It cannot tell you whether `"Sign in"` is actually on the page — that still takes a run.

## When you're unsure of the exact keyword

Read [reference.md](reference.md) — it lists every command category with exact syntax. testRigor is forgiving about phrasing, but using the documented keywords (`click`, `enter ... into ...`, `check that page contains ...`, `grab ... and save it as ...`) is the most reliable. If a step is genuinely visual or ambiguous, add `using AI` (e.g. `check that page "shows a success banner" using ai`).
Read [reference.md](reference.md) — it lists every command category with exact syntax. testRigor is forgiving about phrasing, but using the documented keywords (`click`, `enter ... into ...`, `check that page contains ...`, `grab ... and save it as ...`) is the most reliable. If a step is genuinely visual or ambiguous, add `using AI` (e.g. `check that page "shows a success banner" using ai`). When you've guessed at phrasing, settle it with `validate_syntax` (above) rather than finding out mid-run.