diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml.disabled similarity index 100% rename from .github/workflows/ci.yml rename to .github/workflows/ci.yml.disabled diff --git a/10-26 AI Engineering Night/AI Engineering Night - Oct 2026.pptx b/10-26 AI Engineering Night/AI Engineering Night - Oct 2026.pptx new file mode 100644 index 0000000..820e47a Binary files /dev/null and b/10-26 AI Engineering Night/AI Engineering Night - Oct 2026.pptx differ diff --git a/10-26 AI Engineering Night/demo/.env.example b/10-26 AI Engineering Night/demo/.env.example new file mode 100644 index 0000000..a57abe3 --- /dev/null +++ b/10-26 AI Engineering Night/demo/.env.example @@ -0,0 +1,37 @@ +# copy to .env and fill in. The scripts source it; stackql-deploy reads it with +# --env-file .env; the agent loads it itself. Never commit .env. + +# --- the model: OpenAI through the Vercel AI SDK -------------------------------- +OPENAI_API_KEY=sk-... +# gpt-6.1-sol keeps a live run short; gpt-6-astra for a more careful report +OPENAI_MODEL=gpt-6.1-sol + +# --- the estate: one Cloudflare zone, a throwaway one --------------------------- +# a token scoped to THIS zone only: Zone > Zone WAF > Edit, Zone > Zone > Read, +# Account > Account Analytics > Read (for the traffic query). Nothing else. +CLOUDFLARE_API_TOKEN= +CLOUDFLARE_ZONE_ID= +# printed by every script so the room (and you) can see which zone is in play +CLOUDFLARE_ZONE_NAME=stackql.xyz + +# --- the declaration's home on GitHub, for the provenance query ------------------ +# public repo, no token needed. owner/repo and the path of the manifest in it +DECLARATION_REPO=stackql-labs/agentic-platform-engineering +DECLARATION_PATH="10-26 AI Engineering Night/demo/stack/stackql_manifest.yml" + +# --- the platform: StackQL MCP server ------------------------------------------- +# read_only: writes refused by the server. safe: writes pause for operator approval. +# The agent code never starts a full_access server. +STACKQL_MCP_MODE=read_only +# where providers are pulled to; cloudflare and github must be there. Empty: stackql's own +# default (~/.stackql), and no --approot is passed +STACKQL_APPROOT= +# the stackql binary the MCP launcher runs. Empty: `stackql` on the PATH, nothing downloaded +STACKQL_MCP_BIN= +# the server's audit sink (one JSON record per tool call), the transcripts and the last +# proposal. Empty: audit/ under the directory the agent runs from (agent/audit with npm run) +STACKQL_AUDIT_DIR= +# cap on model steps per run +AGENT_MAX_STEPS=12 +# the stackql-deploy entry point. Empty: `stackql-deploy` on the Git Bash PATH +STACKQL_DEPLOY= diff --git a/10-26 AI Engineering Night/demo/.gitignore b/10-26 AI Engineering Night/demo/.gitignore new file mode 100644 index 0000000..cd98d79 --- /dev/null +++ b/10-26 AI Engineering Night/demo/.gitignore @@ -0,0 +1,8 @@ +.env +runs/ +audit/ +.stackql/ +agent/node_modules/ +agent/dist/ +*.log +.stackql-deploy-exports diff --git a/10-26 AI Engineering Night/demo/README.md b/10-26 AI Engineering Night/demo/README.md new file mode 100644 index 0000000..5945a5c --- /dev/null +++ b/10-26 AI Engineering Night/demo/README.md @@ -0,0 +1,103 @@ +# demo: the 3am rate limit + +Demo assets for "Query before mutation: guardrail patterns for agents that touch production infrastructure" (AI Engineering Night, Melbourne, 29 October 2026, 18 minutes including demo and Q&A). The talk makes the argument (the human pause is gone; put the judgement back into the system); this demo is where StackQL appears, as the implementation of the guardrails. + +## the scenario + +One Cloudflare zone (a throwaway, free plan), with one rate limit rule declared in a `stackql-deploy` manifest and asserted against the live API with no state file: block any client over 100 requests per 10 seconds. At 3am someone saw a burst of bot traffic, tightened the rule in the dashboard to 5, left a note in the description, and went back to bed. Every page load with more than five assets now gets a 429. + +That is drift, and it is hurting. A scheduled platform agent reviews the zone with the StackQL MCP server as its only tool surface, reads the live rule, reads when the declaration last changed (GitHub), reads the last hour of traffic, and decides. It can propose exactly one write: the assertion of the declared rule. Whether that write runs is decided by three gates the platform owns and the model cannot change. + +## how it maps to the talk + +| Guardrail (slide) | Where it shows in the demo | +| --- | --- | +| 01 Query before mutation | The agent's first statements are `SELECT`s against `cloudflare.rulesets.phases`, `github.repos.commits`, `cloudflare.zones.http_requests_adaptive_groups`. It proposes nothing until it has read live state, and it reads the declaration's history too. | +| 02 Idempotent assertions | `scripts/build.sh` is one `stackql-deploy build`: the createorupdate anchor runs the same REPLACE whether the rule is missing, drifted or already right, and the statecheck confirms it. The same command creates the rule and converges it after the drift. The agent's one permitted write is that same assertion, so a retry changes nothing. | +| 03 Policy gates | `agent/src/policy.ts`: four rules, in code, checked before the statement leaves the agent process. Then the server mode: `read_only` refuses, `safe` asks. Then a human, bound to the exact statement. | +| 04 Scoped credentials and mutation surfaces | The token is scoped to one zone and lives in the server process, never in the model's context. The mutation surface is one statement on one resource, given to the model verbatim; the policy declines anything else, field for field. | +| 05 Recovery paths | Blocked: the agent reports the statement and stops; it does not rephrase or retry. A human replays it through the gate (`npm run probe` in `safe`), or the declaration is re-asserted (`scripts/restore.sh`). Failed: re-assert; the build is idempotent. | + +## layout + +| Path | What it is | +| --- | --- | +| `stack/stackql_manifest.yml`, `stack/resources/rate_limit.iql` | the estate and the declaration: one rate limit rule, a post-build summary query | +| `agent/src/agent.ts` | the agent. Starts the MCP server in the mode the platform chose, renders the prompts with the declared rule and the permitted write, wraps the write tool with the policy, runs the loop, prints every statement, writes a transcript | +| `agent/src/policy.ts` | the policy gate: a pure function, `check(sql, declared)`, with `agent/test/policy.test.ts` | +| `agent/src/declaration.ts` | the declared rule and the permitted write, rendered from the manifest; `agent/test/declaration.test.ts` checks it against the `.iql` anchor field for field | +| `agent/src/console.ts` | the screen: colour, ASCII boxes and tables, the renderer for the model's markdown report | +| `agent/src/audit.ts` | `npm run audit`: every audit sink of the session as one table | +| `recordings/` | the transcript of the dry run, for `npm run replay` | +| `agent/prompts/system.md` | the briefing: the surface, the resource shapes, the one permitted write | +| `agent/prompts/intent.md` | the intent: the scheduled review, with `{{ placeholders }}` filled from `.env` and the manifest | +| `scripts/_env.sh` | sourced by the scripts: `.env`, defaults, refuses to run without a named zone | +| `scripts/build.sh` | `stackql-deploy build`: the rule, and the convergence | +| `scripts/drift.sh`, `scripts/drift.sql` | the 3am change, one statement | +| `scripts/restore.sh` | `build.sh` under the name the running order uses, then the rule read back | +| `scripts/shell.sh`, `scripts/shell.iql` | the human view of the same estate, for `stackql shell` | +| `scripts/denied.sql` | a plausible write that is not the declaration, for the policy beat | + +## prerequisites + +- `stackql` on PATH ([downloads](https://stackql.io/downloads)), with the `cloudflare` and `github` providers pulled (`stackql exec "REGISTRY PULL cloudflare"`, `stackql exec "REGISTRY PULL github"`). stackql's approot is `.stackql` under the directory it runs from: the first `scripts/build.sh` pulls `cloudflare` into the demo root's, `stackql shell` there needs `github` too, and the agent's MCP server uses `~/.stackql` (the launcher's default), which needs both. Or set `STACKQL_APPROOT` in `.env` and keep one store. +- `stackql-deploy` on the Git Bash PATH (`pip install stackql-deploy`); otherwise set `STACKQL_DEPLOY` in `.env` to the exe +- a Cloudflare zone you do not care about, on the free plan, and an API token scoped to that zone: Zone WAF Edit, Zone Read, Account Analytics Read +- Node 24 or later +- an OpenAI API key + +Windows notes: run the scripts from Git Bash. `stackql-deploy` needs `PYTHONUTF8=1` (the scripts set it). + +## setup + +```bash +cp .env.example .env # keys, zone, model +scripts/build.sh # the rule, from nothing, in a few seconds +scripts/drift.sh # the 3am change +cd agent && npm install +``` + +## running order + +Before the talk: rule built, `scripts/drift.sh` applied, `scripts/shell.sh` rendered, a terminal in `agent/` with a wide window. [runsheet.md](runsheet.md) is the on-screen walkthrough. + +| Beat | Command | What the room sees | Guardrail | +| --- | --- | --- | --- | +| The zone at 3am | `stackql shell`, paste from `runs/shell.iql` | threshold 5 (declared 100), the note in the description, `last_updated` at 3am; the declaration's last commit, older; `out_of_band = 1` from one query across two providers; 429s if there has been traffic | 01 | +| The agent, `read_only` | `npm run agent` | the declared table; the statements stream past: rule, commits, traffic; the diagnosis; the one write proposed; the server refuses it; the agent reports and stops | 01, 03, 04, 05 | +| The gate, `safe` | `STACKQL_MCP_MODE=safe npm run probe` | the agent's proposal replayed: policy passes it, the approval box, `y`, the rule is restored | 03, 05 | +| The policy | `npm run probe -- ../scripts/denied.sql` | threshold 1000: declined by policy, `$[0].ratelimit.requests_per_period: 1000 vs declared 100`, before the server or a human is involved | 03, 04 | +| The event log | `npm run audit` | every run of the session in one table: the statement, the mode, the decision, one record each, including the policy decline | 03 | +| Converge (optional) | `scripts/restore.sh` | `stackql-deploy build`: the same REPLACE runs, the statecheck confirms it, the rule is read back where the manifest says | 02, 05 | + +Say which mode the server is in before each run. + +## fallback + +`npm run replay` replays the newest transcript from `agent/audit/` (or `recordings/`) with the pacing of a run, under a banner that says it is a replay. Use it when there is no network or no key. Do not present it as a live run. + +## what it proves, and does not + +- It demonstrates the five guardrails on one public cloud (Cloudflare) plus one SaaS provider (GitHub), with the policy enforced in code, the mode enforced by the server, and the convergence done by a declarative build. +- A query is an observation, not a lock. The agent re-reads after a change; nothing here is an atomic precondition. +- The approval at the terminal is the MCP elicitation flow answered by whoever is at the keyboard. It binds the answer to the exact statement; it is not an authenticated production approval service. +- The agent's blast radius is the token's: one zone, WAF edit. A broader token would widen it silently; the policy would still hold, the token is the backstop. +- The traffic query needs traffic. Zero rows is a valid answer and the agent is told to say so. +- `safe` trusts the MCP client to show the prompt to a person; against an untrusted client use `read_only` or a narrowed tool set. + +## status + +Run end to end on 29 October 2026 on Windows 11 (Git Bash for the scripts, PowerShell for the agent) with stackql v0.12.824, `@stackql/mcp-server` 0.12.824, the cloudflare provider v26.07.00424, github v26.08.00448, stackql-deploy 2.4.0, Node 24.19.0, `ai` 7.0.154, `@ai-sdk/openai` 4.0.102, `@ai-sdk/mcp` 2.0.85 and `gpt-6.1-sol`. `npm run typecheck` and `npm test` (18 tests) pass; the `.iql` anchor and the agent's rendered statement are checked field for field. + +| Beat | Wall time | +| --- | --- | +| `scripts/build.sh`, create or converge | 15 to 17 s, of which 2 to 3 s is the deploy; the rest is stackql-deploy starting its server | +| `scripts/drift.sh` | about 1 s | +| `npm run tools` | about 1 s | +| `npm run agent`, `read_only` | 25 to 30 s, 5 steps, 4 tool calls, about 46k tokens in | +| `npm run probe`, `safe`, approved | about 7 s plus the pause at the prompt | +| `npm run probe -- ../scripts/denied.sql` | about 2 s | +| `npm run audit` | about 1 s | +| `npm run replay` | about 10 s | + +`recordings/agent_dry-run_2026-10-29.txt` is the `read_only` run against the drifted zone, with the GitHub query returning zero rows because the folder was not yet on origin. The zone was left drifted, ready for the talk. The server's three start-up lines on stderr are dropped from the screen; anything else it writes there still shows. diff --git a/10-26 AI Engineering Night/demo/agent/package-lock.json b/10-26 AI Engineering Night/demo/agent/package-lock.json new file mode 100644 index 0000000..9982f80 --- /dev/null +++ b/10-26 AI Engineering Night/demo/agent/package-lock.json @@ -0,0 +1,886 @@ +{ + "name": "edge-agent", + "version": "0.1.0", + "lockfileVersion": 3, + "requires": true, + "packages": { + "": { + "name": "edge-agent", + "version": "0.1.0", + "dependencies": { + "@ai-sdk/mcp": "2.0.85", + "@ai-sdk/openai": "4.0.102", + "@stackql/mcp-server": "0.12.824", + "ai": "7.0.154", + "yaml": "2.9.1" + }, + "devDependencies": { + "@types/node": "24.0.0", + "tsx": "4.20.0", + "typescript": "5.9.3" + }, + "engines": { + "node": ">=24.19.0" + } + }, + "node_modules/@ai-sdk/gateway": { + "version": "4.0.127", + "resolved": "https://registry.npmjs.org/@ai-sdk/gateway/-/gateway-4.0.127.tgz", + "integrity": "sha512-ENcIQNrZxgFnNSx29X2o/mLKZRzMN91c53Qlp/UrF4RQGVSfaROjFQAq4eeJliRPWwjHvEhJYG42IrOwtWUGOQ==", + "license": "Apache-2.0", + "dependencies": { + "@ai-sdk/provider": "4.0.34", + "@ai-sdk/provider-utils": "5.0.69", + "@vercel/oidc": "3.2.0" + }, + "engines": { + "node": ">=22" + }, + "peerDependencies": { + "zod": "^3.25.76 || ^4.1.8" + } + }, + "node_modules/@ai-sdk/mcp": { + "version": "2.0.85", + "resolved": "https://registry.npmjs.org/@ai-sdk/mcp/-/mcp-2.0.85.tgz", + "integrity": "sha512-I3l10JGrsjYvAnliCENjZ40UYes/UfQfkJNUzKnusHCihu051aK56WfYGnUddKaz6GEKMD68dgPJzSJV9zoSiQ==", + "license": "Apache-2.0", + "dependencies": { + "@ai-sdk/provider": "4.0.34", + "@ai-sdk/provider-utils": "5.0.69", + "cross-spawn": "^7.0.6", + "pkce-challenge": "^5.0.1" + }, + "engines": { + "node": ">=22" + }, + "peerDependencies": { + "zod": "^3.25.76 || ^4.1.8" + } + }, + "node_modules/@ai-sdk/openai": { + "version": "4.0.102", + "resolved": "https://registry.npmjs.org/@ai-sdk/openai/-/openai-4.0.102.tgz", + "integrity": "sha512-8hvs5UO6OgGSR0HASptyopqoJYgdCccVfksUlBLeJ8/h2KoKYU2ZiFjArOkS6ciJbmnDz3G+Zoo4KiI4rlQSoA==", + "license": "Apache-2.0", + "dependencies": { + "@ai-sdk/provider": "4.0.34", + "@ai-sdk/provider-utils": "5.0.68" + }, + "engines": { + "node": ">=22" + }, + "peerDependencies": { + "zod": "^3.25.76 || ^4.1.8" + } + }, + "node_modules/@ai-sdk/openai/node_modules/@ai-sdk/provider-utils": { + "version": "5.0.68", + "resolved": "https://registry.npmjs.org/@ai-sdk/provider-utils/-/provider-utils-5.0.68.tgz", + "integrity": "sha512-SASuC1DqSdWoYyYfpxbgPKm9VCnMP1VTahWY7WEYzi2QDmFdbw4runqmH7NKC33iB/INo+XbaN2hH3dXHSNQZw==", + "license": "Apache-2.0", + "dependencies": { + "@ai-sdk/provider": "4.0.34", + "@standard-schema/spec": "^1.1.0", + "@workflow/serde": "4.1.0", + "eventsource-parser": "^3.0.8", + "undici": "^7.29.0" + }, + "engines": { + "node": ">=22" + }, + "peerDependencies": { + "zod": "^3.25.76 || ^4.1.8" + } + }, + "node_modules/@ai-sdk/provider": { + "version": "4.0.34", + "resolved": "https://registry.npmjs.org/@ai-sdk/provider/-/provider-4.0.34.tgz", + "integrity": "sha512-Ip7ZQBq/FiGpM1mmOVseTtX9PBjnzo0bLC41WTgBSBBRUBz6LhNFGbYaNUzI6hPm6ll1oNSPOAZc/q/z+3QZEQ==", + "license": "Apache-2.0", + "dependencies": { + "json-schema": "^0.4.0" + }, + "engines": { + "node": ">=22" + } + }, + "node_modules/@ai-sdk/provider-utils": { + "version": "5.0.69", + "resolved": "https://registry.npmjs.org/@ai-sdk/provider-utils/-/provider-utils-5.0.69.tgz", + "integrity": "sha512-oGCXIoBKLpNrrAzJvltFHdBl20FJMyT+7edD9D4nwxoI8Scbj3y1EgKzjoF2XFhNYkMTcv1oTFDgUzLxdjqSKg==", + "license": "Apache-2.0", + "dependencies": { + "@ai-sdk/provider": "4.0.34", + "@standard-schema/spec": "^1.1.0", + "@workflow/serde": "4.1.0", + "eventsource-parser": "^3.0.8", + "undici": "^7.29.0" + }, + "engines": { + "node": ">=22" + }, + "peerDependencies": { + "zod": "^3.25.76 || ^4.1.8" + } + }, + "node_modules/@esbuild/aix-ppc64": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/aix-ppc64/-/aix-ppc64-0.25.12.tgz", + "integrity": "sha512-Hhmwd6CInZ3dwpuGTF8fJG6yoWmsToE+vYgD4nytZVxcu1ulHpUQRAB1UJ8+N1Am3Mz4+xOByoQoSZf4D+CpkA==", + "cpu": [ + "ppc64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "aix" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/android-arm": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/android-arm/-/android-arm-0.25.12.tgz", + "integrity": "sha512-VJ+sKvNA/GE7Ccacc9Cha7bpS8nyzVv0jdVgwNDaR4gDMC/2TTRc33Ip8qrNYUcpkOHUT5OZ0bUcNNVZQ9RLlg==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/android-arm64": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/android-arm64/-/android-arm64-0.25.12.tgz", + "integrity": "sha512-6AAmLG7zwD1Z159jCKPvAxZd4y/VTO0VkprYy+3N2FtJ8+BQWFXU+OxARIwA46c5tdD9SsKGZ/1ocqBS/gAKHg==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/android-x64": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/android-x64/-/android-x64-0.25.12.tgz", + "integrity": "sha512-5jbb+2hhDHx5phYR2By8GTWEzn6I9UqR11Kwf22iKbNpYrsmRB18aX/9ivc5cabcUiAT/wM+YIZ6SG9QO6a8kg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/darwin-arm64": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/darwin-arm64/-/darwin-arm64-0.25.12.tgz", + "integrity": "sha512-N3zl+lxHCifgIlcMUP5016ESkeQjLj/959RxxNYIthIg+CQHInujFuXeWbWMgnTo4cp5XVHqFPmpyu9J65C1Yg==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/darwin-x64": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/darwin-x64/-/darwin-x64-0.25.12.tgz", + "integrity": "sha512-HQ9ka4Kx21qHXwtlTUVbKJOAnmG1ipXhdWTmNXiPzPfWKpXqASVcWdnf2bnL73wgjNrFXAa3yYvBSd9pzfEIpA==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/freebsd-arm64": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/freebsd-arm64/-/freebsd-arm64-0.25.12.tgz", + "integrity": "sha512-gA0Bx759+7Jve03K1S0vkOu5Lg/85dou3EseOGUes8flVOGxbhDDh/iZaoek11Y8mtyKPGF3vP8XhnkDEAmzeg==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/freebsd-x64": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/freebsd-x64/-/freebsd-x64-0.25.12.tgz", + "integrity": "sha512-TGbO26Yw2xsHzxtbVFGEXBFH0FRAP7gtcPE7P5yP7wGy7cXK2oO7RyOhL5NLiqTlBh47XhmIUXuGciXEqYFfBQ==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-arm": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/linux-arm/-/linux-arm-0.25.12.tgz", + "integrity": "sha512-lPDGyC1JPDou8kGcywY0YILzWlhhnRjdof3UlcoqYmS9El818LLfJJc3PXXgZHrHCAKs/Z2SeZtDJr5MrkxtOw==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-arm64": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/linux-arm64/-/linux-arm64-0.25.12.tgz", + "integrity": "sha512-8bwX7a8FghIgrupcxb4aUmYDLp8pX06rGh5HqDT7bB+8Rdells6mHvrFHHW2JAOPZUbnjUpKTLg6ECyzvas2AQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-ia32": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/linux-ia32/-/linux-ia32-0.25.12.tgz", + "integrity": "sha512-0y9KrdVnbMM2/vG8KfU0byhUN+EFCny9+8g202gYqSSVMonbsCfLjUO+rCci7pM0WBEtz+oK/PIwHkzxkyharA==", + "cpu": [ + "ia32" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-loong64": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/linux-loong64/-/linux-loong64-0.25.12.tgz", + "integrity": "sha512-h///Lr5a9rib/v1GGqXVGzjL4TMvVTv+s1DPoxQdz7l/AYv6LDSxdIwzxkrPW438oUXiDtwM10o9PmwS/6Z0Ng==", + "cpu": [ + "loong64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-mips64el": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/linux-mips64el/-/linux-mips64el-0.25.12.tgz", + "integrity": "sha512-iyRrM1Pzy9GFMDLsXn1iHUm18nhKnNMWscjmp4+hpafcZjrr2WbT//d20xaGljXDBYHqRcl8HnxbX6uaA/eGVw==", + "cpu": [ + "mips64el" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-ppc64": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/linux-ppc64/-/linux-ppc64-0.25.12.tgz", + "integrity": "sha512-9meM/lRXxMi5PSUqEXRCtVjEZBGwB7P/D4yT8UG/mwIdze2aV4Vo6U5gD3+RsoHXKkHCfSxZKzmDssVlRj1QQA==", + "cpu": [ + "ppc64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-riscv64": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/linux-riscv64/-/linux-riscv64-0.25.12.tgz", + "integrity": "sha512-Zr7KR4hgKUpWAwb1f3o5ygT04MzqVrGEGXGLnj15YQDJErYu/BGg+wmFlIDOdJp0PmB0lLvxFIOXZgFRrdjR0w==", + "cpu": [ + "riscv64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-s390x": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/linux-s390x/-/linux-s390x-0.25.12.tgz", + "integrity": "sha512-MsKncOcgTNvdtiISc/jZs/Zf8d0cl/t3gYWX8J9ubBnVOwlk65UIEEvgBORTiljloIWnBzLs4qhzPkJcitIzIg==", + "cpu": [ + "s390x" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-x64": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/linux-x64/-/linux-x64-0.25.12.tgz", + "integrity": "sha512-uqZMTLr/zR/ed4jIGnwSLkaHmPjOjJvnm6TVVitAa08SLS9Z0VM8wIRx7gWbJB5/J54YuIMInDquWyYvQLZkgw==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/netbsd-arm64": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/netbsd-arm64/-/netbsd-arm64-0.25.12.tgz", + "integrity": "sha512-xXwcTq4GhRM7J9A8Gv5boanHhRa/Q9KLVmcyXHCTaM4wKfIpWkdXiMog/KsnxzJ0A1+nD+zoecuzqPmCRyBGjg==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "netbsd" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/netbsd-x64": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/netbsd-x64/-/netbsd-x64-0.25.12.tgz", + "integrity": "sha512-Ld5pTlzPy3YwGec4OuHh1aCVCRvOXdH8DgRjfDy/oumVovmuSzWfnSJg+VtakB9Cm0gxNO9BzWkj6mtO1FMXkQ==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "netbsd" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/openbsd-arm64": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/openbsd-arm64/-/openbsd-arm64-0.25.12.tgz", + "integrity": "sha512-fF96T6KsBo/pkQI950FARU9apGNTSlZGsv1jZBAlcLL1MLjLNIWPBkj5NlSz8aAzYKg+eNqknrUJ24QBybeR5A==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "openbsd" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/openbsd-x64": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/openbsd-x64/-/openbsd-x64-0.25.12.tgz", + "integrity": "sha512-MZyXUkZHjQxUvzK7rN8DJ3SRmrVrke8ZyRusHlP+kuwqTcfWLyqMOE3sScPPyeIXN/mDJIfGXvcMqCgYKekoQw==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "openbsd" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/openharmony-arm64": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/openharmony-arm64/-/openharmony-arm64-0.25.12.tgz", + "integrity": "sha512-rm0YWsqUSRrjncSXGA7Zv78Nbnw4XL6/dzr20cyrQf7ZmRcsovpcRBdhD43Nuk3y7XIoW2OxMVvwuRvk9XdASg==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "openharmony" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/sunos-x64": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/sunos-x64/-/sunos-x64-0.25.12.tgz", + "integrity": "sha512-3wGSCDyuTHQUzt0nV7bocDy72r2lI33QL3gkDNGkod22EsYl04sMf0qLb8luNKTOmgF/eDEDP5BFNwoBKH441w==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "sunos" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/win32-arm64": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/win32-arm64/-/win32-arm64-0.25.12.tgz", + "integrity": "sha512-rMmLrur64A7+DKlnSuwqUdRKyd3UE7oPJZmnljqEptesKM8wx9J8gx5u0+9Pq0fQQW8vqeKebwNXdfOyP+8Bsg==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/win32-ia32": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/win32-ia32/-/win32-ia32-0.25.12.tgz", + "integrity": "sha512-HkqnmmBoCbCwxUKKNPBixiWDGCpQGVsrQfJoVGYLPT41XWF8lHuE5N6WhVia2n4o5QK5M4tYr21827fNhi4byQ==", + "cpu": [ + "ia32" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/win32-x64": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/win32-x64/-/win32-x64-0.25.12.tgz", + "integrity": "sha512-alJC0uCZpTFrSL0CCDjcgleBXPnCrEAhTBILpeAp7M/OFgoqtAetfBzX0xM00MUsVVPpVjlPuMbREqnZCXaTnA==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@stackql/mcp-server": { + "version": "0.12.824", + "resolved": "https://registry.npmjs.org/@stackql/mcp-server/-/mcp-server-0.12.824.tgz", + "integrity": "sha512-JGJrqaX79vTrqWmjBm4QxCneGApqLovbSZy7FkFaZ6EIxI7y65sjqBpUwv2OxTrCjhtw/WEZh09wHrkoo7uVzA==", + "license": "MIT", + "dependencies": { + "adm-zip": "^0.6.1" + }, + "bin": { + "stackql-mcp": "bin/stackql-mcp.js" + }, + "engines": { + "node": "^22.23.4 || ^24.22.1 || >=26.12.1" + } + }, + "node_modules/@standard-schema/spec": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/@standard-schema/spec/-/spec-1.1.0.tgz", + "integrity": "sha512-l2aFy5jALhniG5HgqrD6jXLi/rUWrKvqN/qJx6yoJsgKhblVd+iqqU4RCXavm/jPityDo5TCvKMnpjKnOriy0w==", + "license": "MIT" + }, + "node_modules/@types/node": { + "version": "24.0.0", + "resolved": "https://registry.npmjs.org/@types/node/-/node-24.0.0.tgz", + "integrity": "sha512-yZQa2zm87aRVcqDyH5+4Hv9KYgSdgwX1rFnGvpbzMaC7YAljmhBET93TPiTd3ObwTL+gSpIzPKg5BqVxdCvxKg==", + "dev": true, + "license": "MIT", + "dependencies": { + "undici-types": "~7.8.0" + } + }, + "node_modules/@vercel/oidc": { + "version": "3.2.0", + "resolved": "https://registry.npmjs.org/@vercel/oidc/-/oidc-3.2.0.tgz", + "integrity": "sha512-UycprH3T6n3jH0k44NHMa7pnFHGu/N05MjojYr+Mc6I7obkoLIJujSWwin1pCvdy/eOxrI/l3uDLQsmcrOb4ug==", + "license": "Apache-2.0", + "engines": { + "node": ">= 20" + } + }, + "node_modules/@workflow/serde": { + "version": "4.1.0", + "resolved": "https://registry.npmjs.org/@workflow/serde/-/serde-4.1.0.tgz", + "integrity": "sha512-pav4F2BoirECWR7Nf1TKt+2eETcBj7jj4cBefQ8VXQCA6NPkaKeLfj/zMgi+3zYV5ZIBT4GuUiphsj0/b9hPQQ==", + "license": "Apache-2.0" + }, + "node_modules/adm-zip": { + "version": "0.6.1", + "resolved": "https://registry.npmjs.org/adm-zip/-/adm-zip-0.6.1.tgz", + "integrity": "sha512-Xwrja8nx9e5o2N1my4DsKCeKpdrnACyr1wtbPxBDgGzKzKyE9kRtBFA8mWldI+RVlD7CBZNWY/wQ2+ydwOR6kQ==", + "license": "MIT", + "engines": { + "node": ">=14.0" + } + }, + "node_modules/ai": { + "version": "7.0.154", + "resolved": "https://registry.npmjs.org/ai/-/ai-7.0.154.tgz", + "integrity": "sha512-vpvOZoV2Xcu6vWYM/X2XAFJG93T/ELF+qbslXxRnpqqhKZ6dtxfZf1KHFqYBz784uV/XVjXkNjgs9Zc239NY3Q==", + "license": "Apache-2.0", + "dependencies": { + "@ai-sdk/gateway": "4.0.127", + "@ai-sdk/provider": "4.0.34", + "@ai-sdk/provider-utils": "5.0.69" + }, + "engines": { + "node": ">=22" + }, + "peerDependencies": { + "zod": "^3.25.76 || ^4.1.8" + } + }, + "node_modules/cross-spawn": { + "version": "7.0.6", + "resolved": "https://registry.npmjs.org/cross-spawn/-/cross-spawn-7.0.6.tgz", + "integrity": "sha512-uV2QOWP2nWzsy2aMp8aRibhi9dlzF5Hgh5SHaB9OiTGEyDTiJJyx0uy51QXdyWbtAHNua4XJzUKca3OzKUd3vA==", + "license": "MIT", + "dependencies": { + "path-key": "^3.1.0", + "shebang-command": "^2.0.0", + "which": "^2.0.1" + }, + "engines": { + "node": ">= 8" + } + }, + "node_modules/esbuild": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/esbuild/-/esbuild-0.25.12.tgz", + "integrity": "sha512-bbPBYYrtZbkt6Os6FiTLCTFxvq4tt3JKall1vRwshA3fdVztsLAatFaZobhkBC8/BrPetoa0oksYoKXoG4ryJg==", + "dev": true, + "hasInstallScript": true, + "license": "MIT", + "bin": { + "esbuild": "bin/esbuild" + }, + "engines": { + "node": ">=18" + }, + "optionalDependencies": { + "@esbuild/aix-ppc64": "0.25.12", + "@esbuild/android-arm": "0.25.12", + "@esbuild/android-arm64": "0.25.12", + "@esbuild/android-x64": "0.25.12", + "@esbuild/darwin-arm64": "0.25.12", + "@esbuild/darwin-x64": "0.25.12", + "@esbuild/freebsd-arm64": "0.25.12", + "@esbuild/freebsd-x64": "0.25.12", + "@esbuild/linux-arm": "0.25.12", + "@esbuild/linux-arm64": "0.25.12", + "@esbuild/linux-ia32": "0.25.12", + "@esbuild/linux-loong64": "0.25.12", + "@esbuild/linux-mips64el": "0.25.12", + "@esbuild/linux-ppc64": "0.25.12", + "@esbuild/linux-riscv64": "0.25.12", + "@esbuild/linux-s390x": "0.25.12", + "@esbuild/linux-x64": "0.25.12", + "@esbuild/netbsd-arm64": "0.25.12", + "@esbuild/netbsd-x64": "0.25.12", + "@esbuild/openbsd-arm64": "0.25.12", + "@esbuild/openbsd-x64": "0.25.12", + "@esbuild/openharmony-arm64": "0.25.12", + "@esbuild/sunos-x64": "0.25.12", + "@esbuild/win32-arm64": "0.25.12", + "@esbuild/win32-ia32": "0.25.12", + "@esbuild/win32-x64": "0.25.12" + } + }, + "node_modules/eventsource-parser": { + "version": "3.1.1", + "resolved": "https://registry.npmjs.org/eventsource-parser/-/eventsource-parser-3.1.1.tgz", + "integrity": "sha512-EKN1vKAMcZ8MlYMpaNuxN6R9yakzH6uajHcHVTqWJzvu5pWw9DyhbP35HH8MVBQ+dZjAfDxk+A8NiR9KWaXiyQ==", + "license": "MIT", + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/fsevents": { + "version": "2.3.3", + "resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.3.tgz", + "integrity": "sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==", + "dev": true, + "hasInstallScript": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": "^8.16.0 || ^10.6.0 || >=11.0.0" + } + }, + "node_modules/get-tsconfig": { + "version": "4.14.3", + "resolved": "https://registry.npmjs.org/get-tsconfig/-/get-tsconfig-4.14.3.tgz", + "integrity": "sha512-++QEw4DIY7WGoukz+/+A/8dGYPT9l9yIadnmSgZ8Rjr3YVSVDipQSO9CdnJo9ePqFqUUqh+wk9uIaoiAwsiPkA==", + "dev": true, + "license": "MIT", + "dependencies": { + "resolve-pkg-maps": "^1.0.0" + }, + "funding": { + "url": "https://github.com/privatenumber/get-tsconfig?sponsor=1" + } + }, + "node_modules/isexe": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/isexe/-/isexe-2.0.0.tgz", + "integrity": "sha512-RHxMLp9lnKHGHRng9QFhRCMbYAcVpn69smSGcq3f36xjgVVWThj4qqLbTLlq7Ssj8B+fIQ1EuCEGI2lKsyQeIw==", + "license": "ISC" + }, + "node_modules/json-schema": { + "version": "0.4.0", + "resolved": "https://registry.npmjs.org/json-schema/-/json-schema-0.4.0.tgz", + "integrity": "sha512-es94M3nTIfsEPisRafak+HDLfHXnKBhV3vU5eqPcS3flIWqcxJWgXHXiey3YrpaNsanY5ei1VoYEbOzijuq9BA==", + "license": "(AFL-2.1 OR BSD-3-Clause)" + }, + "node_modules/path-key": { + "version": "3.1.1", + "resolved": "https://registry.npmjs.org/path-key/-/path-key-3.1.1.tgz", + "integrity": "sha512-ojmeN0qd+y0jszEtoY48r0Peq5dwMEkIlCOu6Q5f41lfkswXuKtYrhgoTpLnyIcHm24Uhqx+5Tqm2InSwLhE6Q==", + "license": "MIT", + "engines": { + "node": ">=8" + } + }, + "node_modules/pkce-challenge": { + "version": "5.0.1", + "resolved": "https://registry.npmjs.org/pkce-challenge/-/pkce-challenge-5.0.1.tgz", + "integrity": "sha512-wQ0b/W4Fr01qtpHlqSqspcj3EhBvimsdh0KlHhH8HRZnMsEa0ea2fTULOXOS9ccQr3om+GcGRk4e+isrZWV8qQ==", + "license": "MIT", + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/resolve-pkg-maps": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/resolve-pkg-maps/-/resolve-pkg-maps-1.0.0.tgz", + "integrity": "sha512-seS2Tj26TBVOC2NIc2rOe2y2ZO7efxITtLZcGSOnHHNOQ7CkiUBfw0Iw2ck6xkIhPwLhKNLS8BO+hEpngQlqzw==", + "dev": true, + "license": "MIT", + "funding": { + "url": "https://github.com/privatenumber/resolve-pkg-maps?sponsor=1" + } + }, + "node_modules/shebang-command": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/shebang-command/-/shebang-command-2.0.0.tgz", + "integrity": "sha512-kHxr2zZpYtdmrN1qDjrrX/Z1rR1kG8Dx+gkpK1G4eXmvXswmcE1hTWBWYUzlraYw1/yZp6YuDY77YtvbN0dmDA==", + "license": "MIT", + "dependencies": { + "shebang-regex": "^3.0.0" + }, + "engines": { + "node": ">=8" + } + }, + "node_modules/shebang-regex": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/shebang-regex/-/shebang-regex-3.0.0.tgz", + "integrity": "sha512-7++dFhtcx3353uBaq8DDR4NuxBetBzC7ZQOhmTQInHEd6bSrXdiEyzCvG07Z44UYdLShWUyXt5M/yhz8ekcb1A==", + "license": "MIT", + "engines": { + "node": ">=8" + } + }, + "node_modules/tsx": { + "version": "4.20.0", + "resolved": "https://registry.npmjs.org/tsx/-/tsx-4.20.0.tgz", + "integrity": "sha512-TsmdeXxcZYiJ2MKV7fdq38na0CKyLRtCeMqTeHMmrVXQ/gf5yTeJohh+sgr7MnMGsjeKXzHzy+TwOOTR1arl+Q==", + "dev": true, + "license": "MIT", + "dependencies": { + "esbuild": "~0.25.0", + "get-tsconfig": "^4.7.5" + }, + "bin": { + "tsx": "dist/cli.mjs" + }, + "engines": { + "node": ">=18.0.0" + }, + "optionalDependencies": { + "fsevents": "~2.3.3" + } + }, + "node_modules/typescript": { + "version": "5.9.3", + "resolved": "https://registry.npmjs.org/typescript/-/typescript-5.9.3.tgz", + "integrity": "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw==", + "dev": true, + "license": "Apache-2.0", + "bin": { + "tsc": "bin/tsc", + "tsserver": "bin/tsserver" + }, + "engines": { + "node": ">=14.17" + } + }, + "node_modules/undici": { + "version": "7.30.1", + "resolved": "https://registry.npmjs.org/undici/-/undici-7.30.1.tgz", + "integrity": "sha512-03KrMbjIFDWQR5RNPrSw3bd1r12JpyFQl9/yxjUbJtqkjmllqC5lFjZh6ujeuJCKr8EOQvRW872Oy/az+nLYxg==", + "license": "MIT", + "engines": { + "node": ">=20.18.1" + } + }, + "node_modules/undici-types": { + "version": "7.8.0", + "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-7.8.0.tgz", + "integrity": "sha512-9UJ2xGDvQ43tYyVMpuHlsgApydB8ZKfVYTsLDhXkFL/6gfkp+U8xTGdh8pMJv1SpZna0zxG1DwsKZsreLbXBxw==", + "dev": true, + "license": "MIT" + }, + "node_modules/which": { + "version": "2.0.2", + "resolved": "https://registry.npmjs.org/which/-/which-2.0.2.tgz", + "integrity": "sha512-BLI3Tl1TW3Pvl70l3yq3Y64i+awpwXqsGBYWkkqMtnbXgrMD+yj7rhW0kuEDxzJaYXGjEW5ogapKNMEKNMjibA==", + "license": "ISC", + "dependencies": { + "isexe": "^2.0.0" + }, + "bin": { + "node-which": "bin/node-which" + }, + "engines": { + "node": ">= 8" + } + }, + "node_modules/yaml": { + "version": "2.9.1", + "resolved": "https://registry.npmjs.org/yaml/-/yaml-2.9.1.tgz", + "integrity": "sha512-3NxN8+78OdzbT7C/WjGsyfPAtJaN3FNDsWxv7Y7mcDsT/oOmgW8BpyQQFFBnvZE3j9Y2Sdz1ULFLezL7Eb2yFw==", + "license": "ISC", + "bin": { + "yaml": "bin.mjs" + }, + "engines": { + "node": ">= 14.6" + }, + "funding": { + "url": "https://github.com/sponsors/eemeli" + } + }, + "node_modules/zod": { + "version": "4.6.7", + "resolved": "https://registry.npmjs.org/zod/-/zod-4.6.7.tgz", + "integrity": "sha512-g8W2D2kU6a8untc053fX1uVBKzM7yqAxW/3zwk83+6RyF2+SKOuV+c0wBDhUwzV1PaGapmZH+rmcUzPf+3CGwg==", + "license": "MIT", + "peer": true, + "funding": { + "url": "https://github.com/sponsors/colinhacks" + } + } + } +} diff --git a/10-26 AI Engineering Night/demo/agent/package.json b/10-26 AI Engineering Night/demo/agent/package.json new file mode 100644 index 0000000..a142e59 --- /dev/null +++ b/10-26 AI Engineering Night/demo/agent/package.json @@ -0,0 +1,31 @@ +{ + "name": "edge-agent", + "version": "0.1.0", + "private": true, + "type": "module", + "description": "A scheduled platform agent: Vercel AI SDK + OpenAI + the StackQL MCP server, behind a policy gate", + "engines": { + "node": ">=24.19.0" + }, + "scripts": { + "typecheck": "tsc --noEmit", + "agent": "tsx src/agent.ts", + "tools": "tsx src/agent.ts --list-tools", + "probe": "tsx src/agent.ts --probe-write", + "replay": "tsx src/agent.ts --replay", + "audit": "tsx src/audit.ts", + "test": "tsx --test test/policy.test.ts test/declaration.test.ts test/console.test.ts" + }, + "dependencies": { + "@ai-sdk/mcp": "2.0.85", + "@ai-sdk/openai": "4.0.102", + "@stackql/mcp-server": "0.12.824", + "ai": "7.0.154", + "yaml": "2.9.1" + }, + "devDependencies": { + "@types/node": "24.0.0", + "tsx": "4.20.0", + "typescript": "5.9.3" + } +} diff --git a/10-26 AI Engineering Night/demo/agent/prompts/intent.md b/10-26 AI Engineering Night/demo/agent/prompts/intent.md new file mode 100644 index 0000000..3950ce9 --- /dev/null +++ b/10-26 AI Engineering Night/demo/agent/prompts/intent.md @@ -0,0 +1,37 @@ +Scheduled review of the rate limit on the Cloudflare zone `{{ zone_name }}` +(`zone_id = '{{ zone_id }}'`), through the `cloudflare` provider. + +Declared intent for this zone, from the stack manifest the platform team maintains. This is what +should be: + +{{ declared_table }} + +The manifest lives at `{{ declaration_path }}` in the GitHub repository `{{ owner }}/{{ repo }}`. + +Intent: establish the live state before anything else, compare it with what is declared, +decide what to do about the difference, and act only where the platform lets you. + +1. Read the live rule: threshold, period, enabled, description, and `last_updated`. +2. Read when the declaration last changed: the newest commits to the manifest path, with + author, date and message. Zero rows means no history is visible for that path; say so and + treat the declaration you were given as current. +3. Read the last hour of traffic by edge status, for the effect of the rule on real clients. + Zero rows means no traffic in the window; say so. + +Then classify the rule: + +- converged: live matches declared. Nothing to do. Report and stop. +- drifted, declared: the declaration itself changed after the rule did (the newest commit is + newer than `last_updated`). Your copy of the declaration may be stale. Do not write. + Escalate: report both timestamps and stop. +- drifted, out of band: live differs from declared and the rule changed after the declaration + last did. Say what changed, when, and what the description says about why. Then remediate: + issue the permitted write once, exactly as given to you, with `run_mutation_query`. The + platform decides whether it runs; report what happened: executed, declined by policy, refused + by the server mode, or declined by the operator. + +Decide, do not just detect: the status table ends with one of remediate, escalate or accept, +and one line of why. + +Scope and guardrails: this zone only. Only SELECTs for diagnosis. One write at most, the one +you were given, and only after you have read the live state. diff --git a/10-26 AI Engineering Night/demo/agent/prompts/system.md b/10-26 AI Engineering Night/demo/agent/prompts/system.md new file mode 100644 index 0000000..332b614 --- /dev/null +++ b/10-26 AI Engineering Night/demo/agent/prompts/system.md @@ -0,0 +1,58 @@ +You are a scheduled platform agent. Nobody is typing to you. Your only tools are the StackQL +MCP server's: cloud and SaaS providers are data sources accessed via SQL. Here that is the +`cloudflare` provider for one zone and the `github` provider for the repository that holds the +zone's declaration. Every step you take is a SQL statement, and the platform writes every +statement to an audit log. + +Working the surface: + +1. `run_select_query` with `format` "json" and read `rows`. Use `validate_select_query` first + only if you are unsure of a statement's shape. You do not need `list_methods` or + `describe_method` for the resources below; their shapes are given here. +2. Zero rows is a valid answer. Do not retry the same statement hoping for a different result, + and never invent a value that no query returned. +3. Flat SELECTs, one per question. No CTEs. + +Resources (`provider.service.resource`) and their shapes: + +- `cloudflare.rulesets.phases`: the rate limit rule. Required predicates + `zone_id = '{{ zone_id }}'` and `ruleset_phase = 'http_ratelimit'`. Columns `id`, + `last_updated`, and `rules`, a JSON array; the rule is `$[0]`: + `json_extract(rules, '$[0].ratelimit.requests_per_period')`, `'$[0].ratelimit.period'`, + `'$[0].enabled'`, `'$[0].description'`. +- `cloudflare.zones.http_requests_adaptive_groups`: traffic. Required predicates + `zone_tag = '{{ zone_id }}'`, `since = '{{ since }}'` and `until = '{{ until }}'` (the last + hour, rendered when this run started; do not compute the window in SQL, `CURRENT_TIMESTAMP` + and `strftime` do not parse here). Columns `edge_response_status`, `requests`. + `GROUP BY edge_response_status`. Rate limited requests are status 429. +- `github.repos.commits`: the declaration's history. Required predicates + `owner = '{{ owner }}'`, `repo = '{{ repo }}'`, and `path = '{{ declaration_path }}'`. + Columns `sha` and `commit` (JSON: `json_extract(commit, '$.author.date')`, + `'$.author.name'`, `'$.message'`). Public repository, no credentials. + +Dialect facts: a method's required parameters go in the WHERE clause as equality predicates, +or the query does not plan. Most columns hold JSON; pull fields with `json_extract`, which +returns numbers for JSON numbers; if a CAST is unavoidable use `AS REAL`, the parser rejects +`AS INTEGER`. ISO timestamps compare as strings. + +Writes: + +- The only write you may make is the one below, verbatim, with `run_mutation_query`. It + asserts the declared rule; it is idempotent, so running it against a converged zone changes + nothing. Do not compose a different statement, do not change a value in it, do not add to + it. The platform's policy checks it field for field and declines anything else. + +```sql +{{ declared_assertion }} +``` + +- Whether that write runs is decided by the platform, not by you. A policy in code checks the + statement first. Then the server you are connected to is in a mode set by the platform team: + in `read_only` mode every write is refused; in `safe` mode a write pauses for a human to + approve or decline. Report the outcome and the statement, and stop. Do not rephrase, retry + or work around a refusal or a decline. + +Output: a short markdown report with a status table (declared | live | verdict | action) whose +cells are a few words each, then the evidence with names, numbers and timestamps, the proposed +(or applied) statement, and the list of statements you ran. Say "not observed" rather than +guessing. Plain ASCII punctuation: hyphens, no em dashes, no special symbols. diff --git a/10-26 AI Engineering Night/demo/agent/src/agent.ts b/10-26 AI Engineering Night/demo/agent/src/agent.ts new file mode 100644 index 0000000..646be06 --- /dev/null +++ b/10-26 AI Engineering Night/demo/agent/src/agent.ts @@ -0,0 +1,448 @@ +/** + * edge-agent + * + * A scheduled platform agent: the Vercel AI SDK, an OpenAI model, and the StackQL MCP + * server as its only tool surface. It is given an intent in prose (prompts/intent.md) + * and the declared state of the estate (read from the stackql-deploy manifest, the same + * file that asserts it), reads the live state through SQL, and decides whether to write. + * + * Whether a write happens is never the model's decision. Three gates, all owned by the + * platform and set here in code: + * + * policy src/policy.ts, checked before the statement leaves this process + * mode the server's: read_only refuses every write, safe asks a human + * human the approval at this terminal, bound to the exact statement + * + * This program never starts a full_access server. + * + * npm run agent the scheduled review, once + * npm run tools the tool surface and the mode, no model + * npm run probe [-- file.sql] a statement sent through the gates; no model + * npm run replay [-- file] replay a recorded transcript; no model, no network + */ +import type { ChildProcess } from 'node:child_process'; +import { appendFileSync, existsSync, mkdirSync, readdirSync, readFileSync, statSync, writeFileSync } from 'node:fs'; +import path from 'node:path'; +import * as readline from 'node:readline/promises'; +import { setTimeout as sleep } from 'node:timers/promises'; +import { fileURLToPath } from 'node:url'; + +import { createMCPClient, ElicitationRequestSchema, type ElicitResult, type MCPClient } from '@ai-sdk/mcp'; +import { Experimental_StdioMCPTransport as StdioMCPTransport } from '@ai-sdk/mcp/mcp-stdio'; +import { createOpenAI } from '@ai-sdk/openai'; +import { generateText, isStepCount, type Tool, type ToolExecutionOptions, type ToolSet } from 'ai'; + +import { bold, box, cyan, dim, green, magenta, markdown, red, rule, strip, table, wrap, yellow } from './console.js'; +import { assertion, readManifest, type Declaration } from './declaration.js'; +import { check, type Declared } from './policy.js'; + +// --------------------------------------------------------------------------------------- +// configuration: .env at the demo root, then the process environment, then defaults +// --------------------------------------------------------------------------------------- + +const here = path.dirname(fileURLToPath(import.meta.url)); +const agentRoot = path.resolve(here, '..'); +const demoRoot = path.resolve(agentRoot, '..'); + +for (const envFile of [path.join(demoRoot, '.env'), path.join(agentRoot, '.env')]) { + if (existsSync(envFile)) process.loadEnvFile(envFile); // does not override what is already set +} + +const env = (name: string, fallback: string): string => { + const v = process.env[name]; + return v === undefined || v === '' ? fallback : v; +}; + +const cfg = { + model: env('OPENAI_MODEL', 'gpt-6.1-sol'), + mode: env('STACKQL_MCP_MODE', 'read_only'), + approot: env('STACKQL_APPROOT', ''), // empty: stackql's own default (~/.stackql), no --approot passed + bin: env('STACKQL_MCP_BIN', 'stackql'), // the server binary: `stackql` on the PATH unless .env names one + auditDir: env('STACKQL_AUDIT_DIR', path.resolve('audit')), // audit/ under the cwd: agent/audit with npm run + recordingsDir: path.join(demoRoot, 'recordings'), + maxSteps: Number(env('AGENT_MAX_STEPS', '12')), + zoneId: env('CLOUDFLARE_ZONE_ID', ''), + zoneName: env('CLOUDFLARE_ZONE_NAME', ''), + declarationRepo: env('DECLARATION_REPO', 'stackql-labs/agentic-platform-engineering'), + declarationPath: env('DECLARATION_PATH', '10-26 AI Engineering Night/demo/stack/stackql_manifest.yml'), +}; + +// the platform's decision, enforced in code: a model-facing server is read_only or safe +const ALLOWED_MODES = new Set(['read_only', 'safe']); +if (!ALLOWED_MODES.has(cfg.mode)) { + console.error(`STACKQL_MCP_MODE=${cfg.mode} is not allowed for a model-facing server (read_only | safe)`); + process.exit(2); +} +if (!cfg.zoneId) { + console.error('CLOUDFLARE_ZONE_ID is not set (see .env.example)'); + process.exit(2); +} + +const WRITE_TOOLS = new Set(['run_mutation_query', 'run_lifecycle_operation', 'pull_provider', 'reload_credentials']); + +// --------------------------------------------------------------------------------------- +// helpers +// --------------------------------------------------------------------------------------- + +let transcriptFile: string | undefined; +const log = (line = '') => { + console.log(line); + if (transcriptFile) appendFileSync(transcriptFile, `${line}\n`); +}; +const elapsed = (from: number) => `${((Date.now() - from) / 1000).toFixed(1)}s`; +/** local time with its offset, the way the server stamps its audit records */ +const localIso = (d = new Date()) => { + const p = (n: number, w = 2) => String(Math.abs(n)).padStart(w, '0'); + const off = -d.getTimezoneOffset(); + return `${d.getFullYear()}-${p(d.getMonth() + 1)}-${p(d.getDate())}T${p(d.getHours())}:${p(d.getMinutes())}:${p(d.getSeconds())}.${p(d.getMilliseconds(), 3)}${off < 0 ? '-' : '+'}${p(Math.trunc(off / 60))}:${p(off % 60)}`; +}; +const oneLine = (s: string, max: number) => { + const flat = s.replace(/\s+/g, ' ').trim(); + return flat.length > max ? `${flat.slice(0, max)} ...` : flat; +}; +const runStamp = new Date().toISOString().replace(/[:.]/g, '-'); +const paintMode = (m: string) => (m === 'safe' ? yellow(m) : green(m)); + +function render(template: string, vars: Record): string { + return template.replace(/\{\{\s*(\w+)\s*\}\}/g, (_, key: string) => { + if (!(key in vars)) throw new Error(`prompt placeholder {{ ${key} }} has no value`); + return vars[key]; + }); +} +const readPrompt = (name: string) => readFileSync(path.join(agentRoot, 'prompts', name), 'utf8'); + +function newest(dir: string, prefix: string, suffix: string): string | undefined { + if (!existsSync(dir)) return undefined; + return readdirSync(dir) + .filter((f) => f.startsWith(prefix) && f.endsWith(suffix)) + .map((f) => ({ file: path.join(dir, f), mtime: statSync(path.join(dir, f)).mtimeMs })) + .sort((a, b) => b.mtime - a.mtime)[0]?.file; +} + +/** the audit sink: the server writes one JSONL record per tool call; the gate adds its declines */ +const auditFile = path.join(cfg.auditDir, `stackql_mcp_server_${runStamp}.jsonl`); +const latestAuditLog = () => (existsSync(auditFile) ? auditFile : newest(cfg.auditDir, 'stackql_mcp_server_', '.jsonl')); +const proposalFile = () => path.join(cfg.auditDir, 'last-proposal.sql'); + +function audit(record: Record): void { + mkdirSync(cfg.auditDir, { recursive: true }); + appendFileSync(auditFile, `${JSON.stringify({ timestamp: localIso(), mode: cfg.mode, ...record })}\n`); +} + +function resultText(output: unknown): string { + if (typeof output === 'string') return output; + if (output && typeof output === 'object') { + const content = (output as { content?: unknown }).content; + if (Array.isArray(content)) { + return content.map((c) => (c && typeof c === 'object' && 'text' in c ? String((c as { text: unknown }).text) : '')).join('\n'); + } + } + return JSON.stringify(output); +} + +// --------------------------------------------------------------------------------------- +// the declaration: what should be, read from the same manifest stackql-deploy asserts +// --------------------------------------------------------------------------------------- + +/** the declared rule and the one write the policy permits (src/declaration.ts), for this zone */ +function declaration(): Declaration { + const manifest = readManifest(path.join(demoRoot, 'stack', 'stackql_manifest.yml')); + return { ...manifest, zoneId: cfg.zoneId, sql: assertion(cfg.zoneId, manifest.phase, manifest) }; +} + +/** the declared rule for the prompt: markdown, which the model reads well */ +function declaredMarkdown(d: Declaration): string { + return [ + '| zone | phase | threshold (requests per period, per IP) | period (s) | enabled | description |', + '| --- | --- | --- | --- | --- | --- |', + `| ${cfg.zoneName || cfg.zoneId} | ${d.phase} | ${d.threshold} | ${d.period} | true | ${d.description} |`, + ].join('\n'); +} + +/** the declared rule for the console */ +const declaredTable = (d: Declaration) => + table(['phase', 'threshold', 'period', 'enabled', 'description'], [[d.phase, String(d.threshold), String(d.period), 'true', d.description]]); + +// --------------------------------------------------------------------------------------- +// the human in the loop: MCP elicitation, raised by the server in safe mode before a write +// --------------------------------------------------------------------------------------- + +type JsonSchema = { properties?: Record }; + +async function elicit(message: string, requestedSchema: unknown): Promise { + log(); + const body = ['the server is in safe mode and a write is waiting', '', ...message.split('\n').flatMap((l) => (l.length <= 88 ? [l] : wrap(l, 88)))]; + for (const l of box(body, 'APPROVAL REQUIRED', yellow)) log(` ${l}`); + + if (!process.stdin.isTTY) { + log(red(' no operator attached (not a TTY): declined')); // a scheduled run has nobody to ask + return { action: 'decline' }; + } + + const rl = readline.createInterface({ input: process.stdin, output: process.stdout }); + try { + const props = ((requestedSchema ?? {}) as JsonSchema).properties ?? {}; + const content: Record = {}; + if (Object.keys(props).length === 0) { + const a = (await rl.question(bold(' approve? [y/N] '))).trim().toLowerCase(); + return a === 'y' || a === 'yes' ? { action: 'accept', content: {} } : { action: 'decline' }; + } + for (const [key, def] of Object.entries(props)) { + const label = def.description ?? def.title ?? key; + if (def.type === 'boolean') { + const a = (await rl.question(bold(` ${label} [y/N] `))).trim().toLowerCase(); + content[key] = a === 'y' || a === 'yes'; + } else if (Array.isArray(def.enum)) { + const a = (await rl.question(` ${label} (${def.enum.join(' | ')}) `)).trim(); + if (!def.enum.map(String).includes(a)) return { action: 'decline' }; + content[key] = a; + } else { + const a = (await rl.question(` ${label}: `)).trim(); + if (a === '') return { action: 'decline' }; + content[key] = def.type === 'number' || def.type === 'integer' ? Number(a) : a; + } + } + return Object.values(content).some((v) => v === false) ? { action: 'decline' } : { action: 'accept', content }; + } finally { + rl.close(); + } +} + +// --------------------------------------------------------------------------------------- +// the platform: the StackQL MCP server, started here with the mode the platform chose +// --------------------------------------------------------------------------------------- + +/** the server's start-up chatter on stderr (launcher version notice, sink path, start line); everything else passes through */ +const SERVER_NOISE = /^(stackql-mcp: WARNING Node\.js|sink file: |\[stackql mcp\] \S+ (starting|stopping|shutting|exiting))/; + +function relayStderr(transport: StdioMCPTransport): void { + const child = (transport as unknown as { process?: ChildProcess }).process; // set by start(), which createMCPClient awaits + const stream = child?.stderr; + if (!stream) return; + let rest = ''; + const relay = (text: string) => { + for (const l of text.split(/\r?\n/)) if (l && !SERVER_NOISE.test(l)) process.stderr.write(`${l}\n`); + }; + stream.on('data', (chunk: Buffer) => { + const lines = (rest + chunk.toString()).split(/\r?\n/); + rest = lines.pop() ?? ''; + relay(lines.join('\n')); + }); + stream.on('end', () => relay(rest)); +} + +async function startPlatform(): Promise { + mkdirSync(cfg.auditDir, { recursive: true }); + const launcher = path.join(agentRoot, 'node_modules', '@stackql', 'mcp-server', 'bin', 'stackql-mcp.js'); + if (!existsSync(launcher)) throw new Error(`StackQL MCP launcher not found at ${launcher}; run npm install`); + + const serverConfig = { server: { mode: cfg.mode, audit: { disabled: false, file: { path: auditFile } } } }; + const processEnv = Object.fromEntries(Object.entries(process.env).filter((e): e is [string, string] => typeof e[1] === 'string')); + processEnv.STACKQL_MCP_BIN = cfg.bin; // the launcher runs this binary instead of downloading one + const approot = cfg.approot ? ['--approot', cfg.approot] : []; // else the launcher's default, ~/.stackql + + const transport = new StdioMCPTransport({ + command: process.execPath, + args: [launcher, ...approot, '--mcp.config', JSON.stringify(serverConfig), '--mcp.log.format=jsonl'], + env: processEnv, // the Cloudflare token lives in the server process, never in the model + cwd: cfg.auditDir, + stderr: 'pipe', // relayed below, minus the start-up chatter + }); + const client = await createMCPClient({ + name: 'edge-agent', + capabilities: { elicitation: {} }, // or a safe-mode server refuses writes outright instead of asking + protocolVersionDiscovery: false, // negotiate 2025-11-25, where a gated write is an elicitation request + transport, + onUncaughtError: (error) => console.error('mcp:', error), + }); + relayStderr(transport); + + client.onElicitationRequest(ElicitationRequestSchema, (request) => elicit(request.params.message, request.params.requestedSchema)); + return client; +} + +// --------------------------------------------------------------------------------------- +// the gate: policy in code, in front of the server, whatever mode the server is in +// --------------------------------------------------------------------------------------- + +function gated(tools: ToolSet, declared: Declared, record = true): ToolSet { + const mutate = tools['run_mutation_query'] as Tool | undefined; + const execute = mutate?.execute; + if (!mutate || !execute) return tools; + const guarded: Tool = { + ...mutate, + execute: async (input: unknown, options: ToolExecutionOptions) => { + const sql = typeof (input as { sql?: unknown })?.sql === 'string' ? (input as { sql: string }).sql : ''; + if (record) writeFileSync(proposalFile(), sql); // the agent's proposal: what `npm run probe` replays + const verdict = check(sql, declared); + if (!verdict.ok) { + audit({ tool: 'run_mutation_query', decision: 'declined_by_policy', duration_ms: 0, rule: verdict.rule, reason: verdict.reason, sql }); + return { content: [{ type: 'text', text: `tool "run_mutation_query" declined by policy (${verdict.rule}): ${verdict.reason}` }] }; + } + return execute(input, options); + }, + }; + return { ...tools, run_mutation_query: guarded }; +} + +// --------------------------------------------------------------------------------------- +// subcommands that need no model: the surface, and a statement sent through the gates +// --------------------------------------------------------------------------------------- + +async function listTools(client: MCPClient): Promise { + const tools = await client.tools(); + log(`${bold('server')} ${client.serverInfo.name} ${client.serverInfo.version ?? ''}`); + log(`${bold('mode')} ${paintMode(cfg.mode)}`); + log(`${bold('zone')} ${cfg.zoneName} ${dim(`(${cfg.zoneId})`)}`); + log(`${bold('tools')} ${Object.keys(tools).length}`); + for (const name of Object.keys(tools).sort()) log(WRITE_TOOLS.has(name) ? ` ${magenta('[write]')} ${magenta(name)}` : ` ${name}`); +} + +async function probeWrite(client: MCPClient, file?: string): Promise { + // a file if given, else the statement the agent proposed on its last run, else the + // declared assertion itself (so even an approved probe changes nothing but drift) + const declared = declaration(); + const source = file ?? (existsSync(proposalFile()) ? proposalFile() : undefined); + const sql = source ? readFileSync(source, 'utf8').replace(/\{\{\s*zone_id\s*\}\}/g, cfg.zoneId).replace(/^\s*--.*$/gm, '').trim() : declared.sql; + log(`${bold('mode')} ${paintMode(cfg.mode)}`); + log(`${bold('source')} ${source ? path.relative(demoRoot, source) : 'the declared assertion'}`); + log(`${bold('write')} ${oneLine(sql, 400)}`); + log(); + const tools = gated(await client.tools(), declared, false); // a probe is not a proposal + const result = await (tools['run_mutation_query'] as Tool).execute!({ sql }, { toolCallId: 'probe', messages: [], context: undefined }); + const text = resultText(result); + const paint = /declined|refused|denied|error/i.test(text) ? red : green; + for (const l of markdown(text)) log(paint(l)); +} + +// --------------------------------------------------------------------------------------- +// the run +// --------------------------------------------------------------------------------------- + +async function run(client: MCPClient): Promise { + const started = Date.now(); + mkdirSync(cfg.auditDir, { recursive: true }); + transcriptFile = path.join(cfg.auditDir, `agent_${runStamp}.txt`); + + const declared = declaration(); + const [owner, repo] = cfg.declarationRepo.split('/'); + const iso = (ms: number) => new Date(ms).toISOString().replace(/\.\d{3}Z$/, 'Z'); + const vars = { + zone_id: cfg.zoneId, + zone_name: cfg.zoneName || cfg.zoneId, + owner: owner ?? '', + repo: repo ?? '', + declaration_path: cfg.declarationPath, + since: iso(started - 60 * 60 * 1000), // the traffic window: the hour before this run + until: iso(started), + declared_table: declaredMarkdown(declared), + declared_assertion: declared.sql, + }; + const system = [render(readPrompt('system.md'), vars), client.instructions ? `# Server instructions\n\n${client.instructions}` : ''] + .filter(Boolean) + .join('\n\n'); + const prompt = render(readPrompt('intent.md'), vars); + const tools = gated(await client.tools(), declared); + + const head = [ + `${bold('model')} ${cfg.model} ${bold('mode')} ${paintMode(cfg.mode)} ${bold('tools')} ${Object.keys(tools).length} ${bold('zone')} ${vars.zone_name}`, + dim(`audit ${path.relative(demoRoot, auditFile)}`), + ]; + for (const l of box(head, 'edge-agent')) log(l); + log(); + log(`${bold('declared')} ${dim('stack/stackql_manifest.yml')}`); + for (const l of declaredTable(declared)) log(` ${l}`); + log('permitted write: one REPLACE that asserts the declared rule (policy.ts); the server mode decides if it runs'); + log(); + + const toolCounts = new Map(); + let stepStarted = Date.now(); + + const result = await generateText({ + model: createOpenAI().responses(cfg.model), // OPENAI_API_KEY from the environment + system, + prompt, + tools, + stopWhen: isStepCount(cfg.maxSteps), + maxOutputTokens: 6000, + providerOptions: { openai: { parallelToolCalls: false, store: false } }, + onStepEnd({ stepNumber, toolCalls, toolResults }) { + const outputs = new Map(toolResults.map((r) => [r.toolCallId, r.output])); + for (const call of toolCalls) { + toolCounts.set(call.toolName, (toolCounts.get(call.toolName) ?? 0) + 1); + const input = (call.input ?? {}) as Record; + const shown = typeof input.sql === 'string' ? input.sql : JSON.stringify(input); + const write = WRITE_TOOLS.has(call.toolName); + log(`${dim('[')}${bold(String(stepNumber + 1).padStart(2, '0'))} ${dim(elapsed(stepStarted).padStart(6))}${dim(']')} ${write ? magenta(call.toolName) : cyan(call.toolName)}`); + log(` ${oneLine(shown, 320)}`); + const text = resultText(outputs.get(call.toolCallId)); + const blocked = write && /refused|declined/i.test(text); + log(` ${dim('->')} ${blocked ? `${red(bold('BLOCKED'))}${red(`: ${oneLine(text, 200)}`)}` : dim(oneLine(text, 200))}`); + } + stepStarted = Date.now(); + }, + }); + + log(); + log(rule('report')); + for (const l of markdown(result.text.trim())) log(l); + log(); + const usage = result.totalUsage; + const calls = [...toolCounts.entries()].sort((a, b) => b[1] - a[1]).map(([n, c]) => `${n}=${c}`).join(' '); + const writes = [...toolCounts.entries()].filter(([n]) => WRITE_TOOLS.has(n)).reduce((s, [, c]) => s + c, 0); + const rel = (f: string | undefined) => (f ? path.relative(demoRoot, f) : cfg.auditDir); + const summary = [ + `${bold('model')} ${cfg.model} ${bold('mode')} ${paintMode(cfg.mode)} ${bold('finish')} ${result.finishReason} ${bold('elapsed')} ${elapsed(started)}`, + `${bold('steps')} ${result.steps.length} ${bold('tool calls')} ${calls || 'none'}`, + `${bold('writes attempted')} ${writes} ${dim('(the gates decided what happened to them)')}`, + `${bold('tokens')} in=${usage.inputTokens ?? '?'} out=${usage.outputTokens ?? '?'} total=${usage.totalTokens ?? '?'}`, + `${bold('audit')} ${rel(latestAuditLog())}`, + `${bold('transcript')} ${rel(transcriptFile)}`, + ]; + for (const l of box(summary, 'run')) log(l); +} + +// --------------------------------------------------------------------------------------- +// the fallback: replay a recorded transcript, clearly labelled, with the pacing of a run +// --------------------------------------------------------------------------------------- + +async function replay(file?: string): Promise { + const target = file ?? newest(cfg.auditDir, 'agent_', '.txt') ?? newest(cfg.recordingsDir, 'agent_', '.txt'); + if (!target || !existsSync(target)) throw new Error(`no transcript to replay (looked in ${cfg.auditDir} and ${cfg.recordingsDir})`); + const recorded = statSync(target).mtime.toISOString().slice(0, 16).replace('T', ' '); + const banner = [`${path.relative(demoRoot, target)} ${dim(`recorded ${recorded} UTC`)}`, bold('not a live run')]; + for (const l of box(banner, 'REPLAY', yellow)) console.log(l); + console.log(); + for (const line of readFileSync(target, 'utf8').split('\n')) { + if (/^\[\d\d /.test(strip(line))) await sleep(900); + console.log(line); + await sleep(40); + } + console.log(); + console.log(dim(`(end of replay: ${path.relative(demoRoot, target)})`)); +} + +// --------------------------------------------------------------------------------------- + +async function main(): Promise { + const argv = process.argv.slice(2); + const fileAfter = (flag: string) => { + const i = argv.indexOf(flag); + return i >= 0 && argv[i + 1] && !argv[i + 1].startsWith('--') ? path.resolve(argv[i + 1]) : undefined; + }; + if (argv.includes('--replay')) return replay(fileAfter('--replay')); + let client: MCPClient | undefined; + try { + client = await startPlatform(); + if (argv.includes('--list-tools')) await listTools(client); + else if (argv.includes('--probe-write')) await probeWrite(client, fileAfter('--probe-write')); + else await run(client); + } finally { + await client?.close(); + } +} + +main().catch((error) => { + console.error(error instanceof Error ? error.message : error); + process.exit(1); +}); diff --git a/10-26 AI Engineering Night/demo/agent/src/audit.ts b/10-26 AI Engineering Night/demo/agent/src/audit.ts new file mode 100644 index 0000000..e2d953c --- /dev/null +++ b/10-26 AI Engineering Night/demo/agent/src/audit.ts @@ -0,0 +1,89 @@ +/** + * npm run audit [file] + * + * The StackQL MCP audit sink as a table: one record per tool call, with the mode the server + * was in and what the gate decided. The agent is an event source; this is the event log. + * Each run starts its own sink in STACKQL_AUDIT_DIR (default audit/ under the cwd, so + * agent/audit with npm run); without a file argument this reads all of them, oldest first, + * so the agent's run, the replayed proposal and the policy decline appear in one log. + */ +import { existsSync, readdirSync, readFileSync, statSync } from 'node:fs'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; + +import { bold, dim, green, red, table, yellow } from './console.js'; + +const here = path.dirname(fileURLToPath(import.meta.url)); +const demoRoot = path.resolve(here, '..', '..'); +const envFile = path.join(demoRoot, '.env'); +if (existsSync(envFile)) process.loadEnvFile(envFile); +const auditDir = process.env.STACKQL_AUDIT_DIR || path.resolve('audit'); + +const isSink = (f: string) => f.startsWith('stackql_mcp_server_') && (f.endsWith('.jsonl') || f.endsWith('.log')); +const files = process.argv[2] + ? [path.resolve(process.argv[2])] + : existsSync(auditDir) + ? readdirSync(auditDir) + .filter(isSink) + .sort() + .map((f) => path.join(auditDir, f)) + .filter((f) => statSync(f).size > 0) // a run that made no tool call leaves an empty sink + : []; +if (files.length === 0 || !files.every((f) => existsSync(f))) { + console.error(`no audit log found in ${auditDir}; run the agent first`); + process.exit(1); +} + +type Rec = Record; +const parse = (l: string): Rec[] => { + try { + return [JSON.parse(l) as Rec]; + } catch { + return []; + } +}; +const records = files.flatMap((f) => + readFileSync(f, 'utf8') + .split('\n') + .filter((l) => l.trim()) + .flatMap(parse), +); + +const str = (v: unknown): string => (v === undefined || v === null ? '' : typeof v === 'string' ? v : JSON.stringify(v)); +const flat = (s: string) => s.replace(/\s+/g, ' ').trim(); +const fit = (s: string, n: number) => (s.length > n ? `${s.slice(0, n - 3)}...` : s); +const statementOf = (r: Rec): string => { + const args = (r.args ?? {}) as Rec; + return flat(str(r.sql ?? r.query ?? args.sql ?? (Object.keys(args).length ? args : ''))); +}; +const timeOf = (r: Rec): string => { + const t = str(r.timestamp ?? r.time ?? r.ts); + const d = new Date(t); + if (!Number.isNaN(d.getTime())) return d.toTimeString().slice(0, 8); // local time, as the server stamps it + const m = t.match(/T(\d\d:\d\d:\d\d)/); + return m ? m[1] : t.slice(0, 8); +}; +const paintMode = (m: string) => (m === 'safe' ? yellow(m) : m === 'full_access' ? red(m) : m ? green(m) : ''); +const paintDecision = (d: string) => (/declin|refus|denied|error/i.test(d) ? red(d) : d ? green(d) : dim('?')); + +const modes = [...new Set(records.map((r) => str(r.mode)).filter(Boolean))]; +const decisions = new Map(); +for (const r of records) decisions.set(str(r.decision) || '?', (decisions.get(str(r.decision) || '?') ?? 0) + 1); +const width = Math.max(40, (process.stdout.columns ?? 140) - 92); + +console.log(`${bold('audit log')} ${files.length === 1 ? path.relative(demoRoot, files[0]) : `${files.length} sinks in ${path.relative(demoRoot, auditDir)}`}`); +console.log( + `${bold('records')} ${records.length} ${bold('modes')} ${modes.map(paintMode).join(', ') || '?'} ${bold('decisions')} ${[...decisions].map(([d, n]) => `${paintDecision(d)}=${n}`).join(' ')}`, +); +console.log(); +const rows = records.map((r) => [ + timeOf(r), + paintMode(str(r.mode)), + str(r.tool), + paintDecision(str(r.decision)), + r.duration_ms !== undefined ? String(r.duration_ms) : '', + fit(statementOf(r), width), +]); +for (const l of table(['time', 'mode', 'tool', 'decision', 'ms', 'statement'], rows)) console.log(l); +console.log(); +console.log(dim('every statement the agent ran, the mode the server was in, and what the gate decided: one record each, never the result rows.')); diff --git a/10-26 AI Engineering Night/demo/agent/src/console.ts b/10-26 AI Engineering Night/demo/agent/src/console.ts new file mode 100644 index 0000000..85692bf --- /dev/null +++ b/10-26 AI Engineering Night/demo/agent/src/console.ts @@ -0,0 +1,129 @@ +/** + * Console output: colour, ASCII boxes and tables, and a renderer for the model's markdown + * report. ASCII only, so it survives any terminal font and code page. Colour follows the + * terminal: on for a TTY, off when piped or NO_COLOR is set, on regardless with + * FORCE_COLOR=1. Transcripts keep the colour, so a replay looks like the run. + */ + +const colour = + process.env.FORCE_COLOR !== undefined && process.env.FORCE_COLOR !== '' + ? process.env.FORCE_COLOR !== '0' + : process.env.NO_COLOR === undefined && Boolean(process.stdout.isTTY); +const sgr = (open: number, close: number) => (s: string) => (colour ? `\x1b[${open}m${s}\x1b[${close}m` : s); + +export const bold = sgr(1, 22); +export const dim = sgr(2, 22); +export const red = sgr(31, 39); +export const green = sgr(32, 39); +export const yellow = sgr(33, 39); +export const magenta = sgr(35, 39); +export const cyan = sgr(36, 39); + +/** the text without its colour codes */ +export const strip = (s: string) => s.replace(/\x1b\[[0-9;]*m/g, ''); +const len = (s: string) => strip(s).length; +const pad = (s: string, n: number) => s + ' '.repeat(Math.max(0, n - len(s))); + +/** a section rule: --- title ------ */ +export const rule = (title: string, width = 80) => `${dim('--- ')}${bold(title)} ${dim('-'.repeat(Math.max(4, width - len(title) - 5)))}`; + +/** greedy word wrap; a word longer than the width is split */ +export function wrap(s: string, width: number): string[] { + const out: string[] = []; + let line = ''; + for (const word of s.split(/\s+/).filter(Boolean)) { + let w = word; + while (len(w) > width) { + if (line) { + out.push(line); + line = ''; + } + out.push(w.slice(0, width)); + w = w.slice(width); + } + if (!line) line = w; + else if (len(line) + 1 + len(w) <= width) line += ` ${w}`; + else { + out.push(line); + line = w; + } + } + if (line) out.push(line); + return out; +} + +/** a box around lines, the title in the top edge, the border painted with `edge` */ +export function box(lines: string[], title = '', edge: (s: string) => string = dim): string[] { + const inner = Math.max(...lines.map(len), len(title) + 6, 24); + const top = title + ? `${edge('+-- ')}${bold(title)}${edge(` ${'-'.repeat(inner - len(title) - 2)}+`)}` + : edge(`+${'-'.repeat(inner + 2)}+`); + return [top, ...lines.map((l) => `${edge('|')} ${pad(l, inner)} ${edge('|')}`), edge(`+${'-'.repeat(inner + 2)}+`)]; +} + +/** an ASCII table: header bold, borders dim; cells may carry colour of their own */ +export function table(headers: string[], rows: string[][]): string[] { + const widths = headers.map((h, i) => Math.max(len(h), ...rows.map((r) => len(r[i] ?? '')))); + const edge = dim(`+${widths.map((w) => '-'.repeat(w + 2)).join('+')}+`); + const row = (cells: string[], paint = (s: string) => s) => + `${dim('|')} ${widths.map((w, i) => pad(paint(cells[i] ?? ''), w)).join(` ${dim('|')} `)} ${dim('|')}`; + return [edge, row(headers, bold), edge, ...rows.map((r) => row(r)), edge]; +} + +/** the model's punctuation in ASCII: em and en dashes, curly quotes, the multiplication sign, bullets */ +export const ascii = (s: string) => + s + .replace(/[‒–—−]/g, '-') + .replace(/[‘’]/g, "'") + .replace(/[“”]/g, '"') + .replace(/…/g, '...') + .replace(/×/g, 'x') + .replace(/•/g, '-') + .replace(/ /g, ' '); + +/** + * The model's markdown report for the console: headings bold, pipe tables as ASCII tables + * (or as label: value lines when the table would not fit the terminal), code dim. + */ +export function markdown(text: string, columns = process.stdout.columns ?? 120): string[] { + const inline = (s: string) => + s.replace(/\*\*([^*]+)\*\*/g, (_, t: string) => bold(t)).replace(/`([^`]+)`/g, (_, t: string) => cyan(t)); + const isSeparator = (l: string) => /^\s*\|?[\s:|-]*-[\s:|-]*\|?\s*$/.test(l); + const cells = (l: string) => + l + .trim() + .replace(/^\|/, '') + .replace(/\|$/, '') + .split('|') + .map((c) => inline(c.trim())); + const out: string[] = []; + const lines = ascii(text).split('\n'); + for (let i = 0; i < lines.length; i++) { + const l = lines[i]; + if (/^\s*```/.test(l)) { + while (++i < lines.length && !/^\s*```/.test(lines[i])) out.push(dim(` ${lines[i]}`)); + } else if (/^\s*\|/.test(l)) { + const block: string[] = []; + while (i < lines.length && /^\s*\|/.test(lines[i])) block.push(lines[i++]); + i--; + const rows = block.filter((r) => !isSeparator(r)).map(cells); + if (rows.length === 0) continue; + const t = table(rows[0], rows.slice(1)); + if (len(t[0]) <= columns) out.push(...t); + else { + const label = Math.max(...rows[0].map(len)); + rows.slice(1).forEach((r, n) => { + if (n > 0) out.push(''); + r.forEach((cell, c) => out.push(`${bold(pad(rows[0][c] ?? '', label))} ${cell}`)); + }); + } + } else if (/^#{1,6}\s/.test(l)) { + out.push(bold(l.replace(/^#{1,6}\s+/, ''))); + } else if (/^\s*([-*_])\s*(\1\s*){2,}$/.test(l)) { + out.push(dim('-'.repeat(40))); + } else { + out.push(inline(l)); + } + } + return out; +} diff --git a/10-26 AI Engineering Night/demo/agent/src/declaration.ts b/10-26 AI Engineering Night/demo/agent/src/declaration.ts new file mode 100644 index 0000000..8b5e996 --- /dev/null +++ b/10-26 AI Engineering Night/demo/agent/src/declaration.ts @@ -0,0 +1,50 @@ +/** + * The declaration: the rate limit rule as stack/stackql_manifest.yml declares it, and the + * one write the policy permits, the REPLACE that asserts it. Rendered here from the same + * values stackql-deploy renders into stack/resources/rate_limit.iql, so the statement in + * the briefing, the one the gate compares against, and the .iql anchor are the same rule + * field for field. test/declaration.test.ts checks that against the .iql. + */ +import { readFileSync } from 'node:fs'; + +import { parse as parseYaml } from 'yaml'; + +import type { Declared } from './policy.js'; + +export type Rule = { description: string; threshold: number; period: number }; +export type Declaration = Declared & Rule; + +type Manifest = { + globals?: Array<{ name: string; value?: unknown }>; + resources?: Array<{ name: string; props?: Array<{ name: string; value?: unknown }> }>; +}; + +/** the rate_limit resource's props and the ruleset phase, from the manifest */ +export function readManifest(file: string): Rule & { phase: string } { + const manifest = parseYaml(readFileSync(file, 'utf8')) as Manifest; + const phase = String(manifest.globals?.find((g) => g.name === 'ruleset_phase')?.value ?? 'http_ratelimit'); + const res = manifest.resources?.find((r) => r.name === 'rate_limit'); + const prop = (n: string) => String(res?.props?.find((p) => p.name === n)?.value ?? ''); + return { phase, description: prop('description'), threshold: Number(prop('threshold')), period: Number(prop('period')) }; +} + +/** + * The declared rule as the REPLACE that asserts it. The expression's quotes are escaped + * for the SQL parser (\\"), the same way the .iql writes them. + */ +export function assertion(zoneId: string, phase: string, rule: Rule): string { + const r = { + action: 'block', + description: rule.description, + enabled: true, + expression: '(starts_with(http.request.uri.path, "/"))', + ratelimit: { + characteristics: ['ip.src', 'cf.colo.id'], + period: rule.period, + requests_per_period: rule.threshold, + mitigation_timeout: rule.period, + }, + }; + const rules = JSON.stringify([r], null, 2).replace(/\\"/g, '\\\\"'); + return `REPLACE cloudflare.rulesets.phases\nSET rules = '${rules}'\nWHERE zone_id = '${zoneId}' AND ruleset_phase = '${phase}'`; +} diff --git a/10-26 AI Engineering Night/demo/agent/src/policy.ts b/10-26 AI Engineering Night/demo/agent/src/policy.ts new file mode 100644 index 0000000..17e2222 --- /dev/null +++ b/10-26 AI Engineering Night/demo/agent/src/policy.ts @@ -0,0 +1,67 @@ +/** + * The policy gate: the platform's rules for a write, checked in code before the + * statement reaches the server or a human. Pure: a statement and the declaration in, + * a verdict out. The model never sees this file and cannot change what it decides. + * + * Four rules, in order: + * one statement no chaining + * mutation surface only REPLACE cloudflare.rulesets.phases SET rules + * scope only this zone, only the http_ratelimit phase + * declaration the rule it writes is the declared rule, field for field + * + * The last rule is the point: the agent decides whether to write, the declaration + * decides what gets written. + */ + +export type Verdict = { ok: true } | { ok: false; rule: string; reason: string }; +export type Declared = { sql: string; zoneId: string; phase: string }; + +const no = (rule: string, reason: string): Verdict => ({ ok: false, rule, reason }); +const flat = (s: string) => s.replace(/\s+/g, ' ').replace(/;\s*$/, '').trim(); +const literal = (s: string) => s.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); + +/** the JSON array in SET rules = '...', unescaped the way the SQL parser would */ +export function rulesOf(sql: string): unknown { + const m = sql.match(/SET\s+rules\s*=\s*'([\s\S]*?)'\s*WHERE/i); + if (!m) return undefined; + try { + return JSON.parse(m[1].replace(/\\(["\\])/g, '$1')); + } catch { + return undefined; + } +} + +/** the first place two JSON values differ, as a path, or undefined when they match */ +export function firstDiff(a: unknown, b: unknown, path = '$'): string | undefined { + const show = (v: unknown) => JSON.stringify(v) ?? 'missing'; + if (typeof a !== typeof b || Array.isArray(a) !== Array.isArray(b)) return `${path}: ${show(a)} vs declared ${show(b)}`; + if (a && b && typeof a === 'object') { + const ra = a as Record; + const rb = b as Record; + for (const k of [...new Set([...Object.keys(ra), ...Object.keys(rb)])].sort()) { + const d = firstDiff(ra[k], rb[k], Array.isArray(a) ? `${path}[${k}]` : `${path}.${k}`); + if (d) return d; + } + return undefined; + } + return a === b ? undefined : `${path}: ${show(a)} vs declared ${show(b)}`; +} + +export function check(sql: string, declared: Declared): Verdict { + const s = flat(sql); + if (s.includes(';')) return no('one statement', 'more than one statement'); + if (!/^REPLACE\s+cloudflare\.rulesets\.phases\s+SET\s+rules\s*=/i.test(s)) { + return no('mutation surface', 'only REPLACE cloudflare.rulesets.phases SET rules is permitted'); + } + if (!new RegExp(`zone_id\\s*=\\s*'${literal(declared.zoneId)}'`, 'i').test(s)) { + return no('scope', 'zone_id is not the zone this agent is scoped to'); + } + if (!new RegExp(`ruleset_phase\\s*=\\s*'${literal(declared.phase)}'`, 'i').test(s)) { + return no('scope', `ruleset_phase is not ${declared.phase}`); + } + const proposed = rulesOf(s); + if (proposed === undefined) return no('declaration', 'SET rules is not a JSON array'); + const diff = firstDiff(proposed, rulesOf(declared.sql)); + if (diff) return no('declaration', `not the declared rule: ${diff}`); + return { ok: true }; +} diff --git a/10-26 AI Engineering Night/demo/agent/test/console.test.ts b/10-26 AI Engineering Night/demo/agent/test/console.test.ts new file mode 100644 index 0000000..9faa67d --- /dev/null +++ b/10-26 AI Engineering Night/demo/agent/test/console.test.ts @@ -0,0 +1,43 @@ +import assert from 'node:assert/strict'; +import { test } from 'node:test'; + +import { box, markdown, strip, table, wrap } from '../src/console.js'; + +test('a box is rectangular, whatever the content', () => { + const lines = box(['short', 'a much longer line of text'], 'title').map(strip); + for (const l of lines) assert.equal(l.length, lines[0].length, l); + assert.match(lines[0], /^\+-- title -+\+$/); + assert.match(lines[lines.length - 1], /^\+-+\+$/); +}); + +test('a table aligns its columns', () => { + const lines = table(['a', 'bb'], [['1', '22'], ['333', '4']]).map(strip); + assert.deepEqual(lines, ['+-----+----+', '| a | bb |', '+-----+----+', '| 1 | 22 |', '| 333 | 4 |', '+-----+----+']); +}); + +test('markdown tables become ASCII tables; headings, code and bullets survive', () => { + const report = ['## status', '', '| declared | live |', '| --- | --- |', '| 100 | 5 |', '', '```sql', 'SELECT 1', '```', '- a **bold** item']; + assert.deepEqual(markdown(report.join('\n')).map(strip), [ + 'status', + '', + '+----------+------+', + '| declared | live |', + '+----------+------+', + '| 100 | 5 |', + '+----------+------+', + '', + ' SELECT 1', + '- a bold item', + ]); +}); + +test('the report comes out in ASCII, and a table too wide for the terminal becomes label: value lines', () => { + assert.deepEqual(markdown('a — b × c … “d” ‘e’').map(strip), ['a - b x c ... "d" \'e\'']); + const wide = ['| declared | live |', '| --- | --- |', '| one hundred requests | five requests |'].join('\n'); + assert.deepEqual(markdown(wide, 20).map(strip), ['declared one hundred requests', 'live five requests']); +}); + +test('wrap keeps words whole where it can', () => { + assert.deepEqual(wrap('one two three four', 9), ['one two', 'three', 'four']); + assert.deepEqual(wrap('abcdefghij', 4), ['abcd', 'efgh', 'ij']); +}); diff --git a/10-26 AI Engineering Night/demo/agent/test/declaration.test.ts b/10-26 AI Engineering Night/demo/agent/test/declaration.test.ts new file mode 100644 index 0000000..3a66453 --- /dev/null +++ b/10-26 AI Engineering Night/demo/agent/test/declaration.test.ts @@ -0,0 +1,46 @@ +import assert from 'node:assert/strict'; +import { readFileSync } from 'node:fs'; +import path from 'node:path'; +import { test } from 'node:test'; +import { fileURLToPath } from 'node:url'; + +import { assertion, readManifest } from '../src/declaration.js'; +import { check, firstDiff, rulesOf } from '../src/policy.js'; + +const demoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..', '..'); +const zoneId = '0123456789abcdef0123456789abcdef'; +const manifest = readManifest(path.join(demoRoot, 'stack', 'stackql_manifest.yml')); + +/** the createorupdate anchor of rate_limit.iql, rendered the way stackql-deploy renders it */ +function iqlAssertion(): string { + const iql = readFileSync(path.join(demoRoot, 'stack', 'resources', 'rate_limit.iql'), 'utf8'); + const m = iql.match(/\/\*\+ createorupdate \*\/([\s\S]*?)(?=\/\*\+)/); + assert.ok(m, 'rate_limit.iql has a createorupdate anchor'); + const vars: Record = { + zone_id: zoneId, + ruleset_phase: manifest.phase, + description: manifest.description, + threshold: String(manifest.threshold), + period: String(manifest.period), + }; + return m[1].replace(/\{\{\s*(\w+)\s*\}\}/g, (_, k: string) => vars[k] ?? assert.fail(`no value for {{ ${k} }}`)).trim(); +} + +test('the manifest declares the rule the demo talks about', () => { + assert.equal(manifest.phase, 'http_ratelimit'); + assert.equal(manifest.threshold, 100); + assert.equal(manifest.period, 10); + assert.ok(manifest.description); +}); + +test('the .iql anchor and the rendered assertion are the same rule, field for field', () => { + const rendered = assertion(zoneId, manifest.phase, manifest); + const iql = iqlAssertion(); + assert.equal(firstDiff(rulesOf(iql), rulesOf(rendered)), undefined); + assert.deepEqual(check(iql, { sql: rendered, zoneId, phase: manifest.phase }), { ok: true }); + assert.deepEqual(check(rendered, { sql: iql, zoneId, phase: manifest.phase }), { ok: true }); +}); + +test('both escape the expression quotes for the SQL parser, as \\\\"/\\\\"', () => { + for (const sql of [iqlAssertion(), assertion(zoneId, manifest.phase, manifest)]) assert.match(sql, /\\\\"\/\\\\"/); +}); diff --git a/10-26 AI Engineering Night/demo/agent/test/policy.test.ts b/10-26 AI Engineering Night/demo/agent/test/policy.test.ts new file mode 100644 index 0000000..630f8cd --- /dev/null +++ b/10-26 AI Engineering Night/demo/agent/test/policy.test.ts @@ -0,0 +1,86 @@ +import assert from 'node:assert/strict'; +import { test } from 'node:test'; + +import { check, firstDiff, rulesOf, type Declared } from '../src/policy.js'; + +const zoneId = '0123456789abcdef0123456789abcdef'; +const rule = (threshold: number, description = 'edge-estate rate limit (managed by stackql-deploy)') => `[ + { + "action": "block", + "description": "${description}", + "enabled": true, + "expression": "(starts_with(http.request.uri.path, \\\\"/\\\\"))", + "ratelimit": { "characteristics": ["ip.src", "cf.colo.id"], "period": 10, "requests_per_period": ${threshold}, "mitigation_timeout": 10 } + } +]`; +const replace = (rules: string, zone = zoneId, phase = 'http_ratelimit') => + `REPLACE cloudflare.rulesets.phases SET rules = '${rules}' WHERE zone_id = '${zone}' AND ruleset_phase = '${phase}'`; + +const declared: Declared = { sql: replace(rule(100)), zoneId, phase: 'http_ratelimit' }; + +test('the declared assertion passes, whatever the whitespace', () => { + assert.deepEqual(check(declared.sql, declared), { ok: true }); + assert.deepEqual(check(` ${declared.sql.replace(/\s+/g, '\n')} ;`, declared), { ok: true }); +}); + +test('rulesOf unescapes the expression the way the SQL parser does', () => { + const rules = rulesOf(declared.sql) as Array<{ expression: string }>; + assert.equal(rules[0].expression, '(starts_with(http.request.uri.path, "/"))'); +}); + +test('a different threshold is not the declaration', () => { + const v = check(replace(rule(1000)), declared); + assert.equal(v.ok, false); + if (!v.ok) { + assert.equal(v.rule, 'declaration'); + assert.match(v.reason, /requests_per_period: 1000 vs declared 100/); + } +}); + +test('a different description is not the declaration', () => { + const v = check(replace(rule(100, 'TEMP bot scare 03:12 - put back later')), declared); + assert.equal(v.ok, false); + if (!v.ok) assert.match(v.reason, /description/); +}); + +test('another zone is out of scope', () => { + const v = check(replace(rule(100), 'ffffffffffffffffffffffffffffffff'), declared); + assert.equal(v.ok, false); + if (!v.ok) assert.equal(v.rule, 'scope'); +}); + +test('another phase is out of scope', () => { + const v = check(replace(rule(100), zoneId, 'http_request_firewall_custom'), declared); + assert.equal(v.ok, false); + if (!v.ok) assert.equal(v.rule, 'scope'); +}); + +test('another resource, or another verb, is off the mutation surface', () => { + for (const sql of [ + `DELETE FROM cloudflare.dns.records WHERE zone_id = '${zoneId}' AND dns_record_id = 'x'`, + `UPDATE cloudflare.rulesets.phases SET rules = '[]' WHERE zone_id = '${zoneId}' AND ruleset_phase = 'http_ratelimit'`, + `INSERT INTO cloudflare.dns.records(zone_id, name) SELECT '${zoneId}', 'x'`, + ]) { + const v = check(sql, declared); + assert.equal(v.ok, false, sql); + if (!v.ok) assert.equal(v.rule, 'mutation surface', sql); + } +}); + +test('two statements are one too many', () => { + const v = check(`${declared.sql}; ${declared.sql}`, declared); + assert.equal(v.ok, false); + if (!v.ok) assert.equal(v.rule, 'one statement'); +}); + +test('a rules value that is not JSON is declined, not thrown', () => { + const v = check(replace('not json'), declared); + assert.equal(v.ok, false); + if (!v.ok) assert.equal(v.rule, 'declaration'); +}); + +test('firstDiff names the first path that differs', () => { + assert.equal(firstDiff({ a: 1, b: [1, 2] }, { a: 1, b: [1, 2] }), undefined); + assert.equal(firstDiff({ a: 1, b: [1, 3] }, { a: 1, b: [1, 2] }), '$.b[1]: 3 vs declared 2'); + assert.equal(firstDiff({ a: 1 }, { a: 1, c: true }), '$.c: missing vs declared true'); +}); diff --git a/10-26 AI Engineering Night/demo/agent/tsconfig.json b/10-26 AI Engineering Night/demo/agent/tsconfig.json new file mode 100644 index 0000000..95375f2 --- /dev/null +++ b/10-26 AI Engineering Night/demo/agent/tsconfig.json @@ -0,0 +1,16 @@ +{ + "compilerOptions": { + "target": "ES2022", + "module": "NodeNext", + "moduleResolution": "NodeNext", + "lib": ["ES2022"], + "types": ["node"], + "strict": true, + "esModuleInterop": true, + "skipLibCheck": true, + "rootDir": "src", + "outDir": "dist", + "resolveJsonModule": true + }, + "include": ["src/**/*.ts"] +} diff --git a/10-26 AI Engineering Night/demo/recordings/agent_dry-run_2026-10-29.txt b/10-26 AI Engineering Night/demo/recordings/agent_dry-run_2026-10-29.txt new file mode 100644 index 0000000..1ff9932 --- /dev/null +++ b/10-26 AI Engineering Night/demo/recordings/agent_dry-run_2026-10-29.txt @@ -0,0 +1,115 @@ ++-- edge-agent -------------------------------------------------------+ +| model gpt-6.1-sol mode read_only tools 15 zone stackql.xyz | +| audit agent\audit\stackql_mcp_server_2026-10-28T20-34-41-960Z.jsonl | ++---------------------------------------------------------------------+ + +declared stack/stackql_manifest.yml + +----------------+-----------+--------+---------+----------------------------------------------------+ + | phase | threshold | period | enabled | description | + +----------------+-----------+--------+---------+----------------------------------------------------+ + | http_ratelimit | 100 | 10 | true | edge-estate rate limit (managed by stackql-deploy) | + +----------------+-----------+--------+---------+----------------------------------------------------+ +permitted write: one REPLACE that asserts the declared rule (policy.ts); the server mode decides if it runs + +[01  3.7s] run_select_query + SELECT id, last_updated, json_extract(rules, '$[0].ratelimit.requests_per_period') AS requests_per_period, json_extract(rules, '$[0].ratelimit.period') AS period, json_extract(rules, '$[0].enabled') AS enabled, json_extract(rules, '$[0].description') AS description FROM cloudflare.rulesets.phases WHERE zone_id = '30850 ... + -> {"rows":[{"description":"TEMP bot scare 03:12 - put back later","enabled":"1","id":"76a93b9990cc46b88c5d544f3a21389a","last_updated":"2026-10-28T20:31:58.414006Z","period":"10","requests_per_period":" ... +[02  2.9s] run_select_query + SELECT sha, json_extract(commit, '$.author.date') AS author_date, json_extract(commit, '$.author.name') AS author_name, json_extract(commit, '$.message') AS message FROM github.repos.commits WHERE owner = 'stackql-labs' AND repo = 'agentic-platform-engineering' AND path = '10-26 AI Engineering Night/demo/stack/stackql_ ... + -> {"rows":[]} +[03  2.8s] run_select_query + SELECT edge_response_status, SUM(requests) AS requests FROM cloudflare.zones.http_requests_adaptive_groups WHERE zone_tag = '30850494d46bca19a6b11b842bdac9ff' AND since = '2026-10-28T19:34:42Z' AND until = '2026-10-28T20:34:42Z' GROUP BY edge_response_status + -> {"rows":[{"edge_response_status":"200","requests":"11"},{"edge_response_status":"403","requests":"13"}]} +[04  4.8s] run_mutation_query + REPLACE cloudflare.rulesets.phases SET rules = '[ { "action": "block", "description": "edge-estate rate limit (managed by stackql-deploy)", "enabled": true, "expression": "(starts_with(http.request.uri.path, \\"/\\"))", "ratelimit": { "characteristics": [ "ip.src", "cf.colo.id" ], "period": 10, "requests_per_period": 1 ... + -> BLOCKED: tool "run_mutation_query" refused: server is in 'read_only' mode + +--- report --------------------------------------------------------------------- +Rate limit review: stackql.xyz + ++------------------+----------------+-------------------+-----------+ +| declared | live | verdict | action | ++------------------+----------------+-------------------+-----------+ +| 100/10s, enabled | 5/10s, enabled | Out-of-band drift | remediate | ++------------------+----------------+-------------------+-----------+ + +The threshold and description differ. No manifest history is visible, so the supplied declaration is treated as current. Remediation was submitted once but refused by the server's read_only mode. No change was applied. + +Evidence + +- Zone: 30850494d46bca19a6b11b842bdac9ff, phase: http_ratelimit. +- Rule: 76a93b9990cc46b88c5d544f3a21389a. +- Live: 5 requests per IP per 10 seconds, enabled. +- Last updated: 2026-10-28T20:31:58.414006Z. +- Live description: TEMP bot scare 03:12 - put back later. This indicates a temporary bot-related restriction; independent confirmation of the reason was not observed. +- Declared: 100 requests per IP per 10 seconds, enabled, description edge-estate rate limit (managed by stackql-deploy). +- Manifest history: zero rows. Author, commit date, message and SHA: not observed. Timestamp ordering against the declaration cannot be established. +- Traffic window: 2026-10-28T19:34:42Z to 2026-10-28T20:34:42Z: + - HTTP 200: 11 requests. + - HTTP 403: 13 requests. + - HTTP 429: not observed. The 403 responses alone do not establish rate-limit impact. + +Permitted statement submitted + + REPLACE cloudflare.rulesets.phases + SET rules = '[ + { + "action": "block", + "description": "edge-estate rate limit (managed by stackql-deploy)", + "enabled": true, + "expression": "(starts_with(http.request.uri.path, \\"/\\"))", + "ratelimit": { + "characteristics": [ + "ip.src", + "cf.colo.id" + ], + "period": 10, + "requests_per_period": 100, + "mitigation_timeout": 10 + } + } + ]' + WHERE zone_id = '30850494d46bca19a6b11b842bdac9ff' AND ruleset_phase = 'http_ratelimit' + +Statement log + +1. Live rule: + SELECT id, last_updated, + json_extract(rules, '$[0].ratelimit.requests_per_period') AS requests_per_period, + json_extract(rules, '$[0].ratelimit.period') AS period, + json_extract(rules, '$[0].enabled') AS enabled, + json_extract(rules, '$[0].description') AS description + FROM cloudflare.rulesets.phases + WHERE zone_id = '30850494d46bca19a6b11b842bdac9ff' + AND ruleset_phase = 'http_ratelimit' + +2. Manifest history: + SELECT sha, + json_extract(commit, '$.author.date') AS author_date, + json_extract(commit, '$.author.name') AS author_name, + json_extract(commit, '$.message') AS message + FROM github.repos.commits + WHERE owner = 'stackql-labs' + AND repo = 'agentic-platform-engineering' + AND path = '10-26 AI Engineering Night/demo/stack/stackql_manifest.yml' + ORDER BY json_extract(commit, '$.author.date') DESC + LIMIT 5 + +3. Traffic: + SELECT edge_response_status, SUM(requests) AS requests + FROM cloudflare.zones.http_requests_adaptive_groups + WHERE zone_tag = '30850494d46bca19a6b11b842bdac9ff' + AND since = '2026-10-28T19:34:42Z' + AND until = '2026-10-28T20:34:42Z' + GROUP BY edge_response_status + +4. The exact REPLACE statement above, attempted once and refused. No retry or workaround. + ++-- run --------------------------------------------------------------+ +| model gpt-6.1-sol mode read_only finish stop elapsed 23.9s | +| steps 5 tool calls run_select_query=3 run_mutation_query=1 | +| writes attempted 1 (the gates decided what happened to them) | +| tokens in=45990 out=1593 total=47583 | +| audit agent\audit\stackql_mcp_server_2026-10-28T20-34-41-960Z.jsonl | +| transcript agent\audit\agent_2026-10-28T20-34-41-960Z.txt | ++---------------------------------------------------------------------+ diff --git a/10-26 AI Engineering Night/demo/runsheet.md b/10-26 AI Engineering Night/demo/runsheet.md new file mode 100644 index 0000000..a05477c --- /dev/null +++ b/10-26 AI Engineering Night/demo/runsheet.md @@ -0,0 +1,212 @@ +# Runsheet + +Terminal 1 is Git Bash (the scripts), Terminal 2 is PowerShell (the agent). No cluster, no proxy: the estate is one Cloudflare zone. + +## Prerequisites + +Before the day: push this folder to origin. The commits query and `out_of_band` read the manifest's history from GitHub and return nothing until it is there. Drift the zone after the push, so the rule's `last_updated` is newer than the newest commit. + +**Terminal 1 (Git Bash):** + +```bash +cd "/c/LocalGitRepos/stackql/demo-repos/agentic-platform-engineering/10-26 AI Engineering Night/demo" +scripts/shell.sh # renders runs/shell.iql from .env +stackql shell # from this directory: its .stackql holds the cloudflare and github providers +``` + +**Terminal 2 (PowerShell, 120 columns or wider):** + +```powershell +cd "C:\LocalGitRepos\stackql\demo-repos\agentic-platform-engineering\10-26 AI Engineering Night\demo\agent" +npm run tools +# about a second: mode read_only, 15 tools, four marked [write] +``` + +The zone should already be drifted (threshold 5, description `TEMP bot scare 03:12 - put back later`). If it is not, in a Git Bash window: + +```bash +scripts/restore.sh && scripts/drift.sh +``` + +## StackQL Background + +**StackQL** is a SQL interface to cloud and SaaS APIs: + +- providers are versioned interface documents pulled from a registry +- every query is a live API call +- there is no state file + +The **StackQL MCP server** puts that surface in front of an agent as a small, fixed set of tools: + +- **universal**: same query shape across every provider +- **self-describing**: the agent discovers the one contract it needs, not a tool per API +- **gated**: the server mode decides whether a write runs, not the prompt +- **audited**: every statement is an event + +### Quick demo, Terminal 1 (`stackql shell`, paste from `runs/shell.iql`): + +```sql +-- What is live: the rate limit rule on the zone +SELECT json_extract(rules, '$[0].description') AS description, + json_extract(rules, '$[0].ratelimit.requests_per_period') AS threshold, + json_extract(rules, '$[0].ratelimit.period') AS period, + json_extract(rules, '$[0].enabled') AS enabled, + last_updated +FROM cloudflare.rulesets.phases +WHERE zone_id = '' AND ruleset_phase = 'http_ratelimit'; +``` + +Threshold 5 against a declared 100, and a description that says someone meant to put it back. The `last_updated` is the 3am. (The shell wraps the description over two lines; the numbers are on the first.) + +Same query shape, different provider: + +```sql +-- What is declared, and when it last changed +SELECT substr(sha, 1, 7) AS sha, + json_extract(commit, '$.author.date') AS committed, + json_extract(commit, '$.author.name') AS author, + json_extract(commit, '$.message') AS message +FROM github.repos.commits +WHERE owner = 'stackql-labs' AND repo = 'agentic-platform-engineering' + AND path = '10-26 AI Engineering Night/demo/stack/stackql_manifest.yml' +ORDER BY committed DESC LIMIT 3; +``` + +The declaration last changed before the rule did. Nobody changed the intent; someone changed the zone. (Zero rows means the folder is not on origin yet; the agent says "not observed" and carries on.) + +One question across the two (`CROSS JOIN`; the comma form of the join does not plan): + +```sql +-- Did the rule change after the declaration did? +SELECT r.last_updated AS rule_changed, + c.committed AS declaration_changed, + r.last_updated > c.committed AS out_of_band +FROM (SELECT last_updated FROM cloudflare.rulesets.phases + WHERE zone_id = '' AND ruleset_phase = 'http_ratelimit') r +CROSS JOIN + (SELECT max(json_extract(commit, '$.author.date')) AS committed FROM github.repos.commits + WHERE owner = 'stackql-labs' AND repo = 'agentic-platform-engineering' + AND path = '10-26 AI Engineering Night/demo/stack/stackql_manifest.yml') c; +``` + +`out_of_band = 1`: that is the diagnosis, in one query, across two providers. (`null` in both right-hand columns until the folder is pushed.) + +```sql +-- Who is being blocked: requests by edge status in the last hour +SELECT edge_response_status AS status, sum(requests) AS requests +FROM cloudflare.zones.http_requests_adaptive_groups +WHERE zone_tag = '' AND since = '<1h ago>' AND until = '' +GROUP BY edge_response_status ORDER BY requests DESC; +``` + +429s, if there has been traffic. Zero rows is a valid answer; say so. In rehearsal the hour held a few dozen 200s and 403s and no 429s: nothing had tripped the rule, and the agent said so rather than inventing a count. + +## Overview + +This is a scheduled platform agent that... + +- reads the live state of a Cloudflare zone through StackQL +- compares it with what the stack manifest declares, and when that declaration last changed +- decides whether to converge, and proposes exactly one write + +Whether that write runs is decided by three gates the platform owns, none of which the model can change: a policy in code, the server's mode, and a human at the terminal. + +| File | Point at | +| --- | --- | +| [stack/stackql_manifest.yml](stack/stackql_manifest.yml#L31-L46) | The desired state (no state file). | +| [stack/resources/rate_limit.iql](stack/resources/rate_limit.iql#L7-L24) | The assertion: one REPLACE, the same whether it creates or converges. | +| [agent/prompts/intent.md](agent/prompts/intent.md#L11-L37) | The briefing, in prose: live state first, then decide, one write at most. | +| [agent/prompts/system.md](agent/prompts/system.md#L38-L53) | The mutation surface: one statement, given to the model verbatim. | +| [agent/src/policy.ts](agent/src/policy.ts#L1-L14) | The policy gate: four rules, pure function, tested. | +| [agent/src/agent.ts](agent/src/agent.ts#L70-L75) | The platform sets the server mode in code (`read_only` or `safe`, never `full_access`). | +| [agent/src/agent.ts](agent/src/agent.ts#L268-L285) | The gate wraps the write tool: policy first, then the server, then the human. | +| `.env` | The token: scoped to one zone, held by the server process, never seen by the model. | + +## Run + +**Terminal 2 (PowerShell, in `demo/agent`):** + +The agent runs in `read_only` mode: it reads the rule, the commits and the traffic, diagnoses out-of-band drift, proposes the one permitted write, and the server refuses it. The agent reports and stops; it does not retry. + +```powershell +npm run agent +``` + +25 to 30 seconds, 5 steps. On screen, in order: the `edge-agent` box (model, mode, tools, zone), the declared table, then one block per statement: + +- `[01]` `run_select_query`, the rule: `TEMP bot scare 03:12 - put back later`, threshold 5, the `last_updated` +- `[02]` `run_select_query`, the commits: rows once the folder is pushed, `{"rows":[]}` until then +- `[03]` `run_select_query`, the traffic: a row per edge status for the hour +- `[04]` `run_mutation_query`, the REPLACE, then `-> BLOCKED: tool "run_mutation_query" refused: server is in 'read_only' mode` + +Then the report: a four-cell status table ending in `remediate`, the evidence with the rule id and timestamps, the statement "submitted, refused", the statements it ran. Then the `run` box: `steps 5`, `tool calls run_select_query=3 run_mutation_query=1`, `writes attempted 1`, about 46k tokens in, the audit sink and the transcript paths. + +The same proposal, replayed through the gates with the server in `safe` mode: the policy passes it, the approval box appears, type `y`, and the rule is restored. This is the recovery path for a blocked change: a human, the exact statement, nothing else. + +```powershell +$env:STACKQL_MCP_MODE='safe'; npm run probe +``` + +Under ten seconds plus the pause at the prompt. `mode safe`, `source agent\audit\last-proposal.sql`, the write on one line, then the `APPROVAL REQUIRED` box with the server's question, `Approve run_mutation_query (mutation_create)?`, and the full statement. `approve? [y/N]`, type `y`. The result: `Mutation Result`, `The operation was despatched successfully`, and a timestamp. Terminal 1, first query again: threshold 100, the declared description, a new `last_updated`. + +A plausible write that is not the declaration (threshold 1000). The policy declines it before it reaches the server or a human; nobody is asked. The mode is still `safe` in this window; the policy runs first either way. + +```powershell +npm run probe -- ../scripts/denied.sql +``` + +About two seconds, and the one line that matters, in red: + +```text +tool "run_mutation_query" declined by policy (declaration): not the declared rule: $[0].ratelimit.requests_per_period: 1000 vs declared 100 +``` + +No box, no server record: the decline is the gate's own audit record. + +The audit sink: every run of the evening in one table, oldest first, local time: the tool, the mode, the decision, the duration and the statement. Never the rows returned. The policy decline is in there too. + +```powershell +npm run audit +``` + +Expect: three `allow` rows and one `refuse_immediate` from the `read_only` run, one `needs_approval_accepted` from the `safe` run (with its duration), one `declined_by_policy` from the gate. + +[OPTIONAL] The other recovery path: no agent, no approval, re-assert the declaration. The createorupdate anchor runs the same REPLACE whether the rule is missing, drifted or already right, and the statecheck confirms it: `[rate_limit] is in the desired state`. The same command that created it. Git Bash: + +```bash +scripts/restore.sh +``` + +About 17 seconds, most of it stackql-deploy starting its own server. With `--show-queries` every statement is printed in full: the REPLACE, the exists check, the statecheck, the exports; readable SQL, not JSON. The script then reads the rule back: threshold 100, the declared description, a new `last_updated`. + +## Summary + +- the agent queried reality before it acted; + - the live rule, the declaration's history, the traffic, and every decision was grounded in what the API returned +- it worked two providers, Cloudflare and GitHub, through one query shape; + - with no SDK and no tool per API +- it could propose exactly one write, an assertion of the declaration; + - idempotent, so a retry or a re-run is safe +- whether that write ran was decided by the platform, three times; + - policy in code, server mode, a human bound to the exact statement +- blocked meant report and stop, not rephrase and retry; + - and recovery was a human through the gate, or the declaration re-asserted +- every statement it ran is already an audit event + +## Full reset + +```bash +cd "/c/LocalGitRepos/stackql/demo-repos/agentic-platform-engineering/10-26 AI Engineering Night/demo" +scripts/restore.sh && scripts/drift.sh +``` + +## Troubleshooting + +- No network or no key: `npm run replay` (banner says replay; say so). It plays the newest transcript in `agent/audit/`, else `recordings/agent_dry-run_2026-10-29.txt`; a file can be given: `npm run replay -- ../recordings/agent_dry-run_2026-10-29.txt`. +- `npm run probe` approves without asking: the server is still `read_only` in this shell, or stdin is not a TTY. Check `$env:STACKQL_MCP_MODE`, use a plain PowerShell window. +- `npm run probe` with no file replays the agent's last proposal, never the denied statement: only the agent writes `agent/audit/last-proposal.sql`. +- Cloudflare rejects the REPLACE (400): the token is missing Zone WAF Edit, or the free-plan constraints were changed (period 10, both characteristics, `starts_with`). +- GitHub query returns zero rows: the demo folder has not been pushed yet, or `DECLARATION_PATH` differs from the path in the repo. +- `stackql shell` cannot find a provider: stackql's approot is `.stackql` under the directory it runs from. Run it from the demo root, where `scripts/build.sh` pulled cloudflare, and pull github there once: `stackql exec "REGISTRY PULL github"`. +- `stackql exec` says `unknown flag: --`: the statement starts with a `--` comment line, which the CLI reads as a flag. The scripts strip comment lines; paste statements without their leading comments. +- The agent takes extra steps: the briefing gives the traffic window and tells it not to use `CURRENT_TIMESTAMP`, `strftime` or `CAST(... AS INTEGER)`, which do not parse. `AGENT_MAX_STEPS` (12) still caps a bad day. diff --git a/10-26 AI Engineering Night/demo/scripts/_env.sh b/10-26 AI Engineering Night/demo/scripts/_env.sh new file mode 100644 index 0000000..df878db --- /dev/null +++ b/10-26 AI Engineering Night/demo/scripts/_env.sh @@ -0,0 +1,39 @@ +# Sourced by the other scripts: loads .env from the demo root, applies defaults, and +# refuses to run without a zone. Values already in the environment win over .env. +# . "$(dirname "$0")/_env.sh" + +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +cd "$ROOT" + +# .env, with anything already in the environment kept over it (an empty line in .env +# would otherwise blank an exported value) +_names="$(sed -nE 's/^[[:space:]]*(export[[:space:]]+)?([A-Za-z_][A-Za-z0-9_]*)=.*/\2/p' "$ROOT/.env" 2>/dev/null || true)" +for _n in $_names; do [ -z "${!_n+x}" ] || printf -v "_keep_$_n" '%s' "${!_n}"; done +set -a +[ -f "$ROOT/.env" ] && . "$ROOT/.env" +set +a +for _n in $_names; do _k="_keep_$_n"; [ -z "${!_k+x}" ] || export "$_n=${!_k}"; done +unset _n _k _names + +# stackql's own approot (~/.stackql) unless .env names one; only then is --approot passed +APPROOT_ARGS=() +if [ -n "${STACKQL_APPROOT:-}" ]; then APPROOT_ARGS=(--approot "$STACKQL_APPROOT"); fi +STACKQL_SHELL="stackql shell${STACKQL_APPROOT:+ --approot \"$STACKQL_APPROOT\"}" +# stackql-deploy is a pip entry point: the one on the Git Bash PATH, unless .env names one +export STACKQL_DEPLOY="${STACKQL_DEPLOY:-stackql-deploy}" +# stackql-deploy draws its progress boxes with characters the default Windows console +# code page cannot encode +export PYTHONUTF8=1 + +# the blast radius is the token's: scoped to one zone. The scripts still refuse to run +# without saying which zone that is, so the room can see it. +[ -n "${CLOUDFLARE_API_TOKEN:-}" ] || { echo "CLOUDFLARE_API_TOKEN is not set (see .env.example)" >&2; exit 2; } +[ -n "${CLOUDFLARE_ZONE_ID:-}" ] || { echo "CLOUDFLARE_ZONE_ID is not set (see .env.example)" >&2; exit 2; } +[ -n "${CLOUDFLARE_ZONE_NAME:-}" ] || { echo "CLOUDFLARE_ZONE_NAME is not set: name the zone you are about to change" >&2; exit 2; } +if [ -z "${_ZONE_SHOWN:-}" ]; then echo "zone: $CLOUDFLARE_ZONE_NAME ($CLOUDFLARE_ZONE_ID)" >&2; export _ZONE_SHOWN=1; fi + +# Run a statement with the local stackql, echoing it so the room can read it. +sql() { + echo "+ stackql exec $*" >&2 + stackql exec ${APPROOT_ARGS[@]+"${APPROOT_ARGS[@]}"} "$@" +} diff --git a/10-26 AI Engineering Night/demo/scripts/build.sh b/10-26 AI Engineering Night/demo/scripts/build.sh new file mode 100644 index 0000000..80c648a --- /dev/null +++ b/10-26 AI Engineering Night/demo/scripts/build.sh @@ -0,0 +1,10 @@ +#!/usr/bin/env bash +# Assert the rule from stack/stackql_manifest.yml against the live API. No state file: +# exists -> createorupdate, statecheck -> createorupdate. Run it to create the rule the +# first time, and run it again to converge whatever drifted since. Extra arguments go +# to stackql-deploy (for example --dry-run). +set -euo pipefail +. "$(dirname "$0")/_env.sh" + +echo "+ stackql-deploy build stack demo --env-file .env --show-queries $*" >&2 +exec "$STACKQL_DEPLOY" build stack demo --env-file .env --show-queries "$@" diff --git a/10-26 AI Engineering Night/demo/scripts/denied.sql b/10-26 AI Engineering Night/demo/scripts/denied.sql new file mode 100644 index 0000000..a02efde --- /dev/null +++ b/10-26 AI Engineering Night/demo/scripts/denied.sql @@ -0,0 +1,20 @@ +-- A plausible write that is not the declaration: same resource, same zone, but a +-- threshold of 1000 instead of the declared 100. The policy declines it before it +-- reaches the server or a human: `npm run probe -- ../scripts/denied.sql` +REPLACE cloudflare.rulesets.phases +SET rules = '[ + { + "action": "block", + "description": "edge-estate rate limit (managed by stackql-deploy)", + "enabled": true, + "expression": "(starts_with(http.request.uri.path, \\"/\\"))", + "ratelimit": { + "characteristics": ["ip.src", "cf.colo.id"], + "period": 10, + "requests_per_period": 1000, + "mitigation_timeout": 10 + } + } +]' +WHERE zone_id = '{{ zone_id }}' + AND ruleset_phase = 'http_ratelimit' diff --git a/10-26 AI Engineering Night/demo/scripts/drift.sh b/10-26 AI Engineering Night/demo/scripts/drift.sh new file mode 100644 index 0000000..56e339d --- /dev/null +++ b/10-26 AI Engineering Night/demo/scripts/drift.sh @@ -0,0 +1,12 @@ +#!/usr/bin/env bash +# The 3am change (scripts/drift.sql): the rate limit tightened from the declared 100 to +# 5 requests per 10 s, with a note in the description, outside the declaration. One +# statement, so it is repeatable on stage. Undo with scripts/restore.sh (the +# declaration wins) or let the agent propose it. +set -euo pipefail +. "$(dirname "$0")/_env.sh" + +sql "$(sed -e "s/{{ zone_id }}/$CLOUDFLARE_ZONE_ID/" -e "/^--/d" "$ROOT/scripts/drift.sql")" + +echo >&2 +echo "the human view in SQL: $STACKQL_SHELL (paste from runs/shell.iql, rendered by scripts/shell.sh)" >&2 diff --git a/10-26 AI Engineering Night/demo/scripts/drift.sql b/10-26 AI Engineering Night/demo/scripts/drift.sql new file mode 100644 index 0000000..08f6339 --- /dev/null +++ b/10-26 AI Engineering Night/demo/scripts/drift.sql @@ -0,0 +1,21 @@ +-- The 3am change. Someone saw a burst of bot traffic, tightened the rule in the +-- dashboard from 100 to 5 requests per 10 s per IP, left a note in the description, +-- and went back to bed. Every page load with more than five assets now gets a 429. +-- Same REPLACE shape as the declaration; the difference is it never went through it. +REPLACE cloudflare.rulesets.phases +SET rules = '[ + { + "action": "block", + "description": "TEMP bot scare 03:12 - put back later", + "enabled": true, + "expression": "(starts_with(http.request.uri.path, \\"/\\"))", + "ratelimit": { + "characteristics": ["ip.src", "cf.colo.id"], + "period": 10, + "requests_per_period": 5, + "mitigation_timeout": 10 + } + } +]' +WHERE zone_id = '{{ zone_id }}' + AND ruleset_phase = 'http_ratelimit'; diff --git a/10-26 AI Engineering Night/demo/scripts/restore.sh b/10-26 AI Engineering Night/demo/scripts/restore.sh new file mode 100644 index 0000000..bd18e5f --- /dev/null +++ b/10-26 AI Engineering Night/demo/scripts/restore.sh @@ -0,0 +1,20 @@ +#!/usr/bin/env bash +# Converge the rule back to the declaration: the idempotent-assertion beat, and the +# recovery path that needs no agent and no approval. It is scripts/build.sh under another +# name, because it is the same operation: the createorupdate anchor runs the same REPLACE +# whether the rule is missing, drifted or already right, and the statecheck confirms the +# result. Then the rule is read back, so the room sees where it ended. +set -euo pipefail +. "$(dirname "$0")/_env.sh" + +"$ROOT/scripts/build.sh" "$@" + +echo >&2 +echo "the rule as live, after the build:" >&2 +sql "SELECT json_extract(rules, '\$[0].description') AS description, + json_extract(rules, '\$[0].ratelimit.requests_per_period') AS threshold, + json_extract(rules, '\$[0].ratelimit.period') AS period, + json_extract(rules, '\$[0].enabled') AS enabled, + last_updated +FROM cloudflare.rulesets.phases +WHERE zone_id = '$CLOUDFLARE_ZONE_ID' AND ruleset_phase = 'http_ratelimit'" diff --git a/10-26 AI Engineering Night/demo/scripts/shell.iql b/10-26 AI Engineering Night/demo/scripts/shell.iql new file mode 100644 index 0000000..4d507d8 --- /dev/null +++ b/10-26 AI Engineering Night/demo/scripts/shell.iql @@ -0,0 +1,36 @@ +-- The human view of the estate, for `stackql shell`. scripts/shell.sh renders the +-- placeholders from .env into runs/shell.iql; paste from there. + +-- What is live: the rate limit rule on the zone +SELECT json_extract(rules, '$[0].description') AS description, + json_extract(rules, '$[0].ratelimit.requests_per_period') AS threshold, + json_extract(rules, '$[0].ratelimit.period') AS period, + json_extract(rules, '$[0].enabled') AS enabled, + last_updated +FROM cloudflare.rulesets.phases +WHERE zone_id = '{{ zone_id }}' AND ruleset_phase = 'http_ratelimit'; + +-- What is declared, and when it last changed: same query shape, different provider +SELECT substr(sha, 1, 7) AS sha, + json_extract(commit, '$.author.date') AS committed, + json_extract(commit, '$.author.name') AS author, + json_extract(commit, '$.message') AS message +FROM github.repos.commits +WHERE owner = '{{ owner }}' AND repo = '{{ repo }}' AND path = '{{ path }}' +ORDER BY committed DESC LIMIT 3; + +-- Did the rule change after the declaration did? One question across two providers. +SELECT r.last_updated AS rule_changed, + c.committed AS declaration_changed, + r.last_updated > c.committed AS out_of_band +FROM (SELECT last_updated FROM cloudflare.rulesets.phases + WHERE zone_id = '{{ zone_id }}' AND ruleset_phase = 'http_ratelimit') r +CROSS JOIN + (SELECT max(json_extract(commit, '$.author.date')) AS committed FROM github.repos.commits + WHERE owner = '{{ owner }}' AND repo = '{{ repo }}' AND path = '{{ path }}') c; + +-- Who is being blocked: requests by edge status in the last hour (zero rows is a valid answer) +SELECT edge_response_status AS status, sum(requests) AS requests +FROM cloudflare.zones.http_requests_adaptive_groups +WHERE zone_tag = '{{ zone_id }}' AND since = '{{ since }}' AND until = '{{ until }}' +GROUP BY edge_response_status ORDER BY requests DESC; diff --git a/10-26 AI Engineering Night/demo/scripts/shell.sh b/10-26 AI Engineering Night/demo/scripts/shell.sh new file mode 100644 index 0000000..955fdf0 --- /dev/null +++ b/10-26 AI Engineering Night/demo/scripts/shell.sh @@ -0,0 +1,20 @@ +#!/usr/bin/env bash +# Render scripts/shell.iql with the values from .env into runs/shell.iql, for pasting +# into `stackql shell` (with --approot only if .env sets one). The traffic window is the last hour. +set -euo pipefail +. "$(dirname "$0")/_env.sh" + +owner="${DECLARATION_REPO%%/*}" +repo="${DECLARATION_REPO##*/}" +since="$(date -u -d '-1 hour' +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || date -u -v-1H +%Y-%m-%dT%H:%M:%SZ)" +until="$(date -u +%Y-%m-%dT%H:%M:%SZ)" + +mkdir -p runs +sed -e "s|{{ zone_id }}|$CLOUDFLARE_ZONE_ID|g" \ + -e "s|{{ owner }}|$owner|g" \ + -e "s|{{ repo }}|$repo|g" \ + -e "s|{{ path }}|$DECLARATION_PATH|g" \ + -e "s|{{ since }}|$since|g" \ + -e "s|{{ until }}|$until|g" \ + scripts/shell.iql > runs/shell.iql +echo "rendered runs/shell.iql; paste from it into: $STACKQL_SHELL" >&2 diff --git a/10-26 AI Engineering Night/demo/stack/resources/rate_limit.iql b/10-26 AI Engineering Night/demo/stack/resources/rate_limit.iql new file mode 100644 index 0000000..bac0491 --- /dev/null +++ b/10-26 AI Engineering Night/demo/stack/resources/rate_limit.iql @@ -0,0 +1,51 @@ +/*+ exists */ +SELECT json_array_length(rules) AS count +FROM cloudflare.rulesets.phases +WHERE zone_id = '{{ zone_id }}' + AND ruleset_phase = '{{ ruleset_phase }}' + +/*+ createorupdate */ +REPLACE cloudflare.rulesets.phases +SET rules = '[ + { + "action": "block", + "description": "{{ description }}", + "enabled": true, + "expression": "(starts_with(http.request.uri.path, \\"/\\"))", + "ratelimit": { + "characteristics": ["ip.src", "cf.colo.id"], + "period": {{ period }}, + "requests_per_period": {{ threshold }}, + "mitigation_timeout": {{ period }} + } + } +]' +WHERE zone_id = '{{ zone_id }}' + AND ruleset_phase = '{{ ruleset_phase }}' + +/*+ statecheck, retries=2, retry_delay=3 */ +SELECT COUNT(*) AS count FROM ( + SELECT json_extract(rules, '$[0].description') AS description, + json_extract(rules, '$[0].ratelimit.requests_per_period') AS threshold, + json_extract(rules, '$[0].ratelimit.period') AS period, + json_extract(rules, '$[0].enabled') AS enabled + FROM cloudflare.rulesets.phases + WHERE zone_id = '{{ zone_id }}' + AND ruleset_phase = '{{ ruleset_phase }}' +) t +WHERE t.description = '{{ description }}' + AND t.threshold = {{ threshold }} + AND t.period = {{ period }} + AND t.enabled = 1 + +/*+ exports */ +SELECT json_extract(rules, '$[0].id') AS rate_limit_rule_id +FROM cloudflare.rulesets.phases +WHERE zone_id = '{{ zone_id }}' + AND ruleset_phase = '{{ ruleset_phase }}' + +/*+ delete */ +REPLACE cloudflare.rulesets.phases +SET rules = '[]' +WHERE zone_id = '{{ zone_id }}' + AND ruleset_phase = '{{ ruleset_phase }}' diff --git a/10-26 AI Engineering Night/demo/stack/stackql_manifest.yml b/10-26 AI Engineering Night/demo/stack/stackql_manifest.yml new file mode 100644 index 0000000..e83d9fa --- /dev/null +++ b/10-26 AI Engineering Night/demo/stack/stackql_manifest.yml @@ -0,0 +1,60 @@ +# +# edge-estate - one rate limit rule on a Cloudflare zone, as a stackql-deploy stack +# +# No state file: every build asserts the rule against the live API (exists -> +# createorupdate, statecheck -> createorupdate), so a re-run converges whatever a +# human or an agent changed in between. +# +# This file is also the "what should be" of the demo: the agent reads the declared +# threshold, period and description from here and compares them with what the zone +# reports. scripts/drift.sh makes the 3am change; `stackql-deploy build` puts it back. +# +# Build: stackql-deploy build stack demo --env-file .env --show-queries +# Dry run: stackql-deploy build stack demo --env-file .env --dry-run +# Teardown: stackql-deploy teardown stack demo --env-file .env (empties the phase) +# +version: 1 +name: edge-estate +description: a rate limit rule on one Cloudflare zone, for the guardrails demo +providers: + - cloudflare::v26.07.00424 + +globals: + - name: zone_id + description: the one zone the token can reach + value: "{{ CLOUDFLARE_ZONE_ID }}" + - name: ruleset_phase + value: http_ratelimit + +resources: + + - name: rate_limit + description: | + block any client over the threshold. The threshold is the value scripts/drift.sh + changes; the statecheck compares it, so a build after the drift runs the + createorupdate anchor and converges the rule back. + props: + - name: description + value: "edge-estate rate limit (managed by stackql-deploy)" + - name: threshold + description: requests per period, per client IP + value: 100 + - name: period + description: seconds. 10 is the only value the free plan allows + value: 10 + exports: + - rate_limit_rule_id + + # what the estate looks like once the stack has converged + - name: estate_summary + type: query + props: [] + sql: | + SELECT json_extract(rules, '$[0].description') AS description, + json_extract(rules, '$[0].ratelimit.requests_per_period') AS threshold, + json_extract(rules, '$[0].ratelimit.period') AS period, + json_extract(rules, '$[0].enabled') AS enabled, + last_updated + FROM cloudflare.rulesets.phases + WHERE zone_id = '{{ zone_id }}' + AND ruleset_phase = '{{ ruleset_phase }}'; diff --git a/11-26 AI Engineer/demo/README.md b/12-26 AI Engineer/demo/README.md similarity index 100% rename from 11-26 AI Engineer/demo/README.md rename to 12-26 AI Engineer/demo/README.md diff --git a/11-26 AI Engineer/demo/fixtures/release.json b/12-26 AI Engineer/demo/fixtures/release.json similarity index 100% rename from 11-26 AI Engineer/demo/fixtures/release.json rename to 12-26 AI Engineer/demo/fixtures/release.json diff --git a/11-26 AI Engineer/demo/package.json b/12-26 AI Engineer/demo/package.json similarity index 100% rename from 11-26 AI Engineer/demo/package.json rename to 12-26 AI Engineer/demo/package.json diff --git a/11-26 AI Engineer/demo/src/demo.js b/12-26 AI Engineer/demo/src/demo.js similarity index 100% rename from 11-26 AI Engineer/demo/src/demo.js rename to 12-26 AI Engineer/demo/src/demo.js diff --git a/11-26 AI Engineer/demo/src/policy.js b/12-26 AI Engineer/demo/src/policy.js similarity index 100% rename from 11-26 AI Engineer/demo/src/policy.js rename to 12-26 AI Engineer/demo/src/policy.js diff --git a/11-26 AI Engineer/demo/test/policy.test.js b/12-26 AI Engineer/demo/test/policy.test.js similarity index 100% rename from 11-26 AI Engineer/demo/test/policy.test.js rename to 12-26 AI Engineer/demo/test/policy.test.js diff --git a/11-26 AI Engineer/rehearsal.md b/12-26 AI Engineer/rehearsal.md similarity index 100% rename from 11-26 AI Engineer/rehearsal.md rename to 12-26 AI Engineer/rehearsal.md diff --git a/11-26 AI Engineer/slides.md b/12-26 AI Engineer/slides.md similarity index 100% rename from 11-26 AI Engineer/slides.md rename to 12-26 AI Engineer/slides.md diff --git a/11-26 AI Engineer/workshop/facilitator-runbook.md b/12-26 AI Engineer/workshop/facilitator-runbook.md similarity index 100% rename from 11-26 AI Engineer/workshop/facilitator-runbook.md rename to 12-26 AI Engineer/workshop/facilitator-runbook.md diff --git a/11-26 AI Engineer/workshop/participant-workbook.md b/12-26 AI Engineer/workshop/participant-workbook.md similarity index 100% rename from 11-26 AI Engineer/workshop/participant-workbook.md rename to 12-26 AI Engineer/workshop/participant-workbook.md diff --git a/11-26 AI Engineer/workshop/release-gates.md b/12-26 AI Engineer/workshop/release-gates.md similarity index 100% rename from 11-26 AI Engineer/workshop/release-gates.md rename to 12-26 AI Engineer/workshop/release-gates.md diff --git a/11-26 AI Engineer/workshop/scenarios/README.md b/12-26 AI Engineer/workshop/scenarios/README.md similarity index 100% rename from 11-26 AI Engineer/workshop/scenarios/README.md rename to 12-26 AI Engineer/workshop/scenarios/README.md diff --git a/11-26 AI Engineer/workshop/scenarios/fixture.json b/12-26 AI Engineer/workshop/scenarios/fixture.json similarity index 100% rename from 11-26 AI Engineer/workshop/scenarios/fixture.json rename to 12-26 AI Engineer/workshop/scenarios/fixture.json diff --git a/11-26 AI Engineer/workshop/scenarios/policy-cases.json b/12-26 AI Engineer/workshop/scenarios/policy-cases.json similarity index 100% rename from 11-26 AI Engineer/workshop/scenarios/policy-cases.json rename to 12-26 AI Engineer/workshop/scenarios/policy-cases.json diff --git a/11-26 AI Engineer/workshop/scenarios/python/openai_agent.py b/12-26 AI Engineer/workshop/scenarios/python/openai_agent.py similarity index 100% rename from 11-26 AI Engineer/workshop/scenarios/python/openai_agent.py rename to 12-26 AI Engineer/workshop/scenarios/python/openai_agent.py diff --git a/11-26 AI Engineer/workshop/scenarios/python/requirements.txt b/12-26 AI Engineer/workshop/scenarios/python/requirements.txt similarity index 100% rename from 11-26 AI Engineer/workshop/scenarios/python/requirements.txt rename to 12-26 AI Engineer/workshop/scenarios/python/requirements.txt diff --git a/11-26 AI Engineer/workshop/scenarios/python/safety.py b/12-26 AI Engineer/workshop/scenarios/python/safety.py similarity index 100% rename from 11-26 AI Engineer/workshop/scenarios/python/safety.py rename to 12-26 AI Engineer/workshop/scenarios/python/safety.py diff --git a/11-26 AI Engineer/workshop/scenarios/python/test_delivery_content.py b/12-26 AI Engineer/workshop/scenarios/python/test_delivery_content.py similarity index 100% rename from 11-26 AI Engineer/workshop/scenarios/python/test_delivery_content.py rename to 12-26 AI Engineer/workshop/scenarios/python/test_delivery_content.py diff --git a/11-26 AI Engineer/workshop/scenarios/python/test_safety.py b/12-26 AI Engineer/workshop/scenarios/python/test_safety.py similarity index 100% rename from 11-26 AI Engineer/workshop/scenarios/python/test_safety.py rename to 12-26 AI Engineer/workshop/scenarios/python/test_safety.py diff --git a/11-26 AI Engineer/workshop/scenarios/python/test_tracks.py b/12-26 AI Engineer/workshop/scenarios/python/test_tracks.py similarity index 100% rename from 11-26 AI Engineer/workshop/scenarios/python/test_tracks.py rename to 12-26 AI Engineer/workshop/scenarios/python/test_tracks.py diff --git a/11-26 AI Engineer/workshop/scenarios/python/tracks.py b/12-26 AI Engineer/workshop/scenarios/python/tracks.py similarity index 100% rename from 11-26 AI Engineer/workshop/scenarios/python/tracks.py rename to 12-26 AI Engineer/workshop/scenarios/python/tracks.py diff --git a/11-26 AI Engineer/workshop/scenarios/tracks.json b/12-26 AI Engineer/workshop/scenarios/tracks.json similarity index 100% rename from 11-26 AI Engineer/workshop/scenarios/tracks.json rename to 12-26 AI Engineer/workshop/scenarios/tracks.json diff --git a/11-26 AI Engineer/workshop/tracks/README.md b/12-26 AI Engineer/workshop/tracks/README.md similarity index 100% rename from 11-26 AI Engineer/workshop/tracks/README.md rename to 12-26 AI Engineer/workshop/tracks/README.md