diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 8f3f941e6..ec7841caf 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -2013,6 +2013,13 @@ jobs: ca-certificates \ openssl + # Signature fixtures are generated on the runner, outside the tested image. + - name: Set up Go for signature fixtures + uses: actions/setup-go@v7.0.0 + if: env.BUILD_IMAGE_REQUIRED == 'true' + with: + go-version-file: go.mod + - name: Setup Python uses: actions/setup-python@v7.0.0 with: diff --git a/README.md b/README.md index db1a6d277..8d70c119a 100644 --- a/README.md +++ b/README.md @@ -309,6 +309,9 @@ docker pull stackql/stackql ## Usage +Caller authentication and permissions: [configuration](docs/access_management.md) and +[four tested walkthroughs](docs/examples/access-management/README.md). + StackQL can be used via the interactive REPL shell, or via the `exec` command or ran as a server using the [Postgres wire protocol](https://www.postgresql.org/docs/current/protocol.html). > ℹ️ StackQL does not require or install a database. diff --git a/docs/access_management.md b/docs/access_management.md new file mode 100644 index 000000000..a42759cf9 --- /dev/null +++ b/docs/access_management.md @@ -0,0 +1,221 @@ +# Opt-in caller access and IaC + +Caller access is disabled by default. Enable it with +`--access.config '{"path":"/path/to/access.json"}'`. The source envelope requires a nonempty `path` +and rejects unknown fields and trailing JSON. The previous bare filename form remains accepted. + +Start with the [four executable walkthroughs](examples/access-management/README.md). +They are run directly by Robot tests and need no cloud account. +Caller credentials are independent of provider credentials and SQL `AUTH` statements. + +## Configuration + +Select one authentication `type`: `jwt`, `bearer`, `mtls` or `local_os`. Omitted `type` means `jwt` +for compatibility. Configuration is strict JSON; unknown fields and fields belonging to another mode +are rejected. All modes use the same principal directory and policy rules. + +JWT uses an explicit issuer, audience and pinned public keys: + +```json +{ + "type": "jwt", + "issuer": "https://identity.example", + "audience": "stackql", + "enforce_session_deadline": false, + "max_session_seconds": 300, + "keys": [ + {"kid": "primary", "algorithm": "EdDSA", "public_key_pem": "-----BEGIN PUBLIC KEY-----\n...\n-----END PUBLIC KEY-----\n"} + ], + "rules": [ + {"subject": "group:readers", "resource": "operation/demo/list/listItems", "action": "execute"}, + {"subject": "group:readers", "resource": "credential/demo", "action": "use"} + ] +} +``` + +For the other modes, keep the same `rules` and optional deadline settings, but replace JWT's +`audience` and `keys` with the following mode-specific fields: + +```json +{ + "type": "bearer", + "issuer": "local-tokens", + "tokens": [{"subject": "alice", "sha256": "<64 hexadecimal characters: SHA-256 of the token>"}], + "rules": [{"subject": "group:readers", "resource": "host/sql/read", "action": "execute"}] +} +``` + +```json +{ + "type": "mtls", + "issuer": "client-certificates", + "client_ca_pem": "-----BEGIN CERTIFICATE-----\n...\n-----END CERTIFICATE-----\n", + "certificate_identity": "uri_san", + "rules": [{"subject": "group:readers", "resource": "host/sql/read", "action": "execute"}] +} +``` + +```json +{ + "type": "local_os", + "issuer": "this-host", + "rules": [{"subject": "group:readers", "resource": "host/sql/read", "action": "execute"}] +} +``` + +The identity subject is the configured bearer subject, a verified client URI/DNS SAN, or the effective +Unix UID / Windows SID. mTLS requires exactly one SAN of the configured kind (`uri_san` or `dns_san`); +no Common Name fallback is used. Local OS mode ignores `STACKQL_CALLER_TOKEN` and is local-only. +It obtains the process identity from the OS, including numeric container UIDs without passwd entries. +Neither OS groups nor token group claims replace directory membership. Static bearer digests must be +unique SHA-256 values; comparisons happen in the SDK. Basic username/password and OAuth/OIDC discovery +are not provided. An external issuer can issue JWTs, but its keys/issuer/audience must be pinned here. + +JWT accepts EdDSA, RS256/384/512, PS256/384/512 and ES256/384/512. RSA keys require at least 2048 bits. + +Use the real compiled method and operation identifiers from trusted provider documents. An operation +wildcard can grant mutation methods as well as reads. Keys are bound to their configured algorithms; +unknown key IDs, invalid signatures, missing/expired expiry and invalid temporal/issuer/audience +claims fail closed. Token-supplied URLs and groups are not used for trust or directory membership. +Configuration is loaded at startup; rotate policy or keys through a controlled restart. +Queries have no authentication-derived execution deadline by default. Credentials are verified and +directory membership is refreshed for each new statement; expired credentials cannot dispatch new +work, but expiry does not interrupt an already authorized query. Explicit caller timeouts and +cancellation still apply. + +Set `"enforce_session_deadline": true` to opt into cancellation at `max_session_seconds` (default +300; maximum 86400), shortened by JWT or mTLS certificate expiry. `max_session_seconds` has no effect +on query execution while enforcement is disabled. This option does not enable token renewal. + +## Principal directory + +The configured SQLite or PostgreSQL backend stores: + +- `stackql_access_principals`: ID and JSON object of string attributes. +- `stackql_access_identities`: verified `(issuer, subject)` to principal ID. +- `stackql_access_groups`: group IDs. +- `stackql_access_memberships`: principal/group bindings. + +Schema creation is explicit: `accessmanagement.InitializeSchema(ctx, engine)` is an administration +API, not an authentication side effect. Provision tables, identities and memberships through a +trusted database administration path before serving protected queries. Unknown identities, absent +schema, malformed data and directory outages deny access. Ordinary protected SQL cannot address +directory tables. SQLite runtime connections enable foreign keys. Protect the database file and +use separate administrative/runtime database privileges where supported. + +For PostgreSQL, create the four tables through your database administration connection, using the +schema shown in [the walkthrough script](examples/access-management/quickstart.py). Use a dedicated +schema/search path and provision identities before starting the runtime. For example: + +```sql +INSERT INTO stackql_access_principals (id, attributes) VALUES ('alice', '{}'); +INSERT INTO stackql_access_groups (id) VALUES ('readers'); +INSERT INTO stackql_access_identities (issuer, subject, principal_id) +VALUES ('https://identity.example', 'alice', 'alice'); +INSERT INTO stackql_access_memberships (principal_id, group_id) VALUES ('alice', 'readers'); +``` + +Use the authentication mode's actual issuer/subject: for local OS this might be `('this-host', '1000')`, +and for mTLS `('client-certificates', 'spiffe://example/alice')`. Select the backend explicitly: + +```sh +stackql --access.config '{"path":"/etc/stackql/access.json"}' \ + --sqlBackend '{"dbEngine":"postgres_tcp","dsn":"postgres://runtime@localhost/stackql?sslmode=require","sqlDialect":"postgres"}' \ + exec 'SELECT whoami(), groups();' +``` + +Runtime credentials need directory SELECT access; keep membership writes in the trusted administration +connection. Revocation is `DELETE FROM stackql_access_memberships WHERE principal_id = 'alice'` followed +by commit. The next statement resolves a new snapshot. SQLite and PostgreSQL Robot scenarios test this. + +The directory produces an immutable per-statement snapshot. Omni receives that snapshot and an +authorizer; database connections, drivers and SQL remain in StackQL. + +## Entry points and permissions + +| Authentication | `exec`, shell, MCP stdio | HTTPS MCP | +| --- | --- | --- | +| JWT | `STACKQL_CALLER_TOKEN` | Request bearer header | +| Static bearer | `STACKQL_CALLER_TOKEN` | Request bearer header | +| Local OS | Effective process UID/SID; no credential | Explicitly refused | +| mTLS | Refused: no verified TLS connection | Verified client chain and unique configured SAN | + +Supply local token credentials through the process environment, never a SQL literal or command-line +argument. Protected `exec` failures return exit status 1 after presenting the error. + +HTTPS MCP obtains credentials from the actual request or TLS connection. Server TLS certificate/key +configuration is mandatory. mTLS additionally requires and verifies client certificates using the configured +client CA. The caller verifier replaces the legacy shared HTTP bearer gate when access is enabled. +Missing request credentials never fall back to the server's environment token. + +Protected PostgreSQL-wire startup and MCP reverse proxy currently refuse explicitly. The current +wire dependency does not expose verified per-connection authentication evidence to the backend. +They must gain that integration before protection can be enabled on those transports. Unprotected +legacy deployments continue to operate as before. Provider SQL `AUTH`/`AUTH REVOKE` administration also refuses in protected mode; provision provider +credentials through trusted host configuration instead. Other standalone helper/registry commands refuse +when caller access is enabled, rather than bypassing the protected SQL path. + +The `readers` directory group grants the host's safe-read SQL capability. SDK operations and provider +credential use still require explicit policy. Other statements require `execute` on +`host/sql/`; reads outside the implicit reader group require `host/sql/read`. +Additional host checks include: + +| Resource | Action | Scope | +| --- | --- | --- | +| `host/iac/run`, `host/iac/destroy` | `execute` | Effectful IaC relations, despite SELECT syntax | +| `host/sql/exec` | `execute` | Native EXEC, including nested EXEC | +| `host/function/` | `execute` | Functions outside the small known-pure allowlist | +| `host/backend` | `read` | Opaque stored/external backend relations | +| `host/credentials` | `reload` | MCP credential reload administration | + +SDK collection, operation, credential, relation and transport checks are additional requirements. +A collection grant does not grant its underlying methods. Stored query expansion is checked before +execution. Non-integrated legacy execution is refused while protection is enabled. Legacy plan and +row-cache reuse is disabled on protected paths. + +Inspect the established snapshot using: + +```sql +SELECT whoami() AS principal, groups() AS memberships; +``` + +## IaC surface + +For a create, rerun, update and destroy sequence, see the [AWS VPC/subnet walkthrough](iac_walkthrough.md). + +Use `stackql_preview.iac.run` and `.destroy` with the existing preview opt-in. Resource JSON uses +Omni's shared parser: `write_once` carries initial request inputs, `desired` carries maintained +state, and typed `input`, `env`, `self` and sibling bindings remain intact. Retired `params` and +`correlationParam` resource fields are rejected. Resource keys must be unique. + +`inputs` and `environment` predicates accept JSON objects. JSON numbers retain their precision; +strings are literal values, not SQL or templates. Remaining predicates are provider execution scope. +Always supply explicit collection and cloud scope. The host-configured state root confines relative +`state` names, including checks against existing symlink escapes. Protected state is additionally +namespaced by the directory's `tenant` attribute, or by principal ID when no tenant is configured. +Only trusted directory administrators should assign tenant membership. + +```sql +-- Explicit partial deletion. An empty selection is an error. +SELECT * FROM stackql_preview.iac.destroy +WHERE collection = 'deployment' AND state = 'production' + AND keys = '["resource-key"]' AND run_id = 'prune-001'; + +-- Full teardown: omit keys. +SELECT * FROM stackql_preview.iac.destroy +WHERE collection = 'deployment' AND state = 'production' AND run_id = 'destroy-001'; +``` + +Omit `run_id` to generate a unique ID automatically. Explicit IDs must be unique within the state +directory; reusing one is rejected before provider execution. Reapply the latest desired-state file: +returning to an older file can request removal of fields it no longer declares. After pruning, remove the deleted declarations before applying again; omission +alone does not delete, and reapplying the original declaration requests recreation. Omni blocks +selected deletion when retained resources depend on it. Failed/error/outstanding result rows become +SQL failures and CLI failure status, while diagnostic rows remain available. `applied` can mean no-op. +Cancellation and authenticated context propagate to SDK execution, completion polls and compensation; +compensation does not retry anonymously after access expires or fails. + +`test/robot/functional/stackql_access.robot` contains separately named scenarios for each authentication +mode, transport, invalid JWT constraint, policy boundary and IaC lifecycle. It exercises all accepted JWT +signing algorithms and runs the published walkthrough script for all four modes. The fixture uses a local +API and the configured SQLite/PostgreSQL directory. Test tokens remain outside Robot arguments and artifacts. diff --git a/docs/examples/access-management/README.md b/docs/examples/access-management/README.md new file mode 100644 index 000000000..50965c051 --- /dev/null +++ b/docs/examples/access-management/README.md @@ -0,0 +1,106 @@ +# Try caller access locally + +Run these commands **from the StackQL repository root**, in your terminal. Use the built binary +`./build/stackql`. Python 3 provisions the demonstration directory; JWT and mTLS also require OpenSSL. +No cloud account is needed and these examples make no cloud requests. + +**`whoami()` and `groups()` require an authenticated StackQL session. They do not work in an ordinary +shell started without `--access.config`.** Start with the local OS example below. + +## 1. Local OS identity: set up, then open an authenticated shell + +Choose an empty directory. This command creates the principal database and access configuration, +maps your effective Unix UID or Windows SID to principal `reader`, and verifies a query: + +```sh +python3 docs/examples/access-management/quickstart.py \ + --stackql ./build/stackql --mode local_os --directory /tmp/access-os +``` + +Expect the confirmation: + +```text +Verified local_os: principal reader; directory group readers. +``` + +Now open a shell using that configuration and database: + +```sh +./build/stackql \ + --access.config '{"path":"/tmp/access-os/access.json"}' \ + --sqlBackend '{"dsn":"/tmp/access-os/principals.db"}' \ + shell +``` + +**In this authenticated StackQL shell**, run: + +```sql +SELECT whoami() AS principal, groups() AS memberships; +``` + +Expect one row: `principal` is `reader`, and `memberships` is `["readers"]`. +You can repeat the query or reopen the same configured shell. No token is required. Directory +memberships determine permissions; OS group membership does not grant access. HTTP callers cannot +use the server's OS identity. + +The setup command refuses to overwrite a nonempty directory. If you already used `/tmp/access-os`, +reuse its configuration with the shell command above, or choose a new empty directory for setup +and use that path in both shell flags. + +## 2. JWT: provision and verify a signed token + +```sh +python3 docs/examples/access-management/quickstart.py \ + --stackql ./build/stackql --mode jwt --directory /tmp/access-jwt +``` + +Expect `Verified jwt: principal reader; directory group readers.` + +The script creates an Ed25519 signing key, pins its public key and `kid`, issues a five-minute JWT, +and supplies it through `STACKQL_CALLER_TOKEN` to one CLI query. Token group claims grant nothing; +the directory supplies `readers` membership. The token is not exported into your terminal, so opening +a separate shell with this configuration alone will not authenticate you. For deployment, obtain +JWTs from your issuer and configure its trusted public keys. + +## 3. Static bearer token: provision and verify a token digest + +```sh +python3 docs/examples/access-management/quickstart.py \ + --stackql ./build/stackql --mode bearer --directory /tmp/access-bearer +``` + +Expect `Verified bearer: principal reader; directory group readers.` + +The script generates a random token, stores only its SHA-256 digest and subject in `access.json`, +and passes the token to one child CLI query through `STACKQL_CALLER_TOKEN`. The token is not exported +into your terminal or printed. For deployment, provision a token digest and supply its plaintext +credential through your secret-management environment. Changing configured digests requires a restart. + +## 4. Mutual TLS: query over HTTPS with a client certificate + +```sh +python3 docs/examples/access-management/quickstart.py \ + --stackql ./build/stackql --mode mtls --directory /tmp/access-mtls +``` + +Expect `Verified mtls: principal reader; directory group readers.` + +The script creates a demonstration CA and server/client certificates, starts an HTTPS MCP server, +queries it with the client certificate while verifying the server CA, then stops the server. +The client URI SAN `spiffe://access-demo/reader` maps to principal `reader`. No bearer token is sent. +The server requires a verified client chain and exactly one configured identity SAN; Common Name +is not a fallback. For deployment, use your managed CA and certificates. + +## What these examples leave behind + +Each directory contains `access.json`, `backend.json` and `principals.db`. JWT/mTLS also create +private keys; mTLS writes `mcp.json`. These credentials are local demonstration credentials. +Use a separate empty directory for each setup command. + +See the [configuration reference](../../access_management.md) for policy, principal-directory +provisioning, PostgreSQL and supported transports. For actual infrastructure deployment, see the +[AWS VPC/subnet walkthrough](../../iac_walkthrough.md). + +Robot tests execute all four setup scripts. The local OS walkthrough test also starts the configured +shell and runs the identity query inside it. Docker tests use the tested container image; Python +runs on the test host and is not added to the standard image. diff --git a/docs/examples/access-management/quickstart.py b/docs/examples/access-management/quickstart.py new file mode 100644 index 000000000..22c082afd --- /dev/null +++ b/docs/examples/access-management/quickstart.py @@ -0,0 +1,155 @@ +#!/usr/bin/env python3 +"""Run a local caller-access walkthrough. Requires StackQL, Python 3 and OpenSSL.""" +import argparse +import base64 +import csv +import hashlib +import json +import os +from pathlib import Path +import secrets +import shutil +import socket +import sqlite3 +import ssl +import subprocess +import tempfile +import time +import urllib.error +import urllib.request + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--mode", choices=["jwt", "bearer", "mtls", "local_os"], required=True) + parser.add_argument("--directory", type=Path) + parser.add_argument("--stackql", default="stackql") + args = parser.parse_args() + root = (args.directory or Path(tempfile.mkdtemp(prefix="stackql-access-demo-"))).resolve() + root.mkdir(mode=0o700, parents=True, exist_ok=True) + if any(root.iterdir()): + parser.error("choose an empty directory; existing credentials are never overwritten") + exe = str(Path(shutil.which(args.stackql) or args.stackql).resolve()) + + def write(name, value): + path = root / name + path.write_text(value) + path.chmod(0o600) + return path + + def openssl(*command): + result = subprocess.run(["openssl", *map(str, command)], check=True, capture_output=True) + return result.stdout + + def private(name): + path = root / (name + ".key") + openssl("genpkey", "-algorithm", "ED25519", "-out", path) + path.chmod(0o600) + return path + + config = {"type": args.mode, "issuer": "access-demo", "max_session_seconds": 300, + "rules": [{"subject": "group:readers", "resource": "host/sql/read", "action": "execute"}]} + env = os.environ.copy() + env.pop("STACKQL_CALLER_TOKEN", None) + subject = "reader" + if args.mode == "jwt": + key = private("signing") + config.update(audience="stackql", keys=[{"kid": "demo", "algorithm": "EdDSA", "public_key_pem": openssl("pkey", "-in", key, "-pubout").decode()}]) + encode = lambda value: base64.urlsafe_b64encode(value).rstrip(b"=") + claims = {"iss": "access-demo", "sub": subject, "aud": "stackql", "iat": int(time.time()), "exp": int(time.time()) + 300} + unsigned = encode(json.dumps({"alg": "EdDSA", "kid": "demo"}).encode()) + b"." + encode(json.dumps(claims).encode()) + message = write("unsigned.jwt", unsigned.decode()) + signature = openssl("pkeyutl", "-sign", "-rawin", "-inkey", key, "-in", message) + env["STACKQL_CALLER_TOKEN"] = (unsigned + b"." + encode(signature)).decode() + elif args.mode == "bearer": + env["STACKQL_CALLER_TOKEN"] = secrets.token_urlsafe(32) + config["tokens"] = [{"subject": subject, "sha256": hashlib.sha256(env["STACKQL_CALLER_TOKEN"].encode()).hexdigest()}] + elif args.mode == "local_os": + subject = next(csv.reader([subprocess.check_output(["whoami", "/user", "/fo", "csv", "/nh"], text=True).strip()]))[1] if os.name == "nt" else str(os.geteuid()) + else: + subject = "spiffe://access-demo/reader" + ca_key = private("ca") + ca = root / "ca.crt" + openssl("req", "-new", "-x509", "-key", ca_key, "-subj", "/CN=Access Demo CA", "-days", "1", "-addext", "basicConstraints=critical,CA:TRUE", "-addext", "keyUsage=critical,keyCertSign,cRLSign", "-addext", "subjectKeyIdentifier=hash", "-out", ca) + for name, extension in [("server", "subjectAltName=IP:127.0.0.1\nextendedKeyUsage=serverAuth\n"), ("client", "subjectAltName=URI:" + subject + "\nextendedKeyUsage=clientAuth\n")]: + key = private(name) + csr = root / (name + ".csr") + openssl("req", "-new", "-key", key, "-subj", "/CN=" + name, "-out", csr) + extensions = write(name + ".ext", extension + "basicConstraints=critical,CA:FALSE\nkeyUsage=critical,digitalSignature\nsubjectKeyIdentifier=hash\nauthorityKeyIdentifier=keyid,issuer\n") + openssl("x509", "-req", "-in", csr, "-CA", ca, "-CAkey", ca_key, "-CAcreateserial", "-days", "1", "-extfile", extensions, "-out", root / (name + ".crt")) + config.update(client_ca_pem=ca.read_text(), certificate_identity="uri_san") + + database = root / "principals.db" + with sqlite3.connect(database) as db: + db.executescript(""" +CREATE TABLE stackql_access_principals (id TEXT PRIMARY KEY, attributes TEXT NOT NULL DEFAULT '{}'); +CREATE TABLE stackql_access_groups (id TEXT PRIMARY KEY); +CREATE TABLE stackql_access_identities (issuer TEXT NOT NULL, subject TEXT NOT NULL, + principal_id TEXT NOT NULL REFERENCES stackql_access_principals(id), PRIMARY KEY (issuer, subject)); +CREATE TABLE stackql_access_memberships (principal_id TEXT NOT NULL REFERENCES stackql_access_principals(id), + group_id TEXT NOT NULL REFERENCES stackql_access_groups(id), PRIMARY KEY (principal_id, group_id)); +INSERT INTO stackql_access_principals VALUES ('reader','{}'); +INSERT INTO stackql_access_groups VALUES ('readers'); +INSERT INTO stackql_access_memberships VALUES ('reader','readers'); +""") + db.execute("INSERT INTO stackql_access_identities VALUES (?,?,?)", ("access-demo", subject, "reader")) + database.chmod(0o600) + access = write("access.json", json.dumps(config, indent=2)) + backend = write("backend.json", json.dumps({"dsn": str(database)})) + command = [exe, "--approot", str(root), "--access.config", json.dumps({"path": str(access)}), "--sqlBackend", backend.read_text(), "--output", "json"] + query = "SELECT whoami() AS principal, groups() AS memberships;" + if args.mode != "mtls": + result = subprocess.run(command + ["exec", query], env=env, cwd=root, capture_output=True, text=True) + if result.returncode: + raise RuntimeError("protected query failed: " + result.stderr) + output = result.stdout + else: + with socket.socket() as listener: + listener.bind(("127.0.0.1", 0)) + port = listener.getsockname()[1] + mcp = {"server": {"transport": "http", "address": "127.0.0.1:" + str(port), "protocol_version": "2026-07-28", "audit": {"disabled": True}, "tls_cert_file": str(root / "server.crt"), "tls_key_file": str(root / "server.key")}, "enabled_tools": ["run_select_query"]} + write("mcp.json", json.dumps(mcp, indent=2)) + context = ssl.create_default_context(cafile=str(root / "ca.crt")) + context.load_cert_chain(str(root / "client.crt"), str(root / "client.key")) + payload = {"jsonrpc": "2.0", "id": 1, "method": "tools/call", "params": {"_meta": {"io.modelcontextprotocol/protocolVersion": "2026-07-28", "io.modelcontextprotocol/clientCapabilities": {}, "io.modelcontextprotocol/clientInfo": {"name": "access-demo", "version": "1"}}, "name": "run_select_query", "arguments": {"sql": query}}} + headers = {"Content-Type": "application/json", "Accept": "application/json, text/event-stream", "MCP-Protocol-Version": "2026-07-28", "Mcp-Method": "tools/call", "Mcp-Name": "run_select_query"} + with (root / "server.log").open("w") as log: + server = subprocess.Popen(command + ["--mcp.config", json.dumps(mcp), "mcp"], env=env, cwd=root, stdout=log, stderr=log) + try: + for attempt in range(100): + if server.poll() is not None: + raise RuntimeError("MCP startup failed: " + (root / "server.log").read_text()) + try: + request = urllib.request.Request("https://127.0.0.1:" + str(port), data=json.dumps(payload).encode(), headers=headers) + with urllib.request.urlopen(request, context=context, timeout=5) as response: + output = response.read().decode() + break + except urllib.error.URLError: + if attempt == 99: + raise + time.sleep(0.1) + finally: + server.terminate() + server.wait(timeout=10) + if args.mode == "mtls": + message = json.loads(output) if output.lstrip().startswith("{") else next(json.loads(line[5:]) for line in output.splitlines() if line.startswith("data:")) + result = message.get("result", {}) + if message.get("error") or result.get("isError"): + raise RuntimeError("protected MCP query failed: " + output) + rows = result.get("structuredContent", {}).get("rows", []) + else: + rows = json.loads(output) + if len(rows) != 1 or rows[0].get("principal") != "reader": + raise RuntimeError("expected the provisioned principal: " + output) + groups = rows[0].get("memberships") + if isinstance(groups, str): + groups = json.loads(groups) + if groups != ["readers"]: + raise RuntimeError("expected exactly the directory readers group: " + output) + print(output.strip()) + print("Verified " + args.mode + ": principal reader; directory group readers.") + print("Configuration: " + str(access)) + + +if __name__ == "__main__": + main() diff --git a/docs/iac_walkthrough.md b/docs/iac_walkthrough.md new file mode 100644 index 000000000..82286e1b1 --- /dev/null +++ b/docs/iac_walkthrough.md @@ -0,0 +1,86 @@ +# AWS VPC and subnet through StackQL + +Run from the StackQL repository root with your AWS credentials already configured. This walkthrough +creates real resources in `ap-southeast-2` (Sydney). It uses the existing Omni example files and prepared +registry on this machine; the registry overlay supplies the update routes required by the example. + +## Start + +```sh +./build/stackql \ + --preview '{"unstable":true}' \ + --registry '{"localDocRoot":"/Users/generalkroll/stackql-labs/omnisdk/cicd/out/aws-update-registry","srcPrefix":"","verifyConfig":{"nopVerify":true}}' \ + shell +``` + +Keep the same working directory, collection, state and region throughout. StackQL retains the ledger +under `.stackql/iac/demo`. Do not remove that directory before teardown. These commands omit `run_id`: +StackQL generates a unique one for each execution. They create a new deployment rather than adopting +resources from a previous Omni CLI deployment. + +## Create and rerun + +Paste into the StackQL shell: + +```sql +SELECT * FROM stackql_preview.iac.run +WHERE resources_file = '/Users/generalkroll/stackql-labs/omnisdk/cicd/examples/aws-vpc-subnet/resources.json' + AND collection = 'stackql-vpc-demo' + AND state = 'demo' + AND region = 'ap-southeast-2'; +``` + +Check for errors before continuing. Record the VPC and subnet IDs. In the AWS console, select Sydney +and find those IDs: the VPC CIDR is `10.42.0.0/16`, and the subnet CIDR is `10.42.1.0/24`. The initial +file does not assign a VPC Name tag. + +Run that exact SQL again now: both IDs must stay the same, with no duplicate resources. + +## Update and rerun + +```sql +SELECT * FROM stackql_preview.iac.run +WHERE resources_file = '/Users/generalkroll/stackql-labs/omnisdk/cicd/examples/aws-vpc-subnet/update-public-ip.json' + AND collection = 'stackql-vpc-demo' + AND state = 'demo' + AND region = 'ap-southeast-2'; +``` + +Verify the same VPC now has Name `omni-demo-vpc`, and the same subnet has automatic public IPv4 +assignment enabled. This does not create instances or internet routing. + +Run this update SQL again to check unchanged convergence. A successful `applied` status can mean +no-op; verify the IDs remain unchanged. **Continue using this file after the update.** Reapplying +`resources.json` is not an unchanged rerun: it drops fields maintained by the update document. + +## Reverse the subnet setting + +```sql +SELECT * FROM stackql_preview.iac.run +WHERE resources_file = '/Users/generalkroll/stackql-labs/omnisdk/cicd/examples/aws-vpc-subnet/update-private-ip.json' + AND collection = 'stackql-vpc-demo' + AND state = 'demo' + AND region = 'ap-southeast-2'; +``` + +Verify automatic public IPv4 assignment is disabled. The VPC Name remains `omni-demo-vpc`. +You can repeat this same SQL; it is now the latest desired-state document. + +## Destroy + +```sql +SELECT * FROM stackql_preview.iac.destroy +WHERE collection = 'stackql-vpc-demo' + AND state = 'demo' + AND region = 'ap-southeast-2'; +``` + +Inspect the results for errors or outstanding resources, and confirm both recorded IDs disappear +from AWS. The subnet is deleted before its VPC. If an operation fails, retain the ledger and inspect +the returned diagnostics; do not switch collection names or erase state to bypass the error. + +## After the earlier reused-run failure + +That failure reported both resources compensated and no outstanding resources. The local ledger +contains tombstones for both keys. Verify the old IDs are absent in AWS before starting the create +step again with the fixed binary. Do not delete the ledger or its old journals. diff --git a/go.mod b/go.mod index 64518a41d..806374de9 100644 --- a/go.mod +++ b/go.mod @@ -8,6 +8,7 @@ require ( github.com/PaesslerAG/gval v1.0.0 github.com/PaesslerAG/jsonpath v0.1.1 github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e + github.com/golang-jwt/jwt/v5 v5.3.1 github.com/google/go-jsonnet v0.17.0 github.com/google/licensecheck v0.3.1 github.com/google/uuid v1.6.0 @@ -22,7 +23,7 @@ require ( github.com/spf13/cobra v1.10.2 github.com/spf13/pflag v1.0.10 github.com/spf13/viper v1.10.1 - github.com/stackql-labs/omnisdk v0.1.8-alpha02 + github.com/stackql-labs/omnisdk v0.1.9-alpha06 github.com/stackql/go-suffix-map v0.0.1-alpha02 github.com/stackql/psql-wire v0.1.2-beta02 github.com/stackql/stackql-parser v0.0.16-beta01 @@ -69,6 +70,9 @@ require ( github.com/aws/aws-sdk-go-v2/service/ssooidc v1.23.4 // indirect github.com/aws/aws-sdk-go-v2/service/sts v1.28.6 // indirect github.com/aws/smithy-go v1.24.2 // indirect + github.com/bmatcuk/doublestar/v4 v4.6.1 // indirect + github.com/casbin/casbin/v2 v2.135.0 // indirect + github.com/casbin/govaluate v1.3.0 // indirect github.com/cespare/xxhash/v2 v2.3.0 // indirect github.com/clbanning/mxj/v2 v2.7.0 // indirect github.com/cncf/xds/go v0.0.0-20260202195803-dba9d589def2 // indirect @@ -84,7 +88,6 @@ require ( github.com/go-jose/go-jose/v4 v4.1.4 // indirect github.com/goccy/go-json v0.10.4 // indirect github.com/godbus/dbus v0.0.0-20190726142602-4481cbc300e2 // indirect - github.com/golang-jwt/jwt/v5 v5.3.1 // indirect github.com/golang/glog v1.2.5 // indirect github.com/golang/protobuf v1.5.4 // indirect github.com/google/flatbuffers v24.12.23+incompatible // indirect diff --git a/go.sum b/go.sum index 903c1c2c3..8b5b7971a 100644 --- a/go.sum +++ b/go.sum @@ -73,8 +73,14 @@ github.com/aws/smithy-go v1.24.2 h1:FzA3bu/nt/vDvmnkg+R8Xl46gmzEDam6mZ1hzmwXFng= github.com/aws/smithy-go v1.24.2/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc= github.com/benbjohnson/clock v1.1.0 h1:Q92kusRqC1XV2MjkWETPvjJVqKetz1OzxZB7mHJLju8= github.com/benbjohnson/clock v1.1.0/go.mod h1:J11/hYXuz8f4ySSvYwY0FKfm+ezbsZBKZxNJlLklBHA= +github.com/bmatcuk/doublestar/v4 v4.6.1 h1:FH9SifrbvJhnlQpztAx++wlkk70QBf0iBWDwNy7PA4I= +github.com/bmatcuk/doublestar/v4 v4.6.1/go.mod h1:xBQ8jztBU6kakFMg+8WGxn0c6z1fTSPVIjEY1Wr7jzc= github.com/bufbuild/protocompile v0.14.1 h1:iA73zAf/fyljNjQKwYzUHD6AD4R8KMasmwa/FBatYVw= github.com/bufbuild/protocompile v0.14.1/go.mod h1:ppVdAIhbr2H8asPk6k4pY7t9zB1OU5DoEw9xY/FUi1c= +github.com/casbin/casbin/v2 v2.135.0 h1:6BLkMQiGotYyS5yYeWgW19vxqugUlvHFkFiLnLR/bxk= +github.com/casbin/casbin/v2 v2.135.0/go.mod h1:FmcfntdXLTcYXv/hxgNntcRPqAbwOG9xsism0yXT+18= +github.com/casbin/govaluate v1.3.0 h1:VA0eSY0M2lA86dYd5kPPuNZMUD9QkWnOCnavGrw9myc= +github.com/casbin/govaluate v1.3.0/go.mod h1:G/UnbIjZk/0uMNaLwZZmFQrR72tYRZWQkO70si/iR7A= github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs= github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= github.com/chzyer/logex v1.2.1 h1:XHDu3E6q+gdHgsdTPH6ImJMIp436vR6MPtH8gP05QzM= @@ -128,6 +134,8 @@ github.com/golang-jwt/jwt/v5 v5.3.1 h1:kYf81DTWFe7t+1VvL7eS+jKFVWaUnK9cB1qbwn63Y github.com/golang-jwt/jwt/v5 v5.3.1/go.mod h1:fxCRLWMO43lRc8nhHWY6LGqRcf+1gQWArsqaEUEa5bE= github.com/golang/glog v1.2.5 h1:DrW6hGnjIhtvhOIiAKT6Psh/Kd/ldepEa81DKeiRJ5I= github.com/golang/glog v1.2.5/go.mod h1:6AhwSGph0fcJtXVM/PEHPqZlFeoLxhs7/t5UDAwmO+w= +github.com/golang/mock v1.4.4 h1:l75CXGRSwbaYNpl/Z2X1XIIAMSCquvXgpVZDhwEIJsc= +github.com/golang/mock v1.4.4/go.mod h1:l3mdAwkq5BuhzHwde/uurv3sEJeZMXNpwsxVWU71h+4= github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek= github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps= github.com/golang/snappy v0.0.4 h1:yAGX7huGHXlcLOEtBnF4w7FQwA26wojNCwOYAEhLjQM= @@ -320,8 +328,8 @@ github.com/spf13/viper v1.10.1 h1:nuJZuYpG7gTj/XqiUwg8bA0cp1+M2mC3J4g5luUYBKk= github.com/spf13/viper v1.10.1/go.mod h1:IGlFPqhNAPKRxohIzWpI5QEy4kuI7tcl5WvR+8qy1rU= github.com/spiffe/go-spiffe/v2 v2.7.0 h1:uXe1MflJoHw58wAUvxVlcM7WpKtijWG7I1UidcGh6g4= github.com/spiffe/go-spiffe/v2 v2.7.0/go.mod h1:47Q0Q9/AqGha8QLHp+kxpH4Wca7X7EnOtlIJy3mxZ3U= -github.com/stackql-labs/omnisdk v0.1.8-alpha02 h1:RHlMimy9wjpEguOEFpJj6d9tRU8U4l1CCssl7bYMvJw= -github.com/stackql-labs/omnisdk v0.1.8-alpha02/go.mod h1:P3pr9kvDTMox+4xCLcFdzOS1wMMe3ho9i6j2Xl+/Yog= +github.com/stackql-labs/omnisdk v0.1.9-alpha06 h1:lHHxlxPrp/VK9l+AZTR7qED/sWnMURLKXbqBwE/02No= +github.com/stackql-labs/omnisdk v0.1.9-alpha06/go.mod h1:pwxphb3OMfEui7uCMmReoB+ABshSBl83dszn4Sf4vcg= github.com/stackql/go-suffix-map v0.0.1-alpha02 h1:JI4aiq/OfHW4a/BiNqK31NbyaR94zlUOhGZof3G/OUo= github.com/stackql/go-suffix-map v0.0.1-alpha02/go.mod h1:6BSMz4K/gIYfqdVRb7V8xL165wiPYyCjD9Kx7hLMd9Y= github.com/stackql/psql-wire v0.1.2-beta02 h1:7DfliBpyyCaUHVJ1Jxk37q3zVwVLyh6+FCHeM0/gbAQ= @@ -462,6 +470,7 @@ golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U= golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno= golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= +golang.org/x/tools v0.0.0-20190425150028-36563e24a262/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q= golang.org/x/tools v0.0.0-20190425163242-31fd60d6bfdc/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q= golang.org/x/tools v0.0.0-20190621195816-6e04913cbbac/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc= golang.org/x/tools v0.0.0-20190823170909-c4a336ef6a2f/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= diff --git a/internal/stackql/accessmanagement/access.go b/internal/stackql/accessmanagement/access.go new file mode 100644 index 000000000..f1328d764 --- /dev/null +++ b/internal/stackql/accessmanagement/access.go @@ -0,0 +1,317 @@ +package accessmanagement + +import ( + "context" + "crypto/rsa" + "crypto/tls" + "crypto/x509" + "encoding/hex" + "encoding/json" + "errors" + "io" + "os" + "strings" + "time" + + "github.com/golang-jwt/jwt/v5" + "github.com/stackql-labs/omnisdk/public/authentication" + "github.com/stackql-labs/omnisdk/public/authorization" + "github.com/stackql-labs/omnisdk/public/principal" + "github.com/stackql/stackql/internal/stackql/sqlengine" +) + +// Config is host-owned configuration, never accepted from SQL or provider predicates. +type Config struct { + Type string `json:"type"` + Tokens []BearerToken `json:"tokens"` + ClientCAPEM string `json:"client_ca_pem"` + CertificateIdentity string `json:"certificate_identity"` + Issuer string `json:"issuer"` + Audience string `json:"audience"` + Keys []VerificationKey `json:"keys"` + Rules []authorization.Rule `json:"rules"` + EnforceSessionDeadline bool `json:"enforce_session_deadline"` + MaxSessionSeconds int `json:"max_session_seconds"` +} +type VerificationKey struct { + ID string `json:"kid"` + Algorithm string `json:"algorithm"` + PEM string `json:"public_key_pem"` +} + +// BearerToken stores only the digest of a provisioned credential. +type BearerToken struct { + Subject string `json:"subject"` + SHA256 string `json:"sha256"` +} + +// Manager establishes a fresh directory snapshot for every statement. +type Manager interface { + Mode() string + TLSConfig() *tls.Config + Establish(context.Context) (authorization.Session, context.Context, context.CancelFunc, error) +} +type manager struct { + mode string + tlsConfig *tls.Config + authenticator authentication.Authenticator + directory principal.Directory + authorizer authorization.Authorizer + lifetime time.Duration + enforceDeadline bool +} + +const defaultSessionLifetime = 5 * time.Minute + +type evidenceKey struct{} + +// WithEvidence marks a transport boundary, including when no credential was supplied. +// Remote transports must never fall back to the server process's environment token. +func WithEvidence(ctx context.Context, evidence authentication.Evidence) context.Context { + return context.WithValue(ctx, evidenceKey{}, evidence) +} + +//nolint:nilnil // No configuration explicitly disables caller protection. +func NewManager(raw string, engine sqlengine.SQLEngine) (Manager, error) { + if raw == "" { + return nil, nil + } + path, err := configurationPath(raw) + if err != nil { + return nil, err + } + data, err := os.ReadFile(path) + if err != nil { + return nil, errors.New("cannot read caller access configuration") + } + var cfg Config + if err = decodeStrict(data, &cfg); err != nil { + return nil, err + } + + directory, err := NewDirectory(engine) + if err != nil { + return nil, err + } + return newManager(cfg, directory) +} + +func configurationPath(raw string) (string, error) { + if !strings.HasPrefix(strings.TrimSpace(raw), "{") { + return raw, nil + } + var source struct { + Path string `json:"path"` + } + if err := decodeStrict([]byte(raw), &source); err != nil { + return "", err + } + if strings.TrimSpace(source.Path) == "" { + return "", errors.New("caller access configuration requires a nonempty path") + } + return source.Path, nil +} + +func decodeStrict(data []byte, target any) error { + decoder := json.NewDecoder(strings.NewReader(string(data))) + decoder.DisallowUnknownFields() + if err := decoder.Decode(target); err != nil { + return errors.New("invalid caller access configuration") + } + var extra any + if err := decoder.Decode(&extra); !errors.Is(err, io.EOF) { + return errors.New("invalid trailing caller access configuration") + } + return nil +} + +func newManager(cfg Config, directory principal.Directory) (Manager, error) { + if len(cfg.Rules) == 0 || cfg.MaxSessionSeconds < 0 || cfg.MaxSessionSeconds > 86400 { + return nil, errors.New("caller access requires policy and a valid session lifetime") + } + if cfg.Type == "" { + cfg.Type = "jwt" + } + verifier, transport, err := configuredAuthenticator(cfg) + if err != nil { + return nil, err + } + policy, err := authorization.NewCasbin(cfg.Rules) + if err != nil { + return nil, err + } + lifetime := time.Duration(cfg.MaxSessionSeconds) * time.Second + if lifetime == 0 { + lifetime = defaultSessionLifetime + } + return &manager{ + mode: cfg.Type, tlsConfig: transport, authenticator: verifier, + directory: directory, authorizer: policy, lifetime: lifetime, enforceDeadline: cfg.EnforceSessionDeadline, + }, nil +} + +func configuredAuthenticator(cfg Config) (authentication.Authenticator, *tls.Config, error) { + switch cfg.Type { + case "jwt": + if len(cfg.Tokens) != 0 || cfg.ClientCAPEM != "" || cfg.CertificateIdentity != "" { + return nil, nil, errors.New("JWT configuration contains another authentication mode") + } + verifier, err := jwtAuthenticator(cfg) + return verifier, nil, err + case "bearer": + if len(cfg.Keys) != 0 || cfg.Audience != "" || cfg.ClientCAPEM != "" || cfg.CertificateIdentity != "" { + return nil, nil, errors.New("bearer configuration contains another authentication mode") + } + tokens := make([]authentication.Token, len(cfg.Tokens)) + for i, token := range cfg.Tokens { + digest, err := hex.DecodeString(token.SHA256) + if err != nil { + return nil, nil, errors.New("invalid bearer SHA-256 digest") + } + tokens[i] = authentication.Token{Subject: token.Subject, Digest: digest} + } + verifier, err := authentication.NewBearer(cfg.Issuer, tokens) + return verifier, nil, err + case "mtls": + if len(cfg.Keys) != 0 || len(cfg.Tokens) != 0 || cfg.Audience != "" { + return nil, nil, errors.New("mTLS configuration contains another authentication mode") + } + return mtlsAuthenticator(cfg) + case "local_os": + if len(cfg.Keys) != 0 || len(cfg.Tokens) != 0 || cfg.Audience != "" || + cfg.ClientCAPEM != "" || cfg.CertificateIdentity != "" { + return nil, nil, errors.New("local OS configuration contains another authentication mode") + } + verifier, err := authentication.NewLocalOS(cfg.Issuer) + return verifier, nil, err + default: + return nil, nil, errors.New("unsupported caller authentication type") + } +} + +func mtlsAuthenticator(cfg Config) (authentication.Authenticator, *tls.Config, error) { + roots := x509.NewCertPool() + if !roots.AppendCertsFromPEM([]byte(cfg.ClientCAPEM)) { + return nil, nil, errors.New("mTLS requires trusted client CA certificates") + } + if cfg.CertificateIdentity != "uri_san" && cfg.CertificateIdentity != "dns_san" { + return nil, nil, errors.New("mTLS certificate_identity must be uri_san or dns_san") + } + verifier, err := authentication.NewMTLS(cfg.Issuer, roots, func(cert *x509.Certificate) (string, error) { + if cfg.CertificateIdentity == "uri_san" && len(cert.URIs) == 1 { + return cert.URIs[0].String(), nil + } + if cfg.CertificateIdentity == "dns_san" && len(cert.DNSNames) == 1 { + return cert.DNSNames[0], nil + } + return "", errors.New("client certificate must have exactly one configured identity SAN") + }) + return verifier, &tls.Config{ + MinVersion: tls.VersionTLS12, ClientAuth: tls.RequireAndVerifyClientCert, ClientCAs: roots, + }, err +} + +func jwtAuthenticator(cfg Config) (authentication.Authenticator, error) { + if len(cfg.Keys) == 0 { + return nil, errors.New("JWT requires trusted keys") + } + keys := map[string]any{} + algorithms := map[string]string{} + var allowed []string + for _, spec := range cfg.Keys { + if spec.ID == "" || keys[spec.ID] != nil { + return nil, errors.New("verification keys require unique nonempty IDs") + } + var key any + var err error + switch spec.Algorithm { + case "EdDSA": + key, err = jwt.ParseEdPublicKeyFromPEM([]byte(spec.PEM)) + case "RS256", "RS384", "RS512", "PS256", "PS384", "PS512": + key, err = jwt.ParseRSAPublicKeyFromPEM([]byte(spec.PEM)) + case "ES256", "ES384", "ES512": + key, err = jwt.ParseECPublicKeyFromPEM([]byte(spec.PEM)) + default: + return nil, errors.New("unsupported caller JWT algorithm") + } + if err != nil { + return nil, errors.New("invalid caller verification key") + } + if value, ok := key.(*rsa.PublicKey); ok && value.N.BitLen() < 2048 { + return nil, errors.New("RSA key must have at least 2048 bits") + } + + keys[spec.ID], algorithms[spec.ID] = key, spec.Algorithm + allowed = append(allowed, spec.Algorithm) + } + verifier, err := authentication.NewJWT(cfg.Issuer, cfg.Audience, allowed, func(token *jwt.Token) (any, error) { + id, ok := token.Header["kid"].(string) + if !ok || keys[id] == nil || algorithms[id] != token.Method.Alg() { + return nil, errors.New("untrusted caller verification key") + } + return keys[id], nil + }) + if err != nil { + return nil, err + } + return verifier, nil +} + +func (m *manager) Mode() string { return m.mode } +func (m *manager) TLSConfig() *tls.Config { + if m.tlsConfig == nil { + return nil + } + return m.tlsConfig.Clone() +} + +func (m *manager) Establish(ctx context.Context) (authorization.Session, context.Context, context.CancelFunc, error) { + evidence, remote := ctx.Value(evidenceKey{}).(authentication.Evidence) + if remote && m.mode == "local_os" { + return nil, nil, nil, errors.New("local OS authentication is unavailable to remote callers") + } + if !remote && m.mode != "local_os" { + evidence.Token = os.Getenv("STACKQL_CALLER_TOKEN") + } + identity, err := m.authenticator.Authenticate(ctx, evidence) + if err != nil { + return nil, nil, nil, errors.New("caller authentication failed") + } + execution, cancel, err := m.executionContext(ctx, evidence) + if err != nil { + return nil, nil, nil, err + } + caller, err := authorization.NewSession(execution, identity, m.directory, m.authorizer) + if err != nil { + cancel() + return nil, nil, nil, errors.New("caller identity resolution failed") + } + return caller, caller.Context(execution), cancel, nil +} + +// executionContext preserves caller timeouts; credential-derived limits are opt-in. +func (m *manager) executionContext( + ctx context.Context, + evidence authentication.Evidence, +) (context.Context, context.CancelFunc, error) { + if !m.enforceDeadline { + execution, cancel := context.WithCancel(ctx) + return execution, cancel, nil + } + deadline := time.Now().Add(m.lifetime) + if m.mode == "jwt" { + // Inspect expiry only after the verifier authenticates this exact token. + claims := jwt.RegisteredClaims{} + if _, _, err := jwt.NewParser().ParseUnverified(evidence.Token, &claims); err != nil || claims.ExpiresAt == nil { + return nil, nil, errors.New("caller expiry is required") + } + if claims.ExpiresAt.Time.Before(deadline) { + deadline = claims.ExpiresAt.Time + } + } + if m.mode == "mtls" && evidence.TLS.PeerCertificates[0].NotAfter.Before(deadline) { + deadline = evidence.TLS.PeerCertificates[0].NotAfter + } + execution, cancel := context.WithDeadline(ctx, deadline) + return execution, cancel, nil +} diff --git a/internal/stackql/accessmanagement/access_test.go b/internal/stackql/accessmanagement/access_test.go new file mode 100644 index 000000000..e4cd8aaaa --- /dev/null +++ b/internal/stackql/accessmanagement/access_test.go @@ -0,0 +1,286 @@ +package accessmanagement //nolint:testpackage // Exercises the private directory implementation. + +import ( + "context" + "crypto/ed25519" + "crypto/rand" + "crypto/sha256" + "crypto/x509" + "database/sql" + "encoding/hex" + "encoding/pem" + "testing" + "testing/synctest" + "time" + + "github.com/golang-jwt/jwt/v5" + "github.com/stackql-labs/omnisdk/public/authentication" + "github.com/stackql-labs/omnisdk/public/authorization" + "github.com/stackql-labs/omnisdk/public/principal" + "github.com/stackql/stackql-parser/go/vt/sqlparser" +) + +//nolint:gocognit // Sequential session lifecycle assertions share an isolated directory fixture. +func TestCallerSessionsRefreshAndFailClosed(t *testing.T) { + pub, key, err := ed25519.GenerateKey(rand.Reader) + if err != nil { + t.Fatal(err) + } + der, _ := x509.MarshalPKIXPublicKey(pub) + db, err := sql.Open("sqlite", ":memory:") + if err != nil { + t.Fatal(err) + } + defer db.Close() + db.SetMaxOpenConns(1) + if err := initializeSchema(context.Background(), db); err != nil { + t.Fatal(err) + } + for _, stmt := range []string{ + `INSERT INTO stackql_access_principals VALUES ('alice','{}')`, + `INSERT INTO stackql_access_groups VALUES ('readers')`, + `INSERT INTO stackql_access_identities VALUES ('issuer','alice','alice')`, + `INSERT INTO stackql_access_memberships VALUES ('alice','readers')`, + } { + if _, err := db.Exec(stmt); err != nil { + t.Fatal(err) + } + } + directory, _ := newDirectory(db) + m, err := newManager(Config{Issuer: "issuer", Audience: "stackql", Keys: []VerificationKey{{ID: "test", Algorithm: "EdDSA", PEM: string(pem.EncodeToMemory(&pem.Block{Type: "PUBLIC KEY", Bytes: der}))}}, Rules: []authorization.Rule{{Subject: "group:readers", Resource: "operation/demo/list/listItems", Action: "execute"}}}, directory) + if err != nil { + t.Fatal(err) + } + sign := func(modify func(jwt.MapClaims), kid string, signingKey any) string { + claims := jwt.MapClaims{"iss": "issuer", "sub": "alice", "aud": "stackql", "exp": time.Now().Add(time.Minute).Unix(), "iat": time.Now().Unix(), "groups": []string{"admins"}} + if modify != nil { + modify(claims) + } + token := jwt.NewWithClaims(jwt.SigningMethodEdDSA, claims) + token.Header["kid"] = kid + value, err := token.SignedString(signingKey) + if err != nil { + t.Fatal(err) + } + return value + } + establish := func(token string) (authorization.Session, error) { + caller, _, cancel, err := m.Establish(WithEvidence(context.Background(), authentication.Evidence{Token: token})) + if cancel != nil { + t.Cleanup(cancel) + } + return caller, err + } + valid := sign(nil, "test", key) + caller, err := establish(valid) + if err != nil { + t.Fatal(err) + } + if len(caller.Principal().Groups()) != 1 || caller.Principal().Groups()[0] != "readers" { + t.Fatal("JWT groups replaced directory membership") + } + for _, query := range []string{"SELECT 1", "SELECT count(*) FROM demo.service.items"} { + stmt, _ := sqlparser.Parse(query) + if err := CheckStatement(caller.Context(context.Background()), caller, stmt); err != nil { + t.Fatal(err) + } + } + for _, query := range []string{"DELETE FROM demo.service.items", "SELECT * FROM stackql_access_principals", "SELECT pg_read_file('/tmp/x')", "SELECT * FROM stackql_preview.iac.run"} { + stmt, err := sqlparser.Parse(query) + if err != nil { + t.Fatal(err) + } + if CheckStatement(caller.Context(context.Background()), caller, stmt) == nil { + t.Fatalf("allowed %s", query) + } + } + _, other, _ := ed25519.GenerateKey(rand.Reader) + for name, token := range map[string]string{ + "missing": "", "signature": sign(nil, "test", other), "unknown key": sign(nil, "other", key), + "issuer": sign(func(c jwt.MapClaims) { c["iss"] = "other" }, "test", key), + "audience": sign(func(c jwt.MapClaims) { c["aud"] = "other" }, "test", key), + "expiry": sign(func(c jwt.MapClaims) { delete(c, "exp") }, "test", key), + "expired": sign(func(c jwt.MapClaims) { c["exp"] = time.Now().Add(-time.Minute).Unix() }, "test", key), + "future": sign(func(c jwt.MapClaims) { c["iat"] = time.Now().Add(time.Hour).Unix() }, "test", key), + "not before": sign(func(c jwt.MapClaims) { c["nbf"] = time.Now().Add(time.Hour).Unix() }, "test", key), + } { + if _, err := establish(token); err == nil { + t.Fatalf("accepted %s", name) + } + } + if _, err := db.Exec(`DELETE FROM stackql_access_memberships`); err != nil { + t.Fatal(err) + } + next, err := establish(valid) + if err != nil { + t.Fatal(err) + } + stmt, _ := sqlparser.Parse("SELECT 1") + if CheckStatement(next.Context(context.Background()), next, stmt) == nil { + t.Fatal("revoked membership retained") + } + if len(caller.Principal().Groups()) != 1 { + t.Fatal("existing snapshot mutated") + } + if _, err := db.Exec(`UPDATE stackql_access_principals SET attributes = 'null'`); err != nil { + t.Fatal(err) + } + if _, err := establish(valid); err == nil { + t.Fatal("accepted malformed attributes") + } + if err := db.Close(); err != nil { + t.Fatal(err) + } + if _, err := establish(valid); err == nil { + t.Fatal("directory outage allowed access") + } +} + +func TestConfigurationSource(t *testing.T) { + for _, raw := range []string{`{"path":"access.json"}`, "access.json"} { + path, err := configurationPath(raw) + if err != nil || path != "access.json" { + t.Fatalf("%q: path=%q error=%v", raw, path, err) + } + } + for _, raw := range []string{`{}`, `{"path":" "}`, `{"path":null}`, `{"path":3}`, `{"path":"access.json","extra":true}`, `{"path":"access.json"}{}`} { + if _, err := configurationPath(raw); err == nil { + t.Fatalf("accepted %q", raw) + } + } +} + +func TestBearerDeadlineAndRemoteBoundary(t *testing.T) { + directory, err := principal.NewMemoryDirectoryFromBytes([]byte(`{"groups":["readers"],"principals":[{"id":"alice","identities":[{"issuer":"tokens","subject":"alice"}],"groups":["readers"]}]}`)) + if err != nil { + t.Fatal(err) + } + digest := sha256.Sum256([]byte("test-token")) + manager, err := newManager(Config{Type: "bearer", Issuer: "tokens", MaxSessionSeconds: 1, EnforceSessionDeadline: true, + Tokens: []BearerToken{{Subject: "alice", SHA256: hex.EncodeToString(digest[:])}}, + Rules: []authorization.Rule{{Subject: "group:readers", Resource: "host/sql/read", Action: "execute"}}, + }, directory) + if err != nil { + t.Fatal(err) + } + t.Setenv("STACKQL_CALLER_TOKEN", "test-token") + caller, ctx, cancel, err := manager.Establish(context.Background()) + if err != nil { + t.Fatal(err) + } + defer cancel() + deadline, bounded := ctx.Deadline() + if !bounded || time.Until(deadline) > time.Second || caller.Principal().ID() != "alice" { + t.Fatal("unbounded or incorrect bearer session") + } + if _, _, _, err := manager.Establish(WithEvidence(context.Background(), authentication.Evidence{})); err == nil { + t.Fatal("remote request inherited the process token") + } + cancel() + if ctx.Err() == nil { + t.Fatal("cancelled caller context remained live") + } +} + +func TestModeConfigurationFailsClosed(t *testing.T) { + for _, cfg := range []Config{ + {Type: "unknown"}, + {Type: "bearer", Issuer: "tokens", Tokens: []BearerToken{{Subject: "alice", SHA256: "00"}}}, + {Type: "local_os", Issuer: "host", Audience: "unexpected"}, + {Type: "jwt", Issuer: "issuer", ClientCAPEM: "unexpected"}, + {Type: "mtls", Issuer: "certs", ClientCAPEM: "not a CA", CertificateIdentity: "uri_san"}, + } { + if _, _, err := configuredAuthenticator(cfg); err == nil { + t.Fatalf("accepted mode %q", cfg.Type) + } + } + manager, err := newManager(Config{Type: "local_os", Issuer: "host", Rules: []authorization.Rule{{Subject: "group:readers", Resource: "host/sql/read", Action: "execute"}}}, nil) + if err != nil { + t.Fatal(err) + } + if _, _, _, err := manager.Establish(WithEvidence(context.Background(), authentication.Evidence{})); err == nil { + t.Fatal("remote request inherited the server OS identity") + } +} + +// Expiry prevents subsequent dispatch, rather than cancelling work already authorized. +// +//nolint:gocognit // Compare default, opted-in and caller deadlines against one authenticated session clock. +func TestQueryContinuesAfterTokenExpiryByDefault(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + pub, key, err := ed25519.GenerateKey(rand.Reader) + if err != nil { + t.Fatal(err) + } + der, err := x509.MarshalPKIXPublicKey(pub) + if err != nil { + t.Fatal(err) + } + directory, err := principal.NewMemoryDirectoryFromBytes([]byte(`{"groups":["readers"],"principals":[{"id":"alice","identities":[{"issuer":"issuer","subject":"alice"}],"groups":["readers"]}]}`)) + if err != nil { + t.Fatal(err) + } + cfg := Config{Issuer: "issuer", Audience: "stackql", MaxSessionSeconds: 1, Keys: []VerificationKey{{ID: "test", Algorithm: "EdDSA", PEM: string(pem.EncodeToMemory(&pem.Block{Type: "PUBLIC KEY", Bytes: der}))}}, Rules: []authorization.Rule{{Subject: "group:readers", Resource: "host/sql/read", Action: "execute"}}} + token := jwt.NewWithClaims(jwt.SigningMethodEdDSA, jwt.MapClaims{"iss": "issuer", "sub": "alice", "aud": "stackql", "exp": time.Now().Add(time.Minute).Unix()}) + token.Header["kid"] = "test" + signed, err := token.SignedString(key) + if err != nil { + t.Fatal(err) + } + m, err := newManager(cfg, directory) + if err != nil { + t.Fatal(err) + } + parent, stop := context.WithCancel(WithEvidence(context.Background(), authentication.Evidence{Token: signed})) + defer stop() + _, execution, cancel, err := m.Establish(parent) + if err != nil { + t.Fatal(err) + } + defer cancel() + if _, bounded := execution.Deadline(); bounded { + t.Fatal("default execution acquired an authentication deadline") + } + explicitParent, stopExplicit := context.WithTimeout(parent, time.Minute) + defer stopExplicit() + _, explicitExecution, cancelExplicit, err := m.Establish(explicitParent) + if err != nil { + t.Fatal(err) + } + defer cancelExplicit() + parentDeadline, _ := explicitParent.Deadline() + deadline, bounded := explicitExecution.Deadline() + if !bounded || !deadline.Equal(parentDeadline) { + t.Fatal("explicit caller deadline was lost") + } + cfg.EnforceSessionDeadline = true + opted, err := newManager(cfg, directory) + if err != nil { + t.Fatal(err) + } + _, limited, stopLimited, err := opted.Establish(parent) + if err != nil { + t.Fatal(err) + } + defer stopLimited() + time.Sleep(8 * time.Hour) + if limited.Err() != context.DeadlineExceeded { + t.Fatal("opt-in session limit did not cancel execution") + } + if explicitExecution.Err() != context.DeadlineExceeded { + t.Fatal("explicit caller timeout did not cancel execution") + } + + if err := authorization.Check(execution, "host/sql/read", "execute"); err != nil { + t.Fatalf("authorized query did not survive expiry: %v", err) + } + if _, _, cancel, err := m.Establish(parent); err == nil { + cancel() + t.Fatal("expired token dispatched a new query") + } + stop() + if execution.Err() != context.Canceled { + t.Fatal("explicit cancellation was lost") + } + }) +} diff --git a/internal/stackql/accessmanagement/directory.go b/internal/stackql/accessmanagement/directory.go new file mode 100644 index 000000000..9257c7e0f --- /dev/null +++ b/internal/stackql/accessmanagement/directory.go @@ -0,0 +1,121 @@ +// Package accessmanagement owns StackQL's persistent caller directory. Provider credentials +// and their storage are independent of these tables. +package accessmanagement + +import ( + "context" + "database/sql" + "encoding/json" + "errors" + "fmt" + + "github.com/stackql-labs/omnisdk/public/principal" + "github.com/stackql/stackql/internal/stackql/sqlengine" +) + +type directory struct{ db *sql.DB } + +// NewDirectory uses the configured StackQL backend; it neither connects to a second database nor +// initializes schema implicitly. Missing schema or identities fail closed. +func NewDirectory(engine sqlengine.SQLEngine) (principal.Directory, error) { + if engine == nil { + return nil, errors.New("access directory requires the configured SQL engine") + } + db, err := engine.GetDB() + if err != nil { + return nil, err + } + return newDirectory(db) +} +func newDirectory(db *sql.DB) (principal.Directory, error) { + if db == nil { + return nil, errors.New("access directory requires a database") + } + return directory{db}, nil +} + +// InitializeSchema is an explicit administration operation, not part of identity resolution. +// The statements are shared by SQLite and PostgreSQL; table names never come from caller input. +func InitializeSchema(ctx context.Context, engine sqlengine.SQLEngine) error { + if engine == nil { + return errors.New("access directory requires the configured SQL engine") + } + db, err := engine.GetDB() + if err != nil { + return err + } + return initializeSchema(ctx, db) +} +func initializeSchema(ctx context.Context, db *sql.DB) error { + if db == nil { + return errors.New("access directory requires a database") + } + tx, err := db.BeginTx(ctx, nil) + if err != nil { + return err + } + defer tx.Rollback() //nolint:errcheck // Rollback after commit is harmless. + for _, stmt := range []string{ + `CREATE TABLE IF NOT EXISTS stackql_access_principals (id TEXT PRIMARY KEY, attributes TEXT NOT NULL DEFAULT '{}')`, + `CREATE TABLE IF NOT EXISTS stackql_access_groups (id TEXT PRIMARY KEY)`, + `CREATE TABLE IF NOT EXISTS stackql_access_identities (issuer TEXT NOT NULL, subject TEXT NOT NULL, + principal_id TEXT NOT NULL REFERENCES stackql_access_principals(id), PRIMARY KEY (issuer, subject))`, + `CREATE TABLE IF NOT EXISTS stackql_access_memberships ( + principal_id TEXT NOT NULL REFERENCES stackql_access_principals(id), + group_id TEXT NOT NULL REFERENCES stackql_access_groups(id), PRIMARY KEY (principal_id, group_id))`, + } { + if _, err = tx.ExecContext(ctx, stmt); err != nil { + return fmt.Errorf("access directory schema: %w", err) + } + } + return tx.Commit() +} +func (d directory) Resolve(ctx context.Context, i principal.Identity) (principal.Principal, error) { + if i == nil || i.Issuer() == "" || i.Subject() == "" { + return nil, errors.New("verified identity is required") + } + // A single SELECT supplies one consistent membership/attribute snapshot per session. + rows, err := d.db.QueryContext(ctx, `SELECT p.id, p.attributes, m.group_id, g.id + FROM stackql_access_identities i + JOIN stackql_access_principals p ON p.id = i.principal_id + LEFT JOIN stackql_access_memberships m ON m.principal_id = p.id + LEFT JOIN stackql_access_groups g ON g.id = m.group_id + WHERE i.issuer = $1 AND i.subject = $2 ORDER BY m.group_id`, i.Issuer(), i.Subject()) + if err != nil { + return nil, fmt.Errorf("resolve access identity: %w", err) + } + defer rows.Close() + var id, raw string + var groups []string + for rows.Next() { + var nextID, nextRaw string + var group, definedGroup sql.NullString + if err = rows.Scan(&nextID, &nextRaw, &group, &definedGroup); err != nil { + return nil, err + } + if id != "" && (id != nextID || raw != nextRaw) { + return nil, errors.New("ambiguous access identity") + } + id, raw = nextID, nextRaw + if group.Valid { + if !definedGroup.Valid { + return nil, errors.New("directory membership references an undefined group") + } + groups = append(groups, group.String) + } + } + if err = rows.Err(); err != nil { + return nil, err + } + if id == "" { + return nil, errors.New("identity is not in the directory") + } + var attributes map[string]string + if err = json.Unmarshal([]byte(raw), &attributes); err != nil { + return nil, fmt.Errorf("invalid principal attributes: %w", err) + } + if attributes == nil { + return nil, errors.New("principal attributes must be a JSON object") + } + return principal.NewPrincipal(id, groups, attributes) +} diff --git a/internal/stackql/accessmanagement/directory_test.go b/internal/stackql/accessmanagement/directory_test.go new file mode 100644 index 000000000..d04a11ecb --- /dev/null +++ b/internal/stackql/accessmanagement/directory_test.go @@ -0,0 +1,63 @@ +package accessmanagement //nolint:testpackage // Exercises the private directory implementation. + +import ( + "context" + "database/sql" + "testing" + + "github.com/stackql-labs/omnisdk/public/principal" + _ "modernc.org/sqlite" +) + +func TestRDBMSDirectory(t *testing.T) { + db, err := sql.Open("sqlite", ":memory:") + if err != nil { + t.Fatal(err) + } + defer db.Close() + db.SetMaxOpenConns(1) + ctx := context.Background() + if err := initializeSchema(ctx, db); err != nil { + t.Fatal(err) + } + for _, q := range []string{`INSERT INTO stackql_access_principals VALUES ('alice','{"team":"ops"}'), ('bob','{}')`, `INSERT INTO stackql_access_groups VALUES ('readers')`, `INSERT INTO stackql_access_identities VALUES ('one','same','alice'),('two','same','bob')`, `INSERT INTO stackql_access_memberships VALUES ('alice','readers')`} { + if _, err := db.Exec(q); err != nil { + t.Fatal(err) + } + } + d, err := newDirectory(db) + if err != nil { + t.Fatal(err) + } + for _, tc := range []struct { + issuer, id string + groups int + }{{"one", "alice", 1}, {"two", "bob", 0}} { + i, _ := principal.NewIdentity(tc.issuer, "same") + p, err := d.Resolve(ctx, i) + if err != nil { + t.Fatal(err) + } + if p.ID() != tc.id || len(p.Groups()) != tc.groups { + t.Fatalf("principal=%s groups=%v", p.ID(), p.Groups()) + } + } + i, _ := principal.NewIdentity("missing", "same") + if _, err := d.Resolve(ctx, i); err == nil { + t.Fatal("missing identity accepted") + } + // No cached membership: a new resolution sees the current directory. + if _, err := db.Exec(`DELETE FROM stackql_access_memberships`); err != nil { + t.Fatal(err) + } + i, _ = principal.NewIdentity("one", "same") + p, err := d.Resolve(ctx, i) + if err != nil || len(p.Groups()) != 0 { + t.Fatalf("stale directory: %v %v", p, err) + } + canceled, cancel := context.WithCancel(ctx) + cancel() + if _, err := d.Resolve(canceled, i); err == nil { + t.Fatal("ignored cancellation") + } +} diff --git a/internal/stackql/accessmanagement/statements.go b/internal/stackql/accessmanagement/statements.go new file mode 100644 index 000000000..de3f17db8 --- /dev/null +++ b/internal/stackql/accessmanagement/statements.go @@ -0,0 +1,83 @@ +package accessmanagement + +import ( + "context" + "fmt" + "slices" + "strings" + + "github.com/stackql-labs/omnisdk/public/authorization" + "github.com/stackql/stackql-parser/go/vt/sqlparser" +) + +// CheckStatement runs before planning or cache lookup. SQL syntax alone never grants SDK effects. +func CheckStatement(ctx context.Context, caller authorization.Session, statement sqlparser.Statement) error { + if caller == nil { + return authorization.ErrDenied + } + switch statement.(type) { + case *sqlparser.Auth, *sqlparser.AuthRevoke: + return fmt.Errorf("%w: provider AUTH administration is not integrated with caller protection", + authorization.ErrDenied) + } + read := false + switch statement.(type) { + case *sqlparser.Select, *sqlparser.Union, *sqlparser.ParenSelect, *sqlparser.DescribeTable, *sqlparser.DescribeMethod: + read = true + } + if show, ok := statement.(*sqlparser.Show); ok { + switch strings.ToUpper(show.Type) { + case "PROVIDERS", "SERVICES", "RESOURCES", "METHODS": + read = true + } + } + if !read || !slices.Contains(caller.Principal().Groups(), "readers") { + action := strings.ToLower(fmt.Sprintf("%T", statement)) + action = strings.TrimPrefix(action, "*sqlparser.") + if read { + action = "read" + } + if err := authorization.Check(ctx, "host/sql/"+action, "execute"); err != nil { + return err + } + } + return CheckNodes(ctx, statement) +} + +// CheckNodes also applies to expanded stored definitions and opaque backend reads. +func CheckNodes(ctx context.Context, node sqlparser.SQLNode) error { + return sqlparser.Walk(func(node sqlparser.SQLNode) (bool, error) { return true, checkNode(ctx, node) }, node) +} +func checkNode(ctx context.Context, node sqlparser.SQLNode) error { + switch n := node.(type) { + case *sqlparser.Exec: + return authorization.Check(ctx, "host/sql/exec", "execute") + case sqlparser.TableName: + return checkTable(ctx, n) + case *sqlparser.FuncExpr: + return checkFunction(ctx, n) + } + return nil +} +func checkTable(ctx context.Context, n sqlparser.TableName) error { + if strings.HasPrefix(strings.ToLower(n.Name.GetRawVal()), "stackql_access_") { + return fmt.Errorf("%w: directory tables require the administration path", authorization.ErrDenied) + } + if strings.EqualFold(n.Qualifier.GetRawVal(), "iac") && + strings.EqualFold(n.QualifierSecond.GetRawVal(), "stackql_preview") { + return authorization.Check(ctx, "host/iac/"+strings.ToLower(n.Name.GetRawVal()), "execute") + } + return nil +} +func checkFunction(ctx context.Context, n *sqlparser.FuncExpr) error { + name := strings.ToLower(n.Name.GetRawVal()) + // Only known pure host functions receive the reader capability. + const pure = "|whoami|groups|count|sum|avg|min|max|coalesce|ifnull|nullif|lower|upper|length|trim|ltrim|rtrim|" + + "substr|substring|replace|abs|round|ceil|floor|strftime|datetime|date|time|julianday|unixepoch|json|" + + "json_extract|json_array|json_object|json_type|json_valid|json_array_length|split_part|regexp_like|" + + "regexp_substr|regexp_replace|json_equal|aws_policy_equal|" + if !n.Qualifier.IsEmpty() || !strings.Contains(pure, "|"+name+"|") { + return authorization.Check(ctx, "host/function/"+name, "execute") + } + return nil +} diff --git a/internal/stackql/acid/tsm_physio/best_effort_coordinator.go b/internal/stackql/acid/tsm_physio/best_effort_coordinator.go index 9896d5a43..ef2364392 100644 --- a/internal/stackql/acid/tsm_physio/best_effort_coordinator.go +++ b/internal/stackql/acid/tsm_physio/best_effort_coordinator.go @@ -125,7 +125,7 @@ func (m *basicBestEffortTransactionCoordinator) Prepare() error { // This is an approximation of 2PC. func (m *basicBestEffortTransactionCoordinator) Execute() internaldto.ExecutorOutput { - outputs, err := m.votingPhase() + outputs, err := m.votingPhase(nil) m.outputs = outputs if err != nil { return internaldto.NewErroneousExecutorOutput(err) @@ -143,13 +143,15 @@ func (m *basicBestEffortTransactionCoordinator) IsExecuted() bool { return m.outputs != nil } -func (m *basicBestEffortTransactionCoordinator) votingPhase() ([]internaldto.ExecutorOutput, error) { +func (m *basicBestEffortTransactionCoordinator) votingPhase( + commitCtx handler.HandlerContext, +) ([]internaldto.ExecutorOutput, error) { defer func() { m.isExecuted = true }() var rv []internaldto.ExecutorOutput for _, stmt := range m.statementSequence { - coDomain := stmt.Execute() + coDomain := executeQueuedStatement(stmt, commitCtx) rv = append(rv, coDomain) err := coDomain.GetError() undoLog, undoLogExists := coDomain.GetUndoLog() @@ -178,8 +180,8 @@ func (m *basicBestEffortTransactionCoordinator) Begin() (Coordinator, error) { return newBasicBestEffortTransactionCoordinator(m.tsmInstance, m.handlerCtx, m, m.maxTxnDepth), nil } -func (m *basicBestEffortTransactionCoordinator) Commit() acid_dto.CommitCoDomain { - rv, err := m.votingPhase() +func (m *basicBestEffortTransactionCoordinator) Commit(commitCtx handler.HandlerContext) acid_dto.CommitCoDomain { + rv, err := m.votingPhase(commitCtx) if err != nil { return acid_dto.NewCommitCoDomain(rv, err, nil) } diff --git a/internal/stackql/acid/tsm_physio/best_effort_orchestrator.go b/internal/stackql/acid/tsm_physio/best_effort_orchestrator.go index bae3091e5..c2429fd09 100644 --- a/internal/stackql/acid/tsm_physio/best_effort_orchestrator.go +++ b/internal/stackql/acid/tsm_physio/best_effort_orchestrator.go @@ -108,7 +108,7 @@ func (orc *bestEffortOrchestrator) processQuery( internaldto.NewNopEmptyExecutorOutput([]string{"OK"}), }, true } else if transactStatement.IsCommit() { - commitCoDomain := orc.txnCoordinator.Commit() + commitCoDomain := orc.txnCoordinator.Commit(handlerCtx) commitErr, commitErrExists := commitCoDomain.GetError() if commitErrExists { return orc.undo([]string{ diff --git a/internal/stackql/acid/tsm_physio/coordinator.go b/internal/stackql/acid/tsm_physio/coordinator.go index d246141ef..57622f191 100644 --- a/internal/stackql/acid/tsm_physio/coordinator.go +++ b/internal/stackql/acid/tsm_physio/coordinator.go @@ -27,7 +27,7 @@ type Coordinator interface { // Begin a new transaction. Begin() (Coordinator, error) // Commit the current transaction. - Commit() acid_dto.CommitCoDomain + Commit(handler.HandlerContext) acid_dto.CommitCoDomain // Rollback the current transaction. Rollback() acid_dto.CommitCoDomain // Enqueue a transaction operation. diff --git a/internal/stackql/acid/tsm_physio/lazy_coordinator.go b/internal/stackql/acid/tsm_physio/lazy_coordinator.go index 38e08b4e2..54a9f9669 100644 --- a/internal/stackql/acid/tsm_physio/lazy_coordinator.go +++ b/internal/stackql/acid/tsm_physio/lazy_coordinator.go @@ -7,6 +7,7 @@ import ( "github.com/stackql/stackql/internal/stackql/acid/acid_dto" "github.com/stackql/stackql/internal/stackql/acid/binlog" "github.com/stackql/stackql/internal/stackql/acid/tsm" + "github.com/stackql/stackql/internal/stackql/handler" "github.com/stackql/stackql/internal/stackql/internal_data_transfer/internaldto" "github.com/stackql/stackql/internal/stackql/primitivegraph" ) @@ -110,7 +111,7 @@ func (m *basicLazyTransactionCoordinator) Prepare() error { // This is an approximation of 2PC. func (m *basicLazyTransactionCoordinator) Execute() internaldto.ExecutorOutput { - outputs, err := m.votingPhase() + outputs, err := m.votingPhase(nil) m.outputs = outputs if err != nil { return internaldto.NewErroneousExecutorOutput(err) @@ -128,13 +129,15 @@ func (m *basicLazyTransactionCoordinator) IsExecuted() bool { return m.outputs != nil } -func (m *basicLazyTransactionCoordinator) votingPhase() ([]internaldto.ExecutorOutput, error) { +func (m *basicLazyTransactionCoordinator) votingPhase( + commitCtx handler.HandlerContext, +) ([]internaldto.ExecutorOutput, error) { defer func() { m.isExecuted = true }() var rv []internaldto.ExecutorOutput for _, stmt := range m.statementSequence { - coDomain := stmt.Execute() + coDomain := executeQueuedStatement(stmt, commitCtx) rv = append(rv, coDomain) err := coDomain.GetError() undoLog, undoLogExists := coDomain.GetUndoLog() @@ -167,8 +170,8 @@ func (m *basicLazyTransactionCoordinator) Begin() (Coordinator, error) { return newBasicLazyTransactionCoordinator(m.tsmInstance, m, m.maxTxnDepth), nil } -func (m *basicLazyTransactionCoordinator) Commit() acid_dto.CommitCoDomain { - rv, err := m.votingPhase() +func (m *basicLazyTransactionCoordinator) Commit(commitCtx handler.HandlerContext) acid_dto.CommitCoDomain { + rv, err := m.votingPhase(commitCtx) if err != nil { return acid_dto.NewCommitCoDomain(rv, err, nil) } diff --git a/internal/stackql/acid/tsm_physio/statement.go b/internal/stackql/acid/tsm_physio/statement.go index 20e08b25e..68c398712 100644 --- a/internal/stackql/acid/tsm_physio/statement.go +++ b/internal/stackql/acid/tsm_physio/statement.go @@ -124,3 +124,15 @@ func (st *basicStatement) GetAST() (sqlparser.Statement, bool) { func (st *basicStatement) GetPrimitiveGraphHolder() (primitivegraph.PrimitiveGraphHolder, bool) { return st.querySubmitter.GetPrimitiveGraphHolder() } + +// Deferred work runs under the COMMIT request, rather than the already completed +// request that queued it. Replanning also refreshes caller authorization. +func executeQueuedStatement(stmt Statement, commitCtx handler.HandlerContext) internaldto.ExecutorOutput { + if queued, ok := stmt.(*basicStatement); ok && commitCtx != nil && !queued.IsExecuted() { + queued.handlerCtx.SetExecutionContext(commitCtx.GetExecutionContext()) + if err := queued.Prepare(); err != nil { + return internaldto.NewErroneousExecutorOutput(err) + } + } + return stmt.Execute() +} diff --git a/internal/stackql/acid/tsm_physio/txn_orchestrator.go b/internal/stackql/acid/tsm_physio/txn_orchestrator.go index 8450aaa57..41b06b770 100644 --- a/internal/stackql/acid/tsm_physio/txn_orchestrator.go +++ b/internal/stackql/acid/tsm_physio/txn_orchestrator.go @@ -111,7 +111,7 @@ func (orc *standardOrchestrator) processQuery( internaldto.NewNopEmptyExecutorOutput([]string{"OK"}), }, true } else if transactStatement.IsCommit() { - commitCoDomain := orc.txnCoordinator.Commit() + commitCoDomain := orc.txnCoordinator.Commit(handlerCtx) commitErr, commitErrExists := commitCoDomain.GetError() if commitErrExists { retVal := []internaldto.ExecutorOutput{ diff --git a/internal/stackql/cmd/access.go b/internal/stackql/cmd/access.go new file mode 100644 index 000000000..d93469c87 --- /dev/null +++ b/internal/stackql/cmd/access.go @@ -0,0 +1,58 @@ +package cmd + +import ( + "context" + "errors" + "net/http" + "strings" + + "github.com/stackql-labs/omnisdk/public/authentication" + "github.com/stackql/stackql/internal/stackql/accessmanagement" + "github.com/stackql/stackql/internal/stackql/handler" + "github.com/stackql/stackql/pkg/mcp_server" +) + +func configureCallerMCP(host handler.HandlerContext, cfg *mcp_server.Config) error { + manager := host.GetAccessManager() + if manager == nil { + return nil + } + if mcpServerType == "reverse_proxy" { + return errors.New("caller access does not support the MCP reverse proxy") + } + if cfg.GetServerTransport() != "http" { + return nil + } + if manager.Mode() == "local_os" { + return errors.New("local OS authentication is unavailable over HTTP") + } + if cfg.Server.TLSCertFile == "" || cfg.Server.TLSKeyFile == "" { + return errors.New("caller access requires HTTPS for MCP HTTP") + } + cfg.TLSConfig = manager.TLSConfig() + cfg.AuthenticateRequest = func(req *http.Request) (context.Context, error) { + if req.TLS == nil { + return nil, errors.New("TLS credential required") + } + credential := authentication.Evidence{TLS: req.TLS} + if manager.Mode() == "mtls" { + if req.Header.Get("Authorization") != "" { + return nil, errors.New("mTLS accepts certificate credentials only") + } + } else { + fields := strings.Fields(req.Header.Get("Authorization")) + if len(fields) != 2 || !strings.EqualFold(fields[0], "Bearer") { + return nil, errors.New("TLS bearer credential required") + } + credential.Token = fields[1] + } + evidence := accessmanagement.WithEvidence(req.Context(), credential) + _, _, cancel, err := manager.Establish(evidence) + if err != nil { + return nil, err + } + cancel() + return evidence, nil + } + return nil +} diff --git a/internal/stackql/cmd/exec.go b/internal/stackql/cmd/exec.go index b2d4cb24c..049295910 100644 --- a/internal/stackql/cmd/exec.go +++ b/internal/stackql/cmd/exec.go @@ -17,6 +17,7 @@ package cmd import ( "bytes" + "errors" "io" "os" "runtime/pprof" @@ -103,5 +104,9 @@ func (cr *commandRunnerImpl) RunCommand(handlerCtx handler.HandlerContext) { stackqlDriver.ProcessDryRun(handlerCtx.GetRawQuery()) return } - stackqlDriver.ProcessQuery(handlerCtx.GetRawQuery()) + queryErr := stackqlDriver.ProcessQueryWithError(handlerCtx.GetRawQuery()) + var iacFailure interface{ IaCFailure() bool } + if queryErr != nil && (handlerCtx.GetRuntimeContext().AccessConfigFile != "" || errors.As(queryErr, &iacFailure)) { + os.Exit(1) // The response handler has already presented the error. + } } diff --git a/internal/stackql/cmd/mcp.go b/internal/stackql/cmd/mcp.go index 4e9664c9a..8c9751d9e 100644 --- a/internal/stackql/cmd/mcp.go +++ b/internal/stackql/cmd/mcp.go @@ -161,6 +161,7 @@ func loadMCPConfig() (*mcp_server.Config, error) { } func runMCPServer(handlerCtx handler.HandlerContext, config *mcp_server.Config) { + iqlerror.PrintErrorAndExitOneIfError(configureCallerMCP(handlerCtx, config)) defer func() { if r := recover(); r != nil { mcpDiagf("panic: %v\n%s", r, debug.Stack()) diff --git a/internal/stackql/cmd/root.go b/internal/stackql/cmd/root.go index b706d1399..92f617298 100644 --- a/internal/stackql/cmd/root.go +++ b/internal/stackql/cmd/root.go @@ -88,7 +88,17 @@ var otelConfigRaw string // //nolint:gochecknoglobals // global vars are a pattern for this lib var rootCmd = &cobra.Command{ - Use: "stackql", + Use: "stackql", + PersistentPreRunE: func(cmd *cobra.Command, _ []string) error { + if runtimeCtx.AccessConfigFile == "" { + return nil + } + switch cmd.Name() { + case "stackql", "exec", "shell", "srv", "mcp", "version": + return nil + } + return fmt.Errorf("caller access: this command is not integrated with caller authorization") + }, Short: "Cloud asset management and automation using SQL", Long: ` __ __ __ @@ -150,6 +160,7 @@ func vcsBuildDate() string { //nolint:lll,funlen,gochecknoinits,mnd // init is a pattern for this lib func init() { + rootCmd.PersistentFlags().StringVar(&runtimeCtx.AccessConfigFile, "access.config", "", "Caller access configuration: JSON {\"path\":\"/path/to/access.json\"}; token credentials use STACKQL_CALLER_TOKEN locally") buildDate := BuildDate if buildDate == "" { buildDate = vcsBuildDate() diff --git a/internal/stackql/driver/driver.go b/internal/stackql/driver/driver.go index dacae46f6..ebd7fdd90 100644 --- a/internal/stackql/driver/driver.go +++ b/internal/stackql/driver/driver.go @@ -7,7 +7,9 @@ import ( "strings" "time" + "github.com/stackql-labs/omnisdk/public/authentication" "github.com/stackql/psql-wire/pkg/sqldata" + "github.com/stackql/stackql/internal/stackql/accessmanagement" "github.com/stackql/stackql/internal/stackql/acid/tsm_physio" "github.com/stackql/stackql/internal/stackql/handler" "github.com/stackql/stackql/internal/stackql/internal_data_transfer/internaldto" @@ -108,6 +110,7 @@ type StackQLDriver interface { sqlbackend.ISQLBackend ProcessDryRun(string) ProcessQuery(string) + ProcessQueryWithError(string) error } func (dr *basicStackQLDriver) ProcessDryRun(query string) { @@ -123,16 +126,22 @@ func (dr *basicStackQLDriver) ProcessDryRun(query string) { responsehandler.HandleResponse(dr.handlerCtx, response) //nolint:errcheck // TODO: investigate } -func (dr *basicStackQLDriver) ProcessQuery(query string) { +func (dr *basicStackQLDriver) ProcessQuery(query string) { _ = dr.ProcessQueryWithError(query) } + +func (dr *basicStackQLDriver) ProcessQueryWithError(query string) error { clonedCtx := dr.handlerCtx.Clone() clonedCtx.SetRawQuery(query) clonedCtx.SetQueryStartTime(time.Now()) responses, ok := dr.processQueryOrQueries(clonedCtx) + var firstErr error if ok { for _, r := range responses { - responsehandler.HandleResponse(clonedCtx, r) //nolint:errcheck // TODO: investigate + if err := responsehandler.HandleResponse(clonedCtx, r); err != nil && firstErr == nil { + firstErr = err + } } } + return firstErr } type stmtMeta struct { @@ -170,6 +179,7 @@ func (dr *basicStackQLDriver) CloneSQLBackend() sqlbackend.ISQLBackend { //nolint:revive // TODO: review func (dr *basicStackQLDriver) HandleSimpleQuery(ctx context.Context, query string) (sqldata.ISQLResultStream, error) { + dr.handlerCtx.SetExecutionContext(accessmanagement.WithEvidence(ctx, authentication.Evidence{})) dr.handlerCtx.SetRawQuery(query) if isEmptyStatement(query) { // Mirrors PostgreSQL's EmptyQueryResponse: a statement containing only diff --git a/internal/stackql/driver/request_context_test.go b/internal/stackql/driver/request_context_test.go new file mode 100644 index 000000000..177527c0e --- /dev/null +++ b/internal/stackql/driver/request_context_test.go @@ -0,0 +1,69 @@ +package driver_test + +import ( + "context" + "errors" + "io" + "testing" + + "github.com/stackql/psql-wire/pkg/sqldata" + lrucache "github.com/stackql/stackql-parser/go/cache" + "github.com/stackql/stackql/internal/stackql/driver" + "github.com/stackql/stackql/internal/stackql/handler" + "github.com/stackql/stackql/internal/test/stackqltestutil" +) + +func TestCommitUsesCurrentRequestContext(t *testing.T) { + runtime, err := stackqltestutil.GetRuntimeCtx("", "json", t.Name()) + if err != nil { + t.Fatal(err) + } + bundle, err := stackqltestutil.BuildInputBundle(*runtime) + if err != nil { + t.Fatal(err) + } + hc, err := handler.NewHandlerCtx("", *runtime, lrucache.NewLRUCache(0), bundle, "test") + if err != nil { + t.Fatal(err) + } + backend, err := driver.NewStackQLDriver(hc) + if err != nil { + t.Fatal(err) + } + // PG wire cancels each command's context as soon as that command completes. + for _, query := range []string{ + "begin;", + "create materialized view request_context_mv as select 'kept' as value;", + "commit;", + "select * from request_context_mv;", + "drop materialized view request_context_mv;", + } { + ctx, cancel := context.WithCancel(context.Background()) + result, queryErr := backend.HandleSimpleQuery(ctx, query) + if queryErr != nil { + cancel() + t.Fatalf("%s: %v", query, queryErr) + } + if query == "select * from request_context_mv;" { + assertCommittedRequestResult(t, result) + } + if result != nil { + if err := result.Close(); err != nil { + cancel() + t.Fatal(err) + } + } + cancel() + } +} + +func assertCommittedRequestResult(t *testing.T, result sqldata.ISQLResultStream) { + t.Helper() + if result == nil { + t.Fatal("committed materialized view returned no result") + } + rows, err := result.Read() + if (err != nil && !errors.Is(err, io.EOF)) || rows == nil || len(rows.ToArr()) != 1 || rows.ToArr()[0]["value"] != "kept" { + t.Fatalf("committed materialized view: result=%v, error=%v", rows, err) + } +} diff --git a/internal/stackql/entryutil/entryutil.go b/internal/stackql/entryutil/entryutil.go index 9eeee66b9..650986551 100644 --- a/internal/stackql/entryutil/entryutil.go +++ b/internal/stackql/entryutil/entryutil.go @@ -43,7 +43,8 @@ func BuildInputBundle(runtimeCtx dto.RuntimeCtx) (bundle.Bundle, error) { if err != nil { return nil, fmt.Errorf("error unmarshalling auth: %w", err) } - se, err := buildSQLEngine(sqlCfg) + + se, err := buildSQLEngine(sqlCfg, runtimeCtx.AccessConfigFile != "") if err != nil { return nil, err } @@ -143,11 +144,12 @@ func initNamespaces( } func buildSQLEngine( - sqlCfg dto.SQLBackendCfg, + sqlCfg dto.SQLBackendCfg, requireForeignKeys bool, ) (sqlengine.SQLEngine, error) { return sqlengine.NewSQLEngine(sqlengine.Config{ DBEngine: sqlCfg.DBEngine, DSN: sqlCfg.GetDSN(), DbInitFilePath: sqlCfg.DbInitFilePath, - InitMaxRetries: sqlCfg.InitMaxRetries, InitRetryInitialDelay: sqlCfg.InitRetryInitialDelay, + RequireForeignKeys: requireForeignKeys, + InitMaxRetries: sqlCfg.InitMaxRetries, InitRetryInitialDelay: sqlCfg.InitRetryInitialDelay, }) } diff --git a/internal/stackql/handler/handler.go b/internal/stackql/handler/handler.go index 7fd872bd9..e79acf340 100644 --- a/internal/stackql/handler/handler.go +++ b/internal/stackql/handler/handler.go @@ -1,6 +1,7 @@ package handler import ( + "context" "encoding/json" "fmt" "io" @@ -11,6 +12,9 @@ import ( "sync" "time" + "github.com/stackql-labs/omnisdk/public/authorization" + "github.com/stackql/stackql/internal/stackql/accessmanagement" + "github.com/stackql-labs/omnisdk/pkg/omnisdk" "github.com/stackql/stackql/internal/stackql/acid/tsm" "github.com/stackql/stackql/internal/stackql/acid/txn_context" @@ -46,6 +50,11 @@ var ( type HandlerContext interface { //nolint:revive // don't mind stuttering this one Clone() HandlerContext + GetExecutionContext() context.Context + SetExecutionContext(context.Context) + GetAccessManager() accessmanagement.Manager + GetCaller() authorization.Session + SetCaller(authorization.Session) // GetStacqklSemver() string GetASTFormatter() sqlparser.NodeFormatter @@ -122,6 +131,9 @@ type HandlerContext interface { //nolint:revive // don't mind stuttering this on } type standardHandlerContext struct { + executionContext context.Context + accessManager accessmanagement.Manager + caller authorization.Session // mutex to protect auth context map // must be a pointer dure to clonability authMapMutex *sync.Mutex @@ -593,6 +605,7 @@ func (hc *standardHandlerContext) Clone() HandlerContext { hc.sessionCtxMutex.Lock() defer hc.sessionCtxMutex.Unlock() rv := standardHandlerContext{ + executionContext: hc.executionContext, accessManager: hc.accessManager, caller: hc.caller, authMapMutex: hc.authMapMutex, sessionCtxMutex: hc.sessionCtxMutex, providersMapMutex: hc.providersMapMutex, @@ -647,6 +660,10 @@ func NewHandlerCtx( inputBundle bundle.Bundle, stackqlSemver string, ) (HandlerContext, error) { + accessManager, err := accessmanagement.NewManager(runtimeCtx.AccessConfigFile, inputBundle.GetSQLEngine()) + if err != nil { + return nil, err + } reg, err := getRegistry(runtimeCtx) if err != nil { return nil, err @@ -699,6 +716,7 @@ func NewHandlerCtx( txnCoordinatorCtx: inputBundle.GetTxnCoordinatorContext(), typCfg: inputBundle.GetTypingConfig(), sessionContext: inputBundle.GetSessionContext(), + accessManager: accessManager, exportNamespace: runtimeCtx.ExportAlias, outfile: outWriter, outErrFile: outErrWriter, @@ -722,3 +740,16 @@ func transformOpenapiStackqlAuthToLocal(auth *omnisdk.ProviderAuth) *dto.AuthCtx } return &dto.AuthCtx{AuthConfig: auth.AuthConfig, Successor: transformOpenapiStackqlAuthToLocal(auth.Successor)} } + +func (hc *standardHandlerContext) GetExecutionContext() context.Context { + if hc.executionContext == nil { + return context.Background() + } + return hc.executionContext +} +func (hc *standardHandlerContext) SetExecutionContext(ctx context.Context) { hc.executionContext = ctx } +func (hc *standardHandlerContext) GetAccessManager() accessmanagement.Manager { + return hc.accessManager +} +func (hc *standardHandlerContext) GetCaller() authorization.Session { return hc.caller } +func (hc *standardHandlerContext) SetCaller(caller authorization.Session) { hc.caller = caller } diff --git a/internal/stackql/intrinsic/access.go b/internal/stackql/intrinsic/access.go new file mode 100644 index 000000000..b50f3af1c --- /dev/null +++ b/internal/stackql/intrinsic/access.go @@ -0,0 +1,91 @@ +package intrinsic + +import ( + "context" + "encoding/json" + "errors" + "strings" + + "github.com/stackql/stackql/internal/stackql/internal_data_transfer/internaldto" + "github.com/stackql/stackql/internal/stackql/util" + + "github.com/stackql-labs/omnisdk/pkg/omnisdk" + "github.com/stackql-labs/omnisdk/public/authorization" + "github.com/stackql/stackql-parser/go/vt/sqlparser" + "github.com/stackql/stackql/internal/stackql/accessmanagement" +) + +func executionContext(ctx queryContext) context.Context { + if host, ok := ctx.(interface{ GetExecutionContext() context.Context }); ok { + return host.GetExecutionContext() + } + return context.Background() +} +func configureCaller(ctx queryContext, args *omnisdk.Args) error { + args.RequireCaller = ctx.GetRuntimeContext().AccessConfigFile != "" + if host, ok := ctx.(interface{ GetCaller() authorization.Session }); ok { + args.Caller = host.GetCaller() + } + if args.RequireCaller && args.Caller == nil { + return errors.New("authenticated caller is required") + } + return nil +} +func checkHost(ctx queryContext, resource, action string) error { + if ctx.GetRuntimeContext().AccessConfigFile == "" { + return nil + } + return authorization.Check(executionContext(ctx), resource, action) +} +func checkSQLNodes(ctx queryContext, node sqlparser.SQLNode) error { + if ctx.GetRuntimeContext().AccessConfigFile == "" { + return nil + } + return accessmanagement.CheckNodes(executionContext(ctx), node) +} + +// identitySelect exposes only the established directory snapshot, never JWT claims. +func identitySelect(ctx queryContext, node *sqlparser.Select) (func() internaldto.ExecutorOutput, bool) { + if !backendOnlySelect(node) { + return nil, false + } + host, hasCaller := ctx.(interface{ GetCaller() authorization.Session }) + if !hasCaller || host.GetCaller() == nil { + return nil, false + } + row := map[string]interface{}{} + var columns []string + for _, expression := range node.SelectExprs { + expr, ok := expression.(*sqlparser.AliasedExpr) + if !ok { + return nil, false + } + function, ok := expr.Expr.(*sqlparser.FuncExpr) + if !ok || len(function.Exprs) != 0 || !function.Qualifier.IsEmpty() { + return nil, false + } + name := strings.ToLower(function.Name.GetRawVal()) + var value string + switch name { + case "whoami": + value = host.GetCaller().Principal().ID() + case "groups": + data, _ := json.Marshal(host.GetCaller().Principal().Groups()) + value = string(data) + default: + return nil, false + } + output := expr.As.GetRawVal() + if output == "" { + output = name + } + columns = append(columns, output) + row[output] = value + } + if len(columns) == 0 { + return nil, false + } + return func() internaldto.ExecutorOutput { + return prepare(ctx, columns, map[string]map[string]interface{}{"0": row}, util.DefaultRowSort) + }, true +} diff --git a/internal/stackql/intrinsic/auth.go b/internal/stackql/intrinsic/auth.go index c0bf3c087..858bbcb88 100644 --- a/internal/stackql/intrinsic/auth.go +++ b/internal/stackql/intrinsic/auth.go @@ -51,7 +51,7 @@ func authRevokeFunc(ctx queryContext, node *sqlparser.AuthRevoke) func() interna if err != nil { return internaldto.NewExecutorOutput(nil, nil, nil, nil, err) } - deadline, cancel := context.WithTimeout(context.Background(), time.Minute) + deadline, cancel := context.WithTimeout(executionContext(ctx), time.Minute) defer cancel() err = omnisdk.NewCredential(node.Provider, *dto.OmniAuth(configured)).Revoke(deadline) if err == nil { diff --git a/internal/stackql/intrinsic/doc.go b/internal/stackql/intrinsic/doc.go index 7929975ef..423e6b511 100644 --- a/internal/stackql/intrinsic/doc.go +++ b/internal/stackql/intrinsic/doc.go @@ -293,7 +293,7 @@ const mutationSuccessMessage = "The operation was despatched successfully" // openDocQuery describes each relation, resolves the query and opens its // cursor, returning it with the alias of the relation it reports. func openDocQuery(ctx queryContext, translated docQuery) (omnisdk.Rows, string, error) { - return openDocQueryIn(ctx, translated, context.Background()) + return openDocQueryIn(ctx, translated, executionContext(ctx)) } func openDocQueryIn(ctx queryContext, translated docQuery, runCtx context.Context) (omnisdk.Rows, string, error) { diff --git a/internal/stackql/intrinsic/exec.go b/internal/stackql/intrinsic/exec.go index ecf33240c..495a93ab8 100644 --- a/internal/stackql/intrinsic/exec.go +++ b/internal/stackql/intrinsic/exec.go @@ -100,7 +100,7 @@ func execFunc(ctx queryContext, node *sqlparser.Exec) func() internaldto.Executo if err != nil { return internaldto.NewErroneousExecutorOutput(err) } - rows, err := plan.Open(context.Background()) + rows, err := plan.Open(executionContext(ctx)) if err != nil { return internaldto.NewErroneousExecutorOutput(err) } diff --git a/internal/stackql/intrinsic/external_relations.go b/internal/stackql/intrinsic/external_relations.go index b38fc0978..3a311a062 100644 --- a/internal/stackql/intrinsic/external_relations.go +++ b/internal/stackql/intrinsic/external_relations.go @@ -52,6 +52,9 @@ func hasExternalRelations(relations map[string]omnisdk.Relation) bool { func (r *storedResolver) registerExternal( table *sqlparser.AliasedTableExpr, name sqlparser.TableName, source sql_datasource.SQLDataSource, ) error { + if err := checkHost(r.ctx, "host/backend", "read"); err != nil { + return err + } reader, ok := source.(sql_datasource.ContextQuerier) if !ok { return fmt.Errorf("external source %s does not support cancellable reads", name.QualifierSecond.GetRawVal()) @@ -61,7 +64,7 @@ func (r *storedResolver) registerExternal( } // SQL identifiers are parsed names and escaped individually; connection names never enter SQL. actual := quoteRelationIdentifier(name.Qualifier.GetRawVal()) + "." + quoteRelationIdentifier(name.Name.GetRawVal()) - discovery, cancel := context.WithTimeout(context.Background(), omnisdk.DefaultRequestTimeout) + discovery, cancel := context.WithTimeout(executionContext(r.ctx), omnisdk.DefaultRequestTimeout) defer cancel() //nolint:gosec // Only individually quoted SQL identifiers enter this metadata query. metadata, err := reader.QueryContext(discovery, "SELECT * FROM "+actual+" WHERE 1=0") diff --git a/internal/stackql/intrinsic/iac.go b/internal/stackql/intrinsic/iac.go index 86c744cd3..8ce8418ac 100644 --- a/internal/stackql/intrinsic/iac.go +++ b/internal/stackql/intrinsic/iac.go @@ -6,15 +6,22 @@ package intrinsic // compensation are the SDK's problem, not stackql's. import ( + "bytes" + "crypto/rand" + "crypto/sha256" "encoding/json" "errors" "fmt" + "io" "io/fs" + "os" "path/filepath" "strings" + "sync" "time" "github.com/stackql-labs/omnisdk/pkg/omnisdk" + "github.com/stackql-labs/omnisdk/public/authorization" "github.com/stackql/stackql/internal/stackql/internal_data_transfer/internaldto" "github.com/stackql/stackql-parser/go/vt/sqlparser" @@ -32,29 +39,6 @@ var errStateEscapesRoot = errors.New("must name a directory inside the iac state // runIDLayout stamps a run that names no run id, as omnisdk's Converge always has. const runIDLayout = "20060102T150405Z" -// resourceDTO is the wire shape of one entry in the 'resources' predicate. It -// is omnisdk's ManagedResource with the interface flattened into data. -type resourceDTO struct { - Key string `json:"key"` - Provider string `json:"provider"` - Address string `json:"address"` - Desired json.RawMessage `json:"desired"` - Params map[string]string `json:"params"` - Inbound []arrivalDTO `json:"inbound"` - ViaType string `json:"viaType"` - ViaProgram string `json:"viaProgram"` - Identity string `json:"identity"` - AddressedBy string `json:"addressedBy"` - CorrelationParam string `json:"correlationParam"` -} - -// arrivalDTO is one value reaching a resource from a sibling in the same -// collection. -type arrivalDTO struct { - From string `json:"from"` - As string `json:"as"` -} - // iacSelectFunc routes a SELECT over stackql_preview.iac. Opening the plan // performs the run, so a select over the converge relation is a mutation in // every sense except its syntax. @@ -63,21 +47,30 @@ func iacSelectFunc( node *sqlparser.Select, resource string, ) (func() internaldto.ExecutorOutput, bool) { + var action func() internaldto.ExecutorOutput switch { case strings.EqualFold(resource, convergeRelation): - return convergeSelectFunc(ctx, node) + action = convergeSelectFunc(ctx, node) case strings.EqualFold(resource, destroyRelation): - return destroySelectFunc(ctx, node) + action = destroySelectFunc(ctx, node) + default: + action = blueprintSelectFunc(resource) } - return blueprintSelectFunc(resource) + return func() internaldto.ExecutorOutput { + output := action() + if err := output.GetError(); err != nil { + return internaldto.NewErroneousExecutorOutput(iacOutcomeError{cause: err}) + } + return output + }, true } // convergeSelectFunc routes a SELECT over the converge relation. -func convergeSelectFunc(ctx queryContext, node *sqlparser.Select) (func() internaldto.ExecutorOutput, bool) { +func convergeSelectFunc(ctx queryContext, node *sqlparser.Select) func() internaldto.ExecutorOutput { relation := relationAddress(iacService, convergeRelation) params, ledger, refusal := ledgerPredicates(ctx, node, relation) if refusal != nil { - return refusal, true + return refusal } handle := popPredicate(params, blueprintPredicate) spec := popPredicate(params, resourcesPredicate) @@ -86,9 +79,13 @@ func convergeSelectFunc(ctx queryContext, node *sqlparser.Select) (func() intern return refuse(fmt.Errorf( "relation '%s' needs exactly one of '%s', '%s' or '%s'; "+ "a blueprint renders the resources, a specification or a file states them", - relation, blueprintPredicate, resourcesPredicate, resourcesFilePredicate)), true + relation, blueprintPredicate, resourcesPredicate, resourcesFilePredicate)) } return func() internaldto.ExecutorOutput { + inputs, environment, err := iacBindings(params) + if err != nil { + return internaldto.NewErroneousExecutorOutput(err) + } resources, err := convergeResources(handle, spec, file, params) if err != nil { return internaldto.NewErroneousExecutorOutput(err) @@ -97,35 +94,49 @@ func convergeSelectFunc(ctx queryContext, node *sqlparser.Select) (func() intern if argsErr != nil { return internaldto.NewErroneousExecutorOutput(argsErr) } + args.Inputs, args.Environment = inputs, environment plan, planErr := omnisdk.Prepare(registryRoot(ctx), omnisdk.DesiredState(resources), ledger, args) if planErr != nil { return internaldto.NewErroneousExecutorOutput(planErr) } return streamPlan(ctx, plan, convergeRelation, node.SelectExprs) - }, true + } } // destroySelectFunc routes a SELECT over the destroy relation. What the collection holds comes // from its ledger alone, so the remaining predicates are only the scope the deletes run in. With // no resources to name a provider, credentials come from each cloud's canonical environment // variables. -func destroySelectFunc(ctx queryContext, node *sqlparser.Select) (func() internaldto.ExecutorOutput, bool) { +func destroySelectFunc(ctx queryContext, node *sqlparser.Select) func() internaldto.ExecutorOutput { relation := relationAddress(iacService, destroyRelation) params, ledger, refusal := ledgerPredicates(ctx, node, relation) if refusal != nil { - return refusal, true + return refusal + } + selection, selected := params["keys"] + delete(params, "keys") + var keys []string + if selected { + if err := json.Unmarshal([]byte(selection), &keys); err != nil || len(keys) == 0 { + return refuse(errors.New("partial destroy requires a nonempty JSON array of keys")) + } + for _, key := range keys { + if strings.TrimSpace(key) == "" { + return refuse(errors.New("partial destroy requires nonempty keys")) + } + } } return func() internaldto.ExecutorOutput { args, argsErr := previewArgs(ctx, "", params) if argsErr != nil { return internaldto.NewErroneousExecutorOutput(argsErr) } - plan, planErr := omnisdk.Prepare(registryRoot(ctx), omnisdk.Destroy(), ledger, args) + plan, planErr := omnisdk.Prepare(registryRoot(ctx), omnisdk.Destroy(keys...), ledger, args) if planErr != nil { return internaldto.NewErroneousExecutorOutput(planErr) } return streamPlan(ctx, plan, destroyRelation, node.SelectExprs) - }, true + } } // ledgerPredicates reads the predicates every ledger backed relation takes: the collection it @@ -152,13 +163,25 @@ func ledgerPredicates( if root == "" { root = filepath.Join(ctx.GetRuntimeContext().ApplicationFilesRootPath, stateDirName) } + if ctx.GetRuntimeContext().AccessConfigFile != "" { + host, ok := ctx.(interface{ GetCaller() authorization.Session }) + if !ok || host.GetCaller() == nil { + return nil, nil, refuse(errors.New("authenticated caller required for IaC state")) + } + principal := host.GetCaller().Principal() + owner := "principal:" + principal.ID() + if tenant := principal.Attributes()["tenant"]; tenant != "" { + owner = "tenant:" + tenant + } + root = filepath.Join(root, fmt.Sprintf("%x", sha256.Sum256([]byte(owner)))) + } state, stateErr := resolveStateDir(root, popPredicate(params, statePredicate)) if stateErr != nil { return nil, nil, refuse(fmt.Errorf("relation '%s': '%s' %w", relation, statePredicate, stateErr)) } runID := popPredicate(params, runIDPredicate) if runID == "" { - runID = time.Now().UTC().Format(runIDLayout) + runID = time.Now().UTC().Format(runIDLayout) + "-" + rand.Text() } return params, omnisdk.NewLedger(collection, state, runID), nil } @@ -233,7 +256,11 @@ func convergeResources( case handle != "": return blueprintResources(handle, params) case file != "": - return omnisdk.LoadResources(file) + data, err := os.ReadFile(file) + if err != nil { + return nil, err + } + return specResources(string(data)) } return specResources(spec) } @@ -259,28 +286,22 @@ func blueprintResources(handle string, params map[string]string) ([]omnisdk.Mana // specResources reads the 'resources' predicate. func specResources(spec string) ([]omnisdk.ManagedResource, error) { - var specs []resourceDTO - if err := json.Unmarshal([]byte(spec), &specs); err != nil { - return nil, fmt.Errorf( - "intrinsic: '%s' is not a valid resource specification: %w", resourcesPredicate, err) + resources, err := omnisdk.ParseResources([]byte(spec)) + if err != nil { + return nil, fmt.Errorf("intrinsic: invalid resource specification: %w", err) } - out := make([]omnisdk.ManagedResource, 0, len(specs)) - for _, resource := range specs { - arrivals := make([]omnisdk.Arrival, 0, len(resource.Inbound)) - for _, arrival := range resource.Inbound { - arrivals = append(arrivals, omnisdk.Arrival{From: arrival.From, As: arrival.As}) + seen := map[string]bool{} + for _, resource := range resources { + if seen[resource.Key()] { + return nil, fmt.Errorf("duplicate resource key %q", resource.Key()) } - out = append(out, omnisdk.NewResource( - resource.Key, resource.Provider, resource.Address, resource.Desired, resource.Params, - arrivals, resource.ViaType, resource.ViaProgram, - resource.Identity, resource.AddressedBy, resource.CorrelationParam)) + seen[resource.Key()] = true } - return out, nil + return resources, nil } -// convergeCloud is the cloud whose credential a run uses. omnisdk takes one -// credential per run, so a collection spanning two clouds carries the first -// resource's and leaves the rest to the canonical environment variables. +// convergeCloud selects the initial provider credential. Additional providers +// currently use SDK credential resolution from their canonical environment. func convergeCloud(resources []omnisdk.ManagedResource) string { if len(resources) == 0 { return "" @@ -345,16 +366,92 @@ func blueprintColumns(blueprint omnisdk.Blueprint) []column { // blueprintSelectFunc refuses a select over a blueprint relation. A blueprint is // a way of building a resource set, not a thing to read: naming it in a converge // run is how it is applied. -func blueprintSelectFunc(resource string) (func() internaldto.ExecutorOutput, bool) { +func blueprintSelectFunc(resource string) func() internaldto.ExecutorOutput { if _, ok := blueprintFor(resource); !ok { return refuse(fmt.Errorf( "'%s.%s' has no relation '%s'; run SHOW RESOURCES IN %s.%s to list them", - ProviderName, iacService, resource, ProviderName, iacService)), true + ProviderName, iacService, resource, ProviderName, iacService)) } return refuse(fmt.Errorf( "'%s' renders a deployment rather than rows; "+ "apply it with SELECT ... FROM %s WHERE %s = '%s' AND %s = ''", relationAddress(iacService, resource), relationAddress(iacService, convergeRelation), - blueprintPredicate, resource, collectionPredicate)), true + blueprintPredicate, resource, collectionPredicate)) } + +// iacBindings keeps typed values separate from provider execution scope. +func iacBindings(params map[string]string) (map[string]any, map[string]any, error) { + var maps [2]map[string]any + for index, name := range []string{"inputs", "environment"} { + raw := popPredicate(params, name) + if raw == "" { + continue + } + decoder := json.NewDecoder(bytes.NewBufferString(raw)) + decoder.UseNumber() + if err := decoder.Decode(&maps[index]); err != nil { + return nil, nil, fmt.Errorf("%s must be a JSON object: %w", name, err) + } + if maps[index] == nil { + return nil, nil, fmt.Errorf("%s must be a JSON object", name) + } + var extra any + if err := decoder.Decode(&extra); err != io.EOF { + return nil, nil, fmt.Errorf("%s must contain one JSON object", name) + } + } + return maps[0], maps[1], nil +} + +// iacRows preserves diagnostic rows while turning unsuccessful outcomes into SQL errors. +type iacRows struct { + omnisdk.Rows + mu sync.Mutex + failure error +} + +func (r *iacRows) Next() bool { + if !r.Rows.Next() { + return false + } + row := r.Rows.Row() + status, _ := row["status"].(string) + failed := strings.Contains(strings.ToLower(status), "fail") || strings.Contains(strings.ToLower(status), "outstanding") + if value := row["error"]; value != nil && fmt.Sprint(value) != "" { + failed = true + } + if value := row["outstanding"]; value != nil && fmt.Sprint(value) != "[]" && fmt.Sprint(value) != "" { + failed = true + } + if failed { + r.mu.Lock() + r.failure = iacOutcomeError{key: fmt.Sprint(row["key"]), status: status} + r.mu.Unlock() + } + return true +} +func (r *iacRows) Err() error { + if err := r.Rows.Err(); err != nil { + return iacOutcomeError{cause: err} + } + r.mu.Lock() + defer r.mu.Unlock() + return r.failure +} + +type iacOutcomeError struct { + key, status string + cause error +} + +func (e iacOutcomeError) Error() string { + if e.cause != nil { + return e.cause.Error() + } + return fmt.Sprintf("IaC operation did not complete successfully for key %s (status %s); inspect the journal", + e.key, e.status) +} +func (iacOutcomeError) IaCFailure() bool { return true } + +func (e iacOutcomeError) Unwrap() error { return e.cause } diff --git a/internal/stackql/intrinsic/iac_test.go b/internal/stackql/intrinsic/iac_test.go index a03198146..e77f1f964 100644 --- a/internal/stackql/intrinsic/iac_test.go +++ b/internal/stackql/intrinsic/iac_test.go @@ -1,10 +1,13 @@ package intrinsic //nolint:testpackage // tests unexported iac state resolution import ( + "encoding/json" "errors" "os" "path/filepath" "testing" + + "github.com/stackql-labs/omnisdk/pkg/omnisdk" ) func TestResolveStateDirDefaultsToRoot(t *testing.T) { @@ -96,3 +99,68 @@ func TestInitReadsIaCStateRoot(t *testing.T) { t.Errorf("getIaCStateRoot() = %q, want %q", got, "/srv/stackql/iac") } } + +func TestIaCBindingsPreserveNumbersAndLiteralStrings(t *testing.T) { + params := map[string]string{ + "inputs": `{"large":9007199254740993,"literal":"${not_a_template}"}`, + "environment": `{"flag":true}`, "region": "east", + } + inputs, environment, err := iacBindings(params) + if err != nil { + t.Fatal(err) + } + if number, ok := inputs["large"].(json.Number); !ok || number.String() != "9007199254740993" { + t.Fatal(inputs) + } + if inputs["literal"] != "${not_a_template}" || environment["flag"] != true || len(params) != 1 { + t.Fatal("binding conversion changed values or scope") + } + for _, invalid := range []string{`null`, `[]`, `{} {}`, `{"x":`} { + if _, _, err := iacBindings(map[string]string{"inputs": invalid}); err == nil { + t.Fatalf("accepted %s", invalid) + } + } +} + +func TestIaCResourcesRetainSDKBindingsAndRejectRetiredFields(t *testing.T) { + resources, err := specResources(`[{"key":"one","provider":"demo","address":"service.items","write_once":{"name":{"from":"env","attribute":"NAME"}},"desired":{}}]`) + if err != nil { + t.Fatal(err) + } + names, err := omnisdk.ResourceEnvironmentVariables(resources) + if err != nil || len(names) != 1 || names[0] != "NAME" { + t.Fatalf("bindings lost: %v, %v", names, err) + } + for _, invalid := range []string{ + `[{"key":"one","params":{}}]`, `[{"key":"one","correlationParam":"tag"}]`, + `[{"key":"one"},{"key":"one"}]`, + } { + if _, err := specResources(invalid); err == nil { + t.Fatalf("accepted retired or duplicate specification: %s", invalid) + } + } +} + +func TestIaCReportedOutcomesAreApplicationFailures(t *testing.T) { + for _, row := range []omnisdk.Row{ + {"key": "one", "status": "failed", "error": "upstream failure"}, + {"key": "one", "status": "outstanding"}, + {"key": "one", "status": "applied", "outstanding": []string{"two"}}, + } { + cursor := &iacRows{Rows: &fakeRows{rows: []omnisdk.Row{row}}} + if !cursor.Next() || cursor.Row()["key"] != "one" { + t.Fatal("diagnostic row lost") + } + for cursor.Next() { + } + if cursor.Err() == nil { + t.Fatalf("unsuccessful outcome accepted: %v", row) + } + } + cursor := &iacRows{Rows: &fakeRows{rows: []omnisdk.Row{{"status": "applied"}}}} + for cursor.Next() { + } + if err := cursor.Err(); err != nil { + t.Fatal(err) + } +} diff --git a/internal/stackql/intrinsic/intrinsic.go b/internal/stackql/intrinsic/intrinsic.go index c66c748e6..872449cd7 100644 --- a/internal/stackql/intrinsic/intrinsic.go +++ b/internal/stackql/intrinsic/intrinsic.go @@ -65,7 +65,7 @@ func GeneratePrimitiveFunc( case *sqlparser.AuthRevoke: return authRevokeFunc(ctx, node), true case *sqlparser.Sleep: - return sleepFunc(node), true + return sleepFunc(ctx, node), true case *sqlparser.Use: return useFunc(ctx, node) case *sqlparser.Show: @@ -97,7 +97,14 @@ func GenerateStreamFunc( return execute, true } if node, isSelect := stmt.(*sqlparser.Select); isSelect { - return selectFunc(ctx, node, ctx.GetCurrentProvider()) + if action, handled := identitySelect(ctx, node); handled { + return action, true + } + action, handled := selectFunc(ctx, node, ctx.GetCurrentProvider()) + if !handled && ctx.GetRuntimeContext().AccessConfigFile != "" && backendOnlySelect(node) { + return statementReadFunc(ctx, node), true + } + return action, handled } if action, handled := storedInsertSelectFunc(ctx, stmt); handled { return action, true diff --git a/internal/stackql/intrinsic/legacy_read_cache.go b/internal/stackql/intrinsic/legacy_read_cache.go index fd9b82448..4233559d4 100644 --- a/internal/stackql/intrinsic/legacy_read_cache.go +++ b/internal/stackql/intrinsic/legacy_read_cache.go @@ -120,6 +120,9 @@ func (c *legacyNamespaceCache) requestEncoding(canonical string, key rowcache.Ke } func configureLegacyReadCache(ctx queryContext, q query.Unresolved, args *omnisdk.Args) { + if ctx.GetRuntimeContext().AccessConfigFile != "" { + return + } cache := legacyCache(ctx) if cache == nil { return diff --git a/internal/stackql/intrinsic/materialized.go b/internal/stackql/intrinsic/materialized.go index f83e21b76..6bba36f3f 100644 --- a/internal/stackql/intrinsic/materialized.go +++ b/internal/stackql/intrinsic/materialized.go @@ -1,7 +1,6 @@ package intrinsic import ( - "context" "errors" "fmt" "io" @@ -57,7 +56,7 @@ func materialize( if err != nil { return err } - bg := context.Background() + bg := executionContext(ctx) rows, err := open(bg, nil) if err != nil { return err diff --git a/internal/stackql/intrinsic/omnisdk.go b/internal/stackql/intrinsic/omnisdk.go index 7ff596180..3704b3455 100644 --- a/internal/stackql/intrinsic/omnisdk.go +++ b/internal/stackql/intrinsic/omnisdk.go @@ -1,7 +1,6 @@ package intrinsic import ( - "context" "encoding/json" "errors" "fmt" @@ -171,11 +170,14 @@ func openStream( InsecureSkipTLSVerify: input.getInsecureSkipTLSVerify() || ctx.GetRuntimeContext().AllowInsecure, Tuning: omnisdk.Tuning{RequestTimeout: apiRequestTimeout(ctx)}, } + if err = configureCaller(ctx, &args); err != nil { + return nil, err + } plan, err := omnisdk.Default().New(method.Path, args) if err != nil { return nil, err } - rows, err := plan.Open(context.Background()) + rows, err := plan.Open(executionContext(ctx)) if err != nil { return nil, err } @@ -277,6 +279,7 @@ func (rs *rowStream) startProducer() { rs.produced = make(chan omnisdk.Row) go func() { defer close(rs.produced) + defer rs.rows.Close() for rs.rows.Next() { rs.produced <- rs.rows.Row() } diff --git a/internal/stackql/intrinsic/preview_services.go b/internal/stackql/intrinsic/preview_services.go index 26e8520fd..2d6f49319 100644 --- a/internal/stackql/intrinsic/preview_services.go +++ b/internal/stackql/intrinsic/preview_services.go @@ -11,7 +11,6 @@ package intrinsic // the registry's curation behind them. import ( - "context" "fmt" "io" "net/http" @@ -132,10 +131,13 @@ func streamPlan( relation string, exprs sqlparser.SelectExprs, ) internaldto.ExecutorOutput { - rows, openErr := plan.Open(context.Background()) + rows, openErr := plan.Open(executionContext(ctx)) if openErr != nil { return internaldto.NewErroneousExecutorOutput(openErr) } + if relation == convergeRelation || relation == destroyRelation { + rows = &iacRows{Rows: rows} + } input := previewCfg stream := &rowStream{ rows: rows, @@ -189,6 +191,9 @@ func previewArgs(ctx queryContext, cloud string, params map[string]string) (omni PageLimit: ctx.GetRuntimeContext().HTTPPageLimit, }, } + if err = configureCaller(ctx, &args); err != nil { + return omnisdk.Args{}, err + } return args, nil } diff --git a/internal/stackql/intrinsic/preview_services_test.go b/internal/stackql/intrinsic/preview_services_test.go index 1ab480f7e..2fe3b0f35 100644 --- a/internal/stackql/intrinsic/preview_services_test.go +++ b/internal/stackql/intrinsic/preview_services_test.go @@ -184,10 +184,9 @@ func TestSpecResourcesMapsOntoSDK(t *testing.T) { "provider": "aws", "address": "ec2.vpcs", "desired": {"CidrBlock": "10.42.0.0/16"}, - "params": {"TagSpecification.1.ResourceType": "vpc"}, + "write_once": {"TagSpecification.1.ResourceType": "vpc"}, "identity": "line_items.VpcId", - "addressedBy": "VpcId", - "correlationParam": "TagSpecification.1.Tag.1.Value" + "addressedBy": "VpcId" }, { "key": "aws/ec2/subnet", "provider": "aws", @@ -211,9 +210,7 @@ func TestSpecResourcesMapsOntoSDK(t *testing.T) { if vpc.Identity() != "line_items.VpcId" || vpc.AddressedBy() != "VpcId" { t.Errorf("vpc identity = %q, addressedBy = %q", vpc.Identity(), vpc.AddressedBy()) } - if vpc.CorrelationParam() != "TagSpecification.1.Tag.1.Value" { - t.Errorf("vpc correlation = %q", vpc.CorrelationParam()) - } + // Desired is opaque: nothing between the predicate and the wire parses it, // so it must arrive as the bytes the caller wrote. if string(vpc.Desired()) != `{"CidrBlock": "10.42.0.0/16"}` { diff --git a/internal/stackql/intrinsic/relation_reads.go b/internal/stackql/intrinsic/relation_reads.go index 8d6ae224a..e1a7d28ce 100644 --- a/internal/stackql/intrinsic/relation_reads.go +++ b/internal/stackql/intrinsic/relation_reads.go @@ -191,7 +191,7 @@ func openStagedRelation(ctx queryContext, staged stagedSelect, runCtx context.Co return nil, err } rows, readErr := materializeStagedRelation(ctx, manager, id, staged, runCtx) - if releaseErr := manager.Release(context.Background(), id); releaseErr != nil && readErr == nil { + if releaseErr := manager.Release(context.WithoutCancel(runCtx), id); releaseErr != nil && readErr == nil { return nil, releaseErr } return rows, readErr diff --git a/internal/stackql/intrinsic/sleep.go b/internal/stackql/intrinsic/sleep.go index ce62b77d5..165f0d257 100644 --- a/internal/stackql/intrinsic/sleep.go +++ b/internal/stackql/intrinsic/sleep.go @@ -1,7 +1,6 @@ package intrinsic import ( - "context" "fmt" "time" @@ -11,7 +10,7 @@ import ( "github.com/stackql/stackql/internal/stackql/parserutil" ) -func sleepFunc(node *sqlparser.Sleep) func() internaldto.ExecutorOutput { +func sleepFunc(ctx queryContext, node *sqlparser.Sleep) func() internaldto.ExecutorOutput { return func() internaldto.ExecutorOutput { duration, err := parserutil.ExtractSleepDuration(node) if err != nil { @@ -21,7 +20,7 @@ func sleepFunc(node *sqlparser.Sleep) func() internaldto.ExecutorOutput { if err != nil { return internaldto.NewExecutorOutput(nil, nil, nil, nil, err) } - rows, err := planned.Open(context.Background()) + rows, err := planned.Open(executionContext(ctx)) if err != nil { return internaldto.NewExecutorOutput(nil, nil, nil, nil, err) } diff --git a/internal/stackql/intrinsic/staged.go b/internal/stackql/intrinsic/staged.go index a31729ea3..61e622211 100644 --- a/internal/stackql/intrinsic/staged.go +++ b/internal/stackql/intrinsic/staged.go @@ -60,7 +60,7 @@ func getStagingManager(ctx queryContext) (omnistaging.Manager, error) { if err != nil { return nil, err } - return omnistaging.NewManager(context.Background(), db, dialect) + return omnistaging.NewManager(executionContext(ctx), db, dialect) } // needsStaging reports whether a SELECT carries SQL omnisdk leaves unapplied. @@ -378,7 +378,7 @@ func rowCount(expr sqlparser.Expr) (int, error) { // it and releases the table. There is no result cache, so the table is // released as soon as the outer result has been read. func runStagedSelect(ctx queryContext, staged stagedSelect) internaldto.ExecutorOutput { - return runStagedSelectIn(ctx, staged, context.Background()) + return runStagedSelectIn(ctx, staged, executionContext(ctx)) } func runStagedSelectIn(ctx queryContext, staged stagedSelect, runCtx context.Context) internaldto.ExecutorOutput { @@ -391,7 +391,7 @@ func runStagedSelectIn(ctx queryContext, staged stagedSelect, runCtx context.Con return internaldto.NewErroneousExecutorOutput(err) } output, runErr := stageAndQueryIn(ctx, manager, id, staged, runCtx) - if releaseErr := manager.Release(context.Background(), id); releaseErr != nil && runErr == nil { + if releaseErr := manager.Release(context.WithoutCancel(runCtx), id); releaseErr != nil && runErr == nil { runErr = releaseErr } if runErr != nil { diff --git a/internal/stackql/intrinsic/statement_reads.go b/internal/stackql/intrinsic/statement_reads.go index ef3bba2a6..5bdafe220 100644 --- a/internal/stackql/intrinsic/statement_reads.go +++ b/internal/stackql/intrinsic/statement_reads.go @@ -96,7 +96,7 @@ func openUnionRead( if beginErr != nil { return nil, beginErr } - defer manager.Release(context.Background(), id) //nolint:errcheck // Preserve the query outcome. + defer manager.Release(context.WithoutCancel(runCtx), id) //nolint:errcheck // Preserve the query outcome. rows, openErr := open(runCtx, nil) if openErr != nil { return nil, openErr @@ -143,7 +143,7 @@ func statementReadFunc(ctx queryContext, stmt sqlparser.SelectStatement) func() return refuse(err) } return func() internaldto.ExecutorOutput { - rows, openErr := open(context.Background(), nil) + rows, openErr := open(executionContext(ctx), nil) if openErr != nil { return internaldto.NewErroneousExecutorOutput(openErr) } diff --git a/internal/stackql/intrinsic/stored_insert.go b/internal/stackql/intrinsic/stored_insert.go index 9b2d738bb..04524935f 100644 --- a/internal/stackql/intrinsic/stored_insert.go +++ b/internal/stackql/intrinsic/stored_insert.go @@ -1,7 +1,6 @@ package intrinsic import ( - "context" "errors" "fmt" "io" @@ -73,7 +72,7 @@ func storedInsertSelectFunc(ctx queryContext, stmt sqlparser.Statement) (func() } func runStoredInsert(ctx queryContext, staged stagedSelect, prefix string) internaldto.ExecutorOutput { - bg := context.Background() + bg := executionContext(ctx) manager, err := getStagingManager(ctx) if err != nil { return internaldto.NewErroneousExecutorOutput(err) diff --git a/internal/stackql/intrinsic/stored_relations.go b/internal/stackql/intrinsic/stored_relations.go index 201898163..003dbd59e 100644 --- a/internal/stackql/intrinsic/stored_relations.go +++ b/internal/stackql/intrinsic/stored_relations.go @@ -128,6 +128,9 @@ func validateDerivedDepth(table *sqlparser.AliasedTableExpr, maximum, depth int) } func (r *storedResolver) selectNode(node *sqlparser.Select, depth int) (*sqlparser.Select, error) { + if err := checkSQLNodes(r.ctx, node); err != nil { + return nil, err + } if depth >= derivedRelationMaxDepth { return nil, fmt.Errorf("stored relation expansion exceeds nesting bound") } @@ -273,6 +276,9 @@ func (r *storedResolver) named(table *sqlparser.AliasedTableExpr, name sqlparser func (r *storedResolver) register( table *sqlparser.AliasedTableExpr, name sqlparser.TableName, metadata internaldto.RelationDTO, ) error { + if err := checkHost(r.ctx, "host/backend", "read"); err != nil { + return err + } handleName := fmt.Sprintf("stored%d", len(r.relations)) handle := UnstablePrefix + "omni_relations.sources." + handleName columns := map[string]omnisdk.ColumnType{} diff --git a/internal/stackql/intrinsic/transaction.go b/internal/stackql/intrinsic/transaction.go index e331853d6..0a411d307 100644 --- a/internal/stackql/intrinsic/transaction.go +++ b/internal/stackql/intrinsic/transaction.go @@ -10,6 +10,7 @@ import ( "sync" "github.com/stackql-labs/omnisdk/pkg/omnisdk" + "github.com/stackql-labs/omnisdk/public/authorization" "github.com/stackql/stackql-parser/go/vt/sqlparser" "github.com/stackql/stackql/internal/stackql/acid/binlog" "github.com/stackql/stackql/internal/stackql/internal_data_transfer/internaldto" @@ -156,7 +157,7 @@ func executeDeclaredInverse( if e != nil { return internaldto.NewErroneousExecutorOutput(e) } - cursor, e := p.Open(context.Background()) + cursor, e := p.Open(executionContext(ctx)) if e != nil { return internaldto.NewErroneousExecutorOutput(e) } @@ -217,3 +218,13 @@ func mutationHTTPFailure(ctx queryContext, verb string, err error) error { ) return mutationPresentationError{cause: err, message: message} } + +func (c observedQueryContext) GetCaller() authorization.Session { + if host, ok := c.queryContext.(interface{ GetCaller() authorization.Session }); ok { + return host.GetCaller() + } + return nil +} +func (c observedQueryContext) GetExecutionContext() context.Context { + return executionContext(c.queryContext) +} diff --git a/internal/stackql/mcpbackend/credential_reload.go b/internal/stackql/mcpbackend/credential_reload.go index c1bb52012..baea120ae 100644 --- a/internal/stackql/mcpbackend/credential_reload.go +++ b/internal/stackql/mcpbackend/credential_reload.go @@ -8,6 +8,8 @@ import ( "sort" "strings" + "github.com/stackql-labs/omnisdk/public/authorization" + dto "github.com/stackql/stackql/internal/stackql/runtimeconfig" "github.com/stackql/stackql/internal/stackql/envfile" @@ -155,9 +157,19 @@ func (b *stackqlMCPService) providerCredentialRow(providerName, priorFingerprint // against the now-current environment. With no env file configured it // degrades to a pure status probe. func (b *stackqlMCPService) ReloadCredentials( - _ context.Context, + ctx context.Context, input mcp_dto.CredentialsReloadInput, ) (mcp_dto.CredentialsReloadDTO, error) { + if manager := b.handlerCtx.GetAccessManager(); manager != nil { + _, protected, cancel, err := manager.Establish(ctx) + if err != nil { + return mcp_dto.CredentialsReloadDTO{}, err + } + defer cancel() + if err = authorization.Check(protected, "host/credentials", "reload"); err != nil { + return mcp_dto.CredentialsReloadDTO{}, err + } + } rv := mcp_dto.CredentialsReloadDTO{ EnvFile: b.envFile, Providers: []mcp_dto.ProviderCredentialStatusDTO{}, diff --git a/internal/stackql/mcpbackend/mcp_service_stackql.go b/internal/stackql/mcpbackend/mcp_service_stackql.go index dea42902c..f46f27e91 100644 --- a/internal/stackql/mcpbackend/mcp_service_stackql.go +++ b/internal/stackql/mcpbackend/mcp_service_stackql.go @@ -322,8 +322,8 @@ func (b *stackqlMCPService) RunQueryJSON(ctx context.Context, input dto.QueryJSO return b.runPreprocessedQueryJSON(ctx, q, input.RowLimit) } -func (b *stackqlMCPService) runPreprocessedQueryJSON(_ context.Context, query string, rowLimit int) ([]map[string]interface{}, error) { - results, extractErr := b.extractQueryResults(query, rowLimit) +func (b *stackqlMCPService) runPreprocessedQueryJSON(ctx context.Context, query string, rowLimit int) ([]map[string]interface{}, error) { + results, extractErr := b.withContext(ctx).extractQueryResults(query, rowLimit) if extractErr != nil { return nil, extractErr } @@ -414,8 +414,8 @@ func classifyBackendError(err error, providerName string) error { // returns a different shape ({timestamp, rows_affected?, last_insert_id?}) // because it goes through database/sql Exec instead of the orchestrator. // Robot assertions that target both backends must rely only on `timestamp`. -func (b *stackqlMCPService) ExecQuery(_ context.Context, query string) (map[string]any, error) { - return b.execQuery(query) +func (b *stackqlMCPService) ExecQuery(ctx context.Context, query string) (map[string]any, error) { + return b.withContext(ctx).execQuery(query) } func (b *stackqlMCPService) ValidateQuery(ctx context.Context, query string) ([]map[string]any, error) { @@ -586,3 +586,10 @@ func (b *stackqlMCPService) PullProvider(ctx context.Context, input dto.Registry } return b.ExecQuery(ctx, q) } + +func (b *stackqlMCPService) withContext(ctx context.Context) *stackqlMCPService { + cloned := *b + cloned.handlerCtx = b.handlerCtx.Clone() + cloned.handlerCtx.SetExecutionContext(ctx) + return &cloned +} diff --git a/internal/stackql/planbuilder/entrypoint.go b/internal/stackql/planbuilder/entrypoint.go index ed2d1ce85..6878ac6b7 100644 --- a/internal/stackql/planbuilder/entrypoint.go +++ b/internal/stackql/planbuilder/entrypoint.go @@ -4,6 +4,7 @@ import ( "fmt" "github.com/stackql/stackql-parser/go/vt/sqlparser" + "github.com/stackql/stackql/internal/stackql/accessmanagement" "github.com/stackql/stackql/internal/stackql/acid/txn_context" "github.com/stackql/stackql/internal/stackql/astanalysis/earlyanalysis" "github.com/stackql/stackql/internal/stackql/handler" @@ -42,6 +43,16 @@ func (pb *standardPlanBuilder) BuildUndoPlanFromContext(_ handler.HandlerContext //nolint:funlen,gocognit,errcheck,gocyclo,cyclop // no big deal func (pb *standardPlanBuilder) BuildPlanFromContext(handlerCtx handler.HandlerContext) (plan.Plan, error) { + if handlerCtx.GetAccessManager() != nil { + statement, err := sqlparser.Parse(handlerCtx.GetQuery()) + if err != nil { + return nil, err + } + if err = accessmanagement.CheckStatement( + handlerCtx.GetExecutionContext(), handlerCtx.GetCaller(), statement); err != nil { + return nil, err + } + } tcc, err := internaldto.NewTxnControlCounters(handlerCtx.GetTxnCounterMgr()) handlerCtx.GetTxnStore().Put(tcc.GetTxnID()) defer handlerCtx.GetTxnStore().Del(tcc.GetTxnID()) @@ -50,7 +61,8 @@ func (pb *standardPlanBuilder) BuildPlanFromContext(handlerCtx handler.HandlerCo return nil, err } planKey := handlerCtx.GetQuery() - if qp, ok := handlerCtx.GetLRUCache().Get(planKey); ok && isPlanCacheEnabled() { + useCache := handlerCtx.GetAccessManager() == nil && isPlanCacheEnabled() + if qp, ok := handlerCtx.GetLRUCache().Get(planKey); ok && useCache { logging.GetLogger().Infoln("retrieving query plan from cache") pl, plOk := qp.(plan.Plan) if plOk { @@ -140,6 +152,9 @@ func (pb *standardPlanBuilder) BuildPlanFromContext(handlerCtx handler.HandlerCo return qPlan, nil } + if handlerCtx.GetAccessManager() != nil { + return nil, fmt.Errorf("caller access: legacy execution is not authorized") + } primitiveGenerator := primitivegenerator.NewRootPrimitiveGenerator( statement, handlerCtx, pGBuilder.getPlanGraphHolder()) diff --git a/internal/stackql/psqlwire/server.go b/internal/stackql/psqlwire/server.go index 209d2deb3..9e77ac7f3 100644 --- a/internal/stackql/psqlwire/server.go +++ b/internal/stackql/psqlwire/server.go @@ -30,6 +30,10 @@ type SimpleWireServer struct { //nolint:gocognit,nestif,nolintlint func MakeWireServer(sbe sqlbackend.SQLBackendFactory, cfg dto.RuntimeCtx) (IWireServer, error) { + if cfg.AccessConfigFile != "" { + return nil, fmt.Errorf("caller access: PostgreSQL wire transport cannot yet supply verified caller evidence; " + + "use local CLI or HTTPS MCP") + } logger := logging.GetLogger() var tlsCfg dto.PgTLSCfg diff --git a/internal/stackql/querysubmit/querysubmit.go b/internal/stackql/querysubmit/querysubmit.go index ee0ca752f..657a30b97 100644 --- a/internal/stackql/querysubmit/querysubmit.go +++ b/internal/stackql/querysubmit/querysubmit.go @@ -1,7 +1,11 @@ package querysubmit import ( + "sync" + + "github.com/stackql/psql-wire/pkg/sqldata" "github.com/stackql/stackql-parser/go/vt/sqlparser" + "github.com/stackql/stackql/internal/stackql/accessmanagement" "github.com/stackql/stackql/internal/stackql/acid/binlog" "github.com/stackql/stackql/internal/stackql/acid/txn_context" "github.com/stackql/stackql/internal/stackql/handler" @@ -39,6 +43,7 @@ type basicQuerySubmitter struct { undoHandlerCtx handler.HandlerContext handlerCtx handler.HandlerContext transactionContext txn_context.ITransactionContext + cancel func() } func (qs *basicQuerySubmitter) GetRedoLog() (binlog.LogEntry, bool) { @@ -81,11 +86,31 @@ func (qs *basicQuerySubmitter) WithTransactionContext( } func (qs *basicQuerySubmitter) PrepareQuery(handlerCtx handler.HandlerContext) error { + if manager := handlerCtx.GetAccessManager(); manager != nil { + caller, runCtx, cancel, err := manager.Establish(handlerCtx.GetExecutionContext()) + if err != nil { + return err + } + qs.cancel = cancel + handlerCtx.SetCaller(caller) + handlerCtx.SetExecutionContext(runCtx) + statement, err := sqlparser.Parse(handlerCtx.GetQuery()) + if err == nil { + err = accessmanagement.CheckStatement(runCtx, caller, statement) + } + if err != nil { + cancel() + return err + } + } qs.handlerCtx = handlerCtx logging.GetLogger().Debugln("PrepareQuery() invoked...") pb := planbuilder.NewPlanBuilder(qs.transactionContext) plan, err := pb.BuildPlanFromContext(handlerCtx) qs.queryPlan = plan + if err != nil && qs.cancel != nil { + qs.cancel() + } return err } @@ -96,7 +121,26 @@ func (qs *basicQuerySubmitter) SubmitQuery() internaldto.ExecutorOutput { qs.handlerCtx.GetOutfile(), qs.handlerCtx.GetOutErrFile(), ) - return qs.queryPlan.GetInstructions().GetPrimitiveGraph().Execute(pl) + result := qs.queryPlan.GetInstructions().GetPrimitiveGraph().Execute(pl) + // A cancelled graph must not be reported as a successful empty result. + if err := qs.handlerCtx.GetExecutionContext().Err(); err != nil && result.GetError() == nil { + if stream := result.GetSQLResult(); stream != nil { + _ = stream.Close() + } + if qs.cancel != nil { + qs.cancel() + } + return internaldto.NewErroneousExecutorOutput(err) + } + if qs.cancel != nil && qs.queryPlan.IsReadOnly() { + if stream := result.GetSQLResult(); stream != nil { + bounded := &callerStream{ISQLResultStream: stream, cancel: qs.cancel} + result.SetSQLResultFn(func() sqldata.ISQLResultStream { return bounded }) + } else { + qs.cancel() + } + } + return result } func (qs *basicQuerySubmitter) PrepareUndoQuery(handlerCtx handler.HandlerContext) error { @@ -117,3 +161,19 @@ func (qs *basicQuerySubmitter) UndoQuery() internaldto.ExecutorOutput { ) return qs.queryPlan.GetInstructions().GetInversePrimitiveGraph().Execute(pl) } + +// Read-only snapshots live through streaming, and release their deadline timer at EOF/close. +type callerStream struct { + sqldata.ISQLResultStream + cancel func() + once sync.Once +} + +func (s *callerStream) Read() (sqldata.ISQLResult, error) { + row, err := s.ISQLResultStream.Read() + if err != nil { + s.once.Do(s.cancel) + } + return row, err +} +func (s *callerStream) Close() error { s.once.Do(s.cancel); return s.ISQLResultStream.Close() } diff --git a/internal/stackql/runtimeconfig/runtime_ctx.go b/internal/stackql/runtimeconfig/runtime_ctx.go index 62cf117db..6b0f88fb1 100644 --- a/internal/stackql/runtimeconfig/runtime_ctx.go +++ b/internal/stackql/runtimeconfig/runtime_ctx.go @@ -3,6 +3,7 @@ package dto import "strconv" type RuntimeCtx struct { + AccessConfigFile string APIRequestTimeout int AuthRaw string CABundle string @@ -220,6 +221,7 @@ func (rc *RuntimeCtx) Set(key string, val string) error { func (rc RuntimeCtx) Copy() RuntimeCtx { return RuntimeCtx{ + AccessConfigFile: rc.AccessConfigFile, APIRequestTimeout: rc.APIRequestTimeout, AuthRaw: rc.AuthRaw, CABundle: rc.CABundle, diff --git a/internal/stackql/sqlengine/config.go b/internal/stackql/sqlengine/config.go index 5141c54c6..4ebe4c7ac 100644 --- a/internal/stackql/sqlengine/config.go +++ b/internal/stackql/sqlengine/config.go @@ -3,6 +3,7 @@ package sqlengine // Config is the host's resolved database connection configuration. Environment // variables are resolved by the caller before constructing it. type Config struct { + RequireForeignKeys bool `json:"-"` DBEngine string `json:"dbEngine"` DSN string `json:"dsn"` DbInitFilePath string `json:"dbInitFilepath"` diff --git a/internal/stackql/sqlengine/sqlite_embedded.go b/internal/stackql/sqlengine/sqlite_embedded.go index 55b024385..cd9077baf 100644 --- a/internal/stackql/sqlengine/sqlite_embedded.go +++ b/internal/stackql/sqlengine/sqlite_embedded.go @@ -77,7 +77,7 @@ func (se *sqLiteEmbeddedEngine) GetTx() (*sql.Tx, error) { func newSQLiteEmbeddedEngine( cfg Config, ) (*sqLiteEmbeddedEngine, error) { - dsn, expectedPragmas, err := buildDSN(cfg.DSN) + dsn, expectedPragmas, err := buildDSNConfig(cfg) eng := &sqLiteEmbeddedEngine{ dsn: dsn, ctrlMutex: &sync.Mutex{}, @@ -408,9 +408,11 @@ var pragmaNamePattern = regexp.MustCompile(`^[A-Za-z_][A-Za-z0-9_]*$`) // injects the 5000ms legacy busy_timeout default when unset. An empty dsn // selects the default shared in-memory database. The returned map records // every pragma the DSN sets, for the startup assertion. -// -//nolint:gocognit,funlen // Preserve the migrated DSN translation rules during backend relocation. -func buildDSN(dsn string) (string, map[string]string, error) { +func buildDSN(dsn string) (string, map[string]string, error) { return buildDSNConfig(Config{DSN: dsn}) } + +//nolint:gocognit,funlen // Preserve the central DSN translation rules and host-required pragmas. +func buildDSNConfig(cfg Config) (string, map[string]string, error) { + dsn := cfg.DSN if dsn == "" { dsn = sqliteDefaultDSN } @@ -484,6 +486,25 @@ func buildDSN(dsn string) (string, map[string]string, error) { directives = append(directives, fmt.Sprintf("%s(%s)", name, translatedPragmas[name])) } directives = append(directives, outParams["_pragma"]...) + if cfg.RequireForeignKeys { + // modernc sorts pragmas, so remove conflicting directives instead of relying on append order. + filtered := directives[:0] + for _, directive := range directives { + name, _, _ := strings.Cut(directive, "(") + name, _, _ = strings.Cut(name, "=") + if !strings.EqualFold(strings.TrimSpace(name), "foreign_keys") { + filtered = append(filtered, directive) + } + } + filtered = append(filtered, "foreign_keys(1)") + directives = filtered + for name := range expectedPragmas { + if strings.EqualFold(name, "foreign_keys") { + delete(expectedPragmas, name) + } + } + expectedPragmas["foreign_keys"] = "1" + } outParams["_pragma"] = directives return base + "?" + outParams.Encode(), expectedPragmas, nil } diff --git a/internal/stackql/sqlengine/sqlite_embedded_test.go b/internal/stackql/sqlengine/sqlite_embedded_test.go index 7ae7216d3..a5f4d96b9 100644 --- a/internal/stackql/sqlengine/sqlite_embedded_test.go +++ b/internal/stackql/sqlengine/sqlite_embedded_test.go @@ -1,6 +1,7 @@ package sqlengine //nolint:testpackage // Exercises private driver and DSN helpers. import ( + "context" "database/sql" "errors" "fmt" @@ -450,3 +451,41 @@ func TestSQLiteErrorPredicates(t *testing.T) { t.Fatal("non-driver errors must not classify as sqlite errors") } } + +func TestCallerDirectoryForeignKeysOnEveryConnection(t *testing.T) { + engine, err := newSQLiteEmbeddedEngine(Config{ + DSN: filepath.Join(t.TempDir(), "directory.db") + "?_foreign_keys=off", + RequireForeignKeys: true, + }) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = engine.db.Close() }) + engine.db.SetMaxOpenConns(2) + ctx := context.Background() + first, err := engine.db.Conn(ctx) + if err != nil { + t.Fatal(err) + } + defer first.Close() + second, err := engine.db.Conn(ctx) + if err != nil { + t.Fatal(err) + } + defer second.Close() + for _, conn := range []*sql.Conn{first, second} { + var enabled int + if err = conn.QueryRowContext(ctx, "PRAGMA foreign_keys").Scan(&enabled); err != nil || enabled != 1 { + t.Fatalf("foreign keys not enforced on pooled connection: %d, %v", enabled, err) + } + } + if _, err = first.ExecContext(ctx, "CREATE TABLE principals (id TEXT PRIMARY KEY)"); err != nil { + t.Fatal(err) + } + if _, err = first.ExecContext(ctx, "CREATE TABLE memberships (principal TEXT REFERENCES principals(id))"); err != nil { + t.Fatal(err) + } + if _, err = second.ExecContext(ctx, "INSERT INTO memberships VALUES ('unknown')"); err == nil { + t.Fatal("second connection accepted an orphan membership") + } +} diff --git a/pkg/mcp_server/config.go b/pkg/mcp_server/config.go index df11c9f0c..d8ecd1ad5 100644 --- a/pkg/mcp_server/config.go +++ b/pkg/mcp_server/config.go @@ -2,9 +2,12 @@ package mcp_server //nolint:revive,stylecheck,mnd // fine for now import ( "bytes" + "context" + "crypto/tls" "encoding/json" "errors" "fmt" + "net/http" "slices" "strings" "time" @@ -29,6 +32,10 @@ const ( // Config represents the complete configuration for the MCP server. type Config struct { + // TLSConfig is supplied by the host for verified client certificate authentication. + TLSConfig *tls.Config `json:"-" yaml:"-"` + // AuthenticateRequest is a host-supplied transport boundary, never deserialized. + AuthenticateRequest func(*http.Request) (context.Context, error) `json:"-" yaml:"-"` // Server contains server-specific configuration. Server ServerConfig `json:"server" yaml:"server"` diff --git a/pkg/mcp_server/server.go b/pkg/mcp_server/server.go index e0af69c81..c6d8a2f46 100644 --- a/pkg/mcp_server/server.go +++ b/pkg/mcp_server/server.go @@ -79,7 +79,8 @@ func (s *simpleMCPServer) runHTTPServer(server *mcp.Server, config *Config) erro //nolint:gosec // TODO: find viable alternative to http.ListenAndServe if config.Server.TLSCertFile != "" && config.Server.TLSKeyFile != "" { s.logger.Infof("Starting HTTPS server on %s", address) - if err := http.ListenAndServeTLS(address, config.Server.TLSCertFile, config.Server.TLSKeyFile, handlerWithLogging); err != nil { + https := &http.Server{Addr: address, Handler: handlerWithLogging, TLSConfig: config.TLSConfig, ReadHeaderTimeout: 10 * time.Second} + if err := https.ListenAndServeTLS(config.Server.TLSCertFile, config.Server.TLSKeyFile); err != nil { s.logger.Errorf("HTTPS Server failed: %v", err) return err } @@ -103,6 +104,17 @@ func secureHTTPHandler(handler http.Handler, config *Config) (http.Handler, erro rejectRequest(w, r, http.StatusForbidden, "cross-origin request rejected") })) handler = protection.Handler(handler) + if config.AuthenticateRequest != nil { + next := handler + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + ctx, err := config.AuthenticateRequest(r) + if err != nil { + rejectRequest(w, r, http.StatusUnauthorized, "caller authentication failed") + return + } + next.ServeHTTP(w, r.WithContext(ctx)) + }), nil + } envVar := config.Server.AuthTokenEnvVar if envVar == "" { if !config.Server.AllowUnauthenticated { diff --git a/test/python/stackql_test_tooling/StackQLInterfaces.py b/test/python/stackql_test_tooling/StackQLInterfaces.py index 69967b525..d85f1d482 100644 --- a/test/python/stackql_test_tooling/StackQLInterfaces.py +++ b/test/python/stackql_test_tooling/StackQLInterfaces.py @@ -1436,11 +1436,7 @@ def should_stackql_exec_inline_equal_stderr( *args, **cfg ) - se = result.stderr - if self._execution_platform == 'docker': - se_split = se.split('\n') - if len(se_split) > 1: - se = se_split[-1] + se = self.application_stderr(result.stderr) return self.should_be_equal(se, expected_output, collapse_spaces=True, formatter='ascii', strip_spaces=True) @@ -1471,11 +1467,7 @@ def should_stackql_exec_inline_contain_stderr( *args, **cfg ) - se = result.stderr - if self._execution_platform == 'docker': - se_split = se.split('\n') - if len(se_split) > 1: - se = se_split[-1] + se = self.application_stderr(result.stderr) return self.should_contain(se, expected_output, collapse_spaces=True, strip_spaces=True) diff --git a/test/python/stackql_test_tooling/access_management_keywords.py b/test/python/stackql_test_tooling/access_management_keywords.py new file mode 100644 index 000000000..f0c06c6de --- /dev/null +++ b/test/python/stackql_test_tooling/access_management_keywords.py @@ -0,0 +1,740 @@ +"""Black-box caller access fixtures. Tokens stay in subprocess environments, never Robot results.""" +import base64 +import hashlib +import secrets +import csv +import yaml +import concurrent.futures +import json +import datetime +import ipaddress +import socket +import shutil +import uuid +from urllib.parse import urlsplit, urlunsplit, parse_qsl, urlencode +import ssl +import urllib.request +import urllib.error +from cryptography import x509 +from cryptography.x509.oid import NameOID +import os +import pathlib +import sqlite3 +import subprocess +import tempfile +import threading +import time +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer + +from cryptography.hazmat.primitives import serialization +from cryptography.hazmat.primitives.asymmetric import ed25519, rsa, ec +from cryptography.hazmat.primitives import hashes +from cryptography.hazmat.primitives.asymmetric import padding +from cryptography.hazmat.primitives.asymmetric.utils import decode_dss_signature + + +class _DockerSQLite: + """Commit fixture writes in the same kernel as the application's SQLite readers. + + Host SQLite locks and page-cache invalidation do not cross a Docker Desktop VM. + The sidecar uses only Python's standard library; the tested image needs no Python. + """ + + def __init__(self, path): + self._path = pathlib.Path(path) + self._pending = [] + + def execute(self, statement): + self._pending.append(statement) + + def commit(self): + if not self._pending: + return + program = ( + "import json, sqlite3, sys\n" + "with sqlite3.connect(sys.argv[1]) as db:\n" + " for statement in json.load(sys.stdin):\n" + " db.execute(statement)\n" + ) + subprocess.run([ + "docker", "run", "--rm", "--interactive", "--network", "none", + "--user", f"{os.getuid()}:{os.getgid()}", + "--volume", str(self._path.parent) + ":" + str(self._path.parent), + "--entrypoint", "python3", "python:3.13-slim-trixie", + "-c", program, str(self._path), + ], input=json.dumps(self._pending), capture_output=True, text=True, check=True, timeout=120) + self._pending.clear() + + def close(self): + self._pending.clear() + + +class AccessManagementKeywords: + ROBOT_LIBRARY_SCOPE = "SUITE" + + def verify_caller_access(self, executable, backend="{}", execution_platform="native", scenario="iac"): + with tempfile.TemporaryDirectory(prefix="stackql-access-") as directory: + root = pathlib.Path(directory) + docker = execution_platform == "docker" + cfg = json.loads(backend) + postgres = cfg.get("dbEngine") == "postgres_tcp" + directory_schema = None + if postgres: + import psycopg + dsn = cfg["dsn"] + if docker: + subprocess.run(["docker", "compose", "-p", "execrun", "up", "-d", "postgres_stackql"], capture_output=True, check=True, timeout=120) + result = subprocess.run(["docker", "compose", "-p", "execrun", "port", "postgres_stackql", "5432"], capture_output=True, text=True, check=True, timeout=30) + published = result.stdout.strip().splitlines()[0].rsplit(":", 1)[1] + url = urlsplit(dsn) + credentials = url.netloc.rsplit("@", 1)[0] + dsn = urlunsplit((url.scheme, credentials + "@127.0.0.1:" + published, url.path, url.query, url.fragment)) + for attempt in range(50): + try: + db = psycopg.connect(dsn, autocommit=True) + break + except psycopg.OperationalError: + if not docker or attempt == 49: + raise + time.sleep(0.2) + directory_schema = "stackql_access_test_" + uuid.uuid4().hex + db.execute(psycopg.sql.SQL("CREATE SCHEMA {}").format(psycopg.sql.Identifier(directory_schema))) + db.execute(psycopg.sql.SQL("SET search_path TO {}, public").format(psycopg.sql.Identifier(directory_schema))) + url = urlsplit(cfg["dsn"]) + options = dict(parse_qsl(url.query)) + options["options"] = "-csearch_path=" + directory_schema + ",public" + cfg["dsn"] = urlunsplit((url.scheme, url.netloc, url.path, urlencode(options), url.fragment)) + else: + cfg = {"dsn": str(root / "access.db")} + db = _DockerSQLite(cfg["dsn"]) if docker else sqlite3.connect(cfg["dsn"], check_same_thread=False) + key = ed25519.Ed25519PrivateKey.generate() + public = key.public_key().public_bytes(serialization.Encoding.PEM, serialization.PublicFormat.SubjectPublicKeyInfo).decode() + config = {"issuer": "access-test", "audience": "stackql", "keys": [{"kid": "local", "algorithm": "EdDSA", "public_key_pem": public}], "rules": [ + {"subject": "group:readers", "resource": "operation/accessdemo/list/listItems", "action": "execute"}, + {"subject": "group:readers", "resource": "credential/accessdemo", "action": "use"}, + {"subject": "group:operators", "resource": "**", "action": "*"}, + ]} + config_file = root / "access.json" + config_file.write_text(json.dumps(config)) + ddl = [ + "CREATE TABLE IF NOT EXISTS stackql_access_principals (id TEXT PRIMARY KEY, attributes TEXT NOT NULL DEFAULT '{}')", + "CREATE TABLE IF NOT EXISTS stackql_access_groups (id TEXT PRIMARY KEY)", + "CREATE TABLE IF NOT EXISTS stackql_access_identities (issuer TEXT NOT NULL, subject TEXT NOT NULL, principal_id TEXT NOT NULL REFERENCES stackql_access_principals(id), PRIMARY KEY (issuer,subject))", + "CREATE TABLE IF NOT EXISTS stackql_access_memberships (principal_id TEXT NOT NULL REFERENCES stackql_access_principals(id), group_id TEXT NOT NULL REFERENCES stackql_access_groups(id), PRIMARY KEY (principal_id,group_id))", + ] + for statement in ddl: + db.execute(statement) + for table in ["memberships", "identities", "principals", "groups"]: + db.execute("DELETE FROM stackql_access_" + table) + for statement in [ + "INSERT INTO stackql_access_principals VALUES ('reader','{}'),('operator','{}'),('denied','{}')", + "INSERT INTO stackql_access_groups VALUES ('readers'),('operators')", + "INSERT INTO stackql_access_identities VALUES ('access-test','reader','reader'),('access-test','operator','operator'),('access-test','denied','denied')", + "INSERT INTO stackql_access_memberships VALUES ('reader','readers'),('operator','operators')", + ]: + db.execute(statement) + if not postgres: + db.commit() + counts = {"read": 0, "mutation": 0} + managed = {} + + class API(BaseHTTPRequestHandler): + def log_message(self, *_): + pass + + def do_GET(self): + if scenario.startswith("query_deadline"): + time.sleep(2) + counts["read"] += 1 + self.send_response(200) + self.send_header("Content-Type", "application/json") + self.end_headers() + identity = self.path.rsplit("/", 1)[-1] + self.wfile.write(json.dumps(managed.get(identity, {}) if self.path.startswith("/items/") else [{"id":"known-item"}]).encode()) + + def do_POST(self): + counts["mutation"] += 1 + body = json.loads(self.rfile.read(int(self.headers.get("Content-Length", "0"))) or b"{}") + if body.get("name") == "upstream-reject": + self.send_response(400) + self.send_header("Content-Type", "application/json") + self.end_headers() + self.wfile.write(b'{"error":"test rejection"}') + return + self.send_response(200) + self.send_header("Content-Type", "application/json") + self.end_headers() + identity = "managed-" + str(counts["mutation"]) + managed[identity] = dict(body, id=identity) + self.wfile.write(json.dumps(managed[identity]).encode()) + + def do_PATCH(self): + counts["mutation"] += 1 + identity = self.path.rsplit("/", 1)[-1] + body = json.loads(self.rfile.read(int(self.headers.get("Content-Length", "0"))) or b"{}") + managed[identity].update(body) + self.send_response(200) + self.send_header("Content-Type", "application/json") + self.end_headers() + self.wfile.write(json.dumps(managed[identity]).encode()) + + def do_DELETE(self): + counts["mutation"] += 1 + managed.pop(self.path.rsplit("/", 1)[-1], None) + self.send_response(204) + self.end_headers() + + server = ThreadingHTTPServer(("0.0.0.0" if docker else "127.0.0.1", 0), API) + thread = threading.Thread(target=server.serve_forever, daemon=True) + thread.start() + doc_dir = root / "src" / "accessdemo" / "v0.0.1" + doc_dir.mkdir(parents=True) + (doc_dir / "provider.yaml").write_text("name: accessdemo\nproviderServices:\n service:\n service: {$ref: service.yaml}\n") + (doc_dir / "service.yaml").write_text("""openapi: 3.0.0 +servers: [{url: 'http://127.0.0.1:PORT'}] +paths: + /items: + get: + operationId: listItems + responses: + '200': + content: + application/json: + schema: {type: array, items: {type: object, properties: {id: {type: string}}}} + post: + operationId: createItem + responses: + '200': + content: + application/json: + schema: {type: object, properties: {id: {type: string}}} +components: + x-stackQL-resources: + items: + methods: + list: + operation: {$ref: '#/paths/~1items/get'} + response: {mediaType: application/json} + create: + operation: {$ref: '#/paths/~1items/post'} + response: {mediaType: application/json} + sqlVerbs: + select: [{$ref: '#/components/x-stackQL-resources/items/methods/list'}] + insert: [{$ref: '#/components/x-stackQL-resources/items/methods/create'}] +""".replace("PORT", str(server.server_port))) + + service_file = doc_dir / "service.yaml" + service = yaml.safe_load(service_file.read_text()) + if docker: + service["servers"][0]["url"] = service["servers"][0]["url"].replace("127.0.0.1", "host.docker.internal") + schema = {"type": "object", "properties": {"id": {"type": "string"}, "name": {"type": "string"}}} + response = {"200": {"content": {"application/json": {"schema": schema}}}} + body = {"required": True, "content": {"application/json": {"schema": schema}}} + service["paths"]["/items"]["post"]["requestBody"] = body + service["paths"]["/items/{id}"] = { + "get": {"operationId": "getItem", "parameters": [{"name": "id", "in": "path", "required": True, "schema": {"type": "string"}}], "responses": response}, + "patch": {"operationId": "patchItem", "parameters": [{"name": "id", "in": "path", "required": True, "schema": {"type": "string"}}], "requestBody": body, "responses": response}, + "delete": {"operationId": "deleteItem", "parameters": [{"name": "id", "in": "path", "required": True, "schema": {"type": "string"}}], "responses": {"204": {"description": "deleted"}}}, + } + resource = service["components"]["x-stackQL-resources"]["items"] + for name, verb in [("get", "get"), ("update", "patch"), ("delete", "delete")]: + resource["methods"][name] = {"operation": {"$ref": "#/paths/~1items~1{id}/" + verb}, "response": {"mediaType": "application/json"}} + resource["sqlVerbs"]["select"].append({"$ref": "#/components/x-stackQL-resources/items/methods/get"}) + resource["sqlVerbs"]["update"] = [{"$ref": "#/components/x-stackQL-resources/items/methods/update"}] + resource["sqlVerbs"]["delete"] = [{"$ref": "#/components/x-stackQL-resources/items/methods/delete"}] + service_file.write_text(yaml.safe_dump(service)) + + def token(subject="reader", updates=None, remove=(), signer=None, kid="local"): + claims = {"iss": "access-test", "sub": subject, "aud": "stackql", "iat": int(time.time()), "exp": int(time.time()) + 300, "groups": ["operators"]} + claims.update(updates or {}) + for field in remove: + claims.pop(field, None) + encode = lambda value: base64.urlsafe_b64encode(json.dumps(value, separators=(",", ":")).encode()).rstrip(b"=") + message = encode({"alg": "EdDSA", "kid": kid}) + b"." + encode(claims) + signature = (signer or key).sign(message) + return (message + b"." + base64.urlsafe_b64encode(signature).rstrip(b"=")).decode() + + prefix = (["docker", "compose", "-p", "execrun", "run", "--rm", "--no-deps", "-T", "--user", f"{os.getuid()}:{os.getgid()}", "--workdir", str(root), "-v", str(root) + ":" + str(root), "-e", "STACKQL_CALLER_TOKEN", "--entrypoint", "stackql", "stackqlsrv"] + if docker else [shutil.which(executable) or str(pathlib.Path(executable).resolve())]) + base = prefix + ["--approot", str(root), "--sqlBackend", json.dumps(cfg), "--registry", json.dumps({"localDocRoot": str(root), "verifyConfig": {"nopVerify": True}}), "--preview", '{"unstable":true}', "--output", "json"] + + def run(sql, credential=None, protected=True): + env = os.environ.copy() + env.pop("STACKQL_CALLER_TOKEN", None) + if credential is not None: + env["STACKQL_CALLER_TOKEN"] = credential + command = base + (["--access.config", json.dumps({"path": str(config_file)})] if protected else []) + ["exec", sql] + result = subprocess.run(command, capture_output=True, text=True, env=env, timeout=45) + # Separate CLI processes initialize the shared PostgreSQL host catalog. + # Retry only this fixture's known read when concurrent initialization deadlocks. + for _ in range(2): + if sql != "SELECT id FROM stackql_unstable_accessdemo.service.items;" or "40P01" not in result.stdout + result.stderr: + break + result = subprocess.run(command, capture_output=True, text=True, env=env, timeout=45) + output = result.stdout + result.stderr + if credential and credential in output: + raise AssertionError("caller token leaked into output") + return result.returncode, output + + def success(sql, credential, expected): + code, output = run(sql, credential) + assert code == 0 and expected in output, "authorized query failed: " + output + + def denied(sql, credential=None, concurrent=False): + before = counts.copy() + code, output = run(sql, credential) + assert code != 0, "protected query unexpectedly succeeded: " + sql + " output: " + output + assert counts["mutation"] == before["mutation"] and (concurrent or counts["read"] == before["read"]), "denied query reached provider: " + sql + + try: + query = "SELECT id FROM stackql_unstable_accessdemo.service.items;" + reader = token() + invalid = { + "missing": None, + "signature": token(signer=ed25519.Ed25519PrivateKey.generate()), + "issuer": token(updates={"iss": "other"}), + "audience": token(updates={"aud": "other"}), + "expiry_required": token(remove=["exp"]), + "expired": token(updates={"exp": int(time.time()) - 5}), + "not_before": token(updates={"nbf": int(time.time()) + 600}), + "issued_at": token(updates={"iat": int(time.time()) + 600}), + "key_id": token(kid="unknown"), + "identity": token(subject="unknown"), + } + if scenario.startswith("bearer"): + tokens = {name: secrets.token_urlsafe(32) for name in ["reader", "operator", "denied"]} + config.pop("keys") + config.pop("audience") + config.update(type="bearer", tokens=[{"subject": name, "sha256": hashlib.sha256(value.encode()).hexdigest()} for name, value in tokens.items()]) + config_file.write_text(json.dumps(config)) + token = lambda subject="reader": tokens[subject] + reader = token() + invalid = {"missing": None, "wrong": secrets.token_urlsafe(32)} + if scenario.startswith("local_os"): + config = {"type": "local_os", "issuer": "access-test", "rules": config["rules"]} + config_file.write_text(json.dumps(config)) + if os.name == "nt": + identity = next(csv.reader([subprocess.check_output(["whoami", "/user", "/fo", "csv", "/nh"], text=True).strip()]))[1] + else: + identity = str(os.geteuid()) + db.execute("UPDATE stackql_access_identities SET subject = '" + identity.replace("'", "''") + "' WHERE principal_id = 'reader'") + if not postgres: + db.commit() + if scenario == "local_os_http_refused": + result = subprocess.run(base + ["--access.config", json.dumps({"path": str(config_file)}), "--mcp.config", json.dumps({"server": {"transport": "http", "tls_cert_file": "unused.crt", "tls_key_file": "unused.key"}}), "mcp"], capture_output=True, text=True, timeout=30) + assert "local OS authentication is unavailable over HTTP" in result.stdout + result.stderr + assert counts == {"read": 0, "mutation": 0} + return + if scenario.endswith("shell") or scenario.endswith("stdio"): + self._verify_local_transport(base, config_file, None, query, scenario.rsplit("_", 1)[-1]) + return + success("SELECT whoami() AS principal, groups() AS memberships;", None, "readers") + success(query, None, "known-item") + success("SELECT whoami() AS principal;", "untrusted-environment-token", "reader") + denied("DELETE FROM stackql_unstable_accessdemo.service.items;") + db.execute("DELETE FROM stackql_access_identities WHERE principal_id = 'reader'") + if not postgres: + db.commit() + denied(query) + return + if scenario in ("mtls_https", "mtls_https_dns"): + identity_kind = "dns_san" if scenario.endswith("_dns") else "uri_san" + roots, clients = self._client_certificates(root, identity_kind) + config = {"type": "mtls", "issuer": "access-test", "client_ca_pem": roots, "certificate_identity": identity_kind, "rules": config["rules"]} + config_file.write_text(json.dumps(config)) + for name in ["reader", "operator", "denied"]: + subject = name + ".example" if identity_kind == "dns_san" else "spiffe://access-test/" + name + db.execute("UPDATE stackql_access_identities SET subject = '" + subject + "' WHERE principal_id = '" + name + "'") + if not postgres: + db.commit() + self._verify_https(base, config_file, root, key, token, [], query, db, postgres, counts, clients) + denied(query, reader) + return + if scenario.startswith("query_deadline"): + config["max_session_seconds"] = 1 + config["enforce_session_deadline"] = scenario == "query_deadline_opt_in" + config_file.write_text(json.dumps(config)) + code, output = run(query, reader) + if config["enforce_session_deadline"]: + assert code != 0 and "deadline" in output.lower(), "opt-in deadline did not cancel query (rc=" + str(code) + "): " + output + else: + assert code == 0 and "known-item" in output, "default deadline interrupted query: " + output + return + if scenario == "config": + self._verify_configuration(base, config_file, config, reader, query, counts) + return + if scenario.startswith("jwt_invalid:"): + denied(query, invalid[scenario.split(":", 1)[1]]) + return + if scenario == "jwt_algorithms": + self._verify_algorithms(config, config_file, token, run, query, counts) + return + if scenario.endswith("shell") or scenario.endswith("stdio"): + self._verify_local_transport(base, config_file, reader, query, scenario.rsplit("_", 1)[-1]) + self._verify_local_transport(base, config_file, None, query, scenario.rsplit("_", 1)[-1], allowed=False) + return + if scenario.endswith("https"): + self._verify_https(base, config_file, root, key, token, list(invalid.values()), query, db, postgres, counts) + return + if scenario in ("jwt_cli", "bearer_cli"): + success("SELECT whoami() AS principal, groups() AS memberships;", reader, "readers") + success(query, reader, "known-item") + success("SELECT 1 AS ordinary_integer;", reader, "ordinary_integer") + for credential in invalid.values(): + denied(query, credential) + denied(query, token(subject="denied")) + return + if scenario == "compatibility": + code, output = run(query, protected=False) + assert code == 0 and "known-item" in output, "unprotected compatibility failed: " + output + return + if scenario == "policy": + for sql in ["DELETE FROM stackql_unstable_accessdemo.service.items;", "INSERT INTO stackql_unstable_accessdemo.service.items (id) VALUES ('new');", "EXEC stackql_unstable_accessdemo.service.items.create;", "SELECT * FROM stackql_preview.iac.run WHERE collection = 'test' AND resources = '[]';", "SELECT * FROM stackql_preview.iac.destroy WHERE collection = 'test';", "SELECT * FROM stackql_access_principals;", "SELECT pg_read_file('/etc/passwd');"]: + denied(sql, reader) + denied(query, token(subject="denied")) + with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor: + futures = [executor.submit(success, query, reader, "known-item"), executor.submit(denied, "SELECT 1", token(subject="denied"), True)] + for future in futures: + future.result() + return + if scenario == "revocation": + db.execute("DELETE FROM stackql_access_memberships WHERE principal_id = 'reader'") + if not postgres: + db.commit() + denied(query, reader) + db.execute("UPDATE stackql_access_principals SET attributes = 'null' WHERE id = 'operator'") + if not postgres: + db.commit() + denied(query, token(subject="operator")) + assert counts["mutation"] == 0, "reader performed provider mutation" + db.execute("UPDATE stackql_access_principals SET attributes = '{}' WHERE id = 'operator'") + if not postgres: + db.commit() + return + assert scenario == "iac", "unknown access scenario: " + scenario + operator = token(subject="operator") + def converge(label, keys): + resources = [{"key": name, "provider": "accessdemo", "address": "service.items", "identity": "id", "addressed_by": "id", "desired": {"name": {"from": "input", "attribute": "label"}}} for name in keys] + spec = json.dumps(resources).replace("'", "''") + inputs = json.dumps({"label": label}) + sql = "SELECT * FROM stackql_preview.iac.run WHERE collection = 'acceptance' AND state = 'audit' AND run_id = '" + label + "' AND resources = '" + spec + "' AND inputs = '" + inputs + "';" + success(sql, operator, "applied") + assert all(item.get("name") == label for item in managed.values()), "IaC reported applied without converging desired values" + converge("first", ["one", "two"]) + assert len(managed) == 2, "IaC create did not retain both resources" + db.execute("INSERT INTO stackql_access_principals VALUES ('operator2','{}')") + db.execute("INSERT INTO stackql_access_identities VALUES ('access-test','operator2','operator2')") + db.execute("INSERT INTO stackql_access_memberships VALUES ('operator2','operators')") + if not postgres: + db.commit() + denied("SELECT * FROM stackql_preview.iac.destroy WHERE collection = 'acceptance' AND state = 'audit' AND keys = '[\"one\"]';", token(subject="operator2")) + denied("SELECT * FROM stackql_preview.iac.destroy WHERE collection = 'acceptance' AND state = '../../escape';", operator) + converge("second", ["one", "two"]) + converge("third", ["one", "two"]) + # Reusing the original create ID must not unwind its old creates after an update. + snapshot = json.loads(json.dumps(managed)) + before = counts.copy() + replay_spec = json.dumps([{"key": name, "provider": "accessdemo", "address": "service.items", "identity": "id", "addressed_by": "id", "desired": {"name": "upstream-reject"}} for name in ["one", "two"]]) + replay = "SELECT * FROM stackql_preview.iac.run WHERE collection = 'acceptance' AND state = 'audit' AND run_id = 'first' AND resources = '" + replay_spec + "';" + code, output = run(replay, operator) + assert code != 0 and "already exists" in output, "reused run ID accepted: " + output + assert managed == snapshot and counts == before, "reused run changed existing resources" + # resources_file is the user-facing path; omitted run IDs must remain safe on repeat. + resource_file = root / "resources.json" + resource_file.write_text(json.dumps({"resources": [{"key": name, "provider": "accessdemo", "address": "service.items", "identity": "id", "addressed_by": "id", "desired": {"name": "third"}} for name in ["one", "two"]]})) + file_sql = "SELECT * FROM stackql_preview.iac.run WHERE collection = 'acceptance' AND state = 'audit' AND resources_file = '" + str(resource_file).replace("'", "''") + "';" + for _ in range(2): + success(file_sql, operator, "applied") + assert managed == snapshot, "unchanged file rerun changed resources" + denied("SELECT * FROM stackql_preview.iac.destroy WHERE collection = 'acceptance' AND state = 'audit' AND keys = '[]';", operator) + success("SELECT * FROM stackql_preview.iac.destroy WHERE collection = 'acceptance' AND state = 'audit' AND run_id = 'prune' AND keys = '[\"two\"]';", operator, "destroyed") + assert len(managed) == 1, "partial delete changed the retained resource set" + converge("fourth", ["one"]) + success("SELECT * FROM stackql_preview.iac.destroy WHERE collection = 'acceptance' AND state = 'audit' AND run_id = 'teardown';", operator, "destroyed") + assert not managed, "IaC teardown left managed resources" + spec = json.dumps([{"key": "reject", "provider": "accessdemo", "address": "service.items", "identity": "id", "addressed_by": "id", "desired": {"name": "upstream-reject"}}]) + sql = "SELECT * FROM stackql_preview.iac.run WHERE collection = 'rejection' AND run_id = 'reject' AND resources = '" + spec + "';" + code, output = run(sql, protected=False) + assert code != 0 and "failed" in output.lower(), "unsuccessful IaC returned success: " + output + assert not managed, "failed create unexpectedly retained resources" + + finally: + server.shutdown() + server.server_close() + if directory_schema: + db.execute(psycopg.sql.SQL("DROP SCHEMA {} CASCADE").format(psycopg.sql.Identifier(directory_schema))) + db.close() + + def _verify_https(self, base, config_file, root, key, token, invalid, query, db, postgres, counts, clients=None): + subject = x509.Name([x509.NameAttribute(NameOID.COMMON_NAME, "localhost")]) + now = datetime.datetime.now(datetime.timezone.utc) + certificate = (x509.CertificateBuilder().subject_name(subject).issuer_name(subject) + .public_key(key.public_key()).serial_number(x509.random_serial_number()) + .not_valid_before(now - datetime.timedelta(minutes=1)) + .not_valid_after(now + datetime.timedelta(hours=1)) + .add_extension(x509.SubjectAlternativeName([x509.IPAddress(ipaddress.ip_address("127.0.0.1"))]), False) + .sign(key, None)) + cert_path, key_path = root / "server.crt", root / "server.key" + cert_path.write_bytes(certificate.public_bytes(serialization.Encoding.PEM)) + key_path.write_bytes(key.private_bytes(serialization.Encoding.PEM, serialization.PrivateFormat.PKCS8, serialization.NoEncryption())) + key_path.chmod(0o600) + with socket.socket() as sock: + sock.bind(("127.0.0.1", 0)) + port = sock.getsockname()[1] + config = {"server": {"transport": "http", "address": "0.0.0.0:" + str(port), + "protocol_version": "2026-07-28", "mode": "full_access", + "tls_cert_file": str(cert_path), "tls_key_file": str(key_path)}, + "enabled_tools": ["run_select_query"]} + env = os.environ.copy() + # Deliberately set a privileged process token: remote clients must never inherit it. + env["STACKQL_CALLER_TOKEN"] = "server-process-token" if clients else token(subject="operator") + container = "stackql-access-" + uuid.uuid4().hex + server_base = list(base) + docker = server_base[0] == "docker" + if docker: + index = server_base.index("stackqlsrv") + server_base[index:index] = ["--publish", "127.0.0.1:" + str(port) + ":" + str(port), "--name", container] + command = server_base + ["--access.config", json.dumps({"path": str(config_file)}), "--mcp.config", json.dumps(config), "mcp"] + tls = ssl.create_default_context(cafile=str(cert_path)) + if clients: + contexts = {} + for name, (cert, private) in clients.items(): + context = ssl.create_default_context(cafile=str(cert_path)) + context.load_cert_chain(str(cert), str(private)) + contexts[name] = context + token = lambda subject="reader": contexts[subject] + invalid = [None] + [contexts[name] for name in ["untrusted", "expired", "ambiguous", "unknown"]] + with (root / "mcp.log").open("w+") as log: + process = subprocess.Popen(command, stdout=log, stderr=log, env=env) + try: + def call(credential, sql=query): + headers = {"Content-Type": "application/json", "Accept": "application/json, text/event-stream", "MCP-Protocol-Version": "2026-07-28", "Mcp-Method": "tools/call", "Mcp-Name": "run_select_query"} + if credential and not clients: + headers["Authorization"] = "Bearer " + credential + payload = {"jsonrpc": "2.0", "id": 1, "method": "tools/call", "params": {"_meta": {"io.modelcontextprotocol/protocolVersion": "2026-07-28", "io.modelcontextprotocol/clientCapabilities": {}, "io.modelcontextprotocol/clientInfo": {"name": "access-test", "version": "1"}}, "name": "run_select_query", "arguments": {"sql": sql}}} + request = urllib.request.Request("https://127.0.0.1:" + str(port), data=json.dumps(payload).encode(), headers=headers) + try: + with urllib.request.urlopen(request, context=credential if clients and credential else tls, timeout=20) as response: + return response.status, response.read().decode() + except urllib.error.HTTPError as error: + return error.code, error.read().decode() + except (urllib.error.URLError, ssl.SSLError) as error: + if clients: return 0, str(error) + raise + for _ in range(100): + if process.poll() is not None: + log.seek(0) + raise AssertionError("protected HTTPS server failed to start: " + log.read()) + try: + status, response = call(token() if clients else None) + if clients and status == 0: raise OSError(response) + assert status == (200 if clients else 401), "HTTP readiness authentication failed" + break + except (OSError, urllib.error.URLError): + time.sleep(0.1) + else: + raise AssertionError("protected HTTPS server did not start") + reader = token() + status, output = call(reader) + assert status == 200 and "known-item" in output, "authorized HTTPS read failed: " + output + for credential in invalid: + before = counts.copy() + status, _ = call(credential) + assert status in ((0, 401) if clients else (401,)) and counts == before, "invalid HTTP credential reached execution" + def read(): + status, output = call(reader) + assert status == 200 and "known-item" in output, "concurrent HTTPS reader failed: " + output + def deny(): + status, output = call(token(subject="denied")) + assert "known-item" not in output and (status != 200 or '"isError":true' in output.replace(" ", "")), "concurrent caller inherited access" + with concurrent.futures.ThreadPoolExecutor(max_workers=2) as pool: + for future in [pool.submit(read), pool.submit(deny)]: + future.result() + db.execute("DELETE FROM stackql_access_memberships WHERE principal_id = 'reader'") + if not postgres: + db.commit() + status, output = call(reader) + assert "known-item" not in output and (status != 200 or '"isError":true' in output.replace(" ", "")), "HTTP membership revocation ignored: " + output + db.execute("INSERT INTO stackql_access_memberships VALUES ('reader','readers')") + if not postgres: + db.commit() + finally: + process.terminate() + try: + process.wait(timeout=10) + except subprocess.TimeoutExpired: + process.kill() + process.wait(timeout=10) + if docker: + subprocess.run(["docker", "rm", "-f", container], capture_output=True, timeout=30) + log.seek(0) + text = log.read() + assert env["STACKQL_CALLER_TOKEN"] not in text, "server token leaked into log" + + def _verify_configuration(self, base, path, config, credential, query, counts): + env = os.environ.copy() + env["STACKQL_CALLER_TOKEN"] = credential + def execute(source): + return subprocess.run(base + ["--access.config", source, "exec", query], env=env, capture_output=True, text=True, timeout=45) + for source in [json.dumps({"path": str(path)}), str(path)]: + result = execute(source) + assert result.returncode == 0 and "known-item" in result.stdout, "valid configuration source failed: " + result.stderr + before = counts.copy() + for source in ['{}', '{"path":""}', '{"path":null}', '{"path":3}', json.dumps({"path": str(path), "unknown": True}), json.dumps({"path": str(path)}) + '{}', json.dumps({"path": str(path) + ".missing"})]: + result = execute(source) + assert result.returncode != 0 and counts == before, "invalid configuration source reached execution" + for changes in [{"unknown": True}, {"type": "unknown"}, {"type": "bearer"}, {"type": "local_os"}, {"type": "mtls"}]: + path.write_text(json.dumps(dict(config, **changes))) + result = execute(json.dumps({"path": str(path)})) + assert result.returncode != 0 and counts == before, "invalid authentication configuration reached execution" + path.write_text(json.dumps(config)) + + def _verify_algorithms(self, config, path, token, run, query, counts): + key = rsa.generate_private_key(public_exponent=65537, key_size=2048) + algorithms = [("RS" + size, key, getattr(hashes, "SHA" + size)()) for size in ["256", "384", "512"]] + algorithms += [("PS" + size, key, getattr(hashes, "SHA" + size)()) for size in ["256", "384", "512"]] + algorithms += [("ES256", ec.generate_private_key(ec.SECP256R1()), hashes.SHA256()), ("ES384", ec.generate_private_key(ec.SECP384R1()), hashes.SHA384()), ("ES512", ec.generate_private_key(ec.SECP521R1()), hashes.SHA512())] + code, output = run(query, token()) + assert code == 0 and "known-item" in output, "EdDSA JWT failed: " + output + encode = lambda value: base64.urlsafe_b64encode(value).rstrip(b"=") + for algorithm, private, digest in algorithms: + config["keys"] = [{"kid": "family", "algorithm": algorithm, "public_key_pem": private.public_key().public_bytes(serialization.Encoding.PEM, serialization.PublicFormat.SubjectPublicKeyInfo).decode()}] + path.write_text(json.dumps(config)) + claims = {"iss": "access-test", "sub": "reader", "aud": "stackql", "exp": int(time.time()) + 300, "iat": int(time.time())} + message = encode(json.dumps({"alg": algorithm, "kid": "family"}).encode()) + b"." + encode(json.dumps(claims).encode()) + if algorithm.startswith("RS"): + signature = private.sign(message, padding.PKCS1v15(), digest) + elif algorithm.startswith("PS"): + signature = private.sign(message, padding.PSS(mgf=padding.MGF1(digest), salt_length=digest.digest_size), digest) + else: + r, value = decode_dss_signature(private.sign(message, ec.ECDSA(digest))) + width = (private.curve.key_size + 7) // 8 + signature = r.to_bytes(width, "big") + value.to_bytes(width, "big") + credential = (message + b"." + encode(signature)).decode() + code, output = run(query, credential) + assert code == 0 and "known-item" in output, algorithm + " JWT failed: " + output + config["keys"][0]["algorithm"] = "RS256" if algorithm != "RS256" else "PS256" + path.write_text(json.dumps(config)) + before = counts.copy() + code, output = run(query, credential) + assert code != 0 and "known-item" not in output and counts == before, "JWT algorithm/key binding was ignored" + + def _verify_local_transport(self, base, path, credential, query, transport, allowed=True): + env = os.environ.copy() + env.pop("STACKQL_CALLER_TOKEN", None) + if credential: + env["STACKQL_CALLER_TOKEN"] = credential + command = base + ["--access.config", json.dumps({"path": str(path)})] + if transport == "shell": + result = subprocess.run(command + ["shell"], input=query + "\nexit\n", env=env, capture_output=True, text=True, timeout=30) + output = result.stdout + result.stderr + else: + command += ["--mcp.config", json.dumps({"server": {"transport": "stdio", "audit": {"disabled": True}}, "enabled_tools": ["run_select_query"]}), "mcp"] + process = subprocess.Popen(command, env=env, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + watchdog = threading.Timer(30, process.kill) + watchdog.start() + lines = [] + try: + def request(value, response_id): + process.stdin.write(json.dumps(value).encode() + b"\n") + process.stdin.flush() + while True: + line = process.stdout.readline() + if not line: + raise AssertionError("MCP stdio closed before its response") + lines.append(line.decode()) + message = json.loads(line) + if message.get("id") == response_id: + return + request({"jsonrpc": "2.0", "id": 1, "method": "initialize", "params": {"protocolVersion": "2025-06-18", "capabilities": {}, "clientInfo": {"name": "access-test", "version": "1"}}}, 1) + process.stdin.write(b'{"jsonrpc":"2.0","method":"notifications/initialized"}\n') + request({"jsonrpc": "2.0", "id": 2, "method": "tools/call", "params": {"name": "run_select_query", "arguments": {"sql": query}}}, 2) + process.stdin.close() + process.stdin = None + stdout, stderr = process.communicate(timeout=10) + output = "".join(lines) + stdout.decode() + stderr.decode() + assert process.returncode == 0, "MCP stdio failed: " + output + finally: + watchdog.cancel() + if process.poll() is None: + process.kill() + process.wait() + assert ("known-item" in output) == allowed, transport + " caller authorization failed: " + output + if not allowed: + assert "caller authentication failed" in output, "missing credential did not fail authentication" + assert not credential or credential not in output, "local transport leaked a credential" + + def _client_certificates(self, root, identity_kind="uri_san"): + now = datetime.datetime.now(datetime.timezone.utc) + issuer = x509.Name([x509.NameAttribute(NameOID.COMMON_NAME, "access-test CA")]) + ca_key = ed25519.Ed25519PrivateKey.generate() + ca = (x509.CertificateBuilder().subject_name(issuer).issuer_name(issuer).public_key(ca_key.public_key()) + .serial_number(x509.random_serial_number()).not_valid_before(now - datetime.timedelta(days=1)) + .not_valid_after(now + datetime.timedelta(days=1)).add_extension(x509.BasicConstraints(ca=True, path_length=0), True) + .sign(ca_key, None)) + clients = {} + for name in ["reader", "operator", "denied", "unknown", "untrusted", "expired", "ambiguous"]: + private = ed25519.Ed25519PrivateKey.generate() + san = lambda subject: x509.DNSName(subject + ".example") if identity_kind == "dns_san" else x509.UniformResourceIdentifier("spiffe://access-test/" + subject) + uris = [san(name)] + if name == "ambiguous": + uris.append(san("reader")) + certificate = (x509.CertificateBuilder().subject_name(x509.Name([x509.NameAttribute(NameOID.COMMON_NAME, name)])) + .issuer_name(issuer).public_key(private.public_key()).serial_number(x509.random_serial_number()) + .not_valid_before(now - datetime.timedelta(days=2)) + .not_valid_after(now - datetime.timedelta(seconds=1) if name == "expired" else now + datetime.timedelta(hours=1)) + .add_extension(x509.SubjectAlternativeName(uris), False) + .add_extension(x509.ExtendedKeyUsage([x509.oid.ExtendedKeyUsageOID.CLIENT_AUTH]), False) + .sign(ed25519.Ed25519PrivateKey.generate() if name == "untrusted" else ca_key, None)) + cert_path, key_path = root / (name + ".crt"), root / (name + ".key") + cert_path.write_bytes(certificate.public_bytes(serialization.Encoding.PEM)) + key_path.write_bytes(private.private_bytes(serialization.Encoding.PEM, serialization.PrivateFormat.PKCS8, serialization.NoEncryption())) + key_path.chmod(0o600) + clients[name] = (cert_path, key_path) + return ca.public_bytes(serialization.Encoding.PEM).decode(), clients + + def verify_access_walkthrough(self, executable, mode, execution_platform="native"): + script = pathlib.Path(__file__).resolve().parents[3] / "docs/examples/access-management/quickstart.py" + with tempfile.TemporaryDirectory(prefix="stackql-access-walkthrough-") as directory: + root = pathlib.Path(directory).resolve() + docker = execution_platform == "docker" + container = "stackql-walkthrough-" + uuid.uuid4().hex + env = os.environ.copy() + env.pop("STACKQL_CALLER_TOKEN", None) + exe = shutil.which(executable) or str(pathlib.Path(executable).resolve()) + if docker: + # Run the published host-side walkthrough against the actual tested image. + # This adapter forwards credentials through Docker's environment, never arguments. + proxy = root / "stackql-proxy" + prefix = ["docker", "run", "--rm", "--interactive", "--user", f"{os.getuid()}:{os.getgid()}", + "--volume", str(root) + ":" + str(root), "--workdir", str(root), + "--env", "STACKQL_CALLER_TOKEN", "--entrypoint", "stackql"] + image = os.environ.get("STACKQL_IMAGE_NAME", "stackql/stackql") + proxy.write_text("#!/usr/bin/env python3\nimport json,os,sys\n" + + "prefix=" + repr(prefix) + "\nargs=sys.argv[1:]\n" + + "if '--mcp.config' in args:\n" + + " i=args.index('--mcp.config')+1; cfg=json.loads(args[i]); port=cfg['server']['address'].rsplit(':',1)[1]\n" + + " cfg['server']['address']='0.0.0.0:'+port; args[i]=json.dumps(cfg)\n" + + " prefix+=['--publish','127.0.0.1:'+port+':'+port,'--name'," + repr(container) + "]\n" + + "os.execvp('docker',prefix+[" + repr(image) + "]+args)\n") + proxy.chmod(0o700) + exe = str(proxy) + command = [shutil.which("python3") or shutil.which("python"), str(script), "--mode", mode, + "--directory", str(root / "demo"), "--stackql", exe if docker else os.path.relpath(exe)] + try: + result = subprocess.run(command, env=env, capture_output=True, text=True, timeout=60) + assert result.returncode == 0, "documented walkthrough failed: " + result.stdout + result.stderr + assert "Verified " + mode + ": principal reader; directory group readers." in result.stdout + if mode == "local_os": + # Exercise the documented next step, not just its setup script. + demo = root / "demo" + shell = subprocess.run([exe, "--access.config", json.dumps({"path": str(demo / "access.json")}), + "--sqlBackend", (demo / "backend.json").read_text(), "--output", "json", "shell"], + input="SELECT whoami() AS principal, groups() AS memberships;\n", env=env, + cwd=root, capture_output=True, text=True, timeout=45) + assert shell.returncode == 0, "documented authenticated shell failed: " + shell.stdout + shell.stderr + assert '"principal":"reader"' in shell.stdout.replace(" ", ""), "shell did not resolve reader: " + shell.stdout + assert "readers" in shell.stdout, "shell did not resolve directory membership: " + shell.stdout + finally: + if docker: + subprocess.run(["docker", "rm", "-f", container], capture_output=True, timeout=30) diff --git a/test/robot/functional/stackql_access.robot b/test/robot/functional/stackql_access.robot new file mode 100644 index 000000000..ba9bee525 --- /dev/null +++ b/test/robot/functional/stackql_access.robot @@ -0,0 +1,154 @@ +*** Settings *** +Resource ${CURDIR}/stackql.resource +Library stackql_test_tooling.access_management_keywords.AccessManagementKeywords + +*** Test Cases *** + +JWT CLI Authenticates And Uses Directory Groups + [Documentation] Authenticate a signed JWT; inspect its principal/groups; read a provider; refuse missing, invalid and unauthorized credentials. + Verify Access Scenario jwt_cli + +JWT Supports Every Configured Signing Algorithm + [Documentation] Read with EdDSA, RSA, RSA-PSS and ECDSA JWTs; reject a valid token when the configured key algorithm differs. + Verify Access Scenario jwt_algorithms + +JWT HTTPS Isolates Callers And Refreshes Membership + [Documentation] Use request bearer credentials over verified HTTPS; deny bad tokens, concurrent caller leakage and revoked membership. + Verify Access Scenario jwt_https + +JWT Shell Uses The Local Credential + [Documentation] Feed a real shell session a provider query; repeat without a token and require authentication denial. + Verify Access Scenario jwt_shell + +JWT MCP Stdio Uses The Local Credential + [Documentation] Initialize a real MCP stdio session and call run_select_query; repeat without a token and require denial. + Verify Access Scenario jwt_stdio + +Static Bearer CLI Verifies A Provisioned Digest + [Documentation] Provision SHA-256 token digests; accept the reader token and deny missing, wrong and unauthorized tokens. + Verify Access Scenario bearer_cli + +Static Bearer HTTPS Isolates Callers And Refreshes Membership + [Documentation] Use actual HTTPS bearer headers; deny invalid tokens and concurrent identity leakage; revoke membership between requests. + Verify Access Scenario bearer_https + +Static Bearer Shell Uses The Local Credential + [Documentation] Read from the shell with a provisioned token; refuse the same query without it. + Verify Access Scenario bearer_shell + +Static Bearer MCP Stdio Uses The Local Credential + [Documentation] Read through MCP stdio with a provisioned token; refuse the same query without it. + Verify Access Scenario bearer_stdio + +MTLS HTTPS Verifies Client Certificates And SAN Identity + [Documentation] Verify the client CA and unique URI SAN; reject missing, untrusted, expired, ambiguous and unmapped certificates; refresh directory groups. + Verify Access Scenario mtls_https + +MTLS HTTPS Supports A Unique DNS SAN Identity + [Documentation] Map a verified unique DNS SAN through the directory; reject invalid chains, expiry, ambiguous SANs and unknown identities. + Verify Access Scenario mtls_https_dns + +Local OS CLI Uses The Effective User And Directory Groups + [Documentation] Map the effective UID/SID to the reader; read a provider; deny mutation and an unprovisioned OS identity. + Verify Access Scenario local_os_cli + +Local OS Shell Uses The Effective User + [Documentation] Read a provider through the real shell using the process identity, without a token. + Verify Access Scenario local_os_shell + +Local OS MCP Stdio Uses The Effective User + [Documentation] Read a provider through MCP stdio using the process identity, without a token. + Verify Access Scenario local_os_stdio + +Local OS Cannot Authenticate Remote HTTP Callers + [Documentation] Refuse HTTP startup rather than authenticate remote callers as the server OS account. + Verify Access Scenario local_os_http_refused + +Access Configuration Accepts A Strict Path Envelope + [Documentation] Use JSON path and legacy filename forms; deny unknown fields, missing paths, trailing data and invalid authentication modes. + Verify Access Scenario config + +Reader Policy Denies Effects And Protects Directory Tables + [Documentation] Deny DML, EXEC, IaC, directory-table access and effectful functions; isolate concurrent principals. + Verify Access Scenario policy + +Directory Revocation And Malformed Attributes Fail Closed + [Documentation] Revoke a reader membership and corrupt operator attributes; deny execution before provider effects. + Verify Access Scenario revocation + +Protected IAC Applies Updates Prunes And Confines State + [Documentation] Apply and update two resources; refuse another principal and path escape; prune one and destroy the rest; require exit failure for unsuccessful IaC. + Verify Access Scenario iac + +Disabled Caller Access Preserves Legacy Queries + [Documentation] Read the mock provider with no access configuration or caller credential. + Verify Access Scenario compatibility + +JWT Rejects Missing Token + [Documentation] Require CLI failure and zero provider requests for this invalid credential. + Verify Access Scenario jwt_invalid:missing + +JWT Rejects Invalid Signature + [Documentation] Require CLI failure and zero provider requests for this invalid credential. + Verify Access Scenario jwt_invalid:signature + +JWT Rejects Wrong Issuer + [Documentation] Require CLI failure and zero provider requests for this invalid credential. + Verify Access Scenario jwt_invalid:issuer + +JWT Rejects Wrong Audience + [Documentation] Require CLI failure and zero provider requests for this invalid credential. + Verify Access Scenario jwt_invalid:audience + +JWT Rejects Missing Expiry + [Documentation] Require CLI failure and zero provider requests for this invalid credential. + Verify Access Scenario jwt_invalid:expiry_required + +JWT Rejects Expired Token + [Documentation] Require CLI failure and zero provider requests for this invalid credential. + Verify Access Scenario jwt_invalid:expired + +JWT Rejects Future Not Before + [Documentation] Require CLI failure and zero provider requests for this invalid credential. + Verify Access Scenario jwt_invalid:not_before + +JWT Rejects Future Issued At + [Documentation] Require CLI failure and zero provider requests for this invalid credential. + Verify Access Scenario jwt_invalid:issued_at + +JWT Rejects Unknown Key ID + [Documentation] Require CLI failure and zero provider requests for this invalid credential. + Verify Access Scenario jwt_invalid:key_id + +JWT Rejects Unknown Identity + [Documentation] Require CLI failure and zero provider requests for this invalid credential. + Verify Access Scenario jwt_invalid:identity + +Documented JWT Walkthrough Works + [Documentation] Execute the published quickstart script; provision the directory and authenticate its reader using this mode. + Verify Access Walkthrough ${STACKQL_EXE} jwt ${EXECUTION_PLATFORM} + +Documented Static Bearer Walkthrough Works + [Documentation] Execute the published quickstart script; provision the directory and authenticate its reader using this mode. + Verify Access Walkthrough ${STACKQL_EXE} bearer ${EXECUTION_PLATFORM} + +Documented MTLS Walkthrough Works + [Documentation] Execute the published quickstart script; provision the directory and authenticate its reader using this mode. + Verify Access Walkthrough ${STACKQL_EXE} mtls ${EXECUTION_PLATFORM} + +Documented Local OS Walkthrough Works + [Documentation] Execute the published quickstart script; provision the directory and authenticate its reader using this mode. + Verify Access Walkthrough ${STACKQL_EXE} local_os ${EXECUTION_PLATFORM} + +Queries Have No Authentication Deadline By Default + [Documentation] Complete a slow provider query beyond max_session_seconds when deadline enforcement is disabled. + Verify Access Scenario query_deadline_default + +Authentication Query Deadline Is Opt In + [Documentation] Enable deadline enforcement and require cancellation of a slow provider query at max_session_seconds. + Verify Access Scenario query_deadline_opt_in + +*** Keywords *** +Verify Access Scenario + [Arguments] ${scenario} + Verify Caller Access ${STACKQL_EXE} ${SQL_BACKEND_CFG_STR_CANONICAL} ${EXECUTION_PLATFORM} ${scenario} diff --git a/test/robot/functional/stackql_mocked_from_cmd_line.robot b/test/robot/functional/stackql_mocked_from_cmd_line.robot index 27bc89040..11d609a52 100644 --- a/test/robot/functional/stackql_mocked_from_cmd_line.robot +++ b/test/robot/functional/stackql_mocked_from_cmd_line.robot @@ -2331,13 +2331,10 @@ Digitalocean Insert Droplet ... stderr=${CURDIR}/tmp/Digitalocean-Insert-Droplet.tmp Transaction Rollback Digitalocean Insert Droplet - ${nativeOutputStr} = Catenate SEPARATOR=\n + ${outputStr} = Catenate SEPARATOR=\n ... OK ... mutating statement queued ... Rollback OK - ${dockerOutputStr} = Catenate SEPARATOR=\n - ... Rollback OK - ${outputStr} = Set Variable If "${EXECUTION_PLATFORM}" == "docker" ${dockerOutputStr} ${nativeOutputStr} Should Stackql Exec Inline Equal Stderr ... ${STACKQL_EXE} ... ${OKTA_SECRET_STR} @@ -2378,7 +2375,7 @@ Transaction Abort Attempted Commit Digitalocean Insert Droplet ... 'ubuntu-20-04-x64', true, true, true, ... '["env:prod", "web"]' ; ... commit; - ${nativeOutputStr} = Catenate SEPARATOR=\n + ${outputStr} = Catenate SEPARATOR=\n ... http${SPACE}response${SPACE}status${SPACE}code:${SPACE}500,${SPACE}response${SPACE}body:${SPACE}{ ... ${SPACE}${SPACE}"id":${SPACE}"server_error", ... ${SPACE}${SPACE}"message":${SPACE}"Unexpected${SPACE}server-side${SPACE}error" @@ -2390,9 +2387,6 @@ Transaction Abort Attempted Commit Digitalocean Insert Droplet ... insert over HTTP error: 500 INTERNAL SERVER ERROR ... UNDO required: Undo the insert on digitalocean.droplets.droplets ... UNDO required: Undo the insert on digitalocean.droplets.droplets - ${dockerOutputStr} = Catenate SEPARATOR=\n - ... UNDO required: Undo the insert on digitalocean.droplets.droplets - ${outputStr} = Set Variable If "${EXECUTION_PLATFORM}" == "docker" ${dockerOutputStr} ${nativeOutputStr} Should Stackql Exec Inline Equal Stderr ... ${STACKQL_EXE} ... ${OKTA_SECRET_STR} @@ -2411,13 +2405,10 @@ Transaction Rollback Eager Idealised Google Admin Directory User ... insert into googleadmin.directory.users(data__primaryEmail) ... values ('somejimbo@grubit.com'); ... rollback; - ${nativeOutputStr} = Catenate SEPARATOR=\n + ${outputStr} = Catenate SEPARATOR=\n ... OK ... The operation was despatched successfully ... Rollback OK - ${dockerOutputStr} = Catenate SEPARATOR=\n - ... Rollback OK - ${outputStr} = Set Variable If "${EXECUTION_PLATFORM}" == "docker" ${dockerOutputStr} ${nativeOutputStr} Should Stackql Exec Inline Equal Stderr ... ${STACKQL_EXE} ... ${OKTA_SECRET_STR} @@ -9540,7 +9531,7 @@ Explain Select Repeatably Generates Messagez ... explain select * from google.storage.buckets; ... explain select * from google.storage.buckets where project = 'stackql-demo'; ... explain select * from google.storage.buckets where project = 'stackql-demo'; - ${outputErrStrNative} = Catenate SEPARATOR=\n + ${outputErrStr} = Catenate SEPARATOR=\n ... Execution plan generated successfully ... OK ... Execution plan generated successfully @@ -9557,7 +9548,6 @@ Explain Select Repeatably Generates Messagez ... OK ... Execution plan generated successfully ... OK - ${outputErrStr} = Set Variable If "${EXECUTION_PLATFORM}" == "docker" OK ${outputErrStrNative} Should Stackql Exec Inline Equal Stderr ... ${STACKQL_EXE} ... ${OKTA_SECRET_STR} @@ -12061,14 +12051,13 @@ Preview Dynamic Graph Wiring Outside Graph Is Refused By Omnisdk ... stderr=${CURDIR}${/}tmp${/}Preview-Dynamic-Graph-Wiring-Outside-stderr.tmp Preview Iac Resource Without Provider Is Refused By Omnisdk - [Documentation] The 'resources' predicate maps onto omnisdk's NewResource - ... one field at a time; this asserts a malformed entry is - ... caught by the SDK before any effect is attempted. + [Documentation] Omnisdk parses the 'resources' predicate and refuses a missing + ... provider before any effect is attempted. ${preview} = Set Variable {"unstable":true} ${query} = Catenate SEPARATOR=${SPACE} ... select * from stackql_preview.iac.run where collection = 'scratch' ... and resources = '[{"key":"a/b/c"}]'; - Should StackQL Exec Inline Equal Stderr + Should StackQL Exec Inline Contain Stderr ... ${STACKQL_EXE} ... ${OKTA_SECRET_STR} ... ${GITHUB_SECRET_STR} @@ -12077,15 +12066,14 @@ Preview Iac Resource Without Provider Is Refused By Omnisdk ... ${AUTH_CFG_STR} ... ${SQL_BACKEND_CFG_STR_CANONICAL} ... ${query} - ... omnisdk: resource "a/b/c" names no provider or address + ... omnisdk: provider "": stackqldoc: no provider "" in registry ... --preview\=${preview} ... stdout=${CURDIR}${/}tmp${/}Preview-Iac-Resource-No-Provider.tmp ... stderr=${CURDIR}${/}tmp${/}Preview-Iac-Resource-No-Provider-stderr.tmp Preview Iac Resources File Without Provider Is Refused By Omnisdk - [Documentation] The 'resources_file' predicate is read by omnisdk's LoadResources, - ... which also takes an object whose 'resources' is the list; a - ... malformed entry is caught before any effect is attempted. + [Documentation] Omnisdk parses 'resources_file', including an object containing + ... a 'resources' list, and refuses a missing provider before effects. ${preview} = Set Variable {"unstable":true} ${nativeResourcesFile} = Replace String ... ${REPOSITORY_ROOT}${/}test${/}assets${/}input${/}iac${/}resource-without-provider.json \\ / @@ -12095,7 +12083,7 @@ Preview Iac Resources File Without Provider Is Refused By Omnisdk ${query} = Catenate SEPARATOR=${SPACE} ... select * from stackql_preview.iac.run where collection = 'scratch' ... and resources_file = '${resourcesFile}'; - Should StackQL Exec Inline Equal Stderr + Should StackQL Exec Inline Contain Stderr ... ${STACKQL_EXE} ... ${OKTA_SECRET_STR} ... ${GITHUB_SECRET_STR} @@ -12104,7 +12092,7 @@ Preview Iac Resources File Without Provider Is Refused By Omnisdk ... ${AUTH_CFG_STR} ... ${SQL_BACKEND_CFG_STR_CANONICAL} ... ${query} - ... omnisdk: resource "a/b/c" names no provider or address + ... omnisdk: provider "": stackqldoc: no provider "" in registry ... --preview\=${preview} ... stdout=${CURDIR}${/}tmp${/}Preview-Iac-Resources-File-No-Provider.tmp ... stderr=${CURDIR}${/}tmp${/}Preview-Iac-Resources-File-No-Provider-stderr.tmp