[azarouski]

Includes #276 and #277: this branch is feat/adb-relay merged with feat/curve-client-key-vars. Review and merge those first, or merge this PR on its own.

Behavior

• ./zebrunner.sh node-invite <label> <address> [<minutes>] asks STF for an invite (POST /api/v1/nodes/invites) and prints the join string once, with a note that it is a secret for ./zebrunner.sh join of mcloud-agent and how many minutes it stays valid:
  mcloud-join://<host>:<port>?label=<label>&grid=<host>:4446&pin=<pin>&token=<invite>
  • host is AGENTS_ADDRESS of .env. When that is 0.0.0.0 (every address), the host of STF_URL is used. stf-proxy, the grid (4446) and the triproxies (7160, 7250, 7260, 7270) are all published on that address.
  • port is STF_PORT.
  • pin is the base64 SHA-256 of the DER SPKI of configuration/stf-proxy/ssl/ssl.crt, made by openssl (LibreSSL works too) or by the new patch/spki_pin.py. In the URL, +, / and = are percent-encoded. The pin is also printed on its own.
  • The label, the address (IPv4 or host name; no loopback, link-local, multicast, scheme or port) and the validity (1-1440 minutes) are checked before any request is made.
  • An http deployment is refused, because the node's only trust is the pin of the certificate.
• ./zebrunner.sh nodes [list] shows a table: label, address, status, last seen, enrolled, and the first characters of the key.
• ./zebrunner.sh nodes revoke <label> revokes the node, then runs docker restart stf-triproxy-app stf-triproxy-dev to end its open ZeroMQ sessions.
• ./zebrunner.sh nodes delete <label> removes a node; STF accepts this for revoked nodes only.
• ./zebrunner.sh nodes legacy-off:
  • warns that every node that is not enrolled will stop working, lists the key variables it will empty and the current nodes, then asks y/n (no by default);
  • empties STF_ZMQ_CURVE_CLIENT_KEYS, every STF_ZMQ_CURVE_CLIENT_KEY_<HOST> and AGENT_HOSTS;
  • sets new NODE_REGISTRATION_TOKEN (48 characters) and STF_STORAGE_TOKEN (64 characters) with random_string. The new values are never printed. variables.env is written in place and stays mode 600;
  • restarts the services.
• All the commands call the STF API through stf-proxy at <scheme of STF_URL>://127.0.0.1:STF_PORT with -k. STF_TOKEN is passed to curl through its stdin config (-K -), so it never appears in the process arguments and is never printed. The API returns generic errors, so the commands print a hint for each status (000, 400, 401, 403, 404, other).
• stf-proxy, /d/<host>/<serial>/<port>/:
  • auth_request goes to an internal location that proxies to stf-api/api/v1/zebrunner/screen-auth. That subrequest gets only Cookie, Authorization, X-Original-URI $request_uri, Host and X-Forwarded-Proto (proxy_pass_request_headers off).
  • The request to the node is also built with proxy_pass_request_headers off and only the WebSocket headers. Cookie "" and Authorization "" are set explicitly. No X-Stf-* header can reach the node.
• The AGENT_HOSTS map and configuration/stf-proxy/agent-hosts.conf are removed. AGENT_HOSTS is now passed to the environment of stf-api as the legacy list.
• limit_req on location ~* ^/api/v1/nodes/enroll: 6 requests a minute per IP, burst 3, nodelay, then 429. The match is case-insensitive and covers any suffix, because the Express router would accept /Enroll/ too. These requests go to stf-api.
• adb-relay auto mode now reads GET /api/v1/nodes/relay and relays each device's connectPort to the registered address of its node. It skips loopback, link-local and multicast addresses as a second check, in addition to STF's. Its token is also passed to curl through stdin.

Decisions

• Relay and legacy nodes: the auto mode relays enrolled nodes only. The legacy path (/api/v1/devices plus AGENT_HOSTS) is removed: there, the device's address comes from the node itself, and the relay runs in the host's network. Legacy nodes go through ADB_RELAY_HOSTS until they are enrolled. The auto mode only exists in the still unreleased feat: optional adb relay of the device hosts in a private network #277, so no deployment depends on the old behavior.
• write_agent_hosts is replaced by check_agent_hosts. It only warns about invalid entries, and start removes the stale agent-hosts.conf.
• Setup: it no longer asks for AGENT_HOSTS, because new deployments use enrollment. The value from the previous setup is kept.
• legacy-off also empties AGENT_HOSTS. The contract does not list it, but it is the same legacy list and finishes the migration.

Migration

The legacy settings (keys, NODE_REGISTRATION_TOKEN, STF_STORAGE_TOKEN, AGENT_HOSTS) keep working next to enrolled nodes until nodes legacy-off:

1. Update the server and the nodes.
2. Enroll each node, using its current STF_PROVIDER_HOST as the address.
3. Run nodes and check that every node is active.
4. Run nodes legacy-off.

AGENT_HOSTS moves from stf-proxy to stf-api. The README section "Migration from the manual settings" has the details.

Security

The new README "Nodes" section describes what enrollment protects and the risks that remain:

• What it protects: per-node keys and tokens; invites that are hashed, single-use, expiring and bound to a label and an address; trust by certificate pin; destinations set by the admin's address for the grid, screens and relay; no user credentials sent to nodes; STF_TOKEN kept out of process arguments and output.
• Remaining risks: the join string is a bearer secret until it is used; a node with root on its host can act as itself; the server talks to nodes over plain HTTP/WS; the legacy settings stay weak until legacy-off; the rate limit is per IP; revocation has a window of up to 60 s.

Settings for the STF registry of the nodes (zebrunner/stf#1124)

• stf-api command: --nodes-connect-sub tcp://${NODES_HOST}:7250 --nodes-connect-push tcp://${NODES_HOST}:7270 --nodes-connect-app-dealer tcp://${NODES_HOST}:7160 --nodes-connect-dev-dealer tcp://${NODES_HOST}:7260.
  • NODES_HOST is a new .env setting that ./zebrunner.sh setup and start write: AGENTS_ADDRESS, or the host of STF_URL when that is 0.0.0.0. This is the same host as in the join string. An .env from an earlier release gets the line added by start.
  • --nodes-stf-url is left at its default, STF_URL of variables.env, which is the public URL.
• stf-api environment: AGENT_HOSTS (legacy) comes from .env. STF_STORAGE_TOKEN and STF_ZMQ_CURVE_SERVER_PUBLIC_KEY already come from variables.env.
• variables.env.original: STF_NODES_API_URL=http://stf-api:3000, for the triproxies and stf-storage-temp. Every STF unit gets it through env_file, and setup keeps it.
• stf-proxy:
  • /d/ also sets X-Stf-Internal-Token "", on top of the header whitelist, Cookie "" and Authorization "".
  • location ~* ^/api/v1/zebrunner/nodes { return 403; } blocks the internal routes of the triproxies and the storage. It is case-insensitive and catches any suffix, double slashes and percent-encoding.
  • /_screen_auth matches the STF version and is stricter: it passes only the cookie, Authorization, X-Original-URI, Host and X-Forwarded-Proto.
• API calls checked against stf#1124:
  • POST /api/v1/nodes/invites {label,address,ttlMinutes} returns 201 {invite,label,address,expiresAt};
  • GET /api/v1/nodes returns {nodes:[{label,address,publicKey,status,createdAt,revokedAt,lastSeenAt}]};
  • POST /api/v1/nodes/<label>/revoke and DELETE /api/v1/nodes/<label> return 200, or 400 for an unknown node or a node that is not revoked;
  • GET /api/v1/nodes/relay returns {devices:[{serial,label,address,connectPort}]}.
• Reserved labels: node-invite refuses enroll, invites, relay and verify.

Dependencies

• feat: node enrollment and per-node credentials stf#1124: node registry, screen-auth, the --nodes-connect-* options and STF_NODES_API_URL. With an STF image that does not have it:
  • screens return 500;
  • the node commands report "no registry of the nodes";
  • the relay logs that the API call failed.
• feat: per-node tokens verified by STF mcloud-grid#216: verifying node tokens (mcn1.) on registration.
• feat: node enrollment with ./zebrunner.sh join and the token of the host mcloud-agent#460: ./zebrunner.sh join reads the join string.
• e2e: E2E_REFS defaults to STF_REF=feat/node-enrollment until stf#1124 is merged; remove the default once it is. The e2e device host is a legacy node, so the test sets AGENT_HOSTS.

Tests

• make lint: ok.
• make test: ok.
  • zebrunner_test gains the node commands with stub curl and docker. It checks:
    • the curl arguments and request bodies, the method and path, and that the Bearer token is only in curl's stdin;
    • that no secret is printed except the one-time join string (the invite appears exactly once), and the exact join string and pin format, including the python fallback;
    • label, address and validity checks; error hints for each status; refusal on http;
    • list formatting; that revoke restarts the triproxies; delete;
    • legacy-off: decline (explicit and default), the rotation (lengths, new values, not printed, written in place, mode 600), empty keys and AGENT_HOSTS, the restart;
    • the generated nginx config: auth_request, cleared headers, the limit zone and burst;
    • compose: AGENT_HOSTS goes to stf-api, and stf-proxy has no agent-hosts mount.
  • adb_relay_test is rewritten for /api/v1/nodes/relay.
• Docker:
  • New tests/stf_proxy_docker_test.sh (in DOCKER_TESTS): nginx -t, then the real generated config with a fake stf-api and node: 401/403 from screen-auth; the node gets no cookie, Authorization or X-Stf-* while the WebSocket headers pass; the subrequest gets no extra headers; enroll goes to stf-api and returns 200 200 200 200 then 429, also for /API/v1/Nodes/Enroll/; other API routes are not limited. All pass.
  • adb_relay_docker_test (updated): all pass.
  • stf_proxy_docker_test now also checks that the internal token never reaches the node, and that /api/v1/zebrunner/nodes/* returns 403 (case variants, //, %6E) while screen-auth is still reachable.
  • zebrunner_test now also checks:
    • docker compose config of a set-up copy gives stf-api the --nodes-connect-* options at NODES_HOST, and STF_NODES_API_URL reaches both triproxies, stf-storage-temp and stf-api;
    • NODES_HOST follows AGENTS_ADDRESS or the STF_URL host, and is added to an old .env;
    • reserved labels are refused.
  • compose_test (updated: screen-auth probe, no agent-hosts mount) could not run locally, because public.ecr.aws/zebrunner/stf:3.1 and mcloud-grid:3.1 are not published yet. That was already the case before this PR.