Add retry-safe live Slack contract tests

.github/workflows/slack-live.yml

@@ -0,0 +1,82 @@
+name: Slack live integration
+
+# Kept out of ci.yml because it needs a concurrency policy the rest of CI
+# must not have. These tests drive one real Slack workspace, and Slack hands
+# each event to exactly one of an app's open Socket Mode connections — so two
+# runs at once steal each other's messages, and collide on reactions and on
+# deleting each other's posts. The workspace is the shared resource, not the
+# branch, so the lock below is global rather than per-ref.
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+  workflow_dispatch:
+
+# Serialize against the shared workspace and let active runs finish so
+# unacknowledged envelopes cannot retry into the next job. GitHub keeps only
+# one queued run per group; a displaced queued check can simply be rerun.
+concurrency:
+  group: slack-live-workspace
+  cancel-in-progress: false
+
+permissions:
+  contents: read
+
+jobs:
+  slack-live-tests:
+    name: Slack live integration
+    runs-on: ubuntu-latest
+    timeout-minutes: 20
+    # The constraint is forks, not pull requests: a fork cannot read the
+    # secrets, so the job would have nothing to run against. A pull request
+    # from a branch in this repository can, and is exactly where the result
+    # is worth having — waiting until merge means finding out too late.
+    #
+    # A clone without the secrets configured skips every test rather than
+    # failing, so this stays green outside ClickHouse/nerve.
+    if: >-
+      github.event_name != 'pull_request' ||
+      github.event.pull_request.head.repo.full_name == github.repository
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v5
+
+      - name: Set up uv
+        uses: astral-sh/setup-uv@v6
+        with:
+          # Kept in step with the backend job above; bump both together.
+          version: "0.12.0"
+          enable-cache: true
+          cache-dependency-glob: "uv.lock"
+
+      - name: Install project and test dependencies
+        run: uv sync --extra test --locked --python "3.13"
+
+      # Inbound first, and in its own process. The outbound module now keeps
+      # an ack-only socket open around all of its Web API mutations, which
+      # prevents it from scheduling retries that poison the next run. Keep
+      # inbound first for retry schedules left by older revisions or an
+      # interrupted job, and keep the process split so the outbound sink can
+      # never steal an inbound test event. `if: always()` on outbound means a
+      # failure in the first step still reports the second.
+      - name: Run live Slack tests (inbound)
+        env:
+          NERVE_SLACK_TEST_BOT_TOKEN: ${{ secrets.NERVE_SLACK_TEST_BOT_TOKEN }}
+          NERVE_SLACK_TEST_APP_TOKEN: ${{ secrets.NERVE_SLACK_TEST_APP_TOKEN }}
+          NERVE_SLACK_TEST_CHANNEL: ${{ secrets.NERVE_SLACK_TEST_CHANNEL }}
+          NERVE_SLACK_TEST_USER_TOKEN: ${{ secrets.NERVE_SLACK_TEST_USER_TOKEN }}
+        # -s exposes the credential-free `SLACK_LIVE {json}` diagnostics from
+        # the harness even on successful runs; pytest otherwise captures them.
+        run: .venv/bin/pytest tests/test_slack_live_inbound.py -v -s
+
+      - name: Run live Slack tests (outbound)
+        if: always()
+        env:
+          NERVE_SLACK_TEST_BOT_TOKEN: ${{ secrets.NERVE_SLACK_TEST_BOT_TOKEN }}
+          NERVE_SLACK_TEST_APP_TOKEN: ${{ secrets.NERVE_SLACK_TEST_APP_TOKEN }}
+          NERVE_SLACK_TEST_CHANNEL: ${{ secrets.NERVE_SLACK_TEST_CHANNEL }}
+          NERVE_SLACK_TEST_USER_TOKEN: ${{ secrets.NERVE_SLACK_TEST_USER_TOKEN }}
+          NERVE_SLACK_TEST_BOT_TOKEN_NO_EMAIL: ${{ secrets.NERVE_SLACK_TEST_BOT_TOKEN_NO_EMAIL }}
+        run: .venv/bin/pytest tests/test_slack_live.py -v -s

docs/testing-slack.md

@@ -0,0 +1,56 @@
+# Live Slack tests
+
+These tests cover behavior a fake cannot verify: Slack API payloads, scopes,
+text rewriting, and end-to-end Socket Mode delivery. They are optional and
+skip when their credentials are absent.
+
+Use a throwaway workspace. The suite creates and deletes messages, reactions,
+and uploads, but an interrupted run can leave test data behind.
+
+## Setup
+
+1. Create the app from the [Slack manifest](config.md#setting-up-the-slack-app).
+2. Add the user token scopes `chat:write`, `im:write`, and `reactions:write`.
+3. In **App Home → Messages Tab**, enable messages and **Allow users to send
+   Slash commands and messages**. The manifest cannot set this option.
+4. Invite both the bot and the user who installed the app to the test channel.
+
+Set these test-only environment variables:
+
+| Variable | Value | Required for |
+|---|---|---|
+| `NERVE_SLACK_TEST_BOT_TOKEN` | Bot token (`xoxb-…`) | All live tests |
+| `NERVE_SLACK_TEST_APP_TOKEN` | App token (`xapp-…`) with `connections:write` | All live tests |
+| `NERVE_SLACK_TEST_CHANNEL` | Test channel ID (`C…`) | All live tests |
+| `NERVE_SLACK_TEST_USER_TOKEN` | User token (`xoxp-…`) | Inbound tests |
+| `NERVE_SLACK_TEST_BOT_TOKEN_NO_EMAIL` | Bot token without `users:read.email` | One scope test |
+
+These variables do not configure Nerve itself. See [config.md](config.md#slack)
+for production configuration.
+
+## Run
+
+Ensure no CI job or other developer is using the same test workspace. Then run
+the inbound suite first and the outbound suite second, as separate processes:
+
+```bash
+uv run --extra test pytest tests/test_slack_live_inbound.py -v -s
+uv run --extra test pytest tests/test_slack_live.py -v -s
+```
+
+Do not combine the modules in one pytest command. Slack sends each event to
+only one active Socket Mode connection. Process isolation prevents the
+outbound suite's acknowledgement-only connection from stealing inbound events
+and prevents unacknowledged events from being retried into later tests.
+
+## CI and failures
+
+The `Slack live integration` workflow uses the same variable names as GitHub
+secrets and serializes runs against the shared workspace.
+
+With `-s`, diagnostic lines prefixed by `SLACK_LIVE` show connection state,
+probe latency, retries, event age, and a final summary. They omit tokens, IDs,
+and message text, so they are safe to share when investigating a failure.
+
+Slash commands require manual testing because Slack provides no API for
+invoking them.

tests/fake_slack.py

@@ -113,22 +113,45 @@ async def wait_connected(self, timeout: float = 5.0) -> None:
         """Block until the bot has opened its socket."""
         await asyncio.wait_for(self._connected.wait(), timeout)
 
-    async def push(self, envelope_type: str, payload: dict[str, Any]) -> str:
+    async def push(
+        self,
+        envelope_type: str,
+        payload: dict[str, Any],
+        *,
+        retry_attempt: int | None = None,
+        retry_reason: str | None = None,
+    ) -> str:
         """Push one Socket Mode envelope at the bot. Returns its envelope id."""
         await self.wait_connected()
         assert self._ws is not None
         envelope_id = str(uuid.uuid4())
-        await self._ws.send_str(json.dumps({
+        envelope = {
             "type": envelope_type,
             "envelope_id": envelope_id,
             "payload": payload,
             "accepts_response_payload": False,
-        }))
+        }
+        if retry_attempt is not None:
+            envelope["retry_attempt"] = retry_attempt
+        if retry_reason is not None:
+            envelope["retry_reason"] = retry_reason
+        await self._ws.send_str(json.dumps(envelope))
         return envelope_id
 
-    async def push_event(self, event: dict[str, Any]) -> str:
+    async def push_event(
+        self,
+        event: dict[str, Any],
+        *,
+        retry_attempt: int | None = None,
+        retry_reason: str | None = None,
+    ) -> str:
         """Push an Events API event (the common case)."""
-        return await self.push("events_api", {"event": event})
+        return await self.push(
+            "events_api",
+            {"event": event},
+            retry_attempt=retry_attempt,
+            retry_reason=retry_reason,
+        )
 
     # -- the Web API ---------------------------------------------------- #