Add initial experiments from latest fork implementations of Goose (host application) and GitHub MCP server

[olaservo]

Adds a simple test harness and initial results from running a modified version of Goose loading skills from a forked GitHub MCP server.

Also adds diagrams for the fork implementations used in this experiment:

• Goose: documents the skills-over-mcp implementation path alongside existing paths for tools, resources, and file-based skills
• GitHub MCP Server: similarly to above - documents the skills-over-mcp implementation path alongside tools and other resources

Note that these are still experimental and will probably change before being released.

[Copilot]

Pull request overview

Adds a Goose-based scenario harness and results for exercising SEP-2640 skills served by the GitHub MCP server and a public demo server.

Changes:

• Adds scenarios and a Python runner for discovery, loading, and supporting files.
• Records initial experiment results.
• Documents host/server flows with READMEs and diagrams.

Reviewed changes

Copilot reviewed 13 out of 15 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
experiments/README.md	Introduces the experiment suite and servers.
experiments/sep-2640-github-mcp-server-surfaces.svg	Diagrams GitHub MCP server skill surfaces.
experiments/goose/README.md	Documents the Goose harness and scenario format.
experiments/goose/sep-2640-goose-surfaces.svg	Diagrams Goose skill discovery and loading.
experiments/goose/run_scenario.py	Implements probing, execution, parsing, and checks.
experiments/goose/scenarios/uri-load.yaml	Tests loading by URI.
experiments/goose/scenarios/discovery-and-load.yaml	Tests GitHub skill discovery and loading.
experiments/goose/scenarios/demo-unlisted-skill.yaml	Tests an unlisted skill.
experiments/goose/scenarios/demo-supporting-file.yaml	Tests supporting-file loading.
experiments/goose/scenarios/demo-discovery-and-load.yaml	Tests public demo discovery and loading.
experiments/goose/results/uri-load.20260827-190213.json	Records URI-loading results.
experiments/goose/results/discovery-and-load.20260827-190206.json	Records GitHub discovery results.
experiments/goose/results/demo-unlisted-skill.20260825-205215.json	Records unlisted-skill results.
experiments/goose/results/demo-supporting-file.20260825-205332.json	Records supporting-file results.
experiments/goose/results/demo-discovery-and-load.20260825-205208.json	Records demo discovery results.

Suppressed comments (4)

experiments/goose/run_scenario.py:51

• The probe sends skills/list without completing the MCP initialization lifecycle. After initialize, an HTTP client must send notifications/initialized, retain any returned MCP-Session-Id, and include both that session ID and the negotiated MCP-Protocol-Version on subsequent requests. A conforming stateful server may reject the current probe even though these two stateless demo servers accept it.

    init = rpc(endpoint, token, "initialize", {
        "protocolVersion": "2026-07-28",
        "capabilities": {},
        "clientInfo": {"name": "sep2640-harness", "version": "0"},
    }, 1)

experiments/goose/run_scenario.py:188

• The subprocess return code is discarded, so a Goose crash can still produce a passing harness result when the configured checks are wire-only (or empty, which the documented scenario format permits). Return and record proc.returncode, and include a mandatory successful-process check in result["passed"].

        env=env,
        capture_output=True,
        timeout=600,
    )
    return (proc.stdout + proc.stderr).decode("utf-8", "replace")

experiments/goose/run_scenario.py:252

• A substring search does not prove the model enumerated the skill. In these scenarios the loaded skill name also appears in the load_skill transcript and later prose, so this check can pass even when that name was omitted from the requested list. Match a standalone enumeration line (or parse assistant reply segments) instead.

        check(f"enumerated by model: {name}", name in output, "searched reply text")

experiments/goose/run_scenario.py:58

• The 16-page guard silently returns a truncated catalog when a valid server has another cursor. That can make discovery checks and recorded results incomplete while still reporting success. Keep the safety bound, but fail explicitly when nextCursor remains after the final allowed page.

        while page < 16:

---

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.