Add a Databricks Genie Code agent-skills host

agent-skills/DESIGN.md

@@ -60,3 +60,8 @@ or nothing. Without the check it falls back on its priors, the failure seen when
 empty skill tree. The agent's access is never the artifact's: the generated code authenticates
 where it runs, so §0 changes no gate and no generated line. Credentials stay with the user —
 the agent may ask them to authenticate a tool, never for a value.
+
+Genie Code is the Databricks counterpart of CoCo: its own SQL, run as the signed-in user, is the
+read tool, and the notebook it generates runs as whoever runs it, through `DatabricksIdentity()`'s
+ambient credentials. So the Genie host names no service principal and no secret scope unless the
+user asks for one; the legacy Databricks host required both, before the SDK had an ambient form.

agent-skills/README.md

@@ -1,12 +1,13 @@
 # Agent skills: `kumo-data-science`
 
 Agent skill for building, training, and running inference with Kumo models. One shared source tree
-serves every host; host-specific behavior lives under `skills/hosts/`. Two hosts:
+serves every host; host-specific behavior lives under `skills/hosts/`. Three hosts:
 
 | Host | Package | For |
 |---|---|---|
 | `hosts/snowflake-coco/` | `coco` | Snowflake Cortex Code (CoCo), inside Snowflake |
 | `hosts/generic/` | `generic` | a coding agent beside the user — Claude Code, Codex — generating code for any of the six Kumo connectors |
+| `hosts/databricks-genie/` | `genie` | Databricks Genie Code, inside a Databricks workspace, generating notebooks for a GPU cluster |
 
 This directory is deliberately not under `.agents/skills/` or `.claude/skills/`: Codex, Claude Code
 and the CoCo CLI load those for repo developers, and this skill triggers on "kumo" and "train a
@@ -39,13 +40,18 @@ path. Build a host package with:
 ```bash
 python3 agent-skills/scripts/package_skill.py coco        # -> agent-skills/dist/coco/
 python3 agent-skills/scripts/package_skill.py generic     # -> agent-skills/dist/generic/skills/
+python3 agent-skills/scripts/package_skill.py genie       # -> agent-skills/dist/genie/skills/
+python3 agent-skills/scripts/package_skill.py genie --deploy   # + upload to your Genie Code skills folder
 python3 agent-skills/scripts/package_skill.py --check     # build every host, verify references
 ```
 
 A package carries exactly one host, and the core reaches it only as `hosts/<host>/`; tests fail if
 a core file names a host. To use the `generic` package, copy
 `dist/generic/skills/kumo-data-science/` into the agent's user-level skills directory (for Claude
-Code, `~/.claude/skills/`) — not into this repository, for the reason above.
+Code, `~/.claude/skills/`) — not into this repository, for the reason above. `genie --deploy`
+uploads the `genie` package to `/Users/<you>/.assistant/skills/kumo-data-science/` in the
+workspace named by `DATABRICKS_HOST` and `DATABRICKS_TOKEN` and reads every file back; a new Genie
+Code chat loads it.
 
 The build stamps the router's `SKILL-BUILD` marker with the source commit
 (`<commit date>-<short sha>`, `-dirty` for uncommitted edits); never edit it by hand. Every build

agent-skills/scripts/package_skill.py

@@ -6,6 +6,8 @@
 
     python3 agent-skills/scripts/package_skill.py coco             # build into agent-skills/dist/
     python3 agent-skills/scripts/package_skill.py generic          # Claude Code, Codex, other agents
+    python3 agent-skills/scripts/package_skill.py genie            # Databricks Genie Code
+    python3 agent-skills/scripts/package_skill.py genie --deploy   # + upload to your Genie skills folder
     python3 agent-skills/scripts/package_skill.py coco --out DIR   # build somewhere else
     python3 agent-skills/scripts/package_skill.py --check          # build every host in a temp dir
 
@@ -18,18 +20,31 @@
 `<commit date>-<short sha>`, suffixed `-dirty` when `skills/` has uncommitted changes. The source
 tree carries `SKILL-BUILD: source`, so a session quoting that value read an unpackaged copy.
 
+`genie --deploy` uploads the package to `/Users/<you>/.assistant/skills/kumo-data-science/`, the
+folder Genie Code reads skills from, through the Workspace API, then reads every file back and
+compares bytes. It authenticates with `DATABRICKS_HOST` (or `DATABRICKS_URL`) and
+`DATABRICKS_TOKEN` from the environment or `--env-file`. Files already in that folder that the
+package does not ship are reported, and deleted with `--prune` (only inside
+`kumo-data-science/`). A new Genie Code chat picks up the change.
+
 Every build is checked: each markdown link must resolve relative to the file that holds it, and
 each backticked path naming a package directory must resolve relative to the package root. A
 package that references a file it does not ship fails the build.
 """
 from __future__ import annotations
 
 import argparse
+import base64
+import json
+import os
 import re
 import shutil
 import subprocess
 import sys
 import tempfile
+import urllib.error
+import urllib.parse
+import urllib.request
 from pathlib import Path
 
 ROOT = Path(__file__).resolve().parents[1]
@@ -55,7 +70,13 @@
     'generic': dict(prefix='skills',
                     hosts=['generic/adapter.md', 'generic/runtime.md'],
                     capability=True),
+    # Databricks Genie Code. Its skills folder holds the one skill directory, like `generic`.
+    'genie': dict(prefix='skills',
+                  hosts=['databricks-genie/adapter.md', 'databricks-genie/runtime.md',
+                         'databricks-genie/job.md'],
+                  capability=True),
 }
+GENIE_SKILLS = '/Users/{user}/.assistant/skills'
 
 MARKER = re.compile(r'SKILL-BUILD: [^`*\s]+')
 MD_LINK = re.compile(r'\]\(([^)\s#]+)(?:#[^)]*)?\)')
@@ -149,14 +170,132 @@ def unresolved(pkg: Path) -> list[str]:
     return errors
 
 
+class Workspace:
+    """The few Databricks REST calls a deploy needs, with a personal access token."""
+
+    def __init__(self, host: str, token: str) -> None:
+        self.host = host if host.startswith('http') else f'https://{host}'
+        self.host = self.host.rstrip('/')
+        self.token = token
+
+    def api(self, method: str, path: str, body: dict | None = None,
+            query: dict | None = None) -> dict:
+        url = self.host + path + ('?' + urllib.parse.urlencode(query) if query else '')
+        req = urllib.request.Request(
+            url, method=method, data=json.dumps(body).encode() if body is not None else None,
+            headers={'Authorization': f'Bearer {self.token}', 'Content-Type': 'application/json'})
+        try:
+            with urllib.request.urlopen(req, timeout=60) as r:
+                return json.loads(r.read() or b'{}')
+        except urllib.error.HTTPError as e:
+            raise SystemExit(f'{method} {path}: HTTP {e.code} {e.read().decode()[:300]}') from None
+
+    def me(self) -> str:
+        return self.api('GET', '/api/2.0/preview/scim/v2/Me')['userName']
+
+    def listing(self, path: str, dirs: bool = False) -> list[str]:
+        """Every file under `path`, recursively — or, with `dirs`, every directory, deepest
+        first; [] when it does not exist."""
+        try:
+            objects = self.api('GET', '/api/2.0/workspace/list', query=dict(path=path))
+        except SystemExit as e:
+            if 'RESOURCE_DOES_NOT_EXIST' in str(e):
+                return []
+            raise
+        out = []
+        for o in objects.get('objects', []):
+            if o['object_type'] == 'DIRECTORY':
+                out += self.listing(o['path'], dirs) + ([o['path']] if dirs else [])
+            elif not dirs:
+                out.append(o['path'])
+        return out
+
+    def delete(self, path: str) -> None:
+        """Delete one file or one empty directory; never recursive."""
+        self.api('POST', '/api/2.0/workspace/delete', dict(path=path, recursive=False))
+
+
+def read_env_file(path: Path) -> None:
+    """Set KEY=VALUE lines from `path` into the environment, without overriding it."""
+    for line in path.read_text().splitlines():
+        key, sep, value = line.strip().partition('=')
+        if sep and not key.startswith('#'):
+            os.environ.setdefault(key.strip(), value.strip().strip('\'"'))
+
+
+def deploy_genie(pkg: Path, ws: Workspace, prune: bool = False) -> str:
+    """Upload `pkg` to the user's Genie Code skills folder and read every file back.
+
+    Files in the target that the package does not ship — an older build's host directory, say —
+    are reported, and with `prune` deleted, along with directories they leave empty. Pruning runs
+    before the upload, so a failed delete leaves the old build rather than two host directories.
+    Only the `kumo-data-science/` folder is touched; other skills beside it never are, and a listed
+    path outside it stops the deploy before anything is deleted. A stale file under `hosts/` is
+    refused without `prune` before anything is written: the router expects exactly one host
+    directory, so a second one left in place would let Genie read the wrong adapter.
+    """
+    target = f'{GENIE_SKILLS.format(user=ws.me())}/{ROUTER}'
+    files = sorted(p for p in pkg.rglob('*') if p.is_file())
+    remote = {f'{target}/{p.relative_to(pkg).as_posix()}': p for p in files}
+
+    def inside(paths: list[str]) -> list[str]:
+        outside = [path for path in paths if not path.startswith(target + '/')]
+        if outside:
+            raise SystemExit(f'workspace listing of {target} returned paths outside it '
+                             f'(e.g. {outside[0]}). Nothing was deleted.')
+        return paths
+
+    stale = sorted(set(inside(ws.listing(target))) - set(remote))
+    stale_hosts = [path for path in stale if path.startswith(f'{target}/hosts/')]
+    if stale_hosts and not prune:
+        raise SystemExit(f'{len(stale_hosts)} file(s) under {target}/hosts/ are not in this package '
+                         f'(e.g. {stale_hosts[0]}); a second host directory would be read. '
+                         f'Rerun with --prune to delete them. Nothing was written.')
+    if prune:
+        for path in stale:
+            ws.delete(path)
+            print(f'  not in this package, deleted: {path}')
+        for d in inside(ws.listing(target, dirs=True)):
+            if not any(p.startswith(d + '/') for p in remote) and not ws.listing(d):
+                ws.delete(d)
+                print(f'  empty directory deleted: {d}')
+    for path, p in remote.items():
+        ws.api('POST', '/api/2.0/workspace/mkdirs', dict(path=path.rsplit('/', 1)[0]))
+        ws.api('POST', '/api/2.0/workspace/import', dict(
+            path=path, format='AUTO', overwrite=True,
+            content=base64.b64encode(p.read_bytes()).decode()))
+    bad = [path for path, p in remote.items()
+           if base64.b64decode(ws.api('GET', '/api/2.0/workspace/export',
+                                      query=dict(path=path, format='AUTO')).get('content', ''))
+           != p.read_bytes()]
+    if bad:
+        raise SystemExit(f'deploy check failed for {len(bad)} file(s): {bad[:5]}')
+    print(f'genie: {len(files)} files deployed and read back byte for byte -> {target}')
+    if not prune and stale:
+        for path in stale:
+            print(f'  not in this package, left in place: {path}')
+        print('  --prune deletes these.')
+    print('Start a new Genie Code chat (agent mode) to load it; ask it to quote SKILL-BUILD.')
+    return target
+
+
 def main() -> None:
     ap = argparse.ArgumentParser(description='Build and check a kumo-data-science skill package.')
     ap.add_argument('host', nargs='?', choices=sorted(HOSTS))
     ap.add_argument('--out', type=Path, default=DIST, help='output root (default: %(default)s)')
     ap.add_argument('--check', action='store_true', help='build and check every host in a temp dir')
+    ap.add_argument('--deploy', action='store_true',
+                    help='genie: upload to your Genie Code skills folder and read it back')
+    ap.add_argument('--prune', action='store_true',
+                    help='genie --deploy: delete files in the target the package does not ship')
+    ap.add_argument('--env-file', type=Path, help='KEY=VALUE file with DATABRICKS_HOST/_TOKEN')
     args = ap.parse_args()
     if not args.check and not args.host:
         ap.error('give a host, or --check')
+    if args.deploy and (args.check or args.host != 'genie'):
+        ap.error('--deploy is for the genie host')
+    if args.prune and not args.deploy:
+        ap.error('--prune goes with --deploy')
 
     failed = False
     with tempfile.TemporaryDirectory() as tmp:
@@ -168,7 +307,16 @@ def main() -> None:
             for e in errors:
                 print(f'  {e}')
             failed |= bool(errors)
-    sys.exit(1 if failed else 0)
+    if failed:
+        sys.exit(1)
+    if args.deploy:
+        if args.env_file:
+            read_env_file(args.env_file)
+        host = os.environ.get('DATABRICKS_HOST') or os.environ.get('DATABRICKS_URL')
+        token = os.environ.get('DATABRICKS_TOKEN')
+        if not host or not token:
+            sys.exit('--deploy needs DATABRICKS_HOST (or DATABRICKS_URL) and DATABRICKS_TOKEN')
+        deploy_genie(pkg, Workspace(host, token), prune=args.prune)
 
 
 if __name__ == '__main__':