experimental/ssh: resolve bare python/pip to the environment interpreter

[anton-107]

Summary

In an interactive databricks ssh connect session, bare python/pip resolved to the wrong interpreter (system or cluster-libraries python) instead of the environment interpreter at $DATABRICKS_VIRTUAL_ENV, so packages installed in the environment (including with --base-environment) could not be imported without extra setup. import <env-package> failed with ModuleNotFoundError. (DECO-27499)

Root cause (two layers)

sshd forwards a snapshot of the environment-first PATH into each session via a SetEnv line, but two things clobbered it in an interactive shell:

1. Login shell — the client ran exec bash -l; -l re-sources /etc/profile, which rebuilds PATH from scratch, dropping the environment bin dir → bare python = system python.
2. Interactive rc — any interactive bash (-i or -l) sources /etc/bash.bashrc, which on serverless runs activate_root_python_environment.sh and prepends .../cluster_libraries/python/bin ahead of the environment interpreter → bare python = cluster-libraries python.

The non-interactive -- <cmd> path was already correct (sshd runs it via $SHELL -c, sourcing neither file).

Fix (two coordinated changes)

• Client (client.go): launch an interactive non-login shell (bash -i) so /etc/profile is no longer re-sourced.
• Server (server.go): seed a guarded, idempotent ~/.bashrc snippet that re-prepends the environment interpreter's bin directory. ~/.bashrc is sourced after /etc/bash.bashrc, so it wins; /etc/bash.bashrc and /etc/profile.d are root-owned and not writable by the non-root serverless user, so ~/.bashrc is the only usable hook. The seed is best-effort — a write failure logs a warning and does not abort the tunnel (mirrors the existing /run/sshd handling).

Why server-side and not client-side

Each --name gets a fresh ephemeral $HOME, so a client-side edit wouldn't persist. A pure client-side re-prepend via bash --rcfile <(...) was rejected because the remote command is parsed by the login shell (/bin/sh/dash on some images), where process substitution is a syntax error and would break connect.

Scope

Applied on all compute, guarded by $DATABRICKS_VIRTUAL_ENV being set. The bootstrap sets that variable to sys.executable (an absolute path) on every compute type, so dirname always yields the correct interpreter's bin directory.

Testing

• go test ./experimental/ssh/... green, including new TestSeedEnvActivation (create / append / trailing-newline / idempotent-preserves-content / guard-present).
• bash -i assertions updated in client_internal_test.go.

Manually verified against real compute

Confirmed end-to-end across all three compute types — in each interactive session, which python resolved to the environment interpreter (dirname($DATABRICKS_VIRTUAL_ENV)) rather than the system or cluster-libraries python, and importing an environment-only package succeeded:

• Serverless CPU — with a custom --base-environment (cowsay); import cowsay worked.
• Serverless GPU (GPU_1xA10) — with the same custom --base-environment; import cowsay worked.
• Dedicated cluster (single-node) — bare which python resolved to the environment interpreter.

This pull request and its description were written by Isaac.

[github-actions]

Waiting for approval

Based on git history, these people are best suited to review:

• @ilia-db -- recent work in experimental/ssh/internal/client/, experimental/ssh/internal/server/, experimental/ssh/
• @pietern -- recent work in ./, experimental/ssh/, experimental/ssh/internal/client/
• @denik -- recent work in ./, experimental/ssh/, experimental/ssh/internal/client/

Eligible reviewers: @andrewnester, @janniklasrose, @renaudhartert-db, @shreyas-goenka, @simonfaltum

_{Suggestions based on git history. See OWNERS for ownership rules.}

[eng-dev-ecosystem-bot]

Integration test report

Commit: 9b622a5

Run: 29030568998

	Env	💚​RECOVERED	🙈​SKIP	✅​pass	🙈​skip	Time
💚​	aws linux	4	4	230	1077	5:00
💚​	aws windows	4	4	232	1075	7:00
💚​	aws-ucws linux	4	4	314	995	6:12
💚​	aws-ucws windows	4	4	316	993	7:17
💚​	azure linux	4	4	230	1076	4:54
💚​	azure windows	4	4	232	1074	6:56
💚​	azure-ucws linux	4	4	316	992	5:52
💚​	azure-ucws windows	4	4	318	990	7:21
💚​	gcp linux	4	4	229	1078	4:56
💚​	gcp windows	4	4	231	1076	7:09

8 interesting tests: 4 RECOVERED, 4 SKIP

	Test Name	aws linux	aws windows	aws-ucws linux	aws-ucws windows	azure linux	azure windows	azure-ucws linux	azure-ucws windows	gcp linux	gcp windows
💚​	TestAccept	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R
🙈​	TestAccept/bundle/invariant/no_drift	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🙈​	TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🙈​	TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🙈​	TestAccept/ssh/connection	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
💚​	TestFetchRepositoryInfoAPI_FromRepo	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R
💚​	TestFetchRepositoryInfoAPI_FromRepo/root	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R
💚​	TestFetchRepositoryInfoAPI_FromRepo/subdir	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R

Top 10 slowest tests (at least 2 minutes):

duration	env	testname
6:19	gcp windows	TestAccept
6:07	aws-ucws windows	TestAccept
5:58	azure windows	TestAccept
5:56	azure-ucws windows	TestAccept
5:54	aws windows	TestAccept
2:59	gcp linux	TestAccept
2:56	azure linux	TestAccept
2:54	aws linux	TestAccept
2:50	aws-ucws linux	TestAccept
2:47	azure-ucws linux	TestAccept