Skip to content

feat(kata): add selectable Kata workspaces - #166

Merged
Hadar301 merged 44 commits into
validatedpatterns-sandbox:mainfrom
Hadar301:codex/kata-mode
Nov 18, 2026
Merged

Hadar301 merged 44 commits into
validatedpatterns-sandbox:mainfrom
Hadar301:codex/kata-mode

Conversation

@Hadar301

@Hadar301 Hadar301 commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Add a separate deployment.mode: kata for APPENG-6912. RHDH, GitOps, and Make use the shared Kubernetes workspace lifecycle developed in PR #153. OpenShell worker Pods use an owner-approved Kata RuntimeClass. The gateway and supervisors use the default runtime.

PR #166 is the merge candidate for both the shared container lifecycle and Kata changes. It contains the shared changes once. PR #153 provides development history; it does not need to merge first.

Cluster preparation

  1. Install the Red Hat OpenShift sandboxed containers Operator from OperatorHub. Check its Subscription and Succeeded CSV.
  2. Select KVM-capable workers. Apply a platform-owned KataConfig YAML for that pool. Wait for the node installation and required reboots.
  3. Review the generated kata RuntimeClass handler, worker selector, overhead, and UID. Approve its placement digest in the owner site Secret.
  4. Build the namespace-accessible notebook image with the documented container-notebook-image target. Add its digest-pinned profile override and scoped ImageStream pull access to the site Secret. Approve the new profile and validation policy digests.
  5. Install the SAW pattern through the documented path in docs/kata-workspaces.md. The SAW YAML binds Kata only to worker Pods and checks Sandbox and Pod admission. Keep the gateway and supervisors on the default runtime.

Gate and failure diagnosis

make saw-preflight performs read-only cluster checks. RHDH checks approved profile and platform inputs before it stores credentials or registers an owner. GitOps checks the owner and Kata platform before projecting an Application. Enrollment checks generated policy resources before it publishes the immutable runtime binding. A first registration has no generated policy to compare until Argo creates it.

The first live runs stopped at policy approval drift, gateway policy RBAC, and a stock notebook image that could not traverse /sandbox under the OpenShift namespace UID. Those causes are fixed or gated in this branch. The notebook image override also changes a literal image reference in the validation admission policy. The live policy's only approval difference was that image reference. The reviewed validation digest was updated. The portal generator reads the site Secret through an environment variable, so it had to restart before the new digest reached the owner Argo Application. The corrected enrollment Job then succeeded in the same registration.

Review fixes on 2026-11-18

  • F1: Preflight now accepts the documented providerAllowHosts input before GitOps derives providerPolicy. If a policy is already present, its hosts must match. Make, RHDH, and the GitOps projector use the same check.
  • F2: The Make helper reads only Helm ownership metadata. New resources use a normal server apply dry run. Resources owned by this release use --force-conflicts in the server dry run. A resource outside this release fails before apply. Field ownership errors get a safe, specific message.
  • The preflight and Make suites passed (78 tests). The full relevant suites passed before the final metadata query adjustment (265 tests); the two new RHDH provider-input cases passed. A chart render test confirms the generated provider policy matches the documented hosts. A live server dry run of an unchanged Helm-owned projector policy passed, and its resource version stayed unchanged.
  • The review's F2 dry-run reproducer now passes on an existing Helm-owned policy. The full two-stage Make workflow has not been repeated with this commit. CI is rerunning for head 75f6b88.

Kata canary on 2026-11-17

  • RHDH task 48f96ee5-ec90-48ff-ac3e-4623eeadd0a9 and PipelineRun saw-create-7xpcf succeeded. RHDH reported Workspace ready.
  • Both data-science worker Pods were Ready with runtimeClassName: kata. Their Pod UIDs mapped to different QEMU PIDs on the worker node.
  • Both supervisors and the gateway were Ready on the default runtime.
  • Both catalog chat links returned fresh NVIDIA model replies.

Remaining live coverage

Tool allow and deny rules, network denial, secret isolation, audit, persistence, restart, stop/start, deletion, GitOps and Make creation, a container regression, and two active owners remain unverified. These checks remain before full qualification.

pmatouse and others added 12 commits November 10, 2026 15:58
Add selectable container workspaces alongside the default VM backend, using
shared BOM/provider/tool-gate configuration and the stock OpenShell Kubernetes
driver. Unify RHDH, static GitOps and Make composition, enrollment, lifecycle,
credential isolation, owner browser access and retained audit handling.

Include offline contracts, build workflows, installation and architecture
guides, and the bounded OpenAI/data-science qualification matrix. Live checks
covered container lifecycle and fresh-workspace replay plus VM inference,
chat, audited tools, model update/restart persistence and RHDH deletion.

Developed with assistance from OpenAI Codex.
Expose Kata in Make, GitOps and Developer Hub. Bind each workspace to a reviewed RuntimeClass, validate sandbox placement, and reuse the container lifecycle for enrollment, browser chat and restart.
Pin built-in NemoClaw references to the verified multi-arch image index so Kata accepts enabled sandbox images.
Regenerate the RHDH and GitOps catalogs from the pinned profile images so both entry points use the same immutable references.
Report firmware for context while requiring KVM on selected workers.

Red Hat docs do not require UEFI for plain Kata.
@Hadar301 Hadar301 self-assigned this Nov 12, 2026
@Hadar301
Hadar301 requested a review from sauagarwa November 12, 2026 21:05
pmatouse and others added 14 commits November 16, 2026 18:19
Retain the shared Rust admission extension for VM and Kubernetes driver configuration, the deployment-mode Make settings, and the PR image publication checks.
Document the OperatorHub and KataConfig setup used for the
qualification cluster. Keep platform resources outside SAW charts.
Keep the configured observer and grant the RHDH generator and portal\nprovisioner read access to the workspace status ConfigMaps. Limit the\nrole to get on the three named ConfigMaps.
Pin the cluster-local interceptor image built from 1cdc9d4 for the live Kata test.
Use the selected OpenShell gateway version in the runtime plan.

Keep the enrollment default aligned with the gateway image pin.
Return fixed reasons for approval, identity, handler, scheduling,
label, and placement failures. Align RHDH and enrollment status output.
Read the owner-scoped public CA ConfigMap through the Kubernetes API
from enrollment and controller pods.

Limit both RBAC grants to that ConfigMap. Remove cross-namespace CA
volume mounts.
The identity chart writes the public CA ConfigMap in the
registration control namespace. Read it there and scope both
ServiceAccounts to that named ConfigMap.
Check approved live admission hashes before portal credentials or Make\napplies a workspace. Require complete hashes for first registrations\nand check generated policies during enrollment.\n\nTest the shared preflight, portal, Make, and projection paths.
Check the approved Kata runtime and existing admission boundaries before
ApplicationSet emits a workspace Application. Grant only the required read
access. Keep first-registration checks in enrollment.
OpenShell 0.1.6 adds OPENSHELL_GATEWAY_TLS_SERVER_NAME to the
supervisor Pod when the gateway endpoint uses HTTPS. Allow this
single stock variable in the strict runtime admission rule.
The OpenShell gateway adds OPENSHELL_GATEWAY_NAME to each supervisor.
Allow that stock variable in the supervisor admission contract.
The stock shared namespace driver creates and reads per-sandbox
NetworkPolicies. Give the gateway those verbs in its namespace.

Keep the grant in the chart so GitOps can apply it.
Reject the pinned OpenClaw image when an OpenShift namespace UID
cannot traverse its workspace directory. Require a digest-pinned
profile override before RHDH, GitOps, or Make provisions a workspace.

Record the documented image build and the live canary failure.
Record the successful RHDH Kata canary and the remaining checks.
Explain that an image override changes the validation policy digest
and that the portal generator must reload its site Secret.
@Hadar301
Hadar301 marked this pull request as ready for review November 17, 2026 11:54
@Hadar301
Hadar301 marked this pull request as draft November 17, 2026 11:57
@Hadar301
Hadar301 marked this pull request as ready for review November 17, 2026 11:58

@mtalvi mtalvi left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Recommendation: request changes for F1 and F2. Functional CLI checks passed on both actual Kata workloads, but full qualification is incomplete. Testing ended at the user's request. This report reviews PR #166 as the sole merge candidate, including the shared container implementation it delivers. PR #153 is historical context and is not a prerequisite merge.

Reviewed head: 5760c1910b6844c3412471c9560ece747a0cddb1. Target snapshot: 561dc781a6c6ab36a86dcffe5b597954cd26908d (an ancestor of the head). Review date: 2026-11-17. Deployment: cluster-dm6p7, OpenShift 4.22.22, amd64. The source is unchanged; site configuration is isolated on codex/pr166-review-dm6p7 in the operator's fork.

Final refresh confirmed the same head/base, PR still open, eleven successful CI checks and four skipped checks. Site-only HEAD is e0a5923e4dbeafc5e4525a9a3563e11e33987873; the candidate checkout is clean. See final PR metadata.

Findings and qualification blockers

F1 — P2: documented provider inputs are rejected before the policy is derived

The first-install guide explicitly says not to author providerPolicy; saw-users/templates/applications.yaml derives it from the governed catalog and operator-owned providerAllowHosts. However, the shared configuration_facts() requires site.providerPolicy.allowHosts to exist before that derivation. Both the Make preflight and RHDH preflight call it on the raw site. A site following the guide is therefore rejected with ProviderEgressRejected, even with the approved NVIDIA host explicitly configured.

Locations: shared preflight, lines 236–241, Make caller, RHDH caller, and installation instruction.

Reproduced against the fresh cluster by removing only providerPolicy from the otherwise complete owner configuration and invoking the candidate's read-only Make preflight. It fails with SAW preflight failed: ProviderEgressRejected before registration.
Derive the catalog policy before checking it, or make the preflight consume the documented operator inputs while preserving the eventual generated-policy checks. Add a first-install test that omits the generated field. Existing fixture-based tests contain a manually populated providerPolicy and miss this case. For continued qualification, this run explicitly retains the matching NVIDIA policy in private site inputs; this is a documented setup workaround, not a fix to the candidate.

F2 — P2: the documented second Make stage fails against its own Helm-owned policy

After a successful foundation install, enabling the inspected approval and rerunning make saw-create fails before Helm upgrade. The helper performs a server-side apply dry run over resources that the first stage installed through Helm. Kubernetes rejects the unchanged projector credential admission policy with conflict with "helm" … .spec.matchConstraints; its API-defaulted atomic match constraints are owned by Helm. The helper reduces this to oc operation failed, hiding the actionable cause. This blocks the documented two-stage Make workflow and subsequent read-only preflight against that foundation.

Location: server-side dry run before Helm upgrade, line 355. Minimal reproduction: export the already-installed saw-projector-a343914741091c751a27-credentials policy from helm get manifest workspace-pr166a -n saw-workspaces-pr166, then submit it with oc apply --server-side --dry-run=server -f <policy>. Even the unchanged manifest conflicts; selecting field manager helm also conflicts with its existing Update ownership.

Accept operator provider hosts before GitOps derives policy.

Dry-run Helm-owned resources without field manager conflicts, while
rejecting resources outside the release. Add regression coverage.
@Hadar301
Hadar301 requested a review from mtalvi November 18, 2026 08:22

@mtalvi mtalvi left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@Hadar301
Hadar301 merged commit f582a11 into validatedpatterns-sandbox:main Nov 18, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants