Repository navigation
feat(bundles): submit a stack together with the files it reads - #70
Merged
Merged
Conversation
A stack script that needs an OpenAPI spec or a CSV could only read them
from the Nebula server's own filesystem, which is a different machine
from the project that defines the stack. Stacks that read files were
therefore only usable when someone had put the file on the server by
hand.
A stack can now be submitted as a bundle: the script plus a map of
relative path -> content. Nebula unpacks the resources into a directory
belonging to that submission, and the stack reads them through
`resources` inside `stack { }`. Providers that read a file (`s3 {
file(path) }`, and API Gateway's `restApiFromFile` when it lands)
resolve a relative path through the bundle; an absolute path keeps its
existing meaning, so stacks written before bundles are unaffected.
Additive throughout:
- `POST /stacks/bundle` and `PUT /stacks/bundle/{id}` sit alongside the
script-only routes, which are unchanged.
- The `/stream/stacks` message gained a `bundles` field next to
`stacks`. Both default to empty and normalise to the same thing, so an
Orbital that predates bundles keeps working, and a bundle with no
resources behaves exactly as a script-only submission does.
Every resource lookup is confined to the bundle directory: absolute
paths and `..` segments are rejected, and the resolved path is turned
into a real path (following symlinks) and re-checked to still be inside
the bundle. The same check runs at unpack time, so a `../../etc/cron.d`
entry is refused before anything is written. Size limits (10MB per file,
50MB per bundle, 500 files) are checked against the decoded bytes before
any file is created, and a rejected submission leaves nothing on disk.
Resources are text by default, with a BASE64 encoding flag for binary
content. Orbital only sends text, because the files reach it through the
schema pipeline, which carries additional sources as text.
StackRunner now dedupes on the script *and* the resource content, so
editing a CSV next to an unchanged script restarts the stack rather than
being skipped as a duplicate. A stack's unpacked bundle is deleted when
it is replaced or removed; stopping a stack keeps it, since a stopped
stack can be started again.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011c7t8dqC37eQzTt9o44x9v
✅ Deploy Preview for orbital-nebula ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
A stack script that reads a file —
s3 { bucket("tickets") { file(path) } }, or API Gateway's upcomingrestApiFromFile(...)— could only read it from the Nebula server's own filesystem. The project that defines the stack lives somewhere else, so the path pointed at a machine where the file does not exist. Stacks that read files were only usable if someone had put the file on the server by hand.What this adds
A stack can now be submitted as a bundle: the script plus a map of relative path → content. Nebula unpacks the resources into a directory belonging to that submission, and the stack reads them through
resourcesinsidestack { }.stack { val sales = resources.readText("data/sales.csv") s3 { bucket("tickets") { file("data/sales.csv") // resolved through the bundle } } }API
Additive throughout — nothing existing changed shape.
POST /stacks/bundleandPUT /stacks/bundle/{id}sit alongside the script-only routes.The
/stream/stacksmessage gained abundlesfield next tostacks:{ "stacks": { "films": "stack {}" } } { "bundles": { "films": { "script": "stack {}", "resources": { "data/sales.csv": { "content": "...", "encoding": "TEXT" } } } } }Both default to empty and normalise to the same type, so an Orbital that predates bundles keeps working. A bundle with no resources behaves exactly as a script-only submission does — no directory is created.
Resources are text by default, with a
BASE64flag for binary content. Orbital only sends text, because a project's files reach it through the schema pipeline as UTF-8 strings.Safety
Every resource lookup is confined to the bundle directory:
..segments rejected,toRealPath()'d — following symlinks — and re-checked to still be inside the bundle.The same check runs at unpack time against the entry names, so a
../../etc/cron.d/evilentry is refused before anything is written (zip-slip), and a rejected bundle leaves nothing on disk. Size limits — 10MB per file, 50MB per bundle, 500 files — are checked against the decoded bytes before any file is created.This confines resource access. A Nebula script is arbitrary Kotlin and can always open a file directly; the guarantee is that a bundle-relative path resolves inside the bundle. That is what protects against a malformed or hostile resource name, which is the part that comes from outside.
Lifecycle
StackRunnernow dedupes on the script and the resource content, so resubmitting an unchanged script with an edited CSV restarts the stack rather than being skipped as a duplicate. An unpacked bundle is deleted when the stack is replaced or removed, and when a submission is rejected. Stopping a stack keeps its bundle, since a stopped stack can be started again.Tests
61 passing across
nebula-dsl(41) andnebula-runtime(20, 1 pre-existing skip):StackResourcesTest— the containment rules, including a symlink planted inside a bundle that points at a file outside it, and a symlinked directory.StackBundleUnpackerTest— hostile entry names, partial bundles leaving nothing behind, size limits, fingerprints.StackBundleExecutionTest— a bundle compiled through the real scripting host, running, and serving a file that shipped with it. This is the one that fails if the bundle is unpacked but never reaches the script.StreamStacksBundleTest— the same over the websocket Orbital actually uses.UpdateStackRequestCompatibilityTest— both message shapes deserialise and normalise identically.Existing
nebula-runtimeand non-containernebula-dsltests pass unchanged.S3ExecutorTesthas one failure on my machine from Docker network-pool exhaustion (all predefined address pools have been fully subnetted), unrelated to this change — the other two in that spec pass.Docs
docs/pages/bundles.mdx(new), plus §8 ofNEBULA-AGENT.md, the S3 provider page, andCLAUDE.md. The@orbital/nebulaconvention inNEBULA-AGENT.mdwidens fromorbital/nebula/*.nebula.ktstoorbital/nebula/**, so the files beside a script reach Orbital in the first place.Companion change
Orbital side:
feat/nebula-stack-bundles— https://gitlab.com/vyne/vyne/-/merge_requests/2231.🤖 Generated with Claude Code
https://claude.ai/code/session_011c7t8dqC37eQzTt9o44x9v