feat(bundles): submit a stack together with the files it reads

CLAUDE.md

@@ -65,6 +65,16 @@ The DSL supports: Kafka, HTTP servers, SQL databases (PostgreSQL/MySQL/Oracle wi
 - Default imports provide common utilities
 - Components implement `InfrastructureComponent<*>` interface
 
+### Stack bundles
+- A stack can be submitted with the files it reads (`StackBundle` in nebula-api)
+  via `POST /stacks/bundle`, `PUT /stacks/bundle/{id}`, or the `bundles` field of
+  the `/stream/stacks` websocket message. The script-only routes are unchanged.
+- Resources unpack into a per-submission temp directory. `StackResources`
+  (nebula-dsl) confines every lookup to that directory; scripts reach it through
+  `resources` inside `stack { }`, and providers that read files resolve relative
+  paths through `StackResources.resolveFilePath`.
+- See `docs/pages/bundles.mdx` and §8 of `NEBULA-AGENT.md`.
+
 ### Testing
 - Uses Kotest framework with descriptive specifications
 - TestContainers for integration testing

NEBULA-AGENT.md

@@ -30,7 +30,7 @@ version: 0.1.0
 sourceRoot: src/
 additionalSources: {
    "@orbital/config" : "orbital/config/*.conf",
-   "@orbital/nebula" : "orbital/nebula/*.nebula.kts"
+   "@orbital/nebula" : "orbital/nebula/**"
 }
 dependencies: {
    "com.orbitalhq/core" : "github:orbitalapi/orbital-core-taxi#0.34.0"
@@ -40,8 +40,8 @@ dependencies: {
 **Rules:**
 - The `@orbital/nebula` key under `additionalSources` is **mandatory**. Without it Orbital will not discover the stack.
 - By convention the file lives at `orbital/nebula/stack.nebula.kts` (preferred) or `nebula/stack.nebula.kts`.
-- Declare the glob with a wildcard (`*.nebula.kts`), not a single explicit file — future stacks in the same folder will be picked up automatically.
-- If `taxi.conf` already exists and lacks the `@orbital/nebula` entry, add it. Do not remove unrelated `additionalSources` keys.
+- **Declare the glob as `orbital/nebula/**`**, not `orbital/nebula/*.nebula.kts`. The whole directory is the stack's bundle: `*.nebula.kts` files are stacks, and every other file is a resource shipped alongside them (see §8). The narrower `*.nebula.kts` glob still works, but then the stack ships no files.
+- If `taxi.conf` already exists with the older `orbital/nebula/*.nebula.kts` entry, widen it to `orbital/nebula/**` when the stack needs to read a file. Do not remove unrelated `additionalSources` keys.
 
 **Minimal project layout:**
 
@@ -52,7 +52,9 @@ my-project/
 │   └── ... (taxi files)
 └── orbital/
     └── nebula/
-        └── stack.nebula.kts
+        ├── stack.nebula.kts
+        └── data/
+            └── sales.csv        <- shipped with the stack, see §8
 ```
 
 ---
@@ -335,7 +337,8 @@ Empire Strikes Back,LXILO,189
 """)
    }
    bucket("reports") {
-      file("/absolute/path/to/local.csv")          // read from disk
+      file("data/sales.csv")                       // shipped with the stack (see §8)
+      file("/absolute/path/to/local.csv")          // read from the Nebula server's disk
       file("large.csv", generateSequence())        // Sequence<String> for large/streamed files
    }
 }
@@ -352,9 +355,11 @@ Inside a `bucket(name)` block:
 ```kotlin
 fun file(name: String, content: String)         // inline content
 fun file(name: String, content: Sequence<String>) // streaming — for large files
-fun file(path: String)                           // read file from local disk; uses the filename as the S3 key
+fun file(path: String)                           // read file from disk; uses the filename as the S3 key
 ```
 
+`file(path)` resolves a **relative** path against the files shipped with the stack (§8), and an **absolute** path against the Nebula server's own filesystem. Prefer the relative form — the Nebula server is a different machine from the project, so an absolute path almost never points where you think it does.
+
 Use the `Sequence<String>` form for files larger than a few MB (the seeder uses S3 multipart upload — minimum part size is 5 MB, which is already handled by default).
 
 ### 5.6 `hazelcast { ... }` — Hazelcast
@@ -507,10 +512,88 @@ stack {
 
 ---
 
-## 8. Quick checklist before returning a stack
+## 8. Shipping files with a stack
+
+A stack often needs a file: an OpenAPI spec to import, a CSV to seed a bucket, a
+fixture to serve. **Those files go in the project's nebula directory, next to the
+script**, and are submitted to Nebula along with it.
+
+This matters because the Nebula server is a different machine from the project.
+A path like `/home/jane/project/sales.csv` in a script means *Nebula's*
+filesystem, where the file does not exist.
+
+### Layout
+
+Everything under the `@orbital/nebula` directory that is **not** a `.nebula.kts`
+file is a resource. Resources are shared by every stack in that directory.
+
+```
+orbital/nebula/
+├── stack.nebula.kts
+├── data/
+│   └── sales.csv
+└── specs/
+    └── orders.yaml
+```
+
+This requires the wider glob in `taxi.conf` (§2):
+
+```hocon
+"@orbital/nebula" : "orbital/nebula/**"
+```
+
+With the older `orbital/nebula/*.nebula.kts` glob the script is still found, but
+no resources are shipped and `resources` will be empty.
+
+### Reading them
+
+Inside `stack { }`, `resources` gives you the files:
+
+```kotlin
+stack {
+   val sales = resources.readText("data/sales.csv")
+
+   http {
+      get("/sales") { call -> call.respondText(sales) }
+   }
+}
+```
+
+| Function | Returns |
+|---|---|
+| `resources.readText(path)` | the file's text, as UTF-8 |
+| `resources.readBytes(path)` | the file's bytes |
+| `resources.path(path)` | a `java.nio.file.Path` to the file |
+| `resources.exists(path)` | whether the stack shipped that file |
+| `resources.names` | every file the stack shipped |
+| `resources.hasBundle` | whether this stack was submitted with files at all |
+
+Paths are relative to the nebula directory — `data/sales.csv` above, not
+`orbital/nebula/data/sales.csv`.
+
+Providers that read a file (`s3 { file(path) }`, and API Gateway's
+`restApiFromFile(...)`) resolve a **relative** path the same way, so you can pass
+`"data/sales.csv"` straight to them.
+
+**Rules:**
+- Use relative paths. Absolute paths mean the Nebula server's own filesystem, and
+  almost never point where you want.
+- A path may not contain `..`, and may not escape the nebula directory. Both are
+  rejected, so don't try to reach a file elsewhere in the project — copy it into
+  the nebula directory instead.
+- Limits: 10 MB per file, 50 MB per stack, 500 files. For anything larger, mount
+  a volume into the Nebula container.
+- Resources are text by default. Binary files work, but Orbital ships a project's
+  files as text, so prefer text formats (CSV, JSON, YAML) for stacks Orbital
+  submits.
+
+---
+
+## 9. Quick checklist before returning a stack
 
 - [ ] File is named `stack.nebula.kts` (unless user specified otherwise) and placed at `orbital/nebula/stack.nebula.kts`.
-- [ ] `taxi.conf` contains `"@orbital/nebula" : "orbital/nebula/*.nebula.kts"` under `additionalSources`.
+- [ ] `taxi.conf` contains `"@orbital/nebula" : "orbital/nebula/**"` under `additionalSources`.
+- [ ] Any file the stack reads sits under `orbital/nebula/`, and is read with a relative path (`resources.readText("data/sales.csv")`, `file("data/sales.csv")`) — never an absolute one.
 - [ ] Exactly one top-level `stack { }` block.
 - [ ] No re-imports of `stack`, `Random`, `io.ktor.server.request.*`, `io.ktor.server.response.*`.
 - [ ] Duplicate component types (two Postgres, two S3, etc.) each have distinct `componentName`s.

docs/pages/_meta.ts

@@ -1,6 +1,7 @@
 export default {
     "index": "Getting started",
     "stacks": "Working with stacks",
+    "bundles": "Shipping files with a stack",
     "cli": "CLI & networking",
     "logging" : "Logging",
     "examples": "Examples",

docs/pages/bundles.mdx

@@ -0,0 +1,152 @@
+---
+title: Shipping files with a stack
+---
+
+## Shipping files with a stack
+
+A stack script often needs a file: an OpenAPI spec to import into API Gateway, a
+CSV to seed an S3 bucket, a fixture to serve from an HTTP endpoint.
+
+Those files live in the project that defines the stack — on a developer's laptop,
+or in a git repo that Orbital has checked out. The Nebula server is somewhere
+else entirely, usually a different container. A path in a script like
+`file("/home/jane/project/sales.csv")` means *the Nebula server's* filesystem,
+which is not where the file is.
+
+A **stack bundle** fixes that. It is the script plus the files it reads,
+submitted together. Nebula unpacks the files into a directory belonging to that
+submission, and the stack reads them from there.
+
+### Submitting a bundle
+
+`POST /stacks/bundle` creates a stack; `PUT /stacks/bundle/{id}` replaces one.
+Both take JSON:
+
+```json
+{
+  "script": "stack {\n   s3 { bucket(\"tickets\") { file(\"data/sales.csv\") } }\n}",
+  "resources": {
+    "data/sales.csv": { "content": "Title,Tickets\nA New Hope,249\n" },
+    "specs/orders.yaml": { "content": "openapi: 3.0.0\n..." }
+  }
+}
+```
+
+```bash
+curl --location 'http://localhost:8099/stacks/bundle' \
+  --header 'Content-Type: application/json' \
+  --data @bundle.json
+```
+
+The script-only routes (`POST /stacks`, `PUT /stacks/{id}`) are unchanged, and a
+bundle with no `resources` behaves exactly as they do.
+
+Orbital submits bundles over the `/stream/stacks` websocket rather than over
+HTTP. That message accepts both shapes:
+
+```json
+{ "stacks":  { "films": "stack {}" } }
+{ "bundles": { "films": { "script": "stack {}", "resources": { ... } } } }
+```
+
+### Resource encoding
+
+Each resource is `{ "content": "...", "encoding": "TEXT" | "BASE64" }`.
+`encoding` defaults to `TEXT`, where the content is the file's text and is
+written out as UTF-8. Binary files (a `.parquet`, an image) are sent as `BASE64`.
+
+Orbital only ever sends `TEXT`: the files reach it through the schema pipeline,
+which carries a project's additional sources as text. Submit `BASE64` directly if
+you need binary content.
+
+### Limits
+
+| Limit | Value |
+|---|---|
+| Largest single file | 10 MB |
+| Largest bundle | 50 MB |
+| Files per bundle | 500 |
+
+Exceeding any of these fails the submission with a message naming the file and
+the limit. Bundles are for fixtures and specs — if you need gigabytes of seed
+data, mount a volume into the Nebula container instead.
+
+### Reading resources from a script
+
+Inside `stack { }`, `resources` gives you the files that were shipped:
+
+```kotlin
+stack {
+   val sales = resources.readText("data/sales.csv")
+   val spec = resources.readText("specs/orders.yaml")
+
+   http {
+      get("/sales") { call -> call.respondText(sales) }
+   }
+}
+```
+
+| Function | Returns |
+|---|---|
+| `resources.readText(path)` | the file's text, as UTF-8 |
+| `resources.readBytes(path)` | the file's bytes |
+| `resources.path(path)` | a `java.nio.file.Path` to the unpacked file |
+| `resources.exists(path)` | whether the stack shipped that file |
+| `resources.names` | every file the stack shipped, as relative paths |
+| `resources.hasBundle` | whether this stack was submitted with a bundle |
+
+Paths are relative to the bundle root, which is the project's nebula directory —
+so `data/sales.csv` in the script means `orbital/nebula/data/sales.csv` in the
+project.
+
+### Providers that read files
+
+Any provider that reads a file resolves a **relative** path through the bundle:
+
+```kotlin
+stack {
+   s3 {
+      bucket("tickets") {
+         file("data/sales.csv")        // read from the bundle
+         file("/mnt/shared/big.csv")   // read from the Nebula server, as before
+      }
+   }
+}
+```
+
+An **absolute** path keeps its existing meaning — a file on the machine running
+Nebula. Stacks written before bundles existed, and stacks run from the CLI
+against local files, are unaffected.
+
+Resolution happens while the script is compiled, so a typo in a resource name
+fails the submission with a message listing what the stack actually shipped,
+rather than failing later, mid-startup.
+
+### What a stack may read
+
+Every resource lookup is confined to the bundle directory:
+
+- absolute paths are rejected,
+- `..` segments are rejected,
+- the resolved path is turned into a real path — following symlinks — and
+  re-checked to still be inside the bundle.
+
+So a bundle entry named `../../etc/cron.d/evil` is rejected before anything is
+written, and a symlink planted inside a bundle that points at
+`/etc/ssh/id_rsa` is refused when the stack tries to read it.
+
+This confines *resource* access. A Nebula script is arbitrary Kotlin and can
+always open a file directly; the guarantee is that a bundle-relative path
+resolves to a file inside the bundle, nothing more and nothing less. Only submit
+stacks you trust to a Nebula server.
+
+### Lifetime
+
+Each submission gets its own directory under the system temp directory. It is
+deleted when the stack is replaced by a different submission, when the stack is
+removed, and when a submission is rejected. Stopping a stack keeps its bundle,
+because a stopped stack can be started again.
+
+A stack is *replaced* when its script **or** any of its resources change — so
+editing a CSV next to an unchanged script restarts the stack, rather than being
+skipped as a duplicate submission.

docs/pages/providers/s3.mdx

@@ -16,6 +16,26 @@ stack {
 }
 ```
 
+### Seeding a bucket from a file
+
+`file(path)` uploads a file from disk, using its filename as the S3 key.
+
+```kotlin
+stack {
+    s3 {
+        bucket("tickets") {
+            file("data/sales.csv")       // shipped with the stack
+            file("/mnt/shared/big.csv")  // on the machine running Nebula
+        }
+    }
+}
+```
+
+A **relative** path is read from the bundle submitted with the stack, so the CSV
+travels with the project rather than having to exist on the Nebula server — see
+[Shipping files with a stack](/bundles). An **absolute** path is read from the
+machine running Nebula, which is what it has always meant.
+
 ### Returned values
 When an `s3` component is declared, the following data is returned:
 

docs/pages/stacks.mdx

@@ -32,6 +32,13 @@ curl --location 'http://localhost:8099/stacks' \
 }'
 ```
 
+### Creating a stack that ships files
+
+A script that reads a file — a CSV to seed a bucket, an OpenAPI spec — needs that
+file to travel with it, because the Nebula server's filesystem is not the
+project's. POST a bundle to `/stacks/bundle` instead, and read the files through
+`resources` inside the script. See [Shipping files with a stack](/bundles).
+
 ### Listing the state of current stacks
 Fetch the current stacks (including their configured variables) by GET `/stacks`