Skip to content

Phase 7: Connections — Postgres, S3 and MongoDB, set up in the app - #67

Merged
LikelyCause merged 46 commits into
mainfrom
phase-7
Nov 16, 2026
Merged

LikelyCause merged 46 commits into
mainfrom
phase-7

Conversation

@LikelyCause

Copy link
Copy Markdown
Owner

What this does

Connections: Postgres, S3 and MongoDB are set up in the app instead of in deployment config.

  • Saved connections are owned by projects. An owner creates, edits, tests and deletes them; contributors browse, preview and pull; viewers see the name and kind.
  • Secrets (passwords, keys, MongoDB URI) are encrypted with RAINFOREST_SECRET_KEY (Fernet, rotatable) and are write-only: no response, log or audit line carries one.
  • Address guard: each address is resolved, checked and pinned before connecting; S3 over https goes through an in-process checking proxy; the API, worker, database, MLflow, pod ranges and cloud metadata addresses are always closed, loopback and link-local by default.
  • Postgres is read-only (read-only transaction, one SELECT, timeouts).
  • The notebook client can list, browse, preview and pull from a connection.
  • Instance-wide connections (admin) with allowed projects and optional allow_cidrs for private ranges; personal spaces are never allowed.
  • RAINFOREST_MONGO_SOURCES is removed. Migration 042 adds the connection tables.
  • Deployments now need a secret key: Compose generates one into the data volume if RAINFOREST_SECRET_KEY is empty; the chart takes it from a secret.
  • UI: Connections page and form, browse/preview/pull drawer, admin instance-wide page, rail link, user guide and docs/connections.md.

Tests

  • API: 2595 passed (full suite in containers); ruff check and format clean
  • Client: 79 passed
  • UI: 599 vitest passed; typecheck, eslint (0 errors), prettier and build clean
  • End to end on a real Compose stack (--profile sources): MongoDB connection tested, browsed, previewed and pulled into a dataset that reached ready; Postgres project connection to a loopback/own-service host refused with 422 address_denied; instance-wide Postgres connection with the stack subnet allowed and an allowed project previews; BASE=... bash deploy/smoke.sh printed smoke: ok.

🤖 Generated with Claude Code

https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS

LikelyCause and others added 30 commits November 15, 2026 12:13
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
… and pinning parts, the redactor, the Source interface

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…NFOREST_MONGO_SOURCES removed

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…, locked rotation with a key-list guard, unguessable Mongo appnames, own-address and IPv6 checks, proxy limits, no secret key in serve, driver lint gaps

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…est's own services

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…es reported, URI options re-checked, one pull deadline

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…registered kinds are a clean answer

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…ses its URL when it runs

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…sing service name took seconds to fail)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…sses never shrink or open, denied Mongo test is a 422, one binding rule, secrets hidden while browsing and previewing

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…e cursor, type map)

PostgresSource: IP-pinned connect (hostaddr), every libpq parameter named so the server's PG* environment, passfile and
service file are never consulted, a required password, timeouts as session options, a read-only transaction checked
with SHOW. Browse schemas and tables, preview with an EXPLAIN estimate, pull a table or one SELECT through a named
cursor into Parquet. pg_types maps Postgres types to polars and nulls (and counts) values polars can't hold.
console.one_query_problem extracts the console's one-statement check. B2's stub Postgres kind is replaced by the real one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…; CSV, Parquet, JSON Lines, gzip)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…tches by cells, write test as a writer

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…onnection node

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…(…).wait() in the client

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…cts refused), gzip and CSV budget bounded while reading, URLs and signatures out of errors

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…y allows only the endpoint's port, gunzip errors caught, written bytes capped

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…refusals for ignored options, preview warnings, forced S3 key

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…anges; connection docs

A range inside loopback, link-local or the metadata addresses is refused when saved; a range changed alone is not
re-checked, so an allowance can always be taken away. (The code change landed inside 9cc6972.)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…ike public()

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
LikelyCause and others added 16 commits November 15, 2026 13:42
…pernets and mapped ranges refused, ranges audited, last test cleared

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…tale test result cleared

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
The existing tests that used the env registry, /api/sources, POST /api/datasets/connect, run_connect and
dataset.connect now use connections, POST /api/connections/{id}/pull, run_pull and source.pull; test_connections_flow
covers create, edit, test and delete per kind, name collisions, the listing, browse paging, a Mongo pull with labels,
match_on and refresh, lineage scope, the S3 bucket field and a pull from the personal space. The route matrix has a row
per connection route. The chart gives the API and worker a pod-reading Role with notebooks off or on, and the dev values
carry a throwaway secret key. deploy/smoke.sh saves, tests and deletes a connection to an unreachable host. Docs: the
connection modules, source.pull and migration 042, pulled datasets, the connection concept, the notebook client's
refusals. ruff is clean (fields.py was only unformatted).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…eview sample, S3 zero match

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…st the dataset connection readers as system helpers

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…and lineage

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…e source panel's unavailable-connection text

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…connection page

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…gh the address guard, for the tests that use them only

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
@LikelyCause
LikelyCause merged commit 3c41d6f into main Nov 16, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant