Repository navigation
Phase 7: Connections — Postgres, S3 and MongoDB, set up in the app - #67
Merged
Merged
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…ation Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
… and pinning parts, the redactor, the Source interface Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…NFOREST_MONGO_SOURCES removed Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…, locked rotation with a key-list guard, unguessable Mongo appnames, own-address and IPv6 checks, proxy limits, no secret key in serve, driver lint gaps Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…est's own services Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…es reported, URI options re-checked, one pull deadline Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…registered kinds are a clean answer Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…ses its URL when it runs Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…sing service name took seconds to fail) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…sses never shrink or open, denied Mongo test is a 422, one binding rule, secrets hidden while browsing and previewing Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…e cursor, type map) PostgresSource: IP-pinned connect (hostaddr), every libpq parameter named so the server's PG* environment, passfile and service file are never consulted, a required password, timeouts as session options, a read-only transaction checked with SHOW. Browse schemas and tables, preview with an EXPLAIN estimate, pull a table or one SELECT through a named cursor into Parquet. pg_types maps Postgres types to polars and nulls (and counts) values polars can't hold. console.one_query_problem extracts the console's one-statement check. B2's stub Postgres kind is replaced by the real one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…; CSV, Parquet, JSON Lines, gzip) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…tches by cells, write test as a writer Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…onnection node Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…(…).wait() in the client Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…cts refused), gzip and CSV budget bounded while reading, URLs and signatures out of errors Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…y allows only the endpoint's port, gunzip errors caught, written bytes capped Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…refusals for ignored options, preview warnings, forced S3 key Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…anges; connection docs A range inside loopback, link-local or the metadata addresses is refused when saved; a range changed alone is not re-checked, so an allowance can always be taken away. (The code change landed inside 9cc6972.) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…ike public() Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…pernets and mapped ranges refused, ranges audited, last test cleared Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…tale test result cleared Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
The existing tests that used the env registry, /api/sources, POST /api/datasets/connect, run_connect and
dataset.connect now use connections, POST /api/connections/{id}/pull, run_pull and source.pull; test_connections_flow
covers create, edit, test and delete per kind, name collisions, the listing, browse paging, a Mongo pull with labels,
match_on and refresh, lineage scope, the S3 bucket field and a pull from the personal space. The route matrix has a row
per connection route. The chart gives the API and worker a pod-reading Role with notebooks off or on, and the dev values
carry a throwaway secret key. deploy/smoke.sh saves, tests and deletes a connection to an unreachable host. Docs: the
connection modules, source.pull and migration 042, pulled datasets, the connection concept, the notebook client's
refusals. ruff is clean (fields.py was only unformatted).
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…eview sample, S3 zero match Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…st the dataset connection readers as system helpers Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…and lineage Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…e source panel's unavailable-connection text Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…connection page Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
…gh the address guard, for the tests that use them only Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Connections: Postgres, S3 and MongoDB are set up in the app instead of in deployment config.
RAINFOREST_SECRET_KEY(Fernet, rotatable) and are write-only: no response, log or audit line carries one.allow_cidrsfor private ranges; personal spaces are never allowed.RAINFOREST_MONGO_SOURCESis removed. Migration 042 adds the connection tables.RAINFOREST_SECRET_KEYis empty; the chart takes it from a secret.docs/connections.md.Tests
--profile sources): MongoDB connection tested, browsed, previewed and pulled into a dataset that reached ready; Postgres project connection to a loopback/own-service host refused with 422address_denied; instance-wide Postgres connection with the stack subnet allowed and an allowed project previews;BASE=... bash deploy/smoke.shprintedsmoke: ok.🤖 Generated with Claude Code
https://claude.ai/code/session_01LqNnEU86Q4fmrJnN2rXovS