Scaffolding: Federation sync app creation by klpoland · Pull Request #298 · spectrumx/sds-code

klpoland · 2026-06-19T15:17:09Z

In this PR:

[NEW] federation/ (v1) application with:
- services:
  - bootstrap.py: for initialization of federated data from home site and peers (including when new peer is added)
  - fed_index.py: for handling (external peer) asset indexing on webhook events
  - local_events.py: for handling redis dispatches to fetch local data
  - peer registry.py: for registering new peers and holding site information from site hellos
  - peer_sync.py: for packaging and sending asset data to peer sites (through their webhook routes)
- routes:
  - health.py: (stub for now) for pinging site health (is connection established?)
  - webhooks.py: for receiving payloads FROM peers to index into LOCAL federated asset indices as well as a site-hello hook to confirm federation between peers
- schemas and models: for type checking and providing data structures to federation configs, events, and documents (how asset docs are represented by FederatedDatasetDoc, FederatedCaptureDoc may warrant further consideration re: anticipating schema flexibility)
- a main app that subscribes to redis events for picking up signals from gateway (handled in PR Scaffolding: Federation gateway setup #299)

Note

High Risk
New cross-site replication path with API-key minting, OpenSearch writes, and peer webhooks; misconfiguration or weak origin checks could affect federated metadata integrity across deployments.

Overview
Introduces a new federation/ FastAPI sync service that keeps federated dataset/capture metadata in shared OpenSearch indices and exchanges updates with peer sites.

Runtime flow: On startup, bootstrap mints or uses a gateway export Api-Key (FEDERATION_SYNC_DRF_TOKEN → get-federation-sync-api-key), pulls local and peer /federation/export/ lists into fed-* indices, then sends site-hello to configured peers. A Redis subscriber on site-scoped federation:events:* channels loads the local doc from OpenSearch (after gateway indexing) and POSTs dataset-updated / capture-updated webhooks to peers; inbound webhooks index peer docs with stale-event guards. Peer sync_service_url can be updated from site-hello; optional peer CA paths support TLS verification.

Ops & tooling: Docker/compose (local + prod), justfile, dev PKI script, /sync/health operational checks, and shared federation-shared.env docs for sync credentials. Repo pre-commit gains federation-specific ruff, deptry, and pyrefly hooks.

Tests: Broad pytest coverage (regression, integration, two-site mesh) without a live gateway.

^{Reviewed by Cursor Bugbot for commit a973a15. Bugbot is set up for automated code reviews on this repo. Configure here.}

semanticdiff-com · 2026-06-19T15:17:12Z

Review changes with

Changed Files

File	Status
.pre-commit-config.yaml	4% smaller
federation-shared.example.env	Unsupported file format
federation/.gitignore	Unsupported file format
federation/Dockerfile	Unsupported file format
federation/certs/.gitignore	Unsupported file format
federation/compose.local.yaml	0% smaller
federation/compose.production.yaml	0% smaller
federation/federation.example.toml	Unsupported file format
federation/justfile	Unsupported file format
federation/pyproject.toml	Unsupported file format
federation/scripts/env-selection.sh	Unsupported file format
federation/scripts/generate-dev-certs.sh	Unsupported file format
federation/scripts/simulate_redis_event.py	0% smaller
federation/sds_federation/__init__.py	0% smaller
federation/sds_federation/main.py	0% smaller
federation/sds_federation/models.py	0% smaller
federation/sds_federation/routes/__init__.py	0% smaller
federation/sds_federation/routes/health.py	0% smaller
federation/sds_federation/routes/webhooks.py	0% smaller
federation/sds_federation/schemas/__init__.py	0% smaller
federation/sds_federation/schemas/webhooks.py	0% smaller
federation/sds_federation/services/__init__.py	0% smaller
federation/sds_federation/services/bootstrap.py	0% smaller
federation/sds_federation/services/fed_index.py	0% smaller
federation/sds_federation/services/fed_search.py	0% smaller
federation/sds_federation/services/local_events.py	0% smaller
federation/sds_federation/services/operational.py	0% smaller
federation/sds_federation/services/peer_registry.py	0% smaller
federation/sds_federation/services/peer_sync.py	0% smaller
federation/sds_federation/services/redis_channel.py	0% smaller
federation/sds_federation/testing/__init__.py	0% smaller
federation/sds_federation/testing/sample_data.py	0% smaller
federation/tests/__init__.py	0% smaller
federation/tests/conftest.py	0% smaller
federation/tests/support/__init__.py	0% smaller
federation/tests/support/federation_mesh.py	0% smaller
federation/tests/support/gateway_export_mock.py	0% smaller
federation/tests/support/mock_opensearch.py	0% smaller
federation/tests/support/mock_peer_registry.py	0% smaller
federation/tests/test_integration_bootstrap.py	0% smaller
federation/tests/test_integration_mesh.py	0% smaller
federation/tests/test_integration_pipeline.py	0% smaller
federation/tests/test_integration_webhooks.py	0% smaller
federation/tests/test_operational.py	0% smaller
federation/tests/test_redis_event_pipeline.py	0% smaller
federation/tests/test_regression_bootstrap_mint.py	0% smaller
federation/tests/test_regression_fed_search.py	0% smaller
federation/tests/test_regression_indexer.py	0% smaller
federation/tests/test_regression_peer_sync.py	0% smaller
federation/tests/test_regression_redis_channel.py	0% smaller
federation/tests/test_regression_schemas.py	0% smaller
federation/uv.lock	Unsupported file format

lucaspar · 2026-06-25T20:02:39Z

I'll add more comments after those are addressed to reduce the noise

…rap services

…ig setup

…set sync events

…ed the same)

cursor

Cursor Bugbot has reviewed your changes using high effort and found 3 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit a973a15. Configure here.}

cursor · 2026-07-24T18:55:15Z

+def _resolve_gateway_api_key(peer: PeerInfo) -> str:
+    if peer.gateway_export_api_key:
+        return peer.gateway_export_api_key
+    return os.environ.get("FEDERATION_SYNC_SERVER_API_KEY", "").strip()


Peer bootstrap uses local API key

High Severity

The _resolve_gateway_api_key function incorrectly falls back to the local FEDERATION_SYNC_SERVER_API_KEY for remote peers when peer.gateway_export_api_key is not explicitly set. This causes 401 authentication errors when bootstrap_all_peers attempts to fetch metadata, as remote gateways expect their own specific API keys, preventing historical data synchronization.

^{Reviewed by Cursor Bugbot for commit a973a15. Configure here.}

cursor · 2026-07-24T18:55:15Z

+        hello = hello.model_copy(update={"timestamp": datetime.now(UTC)})
+
+    _peer_registry(request).register(hello)
+    return {"status": "registered", "site_name": hello.site_name}


Site-hello skips peer data backfill

Medium Severity

site-hello only updates the in-memory PeerRegistry. It never pulls the registering peer’s export lists into local fed-* indices. When an already-running site receives hello from a newly started peer, that peer’s existing public assets are not indexed until a full restart bootstrap.

^{Reviewed by Cursor Bugbot for commit a973a15. Configure here.}

cursor · 2026-07-24T18:55:15Z

+
+COPY pyproject.toml ./
+COPY sds_federation ./sds_federation
+RUN uv sync --no-dev


Docker build omits lockfile

Medium Severity

The image copies only pyproject.toml and source, then runs uv sync --no-dev without uv.lock or --frozen. Gateway Dockerfiles copy the lockfile and sync frozen; federation builds therefore resolve floating dependency versions and can diverge from tested resolves.

^{Reviewed by Cursor Bugbot for commit a973a15. Configure here.}

lucaspar · 2026-07-24T19:11:01Z

let's add a redeploy recipe for consistency with the other services; redeploy: build -> down -> up -> logs.

lucaspar · 2026-07-24T19:11:33Z

+    search-path      = ["sds_federation"]
+
+    python-platform = "linux"
+    python-version  = "3.13"


klpoland self-assigned this Jun 19, 2026

klpoland added feature New feature or request federation Federation related work (sync service, document storage, peer configuration, etc.) labels Jun 19, 2026

klpoland changed the title ~~Scaffolding: Federation Sync App~~ Scaffolding: Federation sync app creation Jun 19, 2026

klpoland marked this pull request as ready for review June 25, 2026 19:50

klpoland requested a review from lucaspar June 25, 2026 19:50

cursor Bot reviewed Jun 25, 2026

View reviewed changes

lucaspar requested changes Jun 25, 2026

View reviewed changes

Comment thread federation/pyproject.toml

Comment thread federation/pyproject.toml Outdated

cursor Bot reviewed Jun 26, 2026

View reviewed changes

Comment thread federation/sds_federation/services/local_events.py Outdated

Comment thread federation/sds_federation/services/peer_sync.py Outdated

Comment thread federation/sds_federation/services/fed_index.py

cursor Bot reviewed Jul 1, 2026

View reviewed changes

Comment thread federation/sds_federation/routes/webhooks.py

Comment thread federation/justfile Outdated

Comment thread federation/sds_federation/services/bootstrap.py

cursor Bot reviewed Jul 1, 2026

View reviewed changes

Comment thread federation/sds_federation/main.py Outdated

Comment thread federation/sds_federation/main.py

cursor Bot reviewed Jul 2, 2026

View reviewed changes

Comment thread federation/sds_federation/services/fed_index.py

cursor Bot reviewed Jul 9, 2026

View reviewed changes

Comment thread federation/sds_federation/services/local_events.py Outdated

Comment thread federation/sds_federation/services/fed_search.py

cursor Bot reviewed Jul 10, 2026

View reviewed changes

Comment thread federation/compose.local.yaml Outdated

Comment thread federation/sds_federation/services/fed_index.py

Comment thread federation/sds_federation/services/bootstrap.py

Comment thread federation/sds_federation/main.py

cursor Bot reviewed Jul 15, 2026

View reviewed changes

Comment thread federation/compose.local.yaml

Comment thread federation/compose.local.yaml

Comment thread federation/Dockerfile Outdated

cursor Bot reviewed Jul 16, 2026

View reviewed changes

Comment thread federation/sds_federation/services/bootstrap.py

Comment thread federation/sds_federation/routes/webhooks.py

Comment thread federation/sds_federation/services/fed_index.py

Comment thread federation/sds_federation/services/peer_registry.py Outdated

klpoland added 12 commits July 24, 2026 08:13

fastapi app init

6892050

app models, doc schemas, webhook endpoints, peer sync/registry/bootst…

3714149

…rap services

test setup first pass

bb0bc30

create federation transport mesh to mock end-to-end peer app sync

ba43150

remove deprecated vars from tests

c0f4574

integrate pyrefly

55941e0

build out health check stub to check redis/opensearch status and conf…

e06c546

…ig setup

cursor bot comments

eb28cd1

production vs local compose, add opensearch network

aa020ff

set federation toml as example file for new sites to cp

15829ac

follow rfc 12: site-prefixed redis channels

7aa3884

remove postgres db queries/dependencies, query opensearch only for as…

eb03556

…set sync events

klpoland added 5 commits July 24, 2026 08:13

add federation shared env to compose

509f345

align with gateway: remove event types, tombstones (all updates handl…

8557f39

…ed the same)

linting

462cda1

cursor comments

da3be39

config fixes

b4788db

klpoland force-pushed the feature-kpoland-federation-sync-scaffolding branch from 7bcbcba to b4788db Compare July 24, 2026 12:13

cursor Bot reviewed Jul 24, 2026

View reviewed changes

Comment thread federation/sds_federation/services/bootstrap.py Outdated

clean up sync refs/callables

345497b

cursor Bot reviewed Jul 24, 2026

View reviewed changes

Comment thread federation/sds_federation/services/peer_registry.py Outdated

Comment thread federation/sds_federation/services/bootstrap.py

Comment thread federation/scripts/local_e2e/.env Outdated

remove unused code

b415820

cursor Bot reviewed Jul 24, 2026

View reviewed changes

Comment thread federation/sds_federation/services/bootstrap.py Outdated

mint api key through sync init

a973a15

cursor Bot reviewed Jul 24, 2026

View reviewed changes

lucaspar approved these changes Jul 24, 2026

View reviewed changes

Uh oh!

Conversation

klpoland commented Jun 19, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

semanticdiff-com Bot commented Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lucaspar commented Jun 25, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor Bot Jul 24, 2026

Choose a reason for hiding this comment

Peer bootstrap uses local API key

Uh oh!

cursor Bot Jul 24, 2026

Choose a reason for hiding this comment

Site-hello skips peer data backfill

Uh oh!

cursor Bot Jul 24, 2026

Choose a reason for hiding this comment

Docker build omits lockfile

Uh oh!

lucaspar Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

lucaspar Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

klpoland commented Jun 19, 2026 •

edited by cursor Bot

Loading

semanticdiff-com Bot commented Jun 19, 2026 •

edited

Loading