feat: add observability_pipelines resource type by michael-richey · Pull Request #725 · DataDog/datadog-sync-cli

michael-richey · 2026-09-25T21:02:06Z

Summary

Adds support for syncing Datadog Observability Pipelines as a new resource type (observability_pipelines), using the v2 API at /api/v2/obs-pipelines/pipelines.

Changes

New model datadog_sync/model/observability_pipelines.py — ObservabilityPipelines(BaseResource) with:
- LIST — paginated GET /api/v2/obs-pipelines/pipelines (uses client.paginated_request, matching the default PaginationConfig of page[size]/page[number] with response_list_accessor="data").
- Import — GET .../pipelines/{id} for the per-ID path; pass-through for the list path. Unwraps the {"data": ...} envelope.
- Create — POST .../pipelines with {"data": resource}. If the source id already exists in the destination map, delegates to update_resource (id-keyed dedup, mirroring logs_metrics).
- Update — PUT .../pipelines/{destination_id} with {"data": resource}. Re-injects the destination id into the resource before the PUT (PUT, not PATCH — matches the OP API and the precedent of logs_indexes/dashboards).
- Delete — DELETE .../pipelines/{destination_id}.
- Config — resource_mapping_key="id", excluded_attributes=["id"] (strips the server-assigned id from create payloads and from diff comparisons via prep_resource).
- No cross-resource resource_connections — an OP pipeline config (sources/destinations/processors) references external systems via embedded connection settings, not sync-cli-managed Datadog resource IDs.
Registration — datadog_sync/models/__init__.py imports ObservabilityPipelines so init_resources discovers it via models.__dict__.
--id-file support — added observability_pipelines to _ID_FILE_IMPORT_SUPPORTED_TYPES. The default import_resource(_id=...) does a real GET, and the default get_resources_by_ids classifies 404/429/5xx/403 without aborting (satisfies the continue-past-errors design).
README — added observability_pipelines to the supported-resources table.
Integration test stub — tests/integration/resources/test_observability_pipelines.py, skipped (OP pipelines hold downstream routing config; destructive cleanup against a real org is unsafe for CI).

Design decisions

Match by id (not attributes.name) — mirrors logs_metrics. Source and destination UUIDs differ, so the first sync of a pipeline always creates it on the destination; re-sync idempotency relies on persisted state. This means a pipeline deleted on the destination between syncs would be re-created rather than updated in place.
PUT for updates — the OP API uses PUT .../pipelines/{pipeline_id}, not PATCH. update_resource re-injects the destination id (stripped by prep_resource) before the PUT.
No --id-file state-load allowlist entry — the state key is the pipeline id (ID-derivable), but the state-load path scopes by --resources intersection, not by _ID_FILE_STATE_LOAD_SUPPORTED_TYPES. This matches the logs_metrics precedent (also absent from the state-load set).

Testing

tests/unit/test_observability_pipelines.py — config contract, registration via init_resources, paginated get_resources, import (pass-through + GET-by-id), create (POST + id-keyed dedup), update (PUT with destination id), delete, prep_resource id-stripping, no-op hooks.
tests/unit/test_observability_pipelines_id_file.py — allowlist membership (import set + union) and import_resource(_id=...) GET-path verification.
All unit tests pass (1483 passed, 8 skipped). Lint (ruff, black at line-length 120) clean.

Test plan

Unit tests pass
ruff and black clean
Integration test against a live org (deferred — requires cassette; stub is skipped)

Add support for syncing Datadog Observability Pipelines via the v2 /api/v2/obs-pipelines/pipelines API: - New ObservabilityPipelines model (BaseResource subclass) with paginated LIST, GET-by-id import, POST create, PUT update, and DELETE. - Match by id (resource_mapping_key='id'), mirroring logs_metrics. The server-assigned id is excluded from create/update payloads via excluded_attributes=['id']; update_resource re-injects the destination id before the PUT. - Register the model in models/__init__.py so init_resources discovers it. - Add 'observability_pipelines' to _ID_FILE_IMPORT_SUPPORTED_TYPES so import --id-file works (default import_resource(_id=...) does a real GET and get_resources_by_ids classifies 404/429/5xx without aborting). - Unit tests cover config contract, registration, all CRUD methods, prep_resource id-stripping, no-op hooks, and id-file allowlist membership.

- Add observability_pipelines row to the supported-resources table in README.md (alphabetical, between notebooks and powerpacks). - Add integration test stub (skipped — OP pipelines hold downstream routing config, so destructive cleanup against a real org is unsafe for CI).

michael-richey

Thanks for pushing this — the model/test coverage is solid overall, but I found one blocking issue before we merge.

Blocking: pagination config likely mismatches OP list response shape

ObservabilityPipelines.get_resources() uses client.paginated_request(client.get) with the default PaginationConfig.

The default remaining_func in custom_client.py expects resp["meta"]["page"]["total_count"]. For Observability Pipelines, the v2 API/SDK schema exposes list metadata as meta.totalCount (not meta.page.total_count).

That means this code can work for orgs with <100 pipelines (single page), but once a page is full it may hit remaining_func and raise on the missing meta.page key, aborting import/sync for this type.

Can we switch this model to an explicit PaginationConfig for the OP response shape and add a unit test that exercises the multi-page path (not just the paginated wrapper call contract)?

The default remaining_func in custom_client.py reads resp["meta"]["page"]["total_count"], but the Observability Pipelines v2 list endpoint returns total count as meta.totalCount (camelCase). For orgs with <100 pipelines the single-page early break hides this, but a full first page would raise KeyError on the missing meta.page key, aborting import/sync for this resource type. Add a custom PaginationConfig with a remaining_func that reads meta.totalCount (with a safe fallback to 0 if meta is absent, so pagination stops gracefully). Update get_resources to pass this config explicitly. Add unit tests for the custom remaining_func (multi-page arithmetic) and for verifying get_resources passes the custom config to paginated_request.

michael-richey · 2026-09-25T21:11:38Z

Good catch — you're right that the default remaining_func reads resp["meta"]["page"]["total_count"], which would raise KeyError on the OP API's meta.totalCount (camelCase) shape once a full page is returned.

Fixed in 171dc9c:

Added a custom PaginationConfig with a remaining_func (_op_remaining_func) that reads resp["meta"]["totalCount"] with a safe .get() fallback to 0 (so pagination stops gracefully if meta is absent, rather than raising).
get_resources now passes pagination_config=self.pagination_config explicitly to paginated_request, instead of relying on the client default.
Added unit tests:
- test_pagination_config_reads_meta_total_count — verifies the custom remaining_func arithmetic (totalCount=150, page_size=100 → remaining=50 after page 0, remaining=-50 after page 1).
- test_get_resources_passes_custom_pagination_config — verifies get_resources passes the model's pagination_config to the inner wrapper, not the client default.

All 1485 unit tests pass; ruff + black clean.

Copilot

Copilot review overview

🟡 Changes recommended

The state-load allowlist contract is incomplete, and all API-backed integration coverage is disabled.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1 · 1

Open (2)

Allow observability_pipelines in ID-file state loading · New
Add API-backed integration coverage for observability_pipelines · New

What changed in this PR

Adds Datadog Observability Pipelines as a syncable resource using the v2 API.

Changes:

Implements list, import, create, update, and delete operations.
Registers and documents the resource with --id-file support.
Adds unit tests and a skipped integration-test stub.

File	Description
`datadog_sync/model/observability_pipelines.py`	Implements the resource model and pagination.
`datadog_sync/models/__init__.py`	Registers the model.
`datadog_sync/utils/configuration.py`	Adds import ID-file support.
`README.md`	Lists the supported resource.
`tests/unit/test_observability_pipelines.py`	Tests model behavior.
`tests/unit/test_observability_pipelines_id_file.py`	Tests ID-file support.
`tests/integration/resources/test_observability_pipelines.py`	Adds a skipped integration-test stub.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Adding observability_pipelines to _ID_FILE_IMPORT_SUPPORTED_TYPES also adds it to the union _ID_FILE_SUPPORTED_TYPES, which _parse_id_file consults for both import and sync --minimize-reads --id-file. The established contract (documented in the dashboards precedent at tests/unit/test_dashboards_id_file.py:43-59) requires every ID-derivable type accepted on the state-load path to be explicitly listed in _ID_FILE_STATE_LOAD_SUPPORTED_TYPES, so the ID-derivability is audited rather than accepted incidentally via the union. The OP pipeline state key is the pipeline id (storage layout: resources/source/observability_pipelines.<id>.json), which is ID-derivable, so it qualifies. Add it to the state-load set and assert membership in a new unit test.

michael-richey · 2026-09-29T13:42:58Z

Note on the test-integrations failure: it is a pre-existing infrastructure timeout, not a code issue. The job exceeds the workflow's 60-minute timeout-minutes ceiling at the "Run integration tests" step (cancelled at 1h5m0s). The same scheduled run on main today (2026-09-29, run 36532942739) timed out identically, confirming this is environment-wide and unrelated to this PR. The observability_pipelines integration test is @pytest.mark.skip (destructive cleanup against a real org is unsafe, same rationale as test_spans_metrics.py), so this PR cannot affect the integration suite outcome. test-integrations is not a required status check — only devflow/mergegate is required, and it passes. All unit-test jobs (ubuntu, ubuntu-arm, macos, windows) and all build jobs pass.

Retriggering again would occupy the queue-mode integration-tests concurrency group (cancel-in-progress: false) for another hour and block other PRs, so I'm leaving it. Ready for human review.

…servability-pipelines

michael-richey added 2 commits September 25, 2026 17:01

michael-richey commented Sep 25, 2026

View reviewed changes

michael-richey marked this pull request as ready for review September 25, 2026 21:08

michael-richey requested a review from a team as a code owner September 25, 2026 21:08

michael-richey added the changelog/Added label Sep 25, 2026

michael-richey requested a balanced review from Copilot September 25, 2026 21:20

Copilot started reviewing on behalf of michael-richey September 25, 2026 21:20 View session

Copilot AI reviewed Sep 25, 2026

View reviewed changes

Comment thread datadog_sync/utils/configuration.py

Comment thread tests/integration/resources/test_observability_pipelines.py

michael-richey added 4 commits September 28, 2026 10:04

chore: retrigger CI (ubuntu-latest runner stuck)

82339d9

chore: retrigger CI (test jobs stuck on runner)

218333b

chore: retrigger CI (integration test suite timed out)

2d44c96

michael-richey added 3 commits September 29, 2026 09:43

chore: retrigger integration tests (queue cleared)

b2a8717

Merge remote-tracking branch 'origin/main' into michael.richey/add-ob…

90a3c83

…servability-pipelines

chore: retrigger integration tests (previous run was cancelled)

d5e6cf1

Navigation Menu

feat: add observability_pipelines resource type - #725

feat: add observability_pipelines resource type#725
michael-richey wants to merge 10 commits into
mainfrom
michael.richey/add-observability-pipelines

michael-richey commented Sep 25, 2026

Uh oh!

michael-richey left a comment

Uh oh!

michael-richey commented Sep 25, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

michael-richey commented Sep 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

michael-richey commented Sep 25, 2026

Summary

Changes

Design decisions

Testing

Test plan

Uh oh!

michael-richey left a comment

Choose a reason for hiding this comment

Blocking: pagination config likely mismatches OP list response shape

Uh oh!

michael-richey commented Sep 25, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

michael-richey commented Sep 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants