Skip to content

feat: NiFi 2.12.0 support and parameter provider helpers - #415

Merged
Chaffelson merged 8 commits into
mainfrom
feature/nifi-2.12-parameter-providers
Oct 22, 2026
Merged

Chaffelson merged 8 commits into
mainfrom
feature/nifi-2.12-parameter-providers

Conversation

@Chaffelson

@Chaffelson Chaffelson commented Oct 19, 2026 •

Copy link
Copy Markdown
Owner

Summary

Uplifts nipyapi to NiFi/Registry 2.12.0 and adds generic helpers for parameter providers, context inheritance and controller-level services. Along the way, a few existing helpers are aligned with library conventions. This PR also carries the 1.8.0 release prep. Full details are in docs/history.rst (1.8.0, dated 2026-10-22) and docs/parameters.rst.

Version: 1.8.0 is a minor bump over 1.7.0. No public function is removed and no signature changes, but some existing functions change behaviour. They are listed under Behaviour Changes at the top of the 1.8.0 release notes, led by get_process_group, which is the highest-traffic of them.

Supported versions and generated clients

  • Both clients are regenerated against the 2.12.0 OpenAPI specs. The surface change is additive: no API methods or models were removed or renamed.
  • Template fixes:
    • ParameterGroupConfigurationEntity.parameter_sensitivities now validates map values rather than keys, and accepts an unset (None) sensitivity. Before this, provider fetch and apply responses could not be deserialized.
    • New Configuration.client_side_validation flag.

New helpers

  • nipyapi.parameters:
    • Provider list and get, type lookup, property update, fetch, apply, and a combined fetch-and-apply.
    • Apply keeps the staged sensitivity and synchronized choices unless the caller overrides them. Unset choices default to sensitive and synchronized, matching NiFi.
    • Apply rejects unknown parameter names in sensitivities, and logs a warning when no selected group is synchronized.
  • set_inherited_parameter_contexts (prepend or replace), get_process_group_parameter_context, and connect_parameter_provider_group.
    • connect_parameter_provider_group inherits a provider-created context and sets local sensitive aliases, in one atomic update.
    • If the source context is already inherited, it keeps its position unless a higher-precedence context shadows one of the mapped names.
    • It rejects non-sensitive source parameters, because NiFi would leave a sensitive alias to them unresolved.
  • canvas.list_all_management_controllers and canvas.get_management_controller, which return controller-level services only.

Behaviour changes to existing functions (full list in the release notes)

  • canvas.get_process_group returns None for a missing ID, instead of raising ValueError("Unable to locate group ..."). That is what its 404 handler intends, and matches get_processor and get_parameter_context. The nipyapi.ci commands already handled None; their error message is now Process group not found: <id>.
  • update_parameter_context (and the rename, upsert and delete-parameter helpers that call it) and verify_controller/verify_processor now share utils.wait_for_request:
    • a request-level failure raises ValueError;
    • a completed request is deleted, including a failed one;
    • a timeout or interruption no longer issues a DELETE, which would cancel the server-side work.
  • canvas.ensure_management_controller matches controller-level services only, and raises if several share the name.
  • The canvas.get_controller docstring is corrected; its behaviour is unchanged.
  • Version guards and branches for NiFi releases before 2.0 are removed. This also drops an extra /flow/about request on every parameter call.

Build tooling

  • make dist, wheel, sdist and install no longer delete the test certificates or coverage reports.
  • check-dist and test-dist reuse an existing build. They still build when dist/ is empty, which CI relies on.
  • rebuild-all failed after its own clean-all, because wait-ready imports nipyapi once the clients are gone. It now removes the generated code just before regenerating it.

Test plan

  • make rebuild-all completes:
    • certs, spec fetch and augmentation, client regeneration;
    • all three profiles: single-user 777 passed / 33 skipped, secure-ldap 784 / 26, secure-mtls 777 / 33;
    • dist build, twine check, the clean-venv import test, and docs.
  • The regenerated clients and the structured docs are byte-identical to the committed ones.
  • Behaviour is tested against the real NiFi 2.12.0 containers wherever the server can produce the condition. That includes the 409 for circular inheritance, the no-synchronized-group apply, stale-revision rejection, argument validation, and cleanup of completed requests (checked by a 404 afterwards).
  • Fakes are limited to cases a live server can't easily produce, such as a completed request with failure_reason. Each one's docstring says why, and they follow NiFi's StandardAsynchronousWebRequest semantics.
  • The built wheel was validated against a remote NiFi 2.x runtime with a cloud-secrets parameter provider: fetch, apply, connect, secret rotation and the CLI one-liner.
  • make lint / pre-commit is clean.

Supported versions and generated clients
- Bump the tested NiFi/Registry version to 2.12.0 and regenerate both
  clients; the generated surface change is additive.
- Validate map enum values rather than keys, accept unset sensitivities,
  add STATELESS_GROUP to AffectedComponentEntity, and add a
  Configuration.client_side_validation switch.

Parameter providers and contexts
- Add provider discovery, type lookup, property updates, fetch, apply and
  combined fetch/apply helpers. Apply keeps staged sensitivity and
  synchronized choices unless overridden, defaults unset ones as NiFi does,
  rejects unknown parameter names, and warns when no group is synchronized.
- Add ordered inheritance updates, bound-context lookup, and connecting a
  provider-created context with local sensitive aliases. Connect keeps an
  already inherited source in place unless a higher context shadows a
  mapped name, rejects non-sensitive sources and keeps alias descriptions.
- Add list/get for controller-level management services.

Behaviour and consistency
- Share async request completion and cleanup in utils.wait_for_request:
  completed requests are deleted, failures raise, and timeouts or
  interruptions no longer cancel the server-side request.
- refresh stays on provider operations that submit the supplied revision;
  inheritance and connect helpers always re-read their target.
- get_process_group returns None for a missing ID, like the other getters.
- ensure_management_controller no longer adopts a process-group service.
- Correct the get_controller docstring.
- Retire version guards for NiFi releases older than 2.0.

Generated with [Snowflake CoCo](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code)

Co-authored-by: Snowflake CoCo <noreply@snowflake.com>
@codecov

codecov Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.27%. Comparing base (9668cc1) to head (94cbb61).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #415      +/-   ##
==========================================
+ Coverage   77.24%   78.27%   +1.03%     
==========================================
  Files          41       41              
  Lines        4966     5115     +149     
==========================================
+ Hits         3836     4004     +168     
+ Misses       1130     1111      -19     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Chaffelson and others added 2 commits October 19, 2026 18:32
Remove the AffectedComponentEntity STATELESS_GROUP augmentation: the value
has not been observed from a live server, and the upstream schema fix is
already proposed. Regenerating without it changes only that enum list.

Also remove a REST diagnostics test that covered behaviour this change does
not touch.

Generated with [Snowflake CoCo](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code)

Co-authored-by: Snowflake CoCo <noreply@snowflake.com>
Add live tests for get_management_controller with a missing ID, duplicate
names in ensure_management_controller, and argument validation in the
parameter helpers. Remove a connect check for braces in source parameter
names: NiFi only allows [A-Za-z0-9 ._-] and the names must already exist,
so it could never trigger.

Generated with [Snowflake CoCo](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code)

Co-authored-by: Snowflake CoCo <noreply@snowflake.com>
@Chaffelson Chaffelson self-assigned this Oct 20, 2026
@ottobackwards

Copy link
Copy Markdown
Collaborator

If we change behavior of functions and or / return values we'll have to make sure the version bump is appropriate

…tive

Release notes: date 1.8.0 as 2026-10-22 and add a Behaviour Changes
section listing every existing function whose behaviour or return value
changes, led by get_process_group returning None for a missing ID.
Simplify the documented CLI form for fetch_and_apply_parameter_provider.

Makefile:
- dist, wheel, sdist and install clean only build output; they no longer
  delete test certificates or coverage reports.
- Certificate removal moves to a new clean-certs target, run by clean-all.
- check-dist and test-dist reuse an existing build instead of rebuilding,
  and still build when dist/ is empty (as CI relies on).
- Generated-code removal moves to clean-generated. rebuild-all runs it
  just before gen-clients, because wait-ready imports nipyapi and failed
  once clean-all had removed the clients.
- rebuild-all no longer augments the specs twice.
- Mark the clean and build targets .PHONY (dist/ and docs/ are real
  directories).
The CI checkout was shallow and had no tags, so setuptools-scm versioned
the wheel built by make test-dist as 0.0.1.dev1+unknown. This has been
true on main as well, not only on this branch. Fetch full history and
tags in that job, and make test_distribution.py fail on a fallback
version so a mis-versioned build can no longer pass.

Move the actions to their Node 24 majors: checkout v5, setup-python v6,
codecov v5 (we already use `files`) and setup-buildx v4 (we pass no
removed inputs). Pin jobs to ubuntu-24.04, because setup-python has no
Python 3.9 build for Ubuntu 26.04, which ubuntu-latest is moving to.
The CI jobs only check out code and run tests; Codecov and the registry
tests use their own secrets. Declaring the permissions resolves the
CodeQL actions/missing-workflow-permissions findings.
checkout v7, setup-python v7 and codecov-action v7. codecov v5 pinned a
Node 20 build of actions/github-script internally; v6+ pins Node 24.
None of the intervening breaking changes affect this workflow: no
pull_request_target, no pip-install input, no post-checkout git
credentials.
@Chaffelson
Chaffelson merged commit 3743318 into main Oct 22, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants