Skip to content

fix(cli): stop gating nginx.conf on the config file plugin list - #13878

Open
AlinsRan wants to merge 6 commits into
apache:masterfrom
AlinsRan:fix/ngx-tpl-plugin-list-independent
Open

fix(cli): stop gating nginx.conf on the config file plugin list#13878
AlinsRan wants to merge 6 commits into
apache:masterfrom
AlinsRan:fix/ngx-tpl-plugin-list-independent

Conversation

@AlinsRan

@AlinsRan AlinsRan commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Description

plugins / stream_plugins in config.yaml is only the boot-time default for the plugin list. /apisix/plugins in etcd replaces it while APISIX runs — through the Admin API, or written directly by whatever owns the configuration — and that happens long after apisix init has rendered nginx.conf. So any directive apisix/cli/ngx_tpl.lua made conditional on that list is missing for a plugin enabled the etcd way, and nginx cannot grow a shared memory zone (or a location, or a mirror) without a reload. The plugin loads, runs, and silently does nothing.

apisix_stream_metrics_zone is the case that surfaced this. The zone was rendered only when stream_plugins named prometheus, so where the plugin list lives in etcd the stream prometheus plugin ran but apisix_stream_active_connections and apisix_stream_bandwidth were never published, while apisix_stream_connection_total and apisix_stream_status, which do not read the zone, kept working. Fixing only that one gate would leave the same trap for limit-conn, limit-count, api-breaker, openid-connect, ocsp-stapling, ext-plugin-*, mcp-bridge, skywalking, error-log-logger, proxy-cache, proxy-mirror and proxy-buffering, so this removes the coupling everywhere.

What changes in the template

  • Every lua_shared_dict that was gated on the plugin list is now rendered unconditionally, as are lua_capture_error_log, the proxy-cache directives, the proxy-mirror locations and mirror, and the @disable_proxy_buffering location.
  • The prometheus export server now follows plugin_attr.prometheus.enable_export_server alone (via prometheus_server_addr), which is the knob that was always meant to govern it. The same applies to the http{} block that hosts it in stream-only mode.
  • Two gates move to a condition that is genuinely about the environment rather than the plugin list: the dubbo directives need mod_dubbo and ngx_multi_upstream_module, so they follow APISIX-Runtime; and prometheus-metrics is declared in the lua{} block whenever the stream subsystem runs, so both subsystems share one dict.
  • enabled_plugins / enabled_stream_plugins are no longer passed to the template.

What changes in the exporter

The same coupling existed in Lua, in two places, and both are fixed here:

  • The prometheus plugin's init hook decided whether to define the L4 metrics from stream_plugins in config.yaml, so where the plugin list comes from etcd, metrics was built without stream_active_connections and collect_stream_zone_metrics() stopped at its first guard for the rest of the process's life: the zone was rendered and filling, and nothing read it.
  • exporter.http_init() returns early outside the init phases (the todo: support hot reload above that guard), so driving it from the plugin's init hook only ever worked when config.yaml already listed prometheus. Enabling the plugin through /apisix/plugins after startup left prometheus nil for the life of the process and /apisix/prometheus/metrics answering {}. plugin.init_prometheus() had the same problem for the exporter timer.

Both are now wired the way the API7 gateway wires them, since its data plane always learns the plugin list from a control plane: the exporter is built in plugin.init_worker() whatever the config file lists, and load() drops it with exporter.destroy() when prometheus is not in the effective list and restores it when it is. A deployment that never enables prometheus is left with a destroyed exporter, so exporter_timer returns at its own if not prometheus and costs nothing. The plugin init hooks are gone, which also stops the stream plugin's destroy hook from reaching across into the HTTP exporter when load_stream() runs in the HTTP subsystem.

Cost

Memory that a trimmed plugin list used to save. With the default plugin list only tracing_buffer, ocsp-stapling and lua_capture_error_log are new; a deployment that lists just a handful of plugins now allocates the full set of dicts. A stream-only deployment also gets the http{} block hosting the export server, which enable_export_server: false still removes.

Tests

  • t/cli/test_http_config.sh and t/cli/test_stream_config.sh: the two "enable shdict on demand" checks asserted exactly the behaviour being removed; they now assert the dict is present whatever the config file lists.
  • t/cli/test_stream_config.sh: stream-only mode renders a second lua_package_path on APISIX-Runtime (the export server's http{}), so the expected count follows the runtime and the case additionally asserts the http proxy is still absent, which is what it was really testing. A second case with enable_export_server: false asserts the block goes away again.
  • t/cli/test_prometheus_stream.sh: a new block seeds the plugin list into etcd with no stream_plugins in config.yaml and the Admin API off, then asserts both that the zone is rendered and that apisix_stream_bandwidth is exported for proxied traffic. Both assertions fail on master.
  • t/cli/test_prometheus.sh: a new block with prometheus in no local list at all and the Admin API off, enabled through etcd once APISIX is already running. It answers {} without the exporter change and exports metrics with it.
  • t/cli/test_zipkin_set_ngx_var.sh: zipkin is now left out of plugins, so the case fails against the old enabled_plugins["zipkin"] gate.

Which issue(s) this PR fixes:

N/A

Checklist

  • I have explained the need for this PR and the problem it solves
  • I have explained the changes or the new features added to this PR
  • I have added tests corresponding to this change
  • I have updated the documentation to reflect this change
  • I have verified that this change is backward compatible (If not, please discuss on the APISIX mailing list first)

`plugins` / `stream_plugins` in config.yaml is only the boot-time default:
`/apisix/plugins` in etcd replaces the list while APISIX runs, long after
`apisix init` has rendered nginx.conf. Any directive the template made
conditional on that list is therefore missing for a plugin enabled the etcd
way, and nginx cannot grow a shared memory zone -- or a location, or a mirror
-- without a reload, so such a plugin silently does nothing.

`apisix_stream_metrics_zone` is the case that surfaced this: the zone was
rendered only when `stream_plugins` named `prometheus`, so on a deployment
whose plugin list lives in etcd the stream prometheus plugin loaded and ran
but `apisix_stream_active_connections` and `apisix_stream_bandwidth` were
never published, while the two metrics that do not read the zone kept working.

Every shared dict, `lua_capture_error_log`, the proxy-cache, proxy-mirror and
proxy-buffering directives are now rendered unconditionally, and the
prometheus export server follows `plugin_attr.prometheus.enable_export_server`
alone. Two gates move to a condition that is actually about the environment
rather than the plugin list: the dubbo directives need `mod_dubbo` and
`ngx_multi_upstream_module`, so they follow APISIX-Runtime, and
`prometheus-metrics` stays in the `lua{}` block whenever the stream subsystem
runs so that both subsystems share it.

The cost is memory that a trimmed plugin list used to save. With the default
list only `tracing_buffer`, `ocsp-stapling` and `lua_capture_error_log` are
new; a deployment that lists only a handful of plugins now allocates the full
set of dicts. A stream-only deployment also gets the http{} block that hosts
the export server, which `enable_export_server: false` still removes.
@dosubot dosubot Bot added the size:M This PR changes 30-99 lines, ignoring generated files. label Aug 26, 2026
…list

The template half of this change renders `apisix_stream_metrics_zone`
whatever config.yaml lists, but the exporter still decided whether to define
the L4 metrics at all from `stream_plugins` in that same file: the prometheus
plugin's `init` hook passed `array_find(local_conf.stream_plugins,
"prometheus")` into `exporter.http_init`. That hook runs whenever the plugin
is loaded -- including when `/apisix/plugins` in etcd loads it -- so on a data
plane whose plugin list comes from etcd `metrics` was rebuilt without
`stream_active_connections`, and `collect_stream_zone_metrics()` stopped at
its first guard for the rest of the process's life. The zone was there and
filling; nothing read it.

It now follows whether the stream subsystem runs, which is a property of the
deployment rather than of a list that changes under APISIX.

This is what `t/cli/test_prometheus_stream.sh` caught: the block added in the
previous commit renders the zone, but exported no `apisix_stream_bandwidth`
until this.
@dosubot dosubot Bot added size:L This PR changes 100-499 lines, ignoring generated files. and removed size:M This PR changes 30-99 lines, ignoring generated files. labels Aug 26, 2026
The zipkin variables and the stream only export server both stopped
depending on the config file plugin list without a case that would notice.
test_zipkin_set_ngx_var.sh now leaves zipkin out of `plugins` (it fails
against the old gate), and test_stream_config.sh asserts the export server's
http{} block goes away again with `enable_export_server: false`.
membphis
membphis previously approved these changes Aug 26, 2026

@membphis membphis left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Decouples generated NGINX configuration from boot-time plugin lists so plugins enabled through etcd retain required resources.

Changes:

  • Renders plugin infrastructure independently of local plugin lists.
  • Bases Prometheus stream setup on proxy mode.
  • Updates documentation and CLI/integration tests.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
apisix/cli/ngx_tpl.lua Removes plugin-list template gates.
apisix/cli/ops.lua Stops passing plugin lists to the template.
apisix/plugins/prometheus.lua Uses stream subsystem state for initialization.
docs/en/latest/plugins/prometheus.md Documents etcd-enabled stream metrics.
docs/zh/latest/plugins/prometheus.md Updates Chinese documentation.
t/cli/test_http_config.sh Verifies unconditional HTTP shared dictionaries.
t/cli/test_stream_config.sh Updates stream rendering assertions.
t/cli/test_prometheus_stream.sh Adds an etcd plugin-list metrics scenario.
t/cli/test_zipkin_set_ngx_var.sh Tests Zipkin attributes without local enablement.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread apisix/plugins/prometheus.lua Outdated
Comment thread t/cli/test_stream_config.sh Outdated
nic-6443
nic-6443 previously approved these changes Aug 27, 2026
The http{} block hosting the export server is rendered inside the template's
`use_apisix_base` guard, so stock OpenResty still renders the stream block
alone. test_stream_config.sh has no exit_if_not_customed_nginx -- the rest of
it is runtime independent -- so the count is what has to be conditional.
`exporter.http_init()` returns early outside the init phases -- the `todo:
support hot reload` above that guard -- so it can restore an exporter that
init_worker built but cannot build one. Driving it from the plugin's own
`init` hook therefore only ever worked when config.yaml already listed
prometheus: a data plane that learns its plugin list from `/apisix/plugins`
kept `prometheus` nil for the life of the process and `/apisix/prometheus/
metrics` answered `{}`. `plugin.init_prometheus()` had the same problem for
the exporter timer.

This is wired the way the API7 gateway has been wiring it: the exporter is
built in `plugin.init_worker()` whatever the config file lists, and `load()`
drops it again when prometheus is not in the effective list and restores it
when it is. A deployment that never enables prometheus is left with a
destroyed exporter, so `exporter_timer` returns at its own `if not prometheus`
and costs nothing. The plugin `init` hooks that used to do this are gone,
which also stops the stream plugin's `destroy` hook from reaching across into
the HTTP exporter when `load_stream()` runs in the HTTP subsystem.

t/cli/test_prometheus.sh gains the case: no prometheus in config.yaml, Admin
API off, enabled through etcd once APISIX is already running. It answers `{}`
without this and exports metrics with it, and disabling it again through etcd
returns it to `{}`.
Whether the xRPC protocol metrics were recorded used to depend on nothing but
`init_stream_metrics()` having run, which is to say on `stream_plugins` in
config.yaml naming prometheus. Now that the exporter is built whatever the
config file lists, that side effect is gone and every xRPC session would be
counted, including on a data plane whose plugin list never enables the stream
prometheus plugin.

sdk.get_metrics() asks the effective plugin list instead, which is the list
that actually decides, and keeps working when /apisix/plugins turns the plugin
on or off later. t/xrpc/prometheus.t TEST 4 covers it: prometheus out of
`stream_plugins`, a redis command through xRPC, and the counter must not move.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants