Skip to content

feat(control-api): report the health checks a plugin owns - #13899

Open
AlinsRan wants to merge 4 commits into
apache:masterfrom
AlinsRan:feat/control-api-plugin-healthcheck
Open

feat(control-api): report the health checks a plugin owns#13899
AlinsRan wants to merge 4 commits into
apache:masterfrom
AlinsRan:feat/control-api-plugin-healthcheck

Conversation

@AlinsRan

@AlinsRan AlinsRan commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Description

/v1/healthcheck only walks checks on upstreams (value.checks / value.upstream.checks), so a route whose real upstreams are LLM instances configured on ai-proxy-multi reports no health state at all — the very state the plugin consults when it skips an unhealthy instance. A monitoring system polling the control API sees those instances as unchecked.

ai-proxy-multi keys each instance checker by <resource key>#plugins['ai-proxy-multi'].instances[<i>], a layout nothing outside the plugin can guess. Rather than hard-coding it in the control API, a plugin now declares its own checkers:

  • apisix/plugins/ai-proxy-multi.lua — new list_healthcheck_targets(conf, resource_key), returning each checker's resource path, its checks and an opaque meta. The path built in pick_target() is extracted into instance_resource_path() and shared, so the checker identity has one definition.
  • apisix/control/v1.luaiter_and_add_healthcheck_info() additionally asks each configured plugin for its checkers; node lookup is factored into get_checker_nodes() so both entry kinds read the same shm.

A plugin entry keeps the shape of an upstream entry and adds two fields:

{
  "name": "/apisix/routes/1#plugins['ai-proxy-multi'].instances[0]",
  "plugin": "ai-proxy-multi",
  "meta": {"instance": "openai"},
  "type": "http",
  "nodes": [
    {"ip": "52.86.68.46", "port": 443, "status": "healthy",
     "counter": {"success": 2, "http_failure": 0, "tcp_failure": 0, "timeout_failure": 0}}
  ]
}

meta is filled by the plugin and reported verbatim — what a checker stands for is the plugin's business, so the control API does not know what an instance is.

New sub-resource: GET /v1/healthcheck/{src_type}/{src_id}/checkers

One resource can now own several checkers — its upstream plus one per plugin instance — which /v1/healthcheck/{src_type}/{src_id} cannot express: it returns a single object, and a route whose upstreams live on ai-proxy-multi answers 404 {"error_msg":"no checker for routes[1]"} there, because its upstream declares no checks.

The sub-resource returns all of them, as an array of the same entries the listing uses, so a client parses one entry shape for both:

GET /v1/healthcheck/routes/1/checkers
[
  {"name": "/apisix/routes/1", "type": "http", "nodes": [...]},
  {"name": "/apisix/routes/1#plugins['ai-proxy-multi'].instances[0]",
   "plugin": "ai-proxy-multi", "meta": {"instance": "openai"},
   "type": "http", "nodes": [...]}
]
  • The existing single-object endpoint is untouched.
  • A resource with no health check at all is not an error here — it owns an empty set, serialized as [] (array_mt, so never {}).
  • 404 only when the resource itself is missing, 400 for an unknown sub-resource.
  • HTML rendering (Accept: text/html) works the same way as on the other two endpoints.

Docs updated in both languages: docs/{en,zh}/latest/control-api.md for the new fields and the sub-resource, and a pointer from docs/{en,zh}/latest/plugins/ai-proxy-multi.md so a reader who configures instances.checks can find where the result shows up.

Which issue(s) this PR fixes

None filed; reported by a user whose monitoring platform polls /v1/healthcheck and cannot see the health of LLM upstreams behind ai-proxy-multi.

Checklist

  • I have explained the need for this PR and the problem it solves
  • I have explained the changes or the new features added to this PR
  • I have added tests corresponding to this change
  • I have updated the documentation to reflect this change
  • I have verified that this change is backward compatible

Notes

  • upstreams cannot carry plugins (schema_def.upstream has no plugins field), so their entries never gain a plugin field and their checkers set holds at most one element.
  • A single-instance ai-proxy-multi config never creates a checker, because pick_ai_instance() short-circuits pick_target(); its entry is therefore listed with an empty node list. Pre-existing plugin behaviour, untouched here.
  • A plugin reaching a route through a plugin_config is not covered, since that merge happens per request; the endpoint does not walk plugin configs today either.

Testing

New t/control/healthcheck-ai-proxy-multi.t, 8 cases: instance health reported as healthy/unhealthy; plugin and upstream checkers coexisting in one listing; disable_upstream_healthcheck leaving every node list unprobed; instances listed before any probe; the checkers sub-resource on an AI route and on a plain upstream route; and its empty-set / 404 / 400 paths.

All 24 assertions pass locally, as does t/control/healthcheck.t. Locally the AI cases need plugin_attr.ai-proxy.http_client: lua-resty-http, because this box has no ngx_http_ffi_client C module — the committed file uses the default client, and the existing t/plugin/ai-proxy-multi-*.t files fail here for the same reason, on master as well as on this branch.

/v1/healthcheck only walked `checks` on upstreams, so a route whose real
upstreams are LLM instances configured on ai-proxy-multi reported no
health state at all -- the very state the plugin consults when it skips
an unhealthy instance.

ai-proxy-multi keys each instance checker by the parent resource key plus
the JSON path of the instance, a layout nothing outside the plugin can
guess, so a plugin now declares its checkers through
list_healthcheck_targets() and the control API asks for them while it
walks the resource configs. Entries keep the shape of an upstream entry
and add `plugin` plus an opaque `meta` the plugin fills: what a checker
stands for is the plugin's business, not the control API's.

Since one resource can now own several checkers, which the single object
returned by /v1/healthcheck/{src_type}/{src_id} cannot express, add
GET /v1/healthcheck/{src_type}/{src_id}/checkers, returning all of them
as an array of the same entries. A resource with no health check owns an
empty set there rather than being an error.
The zh control-api page mirrors the en one entry for entry, so the
plugin-owned checker fields and the checkers sub-resource belong there
too. Also point from both ai-proxy-multi pages at the control API, so a
reader who configures instances.checks can find where the result shows
up.
Conflict in apisix/control/v1.lua with apache#13891, which wraps the node list
in array_mt inside extra_checker_info(); this branch had already moved
the node lookup out of that function into get_checker_nodes(). Resolved
by keeping the wrap and putting it in get_checker_nodes(), so the
checkers a plugin owns report `[]` for an unprobed node list too.
membphis
membphis previously approved these changes Aug 31, 2026

@membphis membphis left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

nic-6443
nic-6443 previously approved these changes Aug 31, 2026
@shreemaan-abhishek
shreemaan-abhishek requested a balanced review from Copilot August 31, 2026 08:44

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds reporting for plugin-owned health checkers, particularly ai-proxy-multi instances.

Changes:

  • Adds a plugin hook for enumerating health-check targets.
  • Adds the /v1/healthcheck/{type}/{id}/checkers endpoint.
  • Adds bilingual documentation and integration tests.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
apisix/control/v1.lua Reports plugin checkers and adds the sub-resource endpoint.
apisix/plugins/ai-proxy-multi.lua Exposes instance health-check targets.
t/control/healthcheck-ai-proxy-multi.t Tests checker reporting and endpoint behavior.
docs/en/latest/control-api.md Documents the API additions.
docs/zh/latest/control-api.md Documents the API additions in Chinese.
docs/en/latest/plugins/ai-proxy-multi.md Links instance checks to the Control API.
docs/zh/latest/plugins/ai-proxy-multi.md Adds the corresponding Chinese guidance.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread apisix/control/v1.lua Outdated
A plugin disabled through `_meta.disable` never runs, so the checkers it
declares never exist -- reporting them lists node sets that stay empty
forever. Worse, plugin.check_schema() keeps such a config even when the
plugin rejects it, so `instances` can be any type at all; handing that to
list_healthcheck_targets() lets one disabled config fail the whole
/v1/healthcheck response (ipairs on a non-table raises).

check_disable() is the existing answer to "does this config ever run", so
export it rather than restating the rule in the control API.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants