send_packet no longer rediscovers routes on APS_NO_ACK; a stale next hop black-holes devices until restart (since #260) #288

Summary

The coordinator's routing table can hold a next hop that still MAC-ACKs but no
longer forwards. Every unicast to that destination then fails with APS_NO_ACK.
Z-Stack does not report NWK_NO_ROUTE in this case. Since #260, send_packet
only rediscovers a route on NWK_NO_ROUTE (#273) and after MAC_TRANSACTION_EXPIRED,
so the route is never repaired. Downlink to the device stays dead until a
restart clears the table. Sleepy end devices eventually go unavailable.

One ZDO.ExtRouteDisc fixes it within 2 s. I tested this 7 times with a sniffer
and it worked every time.

Before #260, APS_NO_ACK was in REQUEST_ROUTING_ERRORS (856bf6a, 2020). The
retry chain then tried the last good route (reversed SrcRtgInd relays), then
route discovery, then IEEE addressing. #260 doesn't mention why this was removed.

Environment

• zigpy-znp 1.1.1, zigpy 2.3.0, ZHA 2.3.1 (HA 2026.10.3 for the first case, 2026.10.4 for the rest; both ship the same ZHA)
• CC2674P10 (SMLIGHT SLZB-Ultima3), Z-Stack 2.7.1, build 20260310
• About 70 devices. Most routers are IKEA TRADFRI GU10 bulbs (EmberZNet), plus TRETAKT plugs, one Hue bulb and one TRADFRI repeater
• A second adapter sniffed channel 15 continuously; the captures were decrypted with the network key

One case in detail (SNZB-02DR2, 0x12B2, 2026-10-25)

1. Until 10:15:44 the path coordinator → 0xE732 (GU10) → 0xA857 → parent 0xDFD9 worked.
2. From 10:31:21 every coordinator frame for 0x12B2 still went to 0xE732. Between 09:57 and 11:15 that was 418 frames. 0xE732 MAC-ACKed 404 of them, forwarded none, and sent no Network Status. It stayed active the whole time (200–290 frames per 10 min).
3. None of these frames were source-routed. Yet the coordinator kept receiving valid route records for 0x12B2 via other routers. Uplink worked, so HA still showed the device as available.
4. zigpy logged APS_NO_ACK at 10:43:56 and 11:12:58. It logged no NWK_NO_ROUTE and started no discovery. The device retried its Time cluster read about 40 times per 10 min (normally 1–8), because no reply reached it.
5. At 11:15:21 I called app._discover_route(0x12B2). A route reply came back via 0xAAAC within 0.1 s. From 11:15:22.6 the coordinator sent via 0xAAAC: frames were delivered and APS-ACKed, the Time retries stopped, and no further APS_NO_ACK appeared.

All cases

Each fix took one discovery. HA last restarted on 10-25 at 15:52 (update to 2026.10.4), so only the first case was affected by a restart. Before I had a probe, only HA restarts cleared this: the Z-Stack soft reset empties the routing table.

Why next hops go stale (partly inferred)

The TRADFRI GU10 firmware (0x23093631, EmberZNet) has a 10-entry route table.
When it is full, the oldest entry at max age is overwritten whatever its status,
so a route in use can be evicted. The bulb drops a route only when it receives a
Network Status. I couldn't determine from the firmware whether it sends a route
error when it can't forward, and none appeared on air. From the coordinator,
the only symptom is APS_NO_ACK.

Related

• poll_control_checkin_callback logs WARNING with full traceback on APS_NO_ACK, causing HA to become unresponsive zigpy#1804 / Don't retry or warn on failed Poll Control check-in responses zigpy#1905 (in zigpy 2.4.1, shipping with HA 2026.11.0): failed Poll Control Check-in Responses are no longer retried and are only logged at debug level, on the assumption that the device stopped listening. The APS_NO_ACKs in the 0x12B2 case were Check-in Responses, but the device was fine; its next hop was dead. With that change this failure leaves no warning, so the fix has to be where the send fails.
• ZHA: LDS ZB-ONOFFPlug-D0005 routers become one-way reachable and remain "available" while commands fail with APS_NO_ACK / NWK_NO_ROUTE home-assistant/core#182014: same symptom on a TI coordinator (routers reachable one way, APS_NO_ACK / NWK_NO_ROUTE). There is no capture there, so the cause is unconfirmed.
• Validate association lookups before dropping or restoring a child #286 changes the association-recovery branch of the same loop. The change below sits next to it.

Proposal

Treat APS_NO_ACK like NWK_NO_ROUTE in send_packet: discover the route and retry once.

if (
    e.response.Status in (t.Status.NWK_NO_ROUTE, t.Status.APS_NO_ACK)
    and device is not None
    and retry_attempt < 1
):
    await self._discover_route(device.nwk)
    continue

_discover_route is already deduplicated per NWK (@combine_concurrent_calls)
and sends a single unicast RREQ. For comparison, zigbee-herdsman's Z-Stack adapter
doesn't retry on APS_NO_ACK either, but when an expected ZCL response times out
it calls discoverRoute() and retries once. That covers requests but not replies
like the Check-in Response above.

Alternative, as before #260: retry once over the reversed relays of the last
SrcRtgInd. In these captures those relays were working paths.

I can share decrypted packet summaries (TSV) for each case, or test a patch.

Related: #289