Skip to content

Reconnect implements that missed a TC restart - #105

Open
gunicsba wants to merge 1 commit into
developfrom
fix/tc-restart-recovery
Open

gunicsba wants to merge 1 commit into
developfrom
fix/tc-restart-recovery

Conversation

@gunicsba

Copy link
Copy Markdown
Contributor

Summary

A field log from 2026-10-13 (Sulky Isolink spreader on a JD tractor) showed the spreader cut off for 20 minutes after the operator restarted the TC three times.

An implement only drops its connection when it sees no TC Status for 6 s. A TC restart takes 2–3 s, and both our TC Status and the stack's own copy (sent from TaskControllerServer::update()) were back on the bus before that. The spreader never noticed the TC had gone away and kept sending process data. The new TC didn't know it, so the stack NACKed every message:

[10:25:19.816] [Warn][TC Server]: NACKing process data command from 0xEF because they are not known to us. Clients must send the working set master message first.

That message appears 4,461 times in the log. The spreader only came back after the tractor was power-cycled. The JD tractor ECU client (0xF0) was stuck the same way.

Changes

  • Startup: for the first 6.5 s the TC doesn't send TC Status and doesn't call tcServer->update(), which also holds the stack's copy. Clients of the previous run time out and reconnect. Their messages wait in the stack's queue.
  • Runtime: MyTCServer registers its own CAN callback for Process Data. If a Client Task or Value message reaches us from a CF that has no active client in the stack, that client still thinks it is connected. The stack ignores its Client Task messages and NACKs its values, so it never recovers on its own. The TC then goes quiet for 6.5 s so the client starts over.
    • At most one quiet period per 30 s, so reconnecting clients can finish.
    • After 3 attempts for the same NAME the TC stops trying, logs it and asks for a power-cycle in AOG. A broken client can't keep disturbing the others.
    • A quiet period disconnects every client, because TC Status is broadcast. They all reconnect within a few seconds.
  • No AgIsoStack changes. The callback only uses get_active_client() and serverControlFunction, which are protected and unchanged in the Bump AgIsoStack++ to upstream main (includes #718) #104 stack pin.

Notes

  • Two TC Status senders still exist: Application::send_task_controller_status_message() and the stack's. That's out of scope here; both are held during a quiet period.
  • 6fe9029 on the integration branch only delays our own TC Status at startup, so it doesn't prevent this. This PR replaces it.
  • Conflicts with the section setpoint and section demand diagnostics PRs in Application::update(). Keep both sides; the resolution is in test/szello-debug.

Testing

  • Release build passes on Windows (MSVC), against both the develop stack pin and Bump AgIsoStack++ to upstream main (includes #718) #104's.
  • Not yet run on a bus: there is no virtual CAN on the build PC. A debug build is going to the same machine. Things to check:
    • after a TC restart the log shows [TC] Holding TC Status for 6500 ms (startup), and the implements reconnect (New client ... detected via WorkingSetMaster) with no NACKing lines;
    • if a client is ever stale at runtime: [TC] Client 0x.. is talking to us as if connected ....

🤖 Generated with Claude Code

An implement only reconnects when it sees no TC Status for 6 s. Restarting the
TC takes 2-3 s, and both our TC Status and the stack's (sent from
TaskControllerServer::update()) were back on the bus before that, so the
implement kept sending process data as if nothing happened. The new TC did not
know it and NACKed every message; in the field a spreader stayed cut off for
20 minutes across three restarts, until the tractor was power-cycled.

- Startup: hold TC Status, and the TC server update that sends the stack's
  copy, for 6.5 s, so clients of the previous run time out and reconnect.
- Runtime: watch for Client Task and Value messages addressed to us from a CF
  the stack has no active client for. Such a client still thinks it is
  connected; hold TC Status for 6.5 s so it starts over. A 30 s cooldown lets
  reconnecting clients finish, and after 3 attempts for the same NAME the TC
  stops and asks for a power-cycle instead of disturbing the other clients.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant