Retry transient remote Kusto entity resolution failures

[Swooo (ShantingLiu)]

Why

Transient cross-cluster Kusto entity-resolution failures currently create query-error Sev3 incidents immediately. These SEM0056 failures can occur even when the remote table exists and resolves on the next request.

Handling this inside individual alert queries with an empty fallback would hide monitoring loss. The alerter should retry the narrowly identified transient failure while preserving fail-loud behavior.

What changed

• Retry once for HTTP 400 SEM0056 errors that specifically report remote entity resolution and failed name or pattern resolution.
• Exclude known permanent OBO, authorization, access-denied, and callout-policy failures.
• Reuse the original query context so the retry evaluates the same alert window.
• Reserve 30 seconds for failure notification.
• If retry fails or cannot safely run, send the existing query-error alert and keep the query unhealthy with a service-error outcome.
• Preserve worker cancellation and notification-throttling behavior.