altool runtests hangs after TestRunnerHub Initialize fails with application authentication

[damian-tomalka]

1. Describe the bug

When running altool runtests against a Business Central Online sandbox using an application access token (client credentials), the tool connects to TestRunnerHub but does not complete or return a useful error.

An isolated diagnostic call using the installed Microsoft AL libraries exposed a server exception from Initialize. Disposing the failed runner also did not complete within a five-second diagnostic deadline.

Using delegated user authentication with the same environment, company, tool version and installed extensions, all ten test methods pass.

We have not confirmed whether application authentication is supported for this endpoint. Regardless, the CLI should report the initialization failure and exit instead of hanging.

2. To Reproduce

1. Use a Business Central Online sandbox with a published AL test extension.
2. Configure an enabled Entra application in BC with:
   • Application API permissions: API.ReadWrite.All and Automation.ReadWrite.All, with tenant consent.
   • BC permission sets: D365 AUTOMATION and EXTEN. MGT. - ADMIN, without a company restriction.
3. Obtain a client-credentials access token for the target tenant and the Business Central resource.
4. Supply it to the CLI through the BC_ACCESS_TOKEN environment variable.
5. Run:

dotnet "<AL-extension-directory>/bin/altool.dll" runtests `
    --testgroups "<path-to-test-groups.json>" `
    --tenant "<tenant-id>" `
    --environmentname "<sandbox-name>" `
    --environmenttype Sandbox `
    --company "<company-name>"

6. Observe that the process connects but does not finish. In our pipeline, we cancelled it after more than five minutes.
7. Repeat using normal delegated AL authentication against the same target. The tests complete successfully.

To isolate the failure, we also used the installed Microsoft AL libraries to connect and invoke only:

Initialize("<company-name>", "", CoverageMode.None)

This reproduced the exception before any test method was invoked.

The original tests belong to a private customer extension. We have not yet reproduced this with a separate minimal public AL test extension. The initialization-only diagnostic isolates the failure from execution of the test methods.

3. Expected behavior

If application authentication is supported, initialization and test execution should complete with the required permissions.

If it is unsupported, or initialization otherwise fails, the CLI should report a clear error and exit with a nonzero exit code. Cleanup should not prevent the error from being reported.

4. Actual behavior

With application authentication:

• Extension publication through the Automation API succeeds.
• Developer metadata retrieval and the TestRunnerHub connection succeed.
• The isolated Initialize call returns:

Microsoft.AspNetCore.SignalR.HubException:
An unexpected error occurred invoking 'Initialize' on the server.

• Disposing the failed runner exceeds a five-second diagnostic deadline.
• The normal CLI remains running without producing a completed test result.

With delegated authentication:

• Initialization succeeds.
• All ten named test methods pass.

We verified the application's BC permission assignments through the Automation API. D365 AUTOMATION includes execute permission on Codeunit ID 0 (all codeunits). However, we have not established equivalent effective permissions or entitlements between the application identity and the delegated administrator.

The Admin Center telemetry query covering the failure returned HTTP 200 with no entries. Application Insights is not configured, so the underlying server exception remains unknown.

An external process timeout now prevents our pipeline from hanging indefinitely, but it does not resolve the initialization failure.

5. Versions:

• AL Language: 18.0.2732683
• Visual Studio Code: Not involved in the failing reproduction; altool.dll was invoked directly using dotnet.
• Business Central: Online sandbox, application version 28.5.54151.54677
• .NET runtime: 10.0.12
• List of Visual Studio Code extensions that you have installed: The reproduction runs outside the VS Code extension host. A separate IDE reproduction with all other extensions disabled has not been performed.
• Operating System:
  • Windows — build 26200
  • Linux
  • MacOS

Final Checklist

• Search the issue repository to ensure you are reporting a new issue
• Reproduce the issue after disabling all extensions except the AL Language extension
• Simplify your code around the issue to better isolate the problem

Isolation performed: reproduced the Initialize failure without invoking any test methods. A minimal public AL test project is not yet available.

[damian-tomalka]

Update: we tested an alternative S2S entry point on the same Business Central Online Sandbox: an OData V4 unbound action invoking a custom AL TestRunner.

All test calls used client-credentials authentication, with API.ReadWrite.All and Automation.ReadWrite.All application roles and no delegated user scope.

Results:

• A simple AL test passes.
• A deliberately failing assertion is correctly reported.
• AutoRollback works: no diagnostic records remain after execution.
• TestPage.OpenEdit() fails with the following server exception:

System.NotSupportedException: Specified method is not supported.
Microsoft.Dynamics.Nav.Runtime.NavSession.CreateNavTestService()
Microsoft.Dynamics.Nav.Runtime.TestServiceConnection.get_Server()
Microsoft.Dynamics.Nav.Runtime.TestServiceConnection.get_ClientApi()
...
Microsoft.Dynamics.Nav.Runtime.NavTestPage.ALOpenEdit()

Separately, inspection of local platform artifacts for BC 28.5.54151.54677 (platform file version 28.0.54476.0) indicates that TestRunnerRuntime.InitializeRuntime creates a WebClient session, while ValidateLicenseForS2SSessionAsync rejects that client type for S2S identities.

This is a hypothesis for the native Initialize failure, not a captured inner exception from SaaS. The OData/TestPage exception above was returned by the actual Sandbox.

Could you please confirm:

1. Does the WebClient/S2S restriction explain the native TestRunnerHub.Initialize failure?
2. Is there a supported application-only execution path for Online Sandbox tests that includes TestPage, Permissions Mock and test isolation?
3. If not, is support planned?

Our CI needs a stable application identity independent of individual developers and their interactive sessions. We do not want to remove TestPage coverage or treat partially executed tests as a successful release gate.

We have a small diagnostic extension and the detailed results available if needed.