Web API started through XML-RPC (start_webapi_server): every request times out after 30 s #403

Describe the bug

When the Web API server is started through XML-RPC (RemoteProxy.start_webapi_server()), every Web API request served by the main thread fails after 30 seconds with HTTP 500 Main thread execution timed out after 30 seconds. This includes requests that should fail immediately: /calc with an unknown function returns this timeout instead of the expected 404.

Cause, from code analysis: RemoteServer.start_webapi_server calls DLMainWindow.start_webapi_server directly in the XML-RPC server thread, so WebApiController.start() creates the WorkspaceAdapter and its MainThreadExecutor in that thread:

• the executor QObject lives in the XML-RPC thread, which has no running event loop: the queued calls are not executed, and the Web API request times out;
• MainThreadExecutor._main_thread is the XML-RPC thread, not the GUI thread;
• pending calls are executed later in the XML-RPC thread, whenever it processes events while waiting for a remote call: Web API operations then run outside the GUI thread, after their request has already failed.

To Reproduce

"""Web API started through XML-RPC: every request times out after 30 s"""

import json
import os
import urllib.error
import urllib.request

os.environ.setdefault("QT_QPA_PLATFORM", "offscreen")

from datalab.env import execenv  # noqa: E402
from datalab.tests import datalab_in_background_context  # noqa: E402


def post_calc(url: str, token: str, name: str) -> tuple[int, str]:
    """Call the /calc endpoint of the Web API"""
    request = urllib.request.Request(
        f"{url}/api/v1/calc",
        data=json.dumps({"name": name}).encode(),
        headers={"Authorization": f"Bearer {token}", "Content-Type": "application/json"},
        method="POST",
    )
    try:
        with urllib.request.urlopen(request, timeout=60) as response:
            return response.status, response.read().decode()
    except urllib.error.HTTPError as exc:
        return exc.code, exc.read().decode()


if __name__ == "__main__":
    execenv.unattended = True
    with datalab_in_background_context() as proxy:
        info = proxy.start_webapi_server()
        print(post_calc(info["url"], info["token"], "unknown_function"))

Output, after 30 seconds:

(500, '{"detail":"Computation error: Main thread execution timed out after 30 seconds"}')

Expected behavior

The request is served by the GUI thread whatever the way the Web API server was started. Here, the expected output is returned immediately:

(404, '{"detail":"Unknown computation function \'unknown_function\': Unknown computation function unknown_function"}')

This is what happens when the Web API is started in the GUI thread, e.g. with DATALAB_WEBAPI_ENABLED=1.

Installation information

• Python package (development checkout), develop branch at c800d5e
• DataLab 1.3.0, Sigima 1.3.0, SigimaX 1.1.0, PlotPy 2.11.1, guidata 3.16.0, PyQt5 5.15.11 (Qt 5.15.2), FastAPI 0.136.0, uvicorn 0.45.0, Python 3.11.6, NumPy 2.4.4, Windows 10.0.26200

Additional context

• Possible fixes: run start_webapi_server and stop_webapi_server in the GUI thread, through a signal as the other remote commands; or make MainThreadExecutor target the application thread (QCoreApplication.instance().thread()) instead of the thread creating it.
• RemoteServer.get_webapi_status also calls the main window directly from the XML-RPC thread.
• Found while investigating Remote load_from_files / load_from_directory intermittently ignored #398 (see Fix remote file loading intermittently ignored at DataLab startup #402), which does not change this behavior.