Skip to content

Async connector jobs need native waiting, capacity metadata, and structured errors #436

Closed

Description

@Broccolito

Describe the bug

Long-running connector jobs expose an incomplete control contract. Status must be polled manually using an advisory recommended polling interval, with no blocking wait or completion-watch operation. Submission can then fail because an undisclosed active-job limit has already been reached.

When a job terminates unsuccessfully, the returned error can be a broad sanitized message covering unrelated causes such as credentials, connectivity, schema, permissions, disk, database execution, or output processing, without a stable error category or retryability. In the observed session these gaps caused repeated status calls, a loop-guard intervention, a shell-sleep workaround, an avoidable capacity rejection, and blind query rewrites.

To reproduce

  1. Submit two long-running connector jobs.
  2. Call the job-status tool while they run and observe that it returns an advisory polling interval but provides no blocking wait or watch parameter or completion notification.
  3. Attempt to submit another job and observe that submission is rejected because two jobs are already active, although the maximum capacity and current availability were not exposed before submission.
  4. Submit or monitor a job that terminates unsuccessfully.
  5. Observe a broad sanitized failure message without a stable error code, a sufficiently actionable failure phase, or retryability guidance.

Expected behavior

The asynchronous-job contract should:

  • Support an efficient native wait or watch operation, or a blocking status call.
  • Expose active and maximum job capacity before submission, or queue excess jobs.
  • Return privacy-safe structured errors with stable error codes, execution phase, and retryability.

Agents should not need rapid polling, shell sleep, trial submissions, or blind query rewrites.

Suspected cause

Medium confidence because the connector implementation itself was outside the checked Biorouter repository, but the session and advertised tool schemas establish the interface gaps. The status tool accepts only a job identifier; it returns a recommended polling interval but offers no wait duration, blocking timeout, or event subscription. The submission tool exposes query and output controls but no capacity information, and no job-list or capacity tool was advertised.

The connector instructions make manual polling the intended workflow. Biorouter's v1.93.2 agent-loop review in docs/history/agent-loop-review/subsystem-reviews/core-loop-and-tool-dispatch.md separately identifies the general lack of adaptive per-tool waiting and partial progress as an architecture weakness. The two-job rejection confirms a connector resource limit that is enforced only at submission.

The terminal ValueError text intentionally withholds raw details, which is reasonable for data protection, but it also collapses distinct actionable causes. This likely needs a shared asynchronous-job result contract in Biorouter plus adoption by the connector: a wait_for_job operation or blocking status, active_jobs and max_active_jobs metadata or queuing, and sanitized error_code, phase, and retryable fields.

Additional context

This report is intentionally limited to the asynchronous job-control contract. It does not cover the working-directory path jail or the separate problem of terminal operation failures appearing as successful outer tool calls.

Do not attach a diagnostics bundle from the originating private chat to the public issue because it contains the conversation unredacted.

Verification status

Reported behavior is being checked against current source with synthetic reproductions. No originating private transcript or diagnostics bundle is attached. Related connector implementations: BaranziniLab/UCSFOMOPAgent and BaranziniLab/CDWAgent.

Activity

  1. Broccolito commented on Oct 24, 2026

    @Broccolito
    CollaboratorAuthor

    Issue #436: Async connector waiting, capacity and structured errors

    Issue · Host PR #454 · merged main source

    Historical desktop/browser/CLI observations used f227 artifacts, accepted via the reviewed production-equivalence comparison; exact-main CI/build/smoke is attributed separately below.

    Problem. Long connector jobs lacked a consistent native wait/capacity/result/error contract, encouraging repeated polling and obscuring failed operations behind successful status lookups.

    Fix and UX. UCSFOMOPAgent PR #4 and CDWAgent PR #6 are merged at their verified mains. Their six-tool contract provides bounded waiting, advisory capacity plus atomic admission, durable job IDs/states, cancellation and safe job-ID result reading. Versioned operation metadata carries stable privacy-safe error fields. Host handling distinguishes a failed job from successful lookup transport; contract tests and metadata-secret redaction cover that distinction. No connector-specific human card is added; normal tool-approval policy still applies.

    Validation. Final actual native C1 OMOP and C3 CDW independently passed 17 matched calls each through normal UI registration and all six tools, with completed, failed and cancelled jobs and a reconstructed 69-byte synthetic CSV. C2/C4 in-place handoffs are separate #437 observations. Parent and independent QA reconciled the 81-pair native cohort. Exact-main host Rust CI, Linux rebuilt storage and UI CI are linked separately; connector mains each had their own green checks.

    Limits. These use actual registrar/job-store/worker logic with deterministic credential-free synthetic workers, not clinical databases or PHI. Native cases did not exercise every capacity/timeout/start-failure category; connector software tests remain separately attributed. Cancellation retained private 69-byte partial files: it does not prove zero bytes generated or automatic deletion. Forty connector RPC records are not forty semantic successes.

    Recommendation. Reuse the bounded wait/capacity/error contract for future connectors, retaining ordinary tool policy rather than adding connector-specific approval friction.

    Merged-main verification. PR454 is merged at 0580110; main Rust CI, frontend CI and related checks finished 20 successes/one skipped (17 push successes plus three landing/deployment checks; these are not security-test counts). Normal main build manifest SHA256: 38cc2979c7da104cb623975bac7f6922a085e834a1be5c18e961d7f26ea811fc; shipping receipt: e6e1d8957d8d0b22a9aae814011a1e028c5d279848ffabbf4c6d0c4edf3c0112. Actual main01 smoke separately recorded four pairs—three reads/one typed Deny—and three unchanged canaries; it does not rerun/recount the historical 81-pair cohort.

    Submission record. Manuscript v1.1 local Supplementary Data S1: Supplementary Data S1 - BioRouter Security Audit - 2026-10-24.zip; manifest SHA256 4f0c0c110ca0b07a29d5281870227f7b36c744a0f37ccb499b8ef1283dea116e. This is a user/submission package, not a claim of public upload. Public PR/source/CI links above provide the public implementation proof.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions