[JFMGC-8] Build failover solution for model serving infrastructure

[yakavaliou]

How it works:

1. You configure several JointFM deployments that serve the same model (e.g. Chevron + research) in one list: JOINTFM_DEPLOYMENT_IDS.
2. On startup / first forecast, the client checks health on each of those deployments.
   • If one is unreachable → log it and continue with the healthy ones.
   • If healthy ones don’t match on model/checkpoint → stop (don’t mix checkpoints for the day’s data).
3. For each forecast/predict request, the client picks the next deployment in turn (round-robin) and calls it.
4. If that call fails in a “deployment is down / warming / gateway” way (e.g. network error, 470, 502/503/504):
   • Log which deployment failed
   • Try the next deployment ID in the list
   • Do not permanently switch to a dedicated backup — just skip the bad one for that request and keep using the pool
5. If all configured deployments fail → the request fails (same as today with one dead deployment).

---

Note

Medium Risk
Changes how all hosted predict/health traffic is routed (round-robin, failover, threading) and adds a new required-env contract for pool mode; behavior is heavily tested but affects production inference paths.

Overview
Adds multi-deployment hosted inference via JOINTFM_DEPLOYMENT_IDS (comma-separated, ≥2 unique IDs, mutually exclusive with other selectors). Settings build a peer instances pool; docs and config.sample.yaml document the new env/YAML wiring.

Introduces JointFMInstancePool with round-robin routing, per-peer health probes (skip bad peers; require matching model_version / checkpoint_version; use minimum max_sample_count), retryable failover (network errors and configured 5xx/470), and short peer cooldowns after transient failures.

JointFMClient routes health(), predict(), and forecasts through the pool when multiple instances are configured. Sample-batch splitting can run in parallel across peers via ThreadPoolExecutor, with one fail-fast transport per peer in production. Public exports include JointFMInstancePool and JointFMInstanceSettings.

^{Reviewed by Cursor Bugbot for commit 34ac5ad. Bugbot is set up for automated code reviews on this repo. Configure here.}

[shackmann]

This is an awesome contribution Yahor, thanks! Could we maybe make one small conceptual change?

Say I request 10K samples but the model has a capacity of 5K, then we were requesting 5K and then again 5K from the same instance. With several models, we not only have failover but we can speed up operations. Say we have 2 instances that can generate 5K samples, we can now run the requests in parallel, using both instances at the same time. Would that be possible?

[mariusvilkas]

I am not very familiar with the jointfm-client, but code does make sense to me.
Is there a way to call client to just get a status on the pool instances - like instance is offline or not reachable or something like that. Otherwise,
LGTM

[yakavaliou]

I am not very familiar with the jointfm-client, but code does make sense to me. Is there a way to call client to just get a status on the pool instances - like instance is offline or not reachable or something like that. Otherwise, LGTM

@mariusvilkas thank you for the review. I'm currently working on Stefan's suggestion above, so the PR will be updated a bit.

But in any case, if you want to check the deployment status, the easiest way is to run uv run jointfm-client health, which should return the status of the "main" instance (first in the list). Something like

{
  "deployment": {
    "deployment_id": "6a7f76b43a393b6b724e0819",
    "deployment_target": null,
    "deployment_url": "https://app.datarobot.com/api/v2/deployments/6a7f76b43a393b6b724e0819",
    "health_url": "https://app.datarobot.com/api/v2/deployments/6a7f76b43a393b6b724e0819/predictionsUnstructured",
    "local_base_url": null,
    "predict_url": "https://app.datarobot.com/api/v2/deployments/6a7f76b43a393b6b724e0819/predictionsUnstructured",
    "selector": "deployment_ids"
  },
  "service": {
    "checkpoint_path": "/models/jointfm.pt",
    "checkpoint_version": "chevron_i576_o288_f16_t8_h16l16_mam7_af_t3r2_cnn_k3l4_hpst_h16l2_studentt_m2cr5df8",
    "data_generation": {
      "max_features": 16,
      "max_targets": 8,
      "min_features": 0,
      "min_targets": 1,
      "n_input": 576,
      "n_output": 288,
      "sampler_type": "fin",
      "t_input": 4.0,
      "t_output": 2.0
    },
    "decoding_strategy": "parallel_dense",
    "device": "cuda",
    "head": "studentt",
    "image_version": "0.2.0",
    "max_sample_count": 5000,
    "model_version": "jointfm-inference:0.2.0+ckpt.chevron_i576_o288_f16_t8_h16l16_mam7_af_t3r2_cnn_k3l4_hpst_h16l2_studentt_m2cr5df8",
    "schema_version": "v1",
    "status": "ok",
    "supported_query_modes": [
      "forecast"
    ],
    "supported_return_modes": [
      "log_prob",
      "mean",
      "quantiles",
      "samples"
    ],
    "supported_time_index_modes": [
      "absolute_datetime",
      "continuous_float",
      "ordinal"
    ],
    "time_index_encoding": "legacy_discrete_grid"
  }
}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit cbad794. Configure here.}

[yakavaliou]

This is an awesome contribution Yahor, thanks! Could we maybe make one small conceptual change?

Say I request 10K samples but the model has a capacity of 5K, then we were requesting 5K and then again 5K from the same instance. With several models, we not only have failover but we can speed up operations. Say we have 2 instances that can generate 5K samples, we can now run the requests in parallel, using both instances at the same time. Would that be possible?

@shackmann I have added parallel execution in 4415f53

The solution doesn’t look architecturally sound, and this kind of separation would be better handled in the API rather than in the clients. However, I hope it will be useful as a temporary solution.

I tested it on a small random dataset against (6a80af033a75b0a67a4df8ab, 6a7f76b43a393b6b724e0819), and the script ran about 1.5x faster.

[shackmann]

LGTM