Overview
Introduce a configurable auto-fallback mechanism when a given model fails to load or process an input (default: false).
Requirements
- If
fallback: true, when a model quantization fails to load or perform an inference, attempt falling back to another allowed quantization for that model.
- Tests must prove compatibility for all of all changes across all impacted flows.
Overview
Introduce a configurable auto-fallback mechanism when a given model fails to load or process an input (default: false).
Requirements
fallback: true, when a model quantization fails to load or perform an inference, attempt falling back to another allowed quantization for that model.