Skip to content

Fix fold split toggles and hide live train toggle for empty metrics - #863

Open
Irozuku wants to merge 2 commits into
feat/exogenous-forecasting-modelsfrom
fix/forecasting-fold-graphs
Open

Fix fold split toggles and hide live train toggle for empty metrics#863
Irozuku wants to merge 2 commits into
feat/exogenous-forecasting-modelsfrom
fix/forecasting-fold-graphs

Conversation

@Irozuku

@Irozuku Irozuku commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

Summary

Fold graphs were unavailable for forecasting cross-validation runs. A forecasting strategy deliberately scores no training partition, since that would mean asking the model about dates it was fitted on, so a forecasting run writes no train fold metrics. Both metrics charts opened on the train split regardless: the fold chart asked the backend for metrics that were never written and rendered "No fold data available - this run may not use cross validation" for a run that did, and the live chart offered a Train toggle that led to an empty panel.

Evaluation strategies now report the partitions they score in the metadata they already send the frontend, and both charts build their split toggles from it instead of assuming all three exist. No strategy name literals, so a new strategy gets the right toggles without either component being touched.


Type of Change

Check all that apply like this [x]:

  • Backend change
  • Frontend change
  • CI / Workflow change
  • Build / Packaging change
  • Bug fix
  • Documentation

Changes (by file)

  • DashAI/back/evaluation/base_evaluation_strategy.py: get_metadata() also reports scored_splits, read off each strategy's existing SCORED_SPLITS.
  • DashAI/front/src/hooks/useStrategyKind.js: new useStrategyMetadata hook returning a strategy's whole metadata, sharing the per name cache; useStrategyKind now derives kind from it.
  • DashAI/front/src/components/models/FoldMetricsChart.jsx: split toggles built from scored_splits, opening on the first partition the strategy actually scores (Train for ordinary CV, Validation for forecasting), and the fetch waits for that metadata so no request goes out against a partition with no metrics.
  • DashAI/front/src/components/models/LiveMetricsChart.jsx: same for the live chart's Train/Validation/Test group, with Test still gated on the session having reserved rows.
  • tests/back/evaluation/test_forecasting_strategies.py: asserts what each strategy declares it scores.

Testing

  • Create a forecasting session over a dated dataset, pick RollingOriginSplitter as its splitter and ForecastingCrossValidationEvaluationStrategy as its strategy, then train a run (ARIMA is enough).
  • Open that run's Fold Metrics tab: it should open on Validation, with one panel per metric and one point per fold, and offer no Train toggle.
  • Switch the chart type through Boxplot, Lines, Q-Q and Histogram: every panel should redraw, none should be empty.
  • Open the same run's live metrics tab: the split group should show Validation and Test only, with no Train toggle and no empty panel behind it.
  • Train a tabular classification run over KFoldSplitter with CrossValidationEvaluationStrategy: its Fold Metrics tab should still open on Train, with both Train and Validation toggles, and its live metrics tab should still show all three splits.
  • Train a forecasting holdout run (TemporalHoldoutSplitter): its live metrics tab should drop the Train toggle too.

A forecasting run scores no train partition, so the chart's train default asked for fold metrics that were never written.
A forecasting strategy writes no train metrics, so the toggle opened on an empty panel.
@Irozuku Irozuku added bug Something isn't working front Frontend work back Backend work labels Sep 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

back Backend work bug Something isn't working front Frontend work

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant