Fix fold split toggles and hide live train toggle for empty metrics - #863
Open
Irozuku wants to merge 2 commits into
Open
Fix fold split toggles and hide live train toggle for empty metrics#863Irozuku wants to merge 2 commits into
Irozuku wants to merge 2 commits into
Conversation
A forecasting run scores no train partition, so the chart's train default asked for fold metrics that were never written.
A forecasting strategy writes no train metrics, so the toggle opened on an empty panel.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fold graphs were unavailable for forecasting cross-validation runs. A forecasting strategy deliberately scores no training partition, since that would mean asking the model about dates it was fitted on, so a forecasting run writes no train fold metrics. Both metrics charts opened on the train split regardless: the fold chart asked the backend for metrics that were never written and rendered "No fold data available - this run may not use cross validation" for a run that did, and the live chart offered a Train toggle that led to an empty panel.
Evaluation strategies now report the partitions they score in the metadata they already send the frontend, and both charts build their split toggles from it instead of assuming all three exist. No strategy name literals, so a new strategy gets the right toggles without either component being touched.
Type of Change
Check all that apply like this [x]:
Changes (by file)
DashAI/back/evaluation/base_evaluation_strategy.py:get_metadata()also reportsscored_splits, read off each strategy's existingSCORED_SPLITS.DashAI/front/src/hooks/useStrategyKind.js: newuseStrategyMetadatahook returning a strategy's whole metadata, sharing the per name cache;useStrategyKindnow deriveskindfrom it.DashAI/front/src/components/models/FoldMetricsChart.jsx: split toggles built fromscored_splits, opening on the first partition the strategy actually scores (Train for ordinary CV, Validation for forecasting), and the fetch waits for that metadata so no request goes out against a partition with no metrics.DashAI/front/src/components/models/LiveMetricsChart.jsx: same for the live chart's Train/Validation/Test group, with Test still gated on the session having reserved rows.tests/back/evaluation/test_forecasting_strategies.py: asserts what each strategy declares it scores.Testing
RollingOriginSplitteras its splitter andForecastingCrossValidationEvaluationStrategyas its strategy, then train a run (ARIMA is enough).KFoldSplitterwithCrossValidationEvaluationStrategy: its Fold Metrics tab should still open on Train, with both Train and Validation toggles, and its live metrics tab should still show all three splits.TemporalHoldoutSplitter): its live metrics tab should drop the Train toggle too.