Repository navigation

Record prediction runs and monitor score drift - #64

Open

Vedaant-J wants to merge 57 commits into

vedaant/mlops-mlflow-trackingfrom

vedaant/mlops-score-sketch

Open

Record prediction runs and monitor score drift#64
Vedaant-J wants to merge 57 commits into
vedaant/mlops-mlflow-trackingfrom
vedaant/mlops-score-sketch

Commits on Oct 29, 2026
- Name Snowflake prediction and monitoring runs as plain identifiers
  
  Vedaant-J
  and
  claude
  committed
  8e60000
- Record and monitor prediction runs of any model's output files
  
  Vedaant-J
  and
  claude
  committed
  1d1276c
- Tag runs by prediction id, write outputs by path, read runs back through the root
  
  Vedaant-J
  and
  claude
  committed
  bd3d86f
- Store prediction runs under on_error like every other artifact
  
  Vedaant-J
  and
  claude
  committed
  2f0838c
- End failed prediction runs failed, type evaluated rows for Model Monitor
  
  Vedaant-J
  and
  claude
  committed
  0c93e17
- Refuse another graph's schema under on_error='raise', gate the bundle's alias
  
  Vedaant-J
  and
  claude
  committed
  1cb0120
- Expect the same-schema graph warning under on_error='raise' too
  
  Vedaant-J
  and
  claude
  committed
  71da4cc
- Read a derived model's training snapshot for feature drift
  
  Vedaant-J
  and
  claude
  committed
  400cb35
- Leave skipped prediction rows to the caller's warning
  
  Vedaant-J
  and
  claude
  committed
  bd1ac49
- Document threshold-free binary scoring and the tested monitor recipes
  
  Vedaant-J
  and
  claude
  committed
  58404e7
- Export the holdout as evaluated rows for a monitor baseline
  
  Vedaant-J
  and
  claude
  committed
  75ff0e2
- Install the databricks-sdk the Data Quality monitor recipe needs
  
  Vedaant-J
  and
  claude
  committed
  ca8e15f
- Document the holdout baseline, the SDK the extra installs and kumo.monitored
  
  Vedaant-J
  and
  claude
  committed
  0238dc4
- Test the holdout baseline against the loaded types without a threshold
  
  Vedaant-J
  and
  claude
  committed
  7264d35
- Say Snowflake allows one model monitor per model version
  
  Vedaant-J
  and
  claude
  committed
  0fdcdb2
- Compare text columns in feature drift; read evaluated rows as a frame
  
  Vedaant-J
  and
  claude
  committed
  8666ee6
- Make the bundle's artifact root a variable; fix example install notes
  
  Vedaant-J
  and
  claude
  committed
  534b84c
- Document evaluation.frame, text-column drift and the PyPI 1.0.0 release
  
  Vedaant-J
  and
  claude
  committed
  f83b864
- Port the SageMaker Pipeline template to SageMaker Python SDK v3
  
  Vedaant-J
  and
  claude
  committed
  b9938e2
- Run the SageMaker pipeline with a least-privilege role; keep SageMaker files out of the root
  
  Vedaant-J
  and
  claude
  committed
  5a31e4a
- Default the job templates to a 7-day query; make it a bundle variable
  
  Vedaant-J
  and
  claude
  committed
  8fd0ee2
- Install the wheel by direct reference in the Snowflake jobs template
  
  Vedaant-J
  and
  claude
  committed
  19e9332
- Document approximate distinct counts and how progress events print
  
  Vedaant-J
  and
  claude
  committed
  cf81088
- Count scores exactly per batch in fixed bins that add up across workers
  
  Vedaant-J
  and
  claude
  committed
  fa063f6
- Count each test split's scores exactly and derive its test curves and metrics
  
  Vedaant-J
  and
  claude
  committed
  3ce2c95
- Log a fit's test curves and metrics, and keep its holdout counts for drift
  
  Vedaant-J
  and
  claude
  committed
  a5bd52a
- Count forecasts per horizon and multilabel tests per label; chart @k over k
  
  Vedaant-J
  and
  claude
  committed
  b87508b
- Document the Snowflake jobs' payload wheel, SageMaker's UI access and notebooks
  
  Vedaant-J
  and
  claude
  committed
  f2f8faa
- Bin a node-regression target in pass B for the split stats' histogram
  
  Vedaant-J
  and
  claude
  committed
  9c3ff10
- Record where a prediction run is stored and name its outputs
  
  Vedaant-J
  and
  claude
  committed
  5714d40
- Keep the trainer's own test metrics where an exact summary has the same name
  
  Vedaant-J
  and
  claude
  committed
  53cd623
- Summarize a prediction run's exact score counts as pred/ and drift/ summaries
  
  Vedaant-J
  and
  claude
  committed
  acadd47
- Count every written prediction batch's scores and merge the workers' counts
  
  Vedaant-J
  and
  claude
  committed
  b90db99
- Record a prediction run's exact score counts in its manifest under MLOps
  
  Vedaant-J
  and
  claude
  committed
  0cbd590
- Log prediction runs' rows and scores as data/ and pred/ summaries, with drift
  
  Vedaant-J
  and
  claude
  committed
  ed3a481
- Remember the registered name, version and alias of a fitted or loaded model
  
  Vedaant-J
  and
  claude
  committed
  f0aabe7
- Name prediction runs after their model's version and tag the fit they came from
  
  Vedaant-J
  and
  claude
  committed
  25bb2ce
- Sample system metrics while a prediction scores and log them on its run
  
  Vedaant-J
  and
  claude
  committed
  b5606c2
- Log a prediction run's scoring graph, artifact locations and datasets
  
  Vedaant-J
  and
  claude
  committed
  218be5a
- Log monitoring runs under drift/ and eval/, named after their model
  
  Vedaant-J
  and
  claude
  committed
  a92b859
- Link a fit's test metrics on the logged model to its training table's dataset
  
  Vedaant-J
  and
  claude
  committed
  e92b475
- Document what each tracked run shows and where it is in the MLflow UI
  
  Vedaant-J
  and
  claude
  committed
  3bfdcf9
- Replace characters metric names reject in class columns' pred/ names
  
  Vedaant-J
  and
  claude
  committed
  92351c7
- Count a forecast's predictions per horizon on the target bins
  
  Vedaant-J
  and
  claude
  committed
  59005b5
- Read a quantile in the bin whose counts reach it, not past empty bins
  
  Vedaant-J
  and
  claude
  committed
  1084ada
- Count multilabel predictions: label shares, labels per row, label histograms
  
  Vedaant-J
  and
  claude
  committed
  c687a9d
- Count recommended items exactly for link prediction and ranking runs
  
  Vedaant-J
  and
  claude
  committed
  ab1671c
- Document the forecast, multilabel and recommendation prediction summaries
  
  Vedaant-J
  and
  claude
  committed
  8c0a961
- Check a multilabel prediction run's label counts against its output
  
  Vedaant-J
  and
  claude
  committed
  ffa1092
- List the forecast, multilabel and recommendation summaries in the run table
  
  Vedaant-J
  and
  claude
  committed
  6c4be47
Commits on Oct 30, 2026
- Bin a forecast's target evenly when its split stats have no histogram
  
  Vedaant-J
  and
  claude
  committed
  39b351d
- Log system metrics sampled after a prediction run starts on that run
  
  Vedaant-J
  and
  claude
  committed
  a41e77a
- Record the artifact root's copy of a prediction run among its outputs
  
  Vedaant-J
  and
  claude
  committed
  833efcf
- Tag every run with the engine version, and monitoring runs with their snapshot and task
  
  Vedaant-J
  and
  claude
  committed
  def1de4
- Document the engine-version tag on every run and prediction runs' output locations
  
  Vedaant-J
  and
  claude
  committed
  ab392d4
- Tag trial runs with their fit's snapshot and task type
  
  Vedaant-J
  and
  claude
  committed
  53f5234
- Keep an absolute output location as is instead of joining it to the root
  
  Vedaant-J
  and
  claude
  committed
  a77599d

Commits

Commits on Oct 29, 2026

Commits on Oct 30, 2026