feat: task eval + training hardening batch (review items 1-10) - #2
Merged
Merged
Conversation
…ix holdout leak and cache bypass - split_with_postcutoff: post-cutoff records now exclusive to test_postcutoff (no more leakage into train); drop dead min_postcutoff param - CacheProvider: drop structured() override that bypassed the sqlite cache - qlora.evaluate: honor quantization setting instead of hardcoded 4-bit - train evaluate/export/task-eval/chat: --model defaults to latest run dir - TrainingConfig: save_steps (checkpointing, keep last 2) and seed fields - train run --resume: TRL native checkpoint resume - train task-eval: verdict accuracy, FP/FN rates, CWE accuracy on holdout - train chat: interactive REPL on adapter or merged model - CI: CPU smoke training job (tiny model, no GPU/bitsandbytes)
…antics, chat cleanup
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Carries the phase-2 training feature set (task eval, checkpoints/resume, chat CLI, CPU smoke, holdout leak + cache fixes) rebased onto current
main, plus the full 10-item quality/security/functional review batch (TDD red->green, no new dependencies, ruff clean).Security:
case.idallowlist^[A-Za-z0-9_-]{1,64}$(path traversal via crafted case files)ProviderConfig.base_urlvalidator: http/https only, rejects link-local + cloud-metadata hosts (SSRF guard); loopback stays allowedMLGW_API_KEY:deadbeefplaceholder (empty default, consistent with other providers)Functional:
train preparedefault output ->datasets/prepared/(directory that run/evaluate/task-eval read by default)latest_run_dir: mtime-based, honors configuredtraining.output_dirtrain run --resumereuses the latest run dir; fails fast with a clear message when no checkpoint existsQuality:
train task-evalCLI path flattens the message list (matches typedgenerate(str | list[str]));prediction_from_textcatches onlyProviderErrorsmoke_cpuskips only when torch is missing; missing deps / unreachable HF hub now fail (exit 1)train chatmisleadingname_or_path+generate_reply_fn, double dataset read indataset splitrun_namegains microsecond precisionNote: replaces
feat/phase2-training-hardening(orphaned history after previous squash merge).Verification
ruff check src tests: cleanpytest tests/unit: all passedbrainforge config schema --check: in syncgitleaks git --all: no leaks found