surface-defect-yolo

Defect detection on the NEU-DET steel surface dataset, run the way I run inspection-model experiments: audit the labels before training, train a controlled model×resolution matrix, then analyze which ground-truth boxes get missed and why, instead of stopping at a single mAP number.

Everything is reproducible on one GPU from four commands (see Reproduce). The dataset (1,800 images, 6 defect classes, 200×200 grayscale) is downloaded from Kaggle at run time and is not redistributed here.

Pipeline

1. Label audit (scripts/audit_labels.py) — before any training: class balance, box-size and aspect distributions, out-of-bounds boxes, near-duplicate boxes. Output: results/label_audit.md.
2. Experiment matrix (scripts/train_matrix.py) — 4 models (YOLOv8n/s, YOLO11n/s) × 2 input sizes (320, 640) = 8 runs, fixed seed, deterministic, early stopping. One config file: configs/matrix.yaml.
3. Evaluation (scripts/eval_matrix.py) — overall and per-class AP on the validation split for every run, written to results/metrics.csv.
4. Failure analysis (scripts/analyze_failures.py) — match predictions to GT boxes one-to-one, then look at the misses: per-class GT recall, montages of missed crops, and a DINOv2 t-SNE of all GT crops colored detected/missed. If misses form their own cluster in a label-free embedding, the class hides a visually distinct sub-type the detector never learned — a taxonomy problem, not a capacity problem.

Results

Validation split, 100-epoch deterministic runs (full tables: results/RESULTS.md, results/metrics.csv).

run	imgsz	params (M)	mAP50	mAP50-95
yolo11n_sz640	640	2.6	0.739	0.397
yolo11s_sz640	640	9.4	0.737	0.397
yolov8n_sz640	640	3.0	0.736	0.387
yolo11n_sz320	320	2.6	0.730	0.390
yolov8n_sz320	320	3.0	0.721	0.388
yolov8s_sz640	640	11.1	0.720	0.381
yolo11s_sz320	320	9.4	0.718	0.378
yolov8s_sz320	320	11.1	0.707	0.376

What the matrix says:

• Upscaling helps a little (640 beats 320 by 1–3 points per model); model capacity does not (n matches or beats s across the board — 1.4k training images can't feed the bigger models). For calibration, recent papers put vanilla YOLOv8n around 0.75–0.78 mAP50 on NEU-DET with longer schedules; a 100-epoch matrix landing at 0.71–0.74 is consistent with that.
• The class spread is the real story: patches 0.92 and scratches 0.85 vs crazing 0.43–0.54 in AP50. Averages hide it.

Failure analysis (best run, conf 0.25)

class	GT boxes	recall
patches	193	0.865
scratches	121	0.851
pitted_surface	87	0.713
inclusion	159	0.704
rolled-in_scale	132	0.545
crazing	162	0.315

For each class, all GT crops are embedded with DINOv2 (a label-free encoder) and projected with t-SNE, colored detected vs missed (results/tsne_*.png). For crazing the missed crops do not form their own cluster — they are interleaved with detected ones across the embedding space:

So crazing's failure is not a hidden visual sub-type the detector never learned (that pattern would show as a separate missed-cluster). Crazing is a diffuse texture annotated with coarse, loosely-bounded boxes — box-level IoU matching is partly ill-posed for it, which caps recall regardless of model capacity. The practical lesson: for diffuse-texture classes, fixing the annotation unit (region/mask level) beats scaling the detector.

Reproduce

pip install -r requirements.txt
python scripts/download_data.py && python scripts/voc2yolo.py
python scripts/audit_labels.py
python scripts/train_matrix.py --device 0        # ~1 h on a single modern GPU
python scripts/eval_matrix.py --device 0
python scripts/make_report.py
python scripts/analyze_failures.py --weights runs/<best>/weights/best.pt --device 0

Notes

• The matrix uses the dataset's provided train/validation split; the same split serves model selection and reporting, so treat absolute numbers as validation scores, not test scores.
• One annotation file (crazing_240.xml) has no matching image in the Kaggle distribution and is skipped by the converter.

License

Code: MIT. NEU-DET data: see the dataset's own terms (research use).