Leaderboard · updated 2026-09-23
Which degradation breaks which segmentation model — and by how much.
Relative mIoU drop on real RELLIS-3D test frames when the same frames are degraded. Lower is more robust. Where an independent (held-out) generator exists we use it, so no model — including ours — has seen the corruption code.
| # | Model | Clean mIoU | DustDust plume | Nightheld-out | Rain on lensRain on lens | Mud / spatterheld-out | Fogheld-out | Motion blurheld-out | Mean drop |
|---|---|---|---|---|---|---|---|---|---|
| 1 | SegFormer-B2Transformer · 27.4M | 42.9 | −47.5% | −53.3% | −24.6% | −28.8% | −17.7% | −24.0% | −32.6% |
| 2 | DeepLabV3-MobileNetV3CNN · 11.0M | 35.0 | −36.4% | −63.6% | −31.9% | −26.2% | −37.5% | −27.4% | −37.2% |
| 3 | SegFormer-B0Transformer · 3.7M | 38.0 | −43.7% | −63.6% | −26.1% | −43.3% | −23.0% | −30.8% | −38.4% |
| – | Mask2Former (Swin-T)Mask transformer | running on our GPU now | |||||||
| – | OneFormer (Swin-T)Universal transformer | needs a larger GPU — queued | |||||||
| – | SAM 2 (Hiera-T encoder + seg head)Foundation encoder | class-agnostic model; needs a trained semantic head — queued | |||||||
reading it
Night is the worst condition on average (−60%). The biggest architecture split is fog: CNN models lose 37%, Transformer models 20%. That’s why a single “robustness score” per model hides what you actually need to fix.
Want your model on this board? Send it — we run the same protocol and publish with your permission. Or get a private failure-map report.
How much a pack wins back
Fine-tuning from each baseline. Control = the same extra training on clean data only, so the gain can’t be extra steps. Held-out degraded mIoU, mean over seeds.
| Model | Control | +200 synthetic frames | + online synthesis | Clean-weather change |
|---|---|---|---|---|
| SegFormer-B21 seed(s) | 29.4 | 32.2 (+2.8) | 33.4 (+3.9) | +0.1 |
| DeepLabV3-MobileNetV33 seed(s) | 21.7 | 24.5 (+2.8) | 25.6 (+3.9) | -0.5 |
| SegFormer-B03 seed(s) | 22.8 | 25.2 (+2.4) | 26.0 (+3.2) | +0.6 |
Methodology
- Real test frames only. No synthetic frame is ever scored as ground truth.
- Held-out generators. Night = Brooks/Cui unprocessing low-light; mud = ImageNet-C spatter; fog and motion blur = ImageNet-C. None is used to make training data.
- Control arm and seeds. Every gain is measured against identical training on clean data, over several seeds.
- Metric. mIoU over the RELLIS-3D classes present in the test split; relative drop = (clean − degraded) / clean, averaged over 3 severities.
limits, stated plainly
- Degraded test inputs are synthetic (held-out generators): RELLIS-3D contains no real dust or night frames.
- Small models at 640×400 on one consumer GPU; absolute mIoU is below full-resolution published numbers.
- Rows marked queued are not measured yet — no numbers are shown for them.
Read the evaluation protocol & code layout.