Blog / 2026-09-23

Night cuts off-road segmentation by 53–64% — and synthetic data only wins part of it back

Public off-road datasets are almost all daytime, dry and clear. We measured how much segmentation models lose when conditions turn bad, and how much synthetic hard cases recover — with a protocol designed not to flatter the result.

The setup

• Data: 1,200 clear RELLIS-3D training frames; 600 real test frames at 640×400, 19 classes.
• Models: SegFormer-B0, SegFormer-B2, DeepLabV3-MobileNetV3.
• Held-out degradations: gains are reported on generators never used to make training data — ImageNet-C fog, spatter and motion blur, and an unprocessing-based low-light model. Testing on the code you trained with is how augmentation results flatter themselves.
• Control arm: every synthetic-data fine-tune is compared with an identical fine-tune on clean data, over several seeds.

1 · Night is the universal failure

Every model loses more than half its mIoU at night. Two independent night models — our headlight-and-sensor-noise simulation and a low-light model from the literature — agree closely on SegFormer-B0: −64.4% vs −63.6%. That agreement is the best evidence the number is real rather than an artefact of one simulator.

2 · Architectures fail differently

Night is the worst condition on average (−60%). The biggest architecture split is fog: CNN models lose 37%, Transformer models 20%. A single “robustness score” per model hides exactly what you need to know: which condition your model can’t handle. The full per-condition table is on the leaderboard.

3 · 200 synthetic frames buy back 2.4–2.8 points, honestly measured

Clear-weather mIoU moved by less than a point, so the robustness wasn’t bought with a regression.

4 · The synthetic-to-real gap, in one number

Night mIoU for SegFormer-B0, same model, same training:

• tested with our own night generator (unseen seeds): 13.7 → 26.2 (+12.5)
• tested with an independent low-light model: 13.3 → 18.3 (+4.9)

The gain shrinks by more than half when the test data comes from a different simulator, and real night footage will differ again. If you validate synthetic augmentation on synthetic data, you will overestimate it. Keep a real adverse-condition validation set that never touches training, and select checkpoints on it.

Limitations

• RELLIS-3D has no real dust or night frames, so degraded test inputs are synthetic (from held-out generators). Validation on real adverse captures is the next step.
• Models are small and run at 640×400 on a single 4 GB consumer GPU; absolute mIoU is below full-resolution published results. The relative drops are the point.
• We don’t synthesise motion blur yet, and it shows: almost no gain there.