Format
- Images: 8-bit sRGB JPEG (quality 95). Full packs 1920×1200; free samples 640×400.
- Labels: single-channel uint8 PNG with RELLIS-3D ontology ids (not train ids).
0= void / ignore. - Colour labels: RGB PNG in the official RELLIS-3D palette, for eyeballing only.
- manifest.json: per frame — condition, generator, severity (1–3), seed, source frame.
- ontology.json: id → name, RGB, train id.
Directory layout
siltframe-<condition>/ ├── images/ 0000.jpg 0001.jpg … ├── labels/ 0000.png … # uint8 RELLIS-3D ids ├── labels_color/ 0000.png … ├── manifest.json ├── ontology.json ├── README.md └── LICENSE.txt
Label mapping
Identical to RELLIS-3D. If you train on train ids 0–18, map with the table below (void → 255, ignored in the loss).
| RELLIS id | Class | Colour (RGB) | Train id |
|---|---|---|---|
| 0 | void | (0, 0, 0) | 255 |
| 1 | dirt | (108, 64, 20) | 0 |
| 3 | grass | (0, 102, 0) | 1 |
| 4 | tree | (0, 255, 0) | 2 |
| 5 | pole | (0, 153, 153) | 3 |
| 6 | water | (0, 128, 255) | 4 |
| 7 | sky | (0, 0, 255) | 5 |
| 8 | vehicle | (255, 255, 0) | 6 |
| 9 | object | (255, 0, 127) | 7 |
| 10 | asphalt | (64, 64, 64) | 8 |
| 12 | building | (255, 0, 0) | 9 |
| 15 | log | (102, 0, 0) | 10 |
| 17 | person | (204, 153, 255) | 11 |
| 18 | fence | (102, 0, 204) | 12 |
| 19 | bush | (255, 153, 204) | 13 |
| 23 | concrete | (170, 170, 170) | 14 |
| 27 | barrier | (41, 121, 255) | 15 |
| 31 | puddle | (134, 255, 239) | 16 |
| 33 | mud | (99, 66, 34) | 17 |
| 34 | rubble | (110, 22, 138) | 18 |
Add it to your training code
Packs use the same image/label pairing as RELLIS-3D, so any RELLIS dataset class works. If you don’t have one, this is enough:
class RellisStyleDataset(torch.utils.data.Dataset):
def __init__(self, img_dir, lab_dir, transform):
self.imgs = sorted(Path(img_dir).glob("*.jpg")); self.lab_dir, self.tf = Path(lab_dir), transform
def __len__(self): return len(self.imgs)
def __getitem__(self, i):
img = np.array(Image.open(self.imgs[i]).convert("RGB"))
lab = LUT[np.array(Image.open(self.lab_dir / (self.imgs[i].stem + ".png")))] # RELLIS id → train id
return self.tf(img, lab)
Then the five lines that matter:
# your existing real dataset stays as is
pack = RellisStyleDataset("siltframe-dust/images", "siltframe-dust/labels", transform=train_tf)
mixed = ConcatDataset([real_train, pack])
w = [0.7 / len(real_train)] * len(real_train) + [0.3 / len(pack)] * len(pack)
loader = DataLoader(mixed, batch_size=8, sampler=WeightedRandomSampler(w, len(real_train)))
That is exactly the +200 frames arm on the benchmark page.
How much synthetic data to mix in
- Start at 20–30 % synthetic per batch. Our measured gains used 30 % (fixed pack) and 50 % (on-the-fly synthesis); both left clean-weather mIoU within ±0.7 points.
- Fine-tune from your current checkpoint at ~½ the original learning rate for a few epochs rather than retraining from scratch.
- We haven’t measured synthetic shares above 50 %. Don’t go past that until your real validation set says it helps.
keep a real validation set — separate
Never select checkpoints or tune the mixing ratio on synthetic frames. Hold out real frames from your own conditions (ideally real dusty / night captures) and report on those. A model that improves on synthetic validation data can stay flat in the field — that is the synthetic-to-real gap, and the only way to see it is a real validation set that never touches training.
How we evaluate
Baseline → control fine-tune (clean only) vs. pack fine-tune, scored on real test frames degraded by held-out generators (ImageNet-C, Brooks/Cui low-light). Details and results: benchmark.