Differentiable Raycasting for Self-supervised Occupancy Forecasting

By Tarasha Khurana*, Peiyun Hu*, Achal Dave, Jason Ziglar, David Held, and Deva Ramanan

* equal contribution

project page | 5-min summary

You can find our paper on ECVA and Springer. If you find our work useful, please consider citing:

@inproceedings{khurana2022differentiable,
  title={Differentiable Raycasting for Self-Supervised Occupancy Forecasting},
  author={Khurana, Tarasha and Hu, Peiyun and Dave, Achal and Ziglar, Jason and Held, David and Ramanan, Deva},
  booktitle={European Conference on Computer Vision},
  pages={353--369},
  year={2022},
  organization={Springer}
}

• Download nuScenes dataset, including the CANBus extension, as we will use the recorded vehicle state data for trajectory sampling. (Tip: the code assumes they are stored under /data/nuScenes.)
• Download ONCE dataset. (Tip: the code assumes they are stored under /data/once)
• Install packages and libraries (via conda if possible), including torch, torchvision, tensorboard, cudatoolkit-11.1, pcl>=1.9, pybind11, eigen3, cmake>=3.10, scikit-image, nuscenes-devkit. For your convenience, and environment.yml file has also been provided. (Tip: verify location of python binary with which python.)
• Compile code for Lidar point cloud ground segmentation under lib/grndseg using CMake.

• Run preprocess.py to generate ground segmentations
• Run precast.py to generate future visible freespace maps and a set of freespace contour points
• Run rasterize.py to generate BEV object occupancy maps and object "shadow" maps.

Refer to train.py, which can be run using train.sh. You might find these arguments useful:

• --dataset: Dataset name
• --data-root: Path to the dataset
• --data-version: Which subset of the dataset
• --sampled-trajectories: Whether to form clothoid-based or data-driven candidate trajectories
• --sample-set: What subset of a dataset to sample the data-driven trajectories from
• --train-on-all-sweeps: Whether to train only on key frames or all frames for nuScenes; to be always used for ONCE
• --nvf-loss-factor: How much weight to apply to the binary cross entropy loss for freespace

Refer to test_once.py and test_nusc.py, which can be run using test_once.sh and test_nusc.sh for the ONCE and nuScenes datasets respectively. You might find these arguments useful.

• --compute-dense-fvf-loss: Compute the dense freespace classification metrics in Table 1 in paper
• --compute-raydist-loss: Compute the error for the predicted distance along every ray in the groundtruth

Planning losses are always computed.

Note that in the provided test_once.py, test_nusc.py, train.py and model.py, the model names refer to the following:

• VanillaNeuralMotionPlanner: An imitation learning baseline that just follows the expert at every way point
• ObjGuidedNeuralMotionPlanner: Best possible re-implementation of the Neural Motion Planner
• VFGuidedNeuralMotionPlanner: Implementation of the Safe Local Motion Planner
• VFExplicitNeuralMotionPlanner: Row (c) in Table 3
• VFSupervisedNeuralMotionPlanner: Row (d) in Table 3
• In order to simulate the two models in Table 1 in paper, just weigh the Lm or max-margin loss with 0 in model.py for the VFGuided and LatOccVFSupervised neural motion planners.

Code heavily adapted from @peiyunh's repository (Safe Local Motion Planning at CVPR '21).