Privately Owned Vehicle Work Group Meeting - 2025/03/03 - Slot 1

[m-zain-khawaja]

Agenda

• Update on Scene3D training
• EgoPath network design discussion
• Update on EgoPath dataset curation
• Update on EgoLanes dataset curation

Scene3D training

@m-zain-khawaja :

The fourth phase Scene3D network suffered from over-fitting on the training data, and although performance was good for KITTI and DDAD datasets, the network produced artefacts in the depth estimation of 'in the wild' images for road scenes as captured by dashcams. Based on further investigation, the likely cause of this was the simulator to real gap induced by relying upon a mixture of simulator images. To remedy this, the network should be trained only on real world data. However, the limitation with real world data is that it is imprecise and noisy. Therefore, the Scene3D network can be trained in a way to have a best of both worlds approach:

1. Create a diverse dataset of only real-world images with relative depth estimation ground truth from SOTA model such as DepthAnythingV2 and perform model distillation to have a fine-tuned, robust and generalizable relative depth estimator which works reliably across diverse real-world scenes and corner case scenarios.

2. Use real-world datasets with ground-truth LIDAR to fine-tune the relative depth estimator model - at this stage it would be worthwhile to consider whether sparse supervision can be included for example, from RADAR to help the network understand the true scene scale at real time.

Last week, I created a first version of a diverse real-world dataset, target is to have over 250K diverse images, I have currently processed 125K images to run a proof-of-concept test and refactored the code to be able to train relative depth.

The Scene3D network is showing very robust generalization capabilities after only 2 epochs of training in the half-size real-world dataset as shown in the test examples below, validating this approach:

Full diverse dataset preparation should be ready this week, with the full dataset network training also expected to begin this week and completion expected by end of next week.

It is planned for the diverse dataset to include data from:

• BDD100K dataset
• Mapillary Vistas dataset
• KITTI dataset
• DDAD dataset
• Driving Stereo Dataset
• ROADWork Dataset
• Zenesact Open Dataset
• Coco Stuff-Things Dataset
• Indian Driving Dataset

Ego Path Network Design

@m-zain-khawaja has completed the Path Context block and EgoPath Head block to predict a Bezier curve for the EgoPath. The proposed loss function is as follows:

Loss = alpha * mAE_endpoints + beta * mAE_derivatives

mAE_endpoints = abs(P1 - P1*) + abs(P4 - P4*), where P1, P4 are predictions of the x,y coordinates of the start and end control points of the cubic Bezier curve defining the Ego Path, and P1* and P4* are ground truth values

• Ensures curve starts and ends at the same location as the ground truth, helping to boost learning of features such as vanishing point, horizon line, perspective, lanes, etc.

mAE_derivates = (1/25) * SUM [t=0 -> t=25] ( Pd|t - Pd*|t), where Pd|t is the derivative at value t along the curve as defined by the predicted control points, and P*d|t is the derivative at value t for the ground truth curve as defined by the ground truth control points

• Ensures the curve has the same shape as the ground truth, acts as a regularizing term to have a smooth prediction, and helps boost learning of features such as road shape, lane curvature, and orientation of on-road and roadside objects

EgoPath Training Tasks

• @TranHuuNhatHuy to update augmentations.py to support keypoints required for EgoPath and load_ego_path.py data loader class to load data for training the network

• Mahmoud Dahmani to implement ego_path_trainer.py class responsible for running the model, calculating the loss, updating gradients and logging results/visualizations

• @m-zain-khawaja to write the train_ego_path.py script responsible for the main training loop

• @TranHuuNhatHuy, @sarun-hub, @devang-marvania, Mahmoud Damani, Augustine Osarogiagbon and @m-zain-khawaja will run various versions of the EgoPath network, experimenting with loss, learning rate, batch size and data mixing

EgoPath Dataset Curation Update

@TranHuuNhatHuy - had a meeting with @m-zain-khawaja to review, parsed dataset is good, auditing in progress and upload to be completed soon.

@docjag - small request to merge latest upstream commits and re-submit PR for ROADWorks dataset since folder structure has changed

Dataset curation tracking

• TuSimple - 6,394 training data samples
• CULane - 15,800 training data samples
• CurveLane - (parsing completed by @TranHuuNhatHuy, data to be uploaded soon)
• BDD100K - 24,500 training samples
• ROADWorks - 4,780 training samples
• Comma2k19 - 31,962 training data samples

EgoLanes Dataset Curation Update

• TuSimple - 6,394 training data samples
• CULane - 15,800 training data samples
• CurveLane - (to be started)
• BDD100K - (in progress by @aadarshkt)

@aadarshkt to share update on BDD100K dataset parsing
@devang-marvania to take on CurveLane dataset parsing for EgoLanes network

Attendees

• @m-zain-khawaja (Senior Tech Lead at AWF and Privately Owned Vehicle Work Group Chair)
• @mitsudome-r (Chair of Autoware Technical Steering Committee)
• Simon Thompson (Director of Engineering, Research at Tier IV)
• @TranHuuNhatHuy (Masters Robotics Student at Tokyo Institute of Technology)
• Mahmoud Dahmani (Robotics Software Engineer at Vecna Robotics)
• @sarun-hub (Undergraduate student at Nagoya University)
• @siddas27 (ADAS Engineer at Canoo)
• @devang-marvania (Lead Engineer - User Experience, Calibration at Gentherm)
• Augustine Osarogiagbon (Canada Revenue Agency, Information Technology Analyst/Developer)

Zoom Meeting Video Recording

Video Meeting Link
Please contact the work group lead (@m-zain-khawaja) to request access to a recording of this meeting.