UNIVERSITY OF WATERLOO · ME 780

LiDAR–Camera Fusion
for Road Segmentation

Camera and LiDAR fusion for semantic road segmentation using a modified SegNet encoder–decoder architecture evaluated on the KITTI road dataset.

Sooyong Kim · Tarun Joseph

Project Overview

Multimodal perception for road detection

The project investigated whether combining RGB camera information with LiDAR-derived depth information could improve semantic road segmentation compared with camera-only perception.

0.9364 Best MaxF · Late Fusion
0.9010 RGB-only MaxF
289 KITTI training samples
96 KITTI test samples
Motivation

Why fuse LiDAR and camera data?

Camera-based semantic segmentation provides dense visual information, but its performance can be affected by illumination, image quality, and environmental conditions.

LiDAR provides complementary geometric and distance information that is less dependent on scene appearance. Combining these modalities therefore provides an opportunity to improve the robustness of road perception.

This project compared camera-only road segmentation with two multimodal architectures: early fusion and late fusion.

Core tasks

  • KITTI camera and LiDAR data processing
  • LiDAR-to-camera coordinate transformation
  • Depth-map generation
  • Modified SegNet development
  • Early-fusion implementation
  • Late-fusion implementation
  • Quantitative performance evaluation
Data Pipeline

LiDAR projection onto the camera plane

KITTI calibration parameters were used to transform Velodyne LiDAR points into the camera coordinate system and project them onto the RGB image plane.

Processing sequence

01 RGB Image
→
02 LiDAR Point Cloud
→
03 Calibration & Projection
→
04 Depth Map
→
05 Fusion Network

The projected LiDAR points were converted into a sparse depth representation aligned with the RGB image and resized for network training and evaluation.

LiDAR-derived depth representation and corresponding KITTI RGB image
LiDAR-derived depth representation aligned with the corresponding KITTI RGB image.
Neural Network

Modified SegNet architecture

The segmentation model was developed from the SegNet encoder–decoder architecture and adapted for binary road segmentation.

The network incorporated convolutional blocks, batch normalization, ReLU activation, dropout, and transposed convolution for spatial reconstruction in the decoder.

Training configuration

  • Framework: Keras / TensorFlow 2
  • Dataset: KITTI Road
  • Optimizer: Adam
  • Learning rate: 0.0001
  • Batch size: 8
  • Epochs: 30
  • Metrics: MaxF, Accuracy, IoU
Modified SegNet encoder-decoder network architecture
Modified SegNet encoder–decoder architecture used for binary road segmentation.
Sensor Fusion

Early fusion vs. late fusion

Two architectures were implemented to investigate how the location of multimodal fusion affected segmentation performance.

Comparison of early and late LiDAR-camera fusion architectures
Early fusion combines RGB and LiDAR information at the input level, whereas late fusion combines features after independent encoder processing.
EARLY FUSION

Fusion at the input level

RGB and LiDAR-derived depth information were concatenated before entering the encoder. The network therefore learned directly from a combined low-level multimodal representation.

LATE FUSION

Fusion after independent encoding

RGB and LiDAR information were processed through independent encoder streams and combined near the end of the encoding stage before reconstruction in the decoder.

Experimental Results

Fusion improved road segmentation

Late fusion produced the strongest overall result among the three sensor configurations evaluated.

Configuration MaxF Accuracy IoU Training Time
RGB only 0.9010 0.7828 0.4755 392 s
Early Fusion 0.9153 0.7875 0.4665 392 s
Late Fusion 0.9364 0.7958 0.4898 584 s
Qualitative road-segmentation comparison of RGB-only, early fusion, and late fusion
Qualitative comparison of RGB-only, early-fusion, and late-fusion road segmentation on KITTI samples.
Network Comparison

Modified SegNet vs. baseline SegNet

Before evaluating multimodal fusion, the modified architecture was compared with the original SegNet using RGB images alone.

0.8142 Original SegNet MaxF
0.9010 Modified SegNet MaxF
Engineering Interpretation

Performance versus computational cost

Early fusion improved MaxF relative to the camera-only configuration without increasing the reported training time.

Late fusion achieved the highest MaxF, accuracy, and IoU, but required two independent encoder streams. Training time therefore increased from 392 seconds to 584 seconds.

Key result

MaxF increased from 0.9010 with RGB-only input to 0.9364 with late LiDAR-camera fusion.

Limitations

Scope of the experiment

The experiments used the KITTI road dataset, which primarily represents favorable road, weather, and lighting conditions.

The results therefore do not establish performance across the full range of adverse conditions encountered in production autonomous-driving environments.

Future work

  • Evaluate more diverse road and weather conditions
  • Investigate real-time segmentation
  • Explore pyramid-based fusion architectures
  • Investigate cross-fusion strategies
Technical Report

Final technical report

The original academic report is being revised into a polished final technical report while preserving the original experiments, quantitative results, technical content, figures, and references.

REPORT

LiDAR–Camera Fusion for Road Segmentation

A revised technical report will provide the complete methodology, mathematical formulation, sensor calibration, network architecture, experimental results, discussion, and references.

View Final Report
← Back to Waterloo Research