Robotics Portfolio · Project 02

ROS 2 Vision Perception Pipeline

Real-time object detection, multi-object tracking, and instance segmentation using a pretrained YOLO model with GPU acceleration in ROS 2 Humble.

Author: Sooyong Kim · Platform: Ubuntu 22.04 / ROS 2 Humble · GPU: NVIDIA GeForce RTX 5060 Laptop GPU

1. Abstract

This project implements a real-time vision perception pipeline for robotics applications in ROS 2. The system receives live camera images, performs object detection, multi-object tracking, and instance segmentation, and republishes an annotated image stream for downstream visualization or robotic decision-making. The pipeline was designed as an integration-focused robotics perception project rather than a model-training exercise; pretrained YOLO weights were used to emphasize ROS 2 integration, real-time inference, tracking continuity, latency measurement, and reproducible system evaluation.

In a 2,136-frame test sequence, the system achieved an average processing latency of 18.26 ms, a P95 processing latency of 24.87 ms, and an average YOLO inference latency of 8.70 ms. With the camera publisher configured at approximately 30 Hz, the pipeline maintained stable real-time operation while processing scenes containing up to 11 detected objects.

Figure 1 — Final unified perception output Insert a representative screenshot showing masks, bounding boxes, track IDs, FPS, latency, and object count.

Recommended width: 1200–1600 px.

Figure 1. Unified ROS 2 perception output generated by the final pipeline.

2. Project Objectives

Integrate a pretrained deep-learning vision model into ROS 2.
Perform real-time object detection on a live image stream.
Maintain object identity across frames using multi-object tracking.
Generate instance-level segmentation masks for detected objects.
Publish annotated results as ROS 2 image topics.
Measure runtime performance using FPS, processing latency, inference latency, and object count.
Provide a reproducible launch configuration and quantitative benchmark workflow.

3. System Architecture

The perception stack is organized as a modular ROS 2 pipeline. A camera publisher acquires frames from the local webcam and publishes them as sensor_msgs/Image messages. The unified perception node subscribes to the image stream, executes YOLO-based tracking and instance segmentation on the GPU, overlays diagnostic information, and republishes the resulting visualization.

Figure 2 — System architecture diagram Suggested content:
Camera → /camera/image_raw → Unified Perception Node → Detection + Tracking + Instance Segmentation + Performance Logging → /vision/unified_image

Figure 2. High-level ROS 2 data flow and perception architecture.

3.1 ROS 2 Nodes

Node	Purpose
`camera_publisher`	Captures webcam frames and publishes them to ROS 2.
`yolo_detector`	Performs object detection and tracking during component-level development.
`segmentation_node`	Performs YOLO instance segmentation during component-level development.
`unified_perception`	Combines tracking, instance segmentation, visualization, and performance logging.

3.2 ROS 2 Topics

Topic	Message Type	Description
`/camera/image_raw`	`sensor_msgs/Image`	Raw camera input.
`/vision/detection_image`	`sensor_msgs/Image`	Detection/tracking visualization.
`/vision/segmentation_image`	`sensor_msgs/Image`	Instance segmentation visualization.
`/vision/unified_image`	`sensor_msgs/Image`	Final combined perception output.

4. Technical Implementation

4.1 Camera Interface

OpenCV is used to acquire webcam frames through cv2.VideoCapture. Each frame is converted to a ROS 2 image message using cv_bridge and published at approximately 30 Hz.

Figure 3 — Camera input / raw ROS 2 image Insert a screenshot of the raw webcam feed or an rqt_image_view capture of /camera/image_raw.

Figure 3. Raw image stream entering the perception pipeline.

4.2 Object Detection and Tracking

The detector uses the Ultralytics YOLO interface with pretrained weights. Tracking is executed through the model's persistent tracking interface, allowing IDs to remain associated with objects across consecutive frames. This provides temporal continuity beyond frame-by-frame detection.

Figure 4 — Object tracking example Insert a frame where multiple objects have visible, persistent tracking IDs.

Figure 4. Multi-object tracking with persistent object IDs.

4.3 Instance Segmentation

The final system uses a YOLO segmentation model to generate instance masks for detected objects. This is instance segmentation, not semantic segmentation: each detected object receives an independent mask, allowing the output to preserve object identity while also describing pixel-level shape.

Figure 5 — Instance segmentation example Insert a representative frame showing object masks overlaid on the live camera image.

Figure 5. Instance-level masks produced by the YOLO segmentation model.

4.4 Unified Perception Node

The final node combines tracking and instance segmentation in a single inference path to avoid redundant model execution. It also overlays FPS, object count, processing latency, and device information before publishing the annotated result.

Design choice: The final pipeline uses one segmentation-capable YOLO model for both object localization and masks, while tracking state is maintained across frames. This reduces unnecessary duplication compared with running separate detection and segmentation models concurrently.

5. Experimental Setup

Hardware

NVIDIA GeForce RTX 5060 Laptop GPU
Webcam input

Software

Ubuntu 22.04
ROS 2 Humble
Python 3.10
PyTorch 2.11.0 + CUDA 12.8
Ultralytics 8.4.147
OpenCV 4.10.0
NumPy 1.26.4

The camera publisher was configured at approximately 30 Hz. Runtime data were logged to CSV for every processed frame, including timestamp, observed FPS, total processing latency, YOLO inference latency, and detected-object count. The first frame was excluded from steady-state statistics because it contained initialization and GPU warm-up overhead.

6. Quantitative Results

Metric	Result
Analyzed frames	2,136
Average observed FPS	31.95 FPS
Minimum observed FPS	16.09 FPS
Maximum observed FPS	80.69 FPS
Average processing latency	18.26 ms
P95 processing latency	24.87 ms
Average YOLO inference latency	8.70 ms
P95 YOLO inference latency	12.52 ms
Average objects per frame	1.83
Maximum objects in a frame	11

Interpretation of FPS: The observed FPS should not be treated as the model's unconstrained maximum throughput. The input publisher was configured near 30 Hz, and ROS 2 callback scheduling can produce short-term timing jitter. The latency metrics are therefore the more meaningful indicators of processing performance.

6.1 Frame-Rate Stability

Figure 6 — FPS over time Insert the Jupyter plot containing instantaneous FPS and the 30-frame moving average.

Figure 6. Observed frame-rate behavior during the experiment. The moving average remains close to the 30 Hz input rate.

6.2 Processing and Inference Latency

Figure 7 — Processing and inference latency over time Insert the latency time-series plot.

Figure 7. End-to-end processing latency compared with YOLO inference latency.

The mean processing latency was 18.26 ms and the P95 processing latency was 24.87 ms. Because a 30 Hz stream has a nominal frame period of approximately 33.3 ms, the measured P95 value remained below the camera frame interval.

6.3 Latency Distribution

Figure 8 — Processing latency distribution Insert the histogram with mean and P95 markers.

Figure 8. Distribution of processing latency after excluding the initialization frame.

The latency distribution shows two visible operating bands rather than a perfectly unimodal distribution. Possible contributors include variations in tracking updates, mask rendering, object count, ROS 2 callback scheduling, and visualization overhead. The available measurements do not isolate one specific cause, so the source of the bimodal pattern should be treated as unresolved rather than attributed to a single subsystem.

6.4 Object Count During Test

Figure 9 — Detected object count over time Insert the object-count plot from Jupyter.

Figure 9. Scene complexity varied during the test, reaching a maximum of 11 detected objects in one frame.

7. Launch and Reproducibility

The complete pipeline is launched with:

ros2 launch vision_perception vision_demo.launch.py

Expected high-level flow:

camera_publisher
    ↓
/camera/image_raw
    ↓
unified_perception
    ↓
/vision/unified_image

8. Project Structure

vision_perception/
├── launch/
│   └── vision_demo.launch.py
├── results/
│   └── vision_results.csv
├── vision_perception/
│   ├── __init__.py
│   ├── camera_publisher.py
│   ├── segmentation_node.py
│   ├── unified_perception.py
│   └── yolo_detector.py
├── package.xml
├── setup.py
├── setup.cfg
└── README.md

9. Engineering Discussion

9.1 What This Project Demonstrates

ROS 2 node and topic integration for deep-learning perception.
GPU-accelerated real-time inference.
Persistent multi-object tracking.
Instance-level segmentation.
Runtime instrumentation and quantitative benchmarking.
Launch-file based orchestration and reproducible execution.

9.2 Limitations

The model uses pretrained general-purpose weights rather than a robotics-specific custom dataset.
The experiment evaluates a local webcam stream rather than a calibrated robot camera or multi-camera system.
Observed FPS is constrained by the publisher rate and should not be interpreted as maximum model throughput.
The current benchmark does not separately profile preprocessing, tracking, rendering, ROS 2 transport, and postprocessing overhead.
Tracking accuracy metrics such as MOTA, HOTA, or IDF1 were not evaluated because no labeled tracking ground truth was used.

9.3 Future Work

Integrate calibrated RGB-D or stereo input.
Publish structured detection and tracking messages in addition to visualization images.
Benchmark CPU and GPU execution separately.
Profile each pipeline stage independently.
Evaluate tracking accuracy on a labeled benchmark.
Fine-tune the model for a robotics-specific dataset if application-specific perception is required.
Fuse visual detections with LiDAR or depth sensing in a later sensor-fusion project.

10. Demo Media

Figure / Video Placeholder A — Main demo Insert the final recorded demonstration or a clickable video thumbnail.

Suggested caption: “Real-time ROS 2 detection, tracking, and instance segmentation demo.”

Figure / Video Placeholder B — Tracking continuity Insert a short sequence or screenshot series emphasizing persistent track IDs.

Figure / Video Placeholder C — Segmentation detail Insert a close-up example showing mask quality around object boundaries.

11. Repository

GitHub portfolio repository: https://github.com/greatsyong/SK

Project directory: 02_vision_perception