Robotics Portfolio Project 02

ROS 2 Vision Perception Pipeline

A real-time ROS 2 perception system that converts live RGB camera data into object-level spatial information through detection, persistent tracking, and instance segmentation, with GPU-accelerated inference and quantitative runtime evaluation.

View source on GitHub →

Demo

Watch on YouTube →

Demo footage is shown for qualitative visualization. Quantitative results below are taken from a separate 2,136-frame benchmark run.

1. Project Objective and Robotics Use Case

A camera produces a high-bandwidth stream of pixels, but a robot cannot make useful decisions from raw RGB values alone. For navigation, manipulation, human-aware motion, scene monitoring, or sensor fusion, the perception layer must convert images into structured information: what objects are present, where they are, which detections belong to the same physical object over time, and which image pixels belong to each object.

The purpose of this project was therefore not simply to run a pretrained neural network. It was to build the vision front-end as a ROS 2 subsystem that could later feed a larger autonomous robot architecture. Detection provides class and bounding-box information; tracking adds temporal identity; instance segmentation adds object-level pixel geometry. Those outputs can subsequently support obstacle reasoning, target selection, human or object tracking, visual servoing, depth/LiDAR association, and higher-level planning.

The implementation was intentionally built as an integration and systems-engineering exercise. Instead of spending the project on custom model training, pretrained weights were used so that the main engineering questions could be examined directly: ROS 2 image transport, GPU inference, temporal tracking, segmentation, visualization, runtime instrumentation, and reproducible benchmark analysis.

2. Technology Selection and Design Rationale

Several perception stacks can solve parts of the same problem. The selected combination was chosen for rapid ROS 2 integration, real-time execution on an NVIDIA laptop GPU, availability of pretrained models, and the ability to perform detection, segmentation, and tracking through one practical Python-facing framework.

2.1 Object Detection / Perception Model

2.2 Why Instance Segmentation?

Bounding boxes indicate approximate object location, but many robotics tasks require object shape at pixel level. Semantic segmentation assigns a class to each pixel but does not inherently distinguish two separate objects of the same class. Instance segmentation provides both class information and a separate mask for each object, which fits naturally with object tracking because each tracked entity can retain both an identity and a spatial footprint.

2.3 PyTorch, CUDA, and Deployment Alternatives

PyTorch was used because the Ultralytics Python stack integrates directly with it and because CUDA-enabled PyTorch can execute model tensors on the NVIDIA GPU without an additional model-conversion stage. For a development project this keeps the path from model loading to ROS 2 inference transparent and easy to debug.

3. System Architecture

Webcam
  │
  ▼
camera_publisher
  │
  └── /camera/image_raw  [sensor_msgs/msg/Image]
                │
                ├──────────────► yolo_detector
                │                    │
                │                    └── /vision/detection_image
                │                        [sensor_msgs/msg/Image]
                │
                ├──────────────► segmentation_node
                │                    │
                │                    └── /vision/segmentation_image
                │                        [sensor_msgs/msg/Image]
                │
                └──────────────► unified_perception
                                     │
                                     ├─ detection / class confidence
                                     ├─ persistent object tracking
                                     ├─ instance-mask generation
                                     ├─ visualization overlays
                                     └─ runtime performance logging
                                              │
                                              ▼
                                  /vision/unified_image
                                  [sensor_msgs/msg/Image]

The component nodes were useful during development because detection/tracking and segmentation could be tested independently. The final demonstration uses unified_perception, which combines the functions into one inference path and publishes a single annotated image. This is why only /camera/image_raw and /vision/unified_image were visible in the final rqt demonstration even though the package defines additional component-level topics.

4. ROS 2 Nodes and Data Interfaces

4.1 Node Responsibilities

4.2 ROS 2 Topics and Message Types

5. Technical Implementation

5.1 Camera Acquisition

OpenCV acquires the webcam stream through cv2.VideoCapture. Each frame is converted to sensor_msgs/msg/Image using cv_bridge. The camera publisher targets approximately 30 Hz, establishing the input rate seen by the downstream perception nodes.

5.2 Detection, Tracking, and Instance Masks

For every incoming frame, the segmentation-capable YOLO model produces object localization, class labels, confidence values, and object masks. Persistent tracking associates detections across frames so that an object can retain a track ID while it moves through the scene. The unified node then renders these outputs together rather than launching separate detection and segmentation inference passes.

5.3 Runtime Instrumentation

The benchmark records an observed frame-rate estimate, total processing latency, YOLO inference latency, and detected-object count for each processed frame. The first frame is excluded from steady-state statistics because model initialization, memory allocation, and GPU warm-up produce a one-time timing condition that is not representative of continued operation.

6. Experimental Setup and Benchmark Method

The camera publisher was configured at approximately 30 Hz. A total of 2,136 steady-state frames were analyzed after excluding the first initialization frame. The benchmark was designed to characterize runtime behavior of the complete ROS 2 perception path rather than to measure detector accuracy against a labeled dataset. Accordingly, the reported metrics quantify timing, throughput behavior, and scene load; they do not represent mAP, segmentation IoU, or tracking-accuracy scores.

7. Quantitative Results and Engineering Interpretation

8. Benchmark Analysis

8.1 Frame-Rate Stability

The instantaneous signal varies much more than the moving average. That is expected when FPS is derived from short inter-frame or callback intervals: operating-system scheduling, ROS 2 executor timing, camera delivery, Python execution, and measurement resolution can perturb individual intervals. Because FPS is the reciprocal of a small time interval, even modest timing variation can produce apparently large instantaneous spikes such as the recorded 80.69 FPS maximum.

The 30-frame moving average remains near the configured camera rate and is therefore a more useful indicator of sustained pipeline behavior. The correct conclusion is not that the model continuously runs at 80 FPS, but that the perception node keeps pace with the approximately 30 Hz input stream while experiencing normal short-term callback jitter.

8.2 Processing and Inference Latency

The inference trace is consistently below the total-processing trace because inference is only one stage of the callback. On average, YOLO inference accounts for 8.70 ms of the 18.26 ms total, or approximately 47.6% of measured processing time. The remaining time is associated with the rest of the perception path: image conversion and transfer, tracking/postprocessing, mask and bounding-box rendering, metric collection, and ROS-facing handling.

Both traces vary over time rather than remaining perfectly constant. GPU kernel scheduling, memory operations, changes in detections/masks, visualization cost, Python/ROS executor timing, and other system activity are plausible contributors. The current instrumentation measures aggregate stages rather than each contributor independently, so the plot supports identification of runtime variability but not a single causal attribution.

8.3 Processing-Latency Distribution

The histogram shows that frame-processing time is concentrated in more than one visible operating band rather than following one narrow unimodal cluster. The mean of 18.26 ms describes the center of overall workload, while the 24.87 ms P95 value is more useful for assessing near-worst-case routine behavior because it captures the upper tail without being dominated by isolated extremes.

A bimodal-like distribution can arise when the pipeline alternates between different execution conditions—for example, different tracking or mask-rendering workloads, GPU/CPU scheduling states, or scene complexity. However, the existing CSV does not separately log all of those internal stages. The defensible conclusion is therefore that multiple timing regimes are visible; identifying their exact cause would require stage-level profiling or controlled experiments.

8.4 Detected Object Count and Scene Load

The object-count trace confirms that the benchmark was not performed under one fixed scene condition. Most frames contained relatively few objects, but the scene periodically became more complex and reached 11 simultaneous detections. This matters because postprocessing, tracking association, mask handling, and rendering work can change with the number and geometry of detections even when the core neural-network input size is fixed.

The current plots should not be used to claim that object count caused the latency variation, because a frame-by-frame correlation or controlled load experiment was not performed. A logical next analysis would align object count and latency per frame and compute correlation or stratified latency statistics for different object-count ranges.

9. Execution and Reproducibility

Unified perception node:

cd ~/robotics_ws
source /opt/ros/humble/setup.bash
source ~/robotics_ws/vision_venv/bin/activate
source install/setup.bash
ros2 run vision_perception unified_perception

The camera publisher runs as a separate ROS 2 process so that /camera/image_raw is available to the perception node. The final rqt demonstration displays the raw camera topic and the final unified image topic.

10. Project Structure

vision_perception/
├── launch/
│   └── vision_demo.launch.py
├── results/
│   └── vision_results.csv
├── vision_perception/
│   ├── __init__.py
│   ├── camera_publisher.py
│   ├── segmentation_node.py
│   ├── unified_perception.py
│   └── yolo_detector.py
├── package.xml
├── setup.py
├── setup.cfg
└── README.md

11. Limitations and Next Steps

This benchmark evaluates runtime integration, not perception accuracy. The system uses pretrained general-purpose weights and no labeled test set was used, so metrics such as detection mAP, segmentation IoU, MOTA, HOTA, or IDF1 cannot be reported from this experiment.

The next engineering step would be to separate and profile preprocessing, host-to-device transfer, neural-network inference, tracking/postprocessing, visualization, and ROS 2 publication. That would allow the two timing bands in the latency histogram to be investigated experimentally rather than inferred from aggregate measurements.

For deployment-oriented optimization, the model could also be exported to ONNX and evaluated with CUDA or TensorRT execution. For robotics functionality, the perception output could be converted from visualization-only images into structured detection, mask, and track messages and then associated with calibrated depth or LiDAR measurements.

12. Key Takeaway

Project by Sooyong Kim · ROS 2 Robotics Portfolio