Getting Started with ROS2: Multimodal Data Collection for ML

A practical introduction to using ROS2 for collecting synchronized camera, IMU, and telemetry data for training machine perception models — based on real work with the Unitree Go1 robot.

Tutorials

Why Synchronized Multimodal Data Matters

Training machine perception models for robotics is fundamentally a data quality problem. Camera frames, IMU readings, and telemetry streams each carry different information — and for models that need to correlate visual input with motion state, synchronization at the millisecond level is critical. Misaligned timestamps are one of the most common sources of poor model performance in robotics ML.

ROS2's message passing architecture with synchronized topic subscribers is the right tool for this problem.

Setting Up a Synchronized Collection Node

The core pattern is a ROS2 node that subscribes to camera, IMU, and telemetry topics using message_filters.ApproximateTimeSynchronizer with a configurable slop window (I use 0.05s for most experiments). When all three streams have matching timestamps, the callback fires and writes a structured record to disk using NumPy for efficient binary serialization.

For the Unitree Go1, the key is configuring the go1-sdk topics correctly and handling the proprietary state estimation messages — the documentation is sparse, but the pattern is straightforward once you know the topic names.

Data Validation with Guardrail-Style Checks

Raw sensor data is noisy. Before any collected frame enters your training dataset, it needs automated quality checks: blur detection on camera frames (Laplacian variance threshold), outlier detection on IMU readings (z-score > 3.5 flags anomalous acceleration), and telemetry completeness checks (missing joint states = discard). These Guardrail-style checks run in a post-processing step using OpenCV and SciPy, reducing corrupted samples from ~8% of collected data to under 0.5%.

Structuring Datasets for CNN and Transformer Training

Once collected and validated, you need a consistent dataset format. I use a HDF5-based structure with separate groups for each modality and a shared timestamp index. This makes it trivial to load any combination of modalities for different model architectures — CNNs that only use image sequences, transformers that use the full multimodal context, or classical models that just need telemetry features. The dataset format is the contract between your collection pipeline and your training code.

Like what you see? There’s more.

Get monthly inspiration, blog updates, and creative process notes — handcrafted for fellow creators.

More to Discover

Tutorials

Mar 15, 2025

Building LLM Pipelines from Scratch: Transformers to Production

A practical walkthrough of building production-ready LLM pipelines — from transformer architecture fundamentals to deploying scalable inference on AWS SageMaker.

Insights

Feb 20, 2025

How I Hit 1700+ Tokens/Sec at the Velric × Foundery Hackathon

A technical breakdown of how our team secured 3rd place in the Agentic AI track — achieving 1700+ tokens/sec throughput across 16 concurrent requests with 11–13ms TTFT via FP8 quantization.