The data behind the challenge

Dataset Introduction

Most large vision-language models still depend almost entirely on RGB data, while modalities such as depth, thermal imaging, IMU, and millimeter-wave radar remain severely underrepresented. The root cause is a lack of large-scale, high-quality paired multimodal datasets.

CUHK-X, built by the AIoT Lab at CUHK, addresses this gap with 64,267 samples across seven fully synchronized modalities collected from 30 participants performing 40 daily activities across two real-world indoor environments. Annotations follow a Ground-Truth First strategy, combining LLM-generated scene descriptions with human review to ensure temporal and logical consistency. The dataset supports three progressive tasks: HAR (action classification), HAU (action understanding), and HARn (action reasoning).

Total Samples
64,267 fully synchronized recordings
Participants
30 (diverse age and gender)
Environments
2 real-world indoor settings
Action Classes
40 daily life activities
Dataset Modalities
RGB · Depth · Thermal · Infrared · Skeleton · IMU (×5) · mmWave
Challenge Modalities
All except RGB
Benchmark Tasks
HAR · HAU · HARn
Annotation Strategy
Ground-Truth First · LLM + human review

Cross-Subject Split — 30 Participants

18
Training Set
user 1–9 · 16–24
4
Public Test Set
user 10,11,25–26
8
Private Test Set
held out
🌐
CUHK-X Dataset Homepage Train & test splits · 30 participants

Modalities — Available in Dataset

RGB (challenge-excluded) Depth Thermal Infrared Skeleton IMU ×5 mmWave Radar

Benchmark Tasks

HAR — Action Classification HAU — Action Understanding HARn — Action Reasoning