CUHK-X OBSERVATORY

// MOBISYS 2026 · MULTIMODAL HUMAN ACTION DATASET

SEVEN WAYS
OF SEEING
A HUMAN ACTION

CUHK-X records people living ordinary moments — brushing teeth, folding clothes, doing squats — through seven synchronized sensor streams. This observatory lets you walk through CUHK-S, the public, privacy-preserving subset: depth, infrared and thermal video with captions, pace labels, action chains and next-action logic.

READ THE PAPER ↗

PARENT DATASET CUHK-X — 64,217 samples · 7 modalities · 30 subjects — available under DUA

SPECTRAL SWEEP SYNC ×3 · —
DEPTH
INFRARED
THERMAL
00:00.0

⟷ DRAG THE DIVIDERS — ONE MOMENT, THREE SENSORS

—VIDEO CLIPS
—SENSOR-STREAM HOURS
—SUBJECTS
—ACTION CLASSES
—SCRIPTED SCENES
—LABEL ROWS

01 STATIONS

Three instruments, one per benchmark. Pick a station to start exploring.

STATION 01 · HAU Sequence Lab Long scripted scenes played back in depth ⊕ infrared ⊕ thermal sync — with captions, pace labels and ordered action chains. 814 TAKES · 7 SCENES · 3 SYNCED STREAMSENTER ▸ STATION 02 · HAR / HARN Action Atlas Every action class on one wall. Hover to scrub film strips, click to inspect dual-stream clips, read the coverage instruments. 40 CLASSES · DEPTH ⊕ IRENTER ▸ STATION 03 · HARN Reasoning Console Watch a depth clip and predict the next action. Real ground truth, real distractors — score yourself against the VLM baselines. 1,492 LOGIC LABELS · BEAT THE VLM BASELINEENTER ▸

02 SENSOR ARRAY

Every sample in CUHK-X is captured simultaneously by seven instruments. Three privacy-preserving streams ship in the public CUHK-S subset.

DEPTH

3-D structure from a Vzense NYX 650 — bodies as geometry, not identity.

● IN CUHK-S

INFRARED

Active IR video that keeps seeing when the lights go out.

● IN CUHK-S

THERMAL

Heat signatures of the body — robust, anonymous, always-on.

● IN CUHK-S

RGB

Reference color video for grounding and annotation.

FULL CUHK-X

MMWAVE

TI radar point clouds — motion sensing through privacy.

FULL CUHK-X

IMU

On-body inertial traces of every gesture.

FULL CUHK-X

SKELETON

3-D pose sequences distilled from the cameras.

FULL CUHK-X

03 ACCESS PROTOCOL

DOWNLOADCUHK-S · open
CUHK-S/
├── HAU/        sequential scenes · depth+IR+thermal
│   ├── data/   user/​scene-env-trial/​*.mp4
│   └── GT/     captions · emotion · ordered chains
└── HARn/       single actions · depth+IR
    ├── data/   40 action classes (+4 extra)
    └── GT/     next-action logic + candidates

Full CUHK-X (7 modalities, 64,217 samples) is available for non-commercial research under a Data Use Agreement. Collection was IRB-approved; public clips are de-identified sensor streams.

CITE
@inproceedings{jiang2026cuhkx,
  title     = {CUHK-X: A Large-Scale Multimodal Dataset
               and Benchmark for Human Action Recognition,
               Understanding and Reasoning},
  author    = {Jiang, Siyang and others},
  booktitle = {Proc. ACM MobiSys},
  year      = {2026}
}

Questions & collaborations — syjiang [at] ie.cuhk.edu.hk