ArXiv version - Video anomaly understanding benchmark

TAU-Bench

From Anomaly Instance Tracking to Fine-Grained Video Anomaly Understanding

Kepeng Yang*, Dongxuan Liu*, Rongxin Gao*, Zixin Su, Rui Wu, Shuzhao Xie, Chenxin Li, Panwang Pan, Yuzhi Huang†, Yue Huang†, Jingyan Jiang

XMU - THU - CUHK - SZTU

Paper PDF BibTeX Code coming soon Dataset coming soon

TAU-Bench connects anomaly instance tracking with fine-grained anomaly understanding. — Figure 1: Overview of TAU-Bench. Most existing benchmarks provide frame-level masks or coarse video-level descriptions, whereas TAU-Bench associates identity-consistent anomaly instance tracks with fine-grained anomaly understanding.

1,118 videos

1,454 identity-consistent tracks

202,438 pixel-level masks

49 / 45 event / scene categories

Highlight

A unified benchmark for evaluating anomaly tracking and anomaly understanding together.

Coupled task formulation

We formulate fine-grained VAU as a coupled tracking and understanding problem, measuring whether spatial, temporal, and semantic predictions remain grounded in the same anomaly instance.

Large-scale benchmark

TAU-Bench provides 1,118 videos, 1,454 identity-consistent anomaly tracks, and 202,438 pixel-level masks across 49 event categories and 45 scene categories.

Scalable data engine

We build TAU-Bench with a scalable construction engine that integrates anomaly video filtering, track construction and verification, hierarchical captioning, and human quality control.

Benchmark

Two capability families, one visual anchor.

Anomaly Instance Tracking

Video-level anomaly detection
Temporal anomaly localization
Anomaly Instance Reasoning Segmentation (A-IRS)

Fine-Grained Understanding

Instance-level identification
Event-level temporal process and location
Scene-level function, violation, and risk reasoning

TAU-Bench comparison with existing video anomaly benchmarks. — Table 1: Comparison with video anomaly detection and understanding benchmarks. TAU-Bench jointly supports video anomaly detection, temporal localization, anomaly instance reasoning segmentation, instance tracks, pixel-level masks, and instance-, event-, and scene-level understanding.

TAU-Bench event examples and category distributions. — Figure 2: Data statistics of TAU-Bench, including anomaly event examples and distributions over instance, event, and scene categories, as well as the histogram of video duration.

Construction

A scalable data engine with human quality control.

Stage 1

Curate and track

Static- and moving-camera anomaly videos are filtered for annotation suitability, then converted into candidate mask tracks.

Stage 2

Annotate hierarchy

Anomaly Evidence Extraction generates instance, event, and scene captions that remain tied to the same focal target.

Review

Verify consistency

Human reviewers correct identity switches, missing masks, ambiguous events, and unsupported causal or risk statements.

TAU-Bench data construction engine. — Figure 3: Data construction engine of TAU-Bench. Stage 1 curates static- and dynamic-camera anomaly videos and constructs identity-consistent anomaly instance tracks. Stage 2 performs Anomaly Evidence Extraction to generate progressive instance-, event-, and scene-level captions, followed by human quality control.

Evaluation

Strong descriptions still do not guarantee grounded tracks.

Fine-grained understanding

No evaluated model dominates all semantic fields. Qwen3-VL-8B-SAMTok is strongest on anomalous instance identification, while VideoChat-R1.5 and InternVL3.5 show stronger event or scene reasoning.

Anomaly instance tracking

Recall is high across models, but temporal localization remains unsaturated. The best recall and best tIoU come from different models, showing that presence detection and temporal recovery are distinct abilities.

A-IRS grounding

GT caption guidance improves J&F by 11.82 points on average over generated captions, but correct semantics alone still does not ensure accurate visual grounding.

Best anomaly recall 98.29

Cue-R1

Best temporal IoU 79.20

InternVL3.5-14B

Best GT-caption A-IRS J&F 55.83

Qwen3-VL-8B-SAMTok

A failure case where a model describes the event plausibly but tracks a distractor vehicle. — Figure 4: A representative failure case on TAU-Bench. The model generates plausible event and scene-level descriptions, yet tracks a distractor vehicle rather than the ground-truth anomaly instance, revealing that linguistic plausibility does not necessarily imply visually grounded anomaly understanding.

Citation

Cite TAU-Bench

@article{yang2026taubench,
  title   = {TAU-Bench: From Anomaly Instance Tracking to Fine-Grained Video Anomaly Understanding},
  author  = {Yang, Kepeng and Liu, Dongxuan and Gao, Rongxin and Su, Zixin and Wu, Rui and Xie, Shuzhao and Li, Chenxin and Pan, Panwang and Huang, Yuzhi and Huang, Yue and Jiang, Jingyan},
  journal = {arXiv preprint},
  eprint  = {2608.05699},
  archivePrefix = {arXiv},
  year    = {2026}
}