STAC: Selective Spatiotemporal Aggregation and Compression for Video Reasoning Segmentation

STAC architecture: a visual tokenizer feeds global and local frame tokens into the STAC module — a temporal state aggregator followed by temporal and spatio-temporal selectors — whose compressed tokens are consumed by a multimodal LLM with LoRA and decoded into segmentation masks by a mask decoder.

Overview. Dense frame tokens are first enriched by the State-informed Spatiotemporal Aggregator, then reduced by the Hierarchical State-adaptive Compression module before reaching the multimodal LLM. Compression decisions are made in a temporally contextualized feature space and are trained end-to-end with the segmentation objective, so the model keeps exactly the tokens the task needs.

Abstract

Video reasoning segmentation demands pixel-accurate object tracking across hundreds of frames under complex natural language queries, producing dense spatiotemporal tokens whose quadratic self-attention cost makes long-video processing prohibitive. Existing methods address this through token compression, yet typically operate on encoder features lacking temporal context, constraining selection before content redundancy can be reliably assessed. Informed compression requires contextual awareness, but acquiring that awareness at full resolution incurs the same quadratic cost compression aims to reduce. State-space models resolve this constraint, as their linear recurrence selectively conditions each token on temporal context at O(T) cost, producing representations where content redundancy becomes assessable. Building on this, Selective SpatioTemporal Aggregation and Compression (STAC) enriches features via decoupled bidirectional spatial and causal temporal scanning, leveraging recurrence-derived redundancy for hierarchical compression with adaptive thresholds optimised with segmentation objective. STAC achieves 85% token reduction and 1.8× speedup while surpassing compression-free baselines on reasoning segmentation benchmarks in a zero-shot streaming-compatible setting.

Highlights

~85%

fewer video tokens

1.8×

faster training & inference

Zero-shot

streaming-compatible

State-informed Spatiotemporal Aggregator

Enriches encoder features through decoupled bidirectional spatial and causal temporal scanning, producing a semantically grounded feature space where redundancy becomes discernible before compression — with streaming deployment supported by the causal temporal design.

Hierarchical State-adaptive Compression

Performs temporal-then-spatial token reduction in the recurrence-enriched feature space, with adaptive thresholds that respond to local content dynamics: static regions are compressed aggressively while motion boundaries are preserved.

Task-Grounded Differentiable Compression

Propagates segmentation gradients directly through discrete compression decisions via straight-through estimation, aligning token retention with downstream mask accuracy and transferring zero-shot to unseen reasoning benchmarks.

Method

Spatial relationships within a frame are non-causal — objects interact with all of their neighbours simultaneously — whereas temporal evolution follows a strictly causal progression. STAC therefore decouples the two: bidirectional scanning aggregates spatial context within each frame, while a causal temporal scan accumulates history across frames at linear cost. Because the recurrence has already absorbed redundant content by the time compression happens, near-identical enriched representations directly signal what can be merged, and adaptive thresholds trained with the segmentation loss decide what is kept.

Decoupled scanning in STAC: bidirectional spatial scans within frames combined with a causal temporal scan across frames.

Decoupled scanning. Bidirectional spatial scans respect the non-causal structure within frames; the causal temporal scan never looks at future frames, which is what makes STAC deployable in a streaming setting.

Results

STAC is trained only on the referring benchmarks MeViS and Refer-YouTube-VOS, and evaluated zero-shot on Ref-DAVIS17 and the reasoning benchmarks ReVOS and ReasonVOS. Across context lengths it surpasses compression-free baselines while processing a fraction of the tokens, and the margin grows as videos get longer — exactly the regime dense attention cannot afford.

Line plot of J&F accuracy versus context length, where STAC maintains the highest accuracy as context grows.

Radar plot of J&F across context lengths C=2 through C=64, with STAC enclosing the competing methods.

Context scaling. Left: compute cost (GPU memory vs. runtime) as temporal context grows — STAC stays in the low-cost corner where dense baselines blow up. Right: J&F across context lengths C=2–64 — STAC (orange) encloses all competing methods.

Qualitative Results

Frame-level qualitative comparison between STAC and prior methods on reasoning segmentation queries.

Frame-level comparison. Qualitative comparison against prior methods on reasoning queries; STAC produces cleaner, temporally stable masks despite discarding ~85% of the tokens.

The clips below compare STAC against GLUS in motion. The Compressed panel visualizes STAC's compression dynamics: a red overlay marks patches being sequentially aggregated, with deeper red indicating heavier aggregation and a colour refresh marking each summary update.

1 · Temporal Consistency & Identity Preservation

“The transportation means with the largest carrying capacity” — night driving scene.

“A person in blue shorts.” — small, fast-moving subject.

“The male individual riding a bike” — identity held through occlusion.

2 · Linguistic Invariance & Cross-Query Consistency

“The relocating / mobile / moving / traveling rabbit” — consistent masks across all four phrasings.

3 · Multi-Object Segmentation

“The young calf and the sheep engaged in a headbutting competition”

“A gathering of pigeons shifting and moving”

BibTeX

@inproceedings{hesham2026stac,
  title     = {{STAC}: Selective Spatiotemporal Aggregation and Compression for Video Reasoning Segmentation},
  author    = {Hesham, Syed Ariff Syed and Liu, Yun and Sun, Guolei and Yang, Jing and Ding, Henghui and Geng, Xue and Jiang, Xudong},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026}
}

Acknowledgements

This research work is supported by the Natural Science Foundation of China (No. 62576176), the Agency for Science, Technology and Research (A*STAR) under its MTC Programmatic Funds (Grant No. M23L7b0021) and the A*STAR Graduate Scholarship. The computational resources are supported by the Supercomputing Center of Nankai University (NKSC).