Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model

Abstract

Standard Vision-Language Models (VLMs) suffer from Moravec's paradox: they excel at complex offline visual reasoning, but fail and suffer computational inefficiency on simple streaming perception tasks. We present Mage-VL, an efficient codec-native streaming foundation model for real-time multimodal understanding and interaction. At its core, our custom tokenizer, Mage-ViT, replaces uniform frame sampling by selectively encoding dynamic, entropy-rich regions using motion vectors and residual energy across sparse anchor (I) and predicted (P) frames. Operating at a 16×16 patch level, this reduces visual token consumption by over 75% while preserving spatio-temporal context. Trained from scratch on merely 100M unlabeled images and videos, Mage-ViT matches or outperforms flagship encoders trained on billions of image–text pairs. We establish AI4AI data pipelines, encompassing prompt-code joint optimization for multimodal captioning and AI-driven performance diagnosis to guide training recipes. Furthermore, through a bio-inspired dual-system architecture—a lightweight System 1 event gate and a causal System 2 decoder—Mage-VL enables proactive streaming perception. Extensive evaluations show that Mage-VL-4B matches Qwen3-VL-4B on static tasks while achieving strong gains in video understanding and 2D/3D spatial reasoning with up to 3.5× wall-clock inference speedup, comprehensively surpassing the 15B Phi-4-Reasoning-Vision baseline. Beyond model artifacts, we deliver seven key empirical findings covering pre-training data efficiency, variable-resolution scaling, codec system acceleration, VideoQA SFT redundancy, motion-spatial synergy, AI4AI data pipelines, and Zero-Vision SFT for multimodal RL.

Codec-Native Encoder

Proactive Streaming

AI4AI Data Pipeline

Codec-native perception

Instead of offline uniform frame sampling with a frozen encoder, Mage-VL reads video the way a codec does. Each stream is split into anchor (I) frames (kept in full) and predicted (P) frames (only motion-salient patches kept) on a 16×16 grid. A single per-patch importance interface is codec-agnostic — it accepts a traditional codec (H.264/HEVC motion + residual) or a neural codec (DCVC-RT rate map) with no change to the encoder — cutting visual tokens by over 75% for up to 3.5× faster inference.

Codec-driven patch selection: HEVC (motion + residual) or DCVC-RT (entropy) → per-frame significance map → top-k patches packed into canvases for Mage-ViT.

Bio-inspired proactive streaming

A single-model System 1 & System 2 design gives low-latency, proactive responses to continuous video. A lightweight System 1 cognition gate watches each rolling codec window and decides when to speak; only when an event is response-worthy does it trigger the full System 2 VLM to generate — no multi-agent pipeline. It stays silent through routine play and speaks the moment an event completes, generalizing to real broadcasts (e.g. 2026 World Cup matches).

Proactive streaming: Mage-ViT features feed an event gate that stays silent until a response-worthy moment, then triggers the LLM decoder to produce event-conditioned commentary.

AI4AI data pipeline

The training data refines itself through an agentic closed loop. A frozen Qwen3-VL-32B captions images; a GPT-5 rubric scorer grades each caption on completeness, redundancy, coherence, and OCR fidelity; and a GitHub Copilot agent co-designs the prompt and harness code (e.g. rendering timestamp overlays so the captioner anchors events by OCR) under a human validation gate. Ten refine–verify iterations lift every downstream OCR / document / chart / perception benchmark (+5.6 InfoVQA, +3.8 OCRBench, +3.8 RealWorldQA); the same paradigm also drives Stage-3 data diagnostics, and this work inspired SkillOpt-Lite.

AI4AI closed loop: frozen Qwen3-VL-32B captions → GPT-5 rubric scorer → Copilot prompt/code modifier → validation gate → improved prompt P_t+1.

Emergent behavior in video caption pipeline. During pipeline iteration, the evaluation sub-agent identified that GPT-4.1 struggled with precise temporal localization when relying solely on generic text prompts. To resolve this, the harness code renders sequential timestamp overlays directly onto sampled frames, while the prompt guides the model to anchor events via visual OCR. This code–prompt co-design converts implicit temporal tracking into a visual grounding task, reducing temporal hallucinations and improving caption consistency across long videos.

AI-guided training diagnostics & recipe optimization. The same AI4AI loop also shaped our training recipe: the evaluation agent surfaced weaknesses in temporal localization — remedied by supplementing high-quality grounding data — and revealed that more frames consistently boost long-video performance, whereas dedicated long-video SFT and resolutions beyond 384 yield only marginal gains. Guided by these findings, we adopt 384-pixel inputs, set Stage-3 temporal length to 384 frames, and raise the RoPE base frequency ($\theta$) to 8M for stable long-context modeling.

LIVE VIDEO · PROACTIVE RESPONSES

Proactive Streaming in the Wild

Mage-VL watches continuously, stays quiet when nothing needs saying, and speaks when the event gate detects a response-worthy moment. Choose a demo and play the video to see each response appear at the moment it is triggered.

England vs. Argentina · 2026 World Cup 03:00 · codec backend · 30-second causal windows

Mage-VL live transcript 00:00

Watching the stream. Mage-VL is silent.

Raw model outputs from Microsoft/Mage-VL, --video_backend codec, --segment_sec 30, followed by standard Transformers generate() on the current segment.

Mage-VL-4B matches Qwen3-VL-4B on general VQA while outperforming it on spatial intelligence and video understanding, and comprehensively surpasses the much larger Phi-4-R-V-15B. Higher is better; per row, the best and second-best results are shaded.

Video Understanding & Temporal Grounding

Benchmark	Mage-VL-4B	Qwen3-VL-4B	Phi-4-MM-5.6B	Phi-4-R-V-15B
Video QA
MV-Bench	65.1	66.7	44.9	49.2
NextQA	83.1	79.8	54.1	69.0
VideoMME	64.0	59.7	44.7	55.3
LongVideoBench	61.3	57.7	41.14	51.2
LVBench	41.8	39.2	25.31	34.4
MLVU-dev	68.7	61.5	44.18	51.8
VideoMME (w/ sub.)	66.3	70.2	45.4	58.3
VideoMME-V2	24.3	24.4	19.2	23.9
VideoEval-Pro	45.2	20.7	14.35	16.8
JumpScore	45.60	3.82	3.61	1.53
MMOU-Test-Mini	39.3	38.1	32.22	51.9
Temporal grounding
Timelens-Charades	50.7	43.1	4.09	20.6
Timelens-ActivityNet	45.4	28.4	2.03	23.0
Timelens-QVHighlight	57.4	34.9	2.47	11.6
Spatial reasoning
VSI-Bench	64.3	53.3	24.09	25.5
Tracking (J&F)
Ref-DAVIS17	25.83	7.48	3.14	2.15
MeViS-ValidU	22.55	3.16	10.28	1.53
ReasonVOS	17.76	9.66	9.50	9.77
Ref-YT-VOS	25.57	5.28	8.64	3.85

Proactive Streaming on OVO-Bench

OVO-Bench measures online video understanding under Real-Time Visual Perception and Backward Tracing. † Qwen2.5-VL-7B + HERMES (4K tokens). Baseline results and table structure follow SimpleStream.

Model	#Frames	Real-Time Visual Perception							Backward Tracing				Avg.
Model	#Frames	OCR	ACR	ATR	STU	FPD	OJR	Avg.	EPM	ASI	HLD	Avg.	Avg.
Human	--	94.0	92.6	94.8	92.7	91.1	94.0	93.2	92.6	93.0	91.4	92.3	92.77
Offline Video LLMs
Qwen2.5-VL-7B	1 fps	67.8	55.1	67.2	42.1	66.3	60.9	59.9	51.5	58.8	23.7	44.7	52.28
LLaVA-Video-7B	64	69.1	58.7	68.8	49.4	74.3	59.8	63.5	56.2	57.4	7.5	40.4	51.95
Qwen3-VL-4B	64	77.9	69.7	77.6	60.1	76.2	75.5	72.8	63.0	66.2	30.1	53.1	63.00
Online / Streaming Video LLMs
VideoLLM-online-8B	2 fps	8.1	23.9	12.1	14.0	45.5	21.2	20.8	22.2	18.8	12.2	17.7	19.26
Flash-VStream-7B	1 fps	24.2	29.4	28.5	33.7	25.7	28.8	28.4	39.1	37.2	5.9	27.4	27.90
Dispider-7B	1 fps	57.7	49.5	62.1	44.9	61.4	51.6	54.6	48.5	55.4	4.3	36.1	45.35
TimeChat-Online-7B	1 fps	75.2	46.8	70.7	47.8	69.3	61.4	61.9	55.9	59.5	9.7	41.7	51.80
StreamForest-7B	1 fps	68.5	53.2	71.6	47.8	65.4	60.9	61.2	58.9	64.9	32.3	52.0	56.60
Streamo-7B	1 fps	79.2	57.8	75.0	49.4	64.4	70.1	66.0	54.6	52.0	31.7	46.1	56.05
HERMES-7B†	1 fps	85.2	64.2	71.6	53.4	74.3	65.2	69.0	48.5	62.2	37.6	49.4	59.20
JoyAI-VL-Interaction-9B	1 fps	72.5	61.5	75.0	59.0	78.2	64.1	68.4	62.0	70.9	12.9	48.6	58.50
Mage-VL-4B	1 fps	94.63	83.49	79.31	74.16	70.30	77.17	79.84	50.17	61.49	32.80	48.15	64.00

Image Understanding & Spatial Intelligence

Benchmark	Mage-VL-4B	Qwen3-VL-4B	Phi-4-MM-5.6B	Phi-4-R-V-15B
Document understanding
DocVQA-val	95.14	94.69	92.79	76.20
InfoVQA-val	80.33	79.50	71.84	55.41
AI2D w/ Mask	83.16	81.54	81.83	82.87
AI2D w/o Mask	91.87	92.20	91.35	93.81
ChartQA	84.88	83.96	83.76	83.40
OCRBench	81.80	81.60	81.70	73.90
CC-OCR Doc	32.25	39.69	4.99	17.65
MultiDocVQA-val	87.46	87.21	46.84	58.35
DUDE	46.44	50.98	3.78	34.34
WebSRC-val	92.80	95.40	71.30	76.60
ChartQAPro	32.57	26.79	0.13	25.38
TextVQA-val	77.28	80.55	39.93	76.06
CharXiv-Descr	75.85	74.40	18.65	75.98
CharXiv-Reason	35.20	26.10	0.10	36.10
General VQA
MMBench-EN-dev	84.02	83.25	65.81	84.19
MMBench-CN-dev	82.04	80.58	75.17	79.47
RealWorldQA	70.46	70.85	60.65	70.72
MMStar	67.32	62.04	61.24	59.63
MME-Perception	1709.54	1703.50	1409.66	1590.21
SeedBench (All)	76.78	75.65	68.28	73.70
SeedBench-Image	79.32	78.87	74.48	77.97
SeedBench2-Plus	69.21	69.65	68.38	69.48
MMT-val	64.47	66.52	59.19	64.89
CV-Bench	87.79	85.37	57.09	81.31
MME-RealWorld	66.52	63.20	32.45	57.80
MME-RealWorld-CN	61.33	63.09	21.19	46.07
Spatial intelligence
CV-Bench-2D	82.13	81.00	56.12	80.11
CV-Bench-3D	94.75	92.30	56.92	82.50
BLINK	65.11	65.10	35.24	57.80
EmbSpatial	82.67	77.50	41.51	72.67
CRPE-Relation	76.12	77.70	34.60	74.46
CrossPoint	80.00	26.90	12.20	47.73
ERQA	36.00	42.30	30.75	40.25
MMSI-Bench	28.20	31.00	28.80	25.70
SAT	67.33	69.30	55.33	66.67

Numbers from the Mage-VL technical report.

Limitations & Path Forward

⚠️ Despite its strong perception and streaming capabilities, Mage-VL still shows a gap to Qwen3-VL on complex agentic tasks. This gap largely stems from tight compute budgets, the absence of joint text-multimodal mixed pre-training, and the omission of large-scale RL post-training.

💡 Zero-Vision SFT as a promising remedy. Our preliminary study suggests a compute-efficient path: skip visual SFT entirely and replace it with high-quality pure-text reasoning data before applying multimodal RL. On the LLaVA-OV-1.5 Quick Start scale, this Zero-Vision SFT + RL pipeline lifts overall accuracy by +5.33% (54.28 vs. 48.96) across 24 benchmarks, wins on 19/24 tasks, and converges with 50.7% fewer RL steps. Reasoning (+7.92% avg.), OCR & Chart (+9.52% avg.), and counting all benefit substantially, indicating that preserving intrinsic textual reasoning during instruction tuning unlocks stronger multimodal RL and offers a practical route to close the agentic gap.

Contributors

Senqiao Yang^*, Kaichen Zhang^*, Zhaoyang Jia^*, Jinghao Guo^*, Yifei Shen^*†, Xinjie Zhang^*†, Xiaoyi Zhang, Haoqing Wang, Xiao Li, Peng Zhang, Xiang An, Yin Xie, Zhening Liu, Xun Guo, Jiahao Li, Shicheng Zheng, Jinglu Wang, Zongyu Guo, Wenxuan Xie, Zihan Zheng, Yuxuan Luo, Bin Li, Yan Lu

^* Equal Contribution. ^† Project Leads (yshenaw@connect.ust.hk, xinjiezhang@microsoft.com).

BibTeX 📚

@article{yang2026mage,
  title={Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model},
  author={Yang, Senqiao and Zhang, Kaichen and Jia, Zhaoyang and Guo, Jinghao and Shen, Yifei and Zhang, Xinjie and Zhang, Xiaoyi and Wang, Haoqing and Li, Xiao and Zhang, Peng and others},
  journal={arXiv preprint arXiv:2607.24904},
  year={2026}
}

Mage-VL An Efficient Codec-Native Streaming
Multimodal Foundation Model