SciForma: Structure-Faithful Generation of Scientific Methodology Diagrams

Yuxuan Luo1, Peng Zhang2, Xinjie Zhang3✉, Xun Guo3, Zhouhui Lian1✉, Yan Lu3
1Wangxuan Institute of Computer Technology, Peking University, 2Zhejiang University, 3Microsoft Research Asia
Corresponding to xinjiezhang@microsoft.com, lianzhouhui@pku.edu.cn

SciForma-9B generated scientific methodology diagrams. Click to expand.

Abstract

We introduce SciForma, a structure-faithful framework for scientific diagram generation built on three universal primitives: Component, Arrow, and Text (C/A/T). Driven by a scaled-up dataset, Multi-Dimensional Conjunctive Preference Optimization (M-DPO), and iterative refinement, it enforces simultaneous correctness across layout, topology, and typography, matching or outperforming leading proprietary models.

Structural Inventory
SciFormaData-700K
SciFormaBench-2K
Structural Inventory

Much like UML and BPMN give software and process diagrams a compact vocabulary of typed nodes and edges, SciForma formalizes scientific figures as a structural inventory over three independently verifiable primitives, Component, Arrow, and Text (C/A/T). The same inventory drives training captions and benchmark rubrics, so what the model is asked to draw is exactly what the judge scores.

Correctness is conjunctive, meaning a figure counts as correct only when all three axes are simultaneously right. A missing arrow or mislabelled block is not offset by a clean layout, so the model cannot coast on averaged scores and must render every axis faithfully.

SciFormaData-700K

726K methodology diagrams from 593K arXiv LaTeX sources (2015–2025), kept after consensus filtering by Qwen3-VL and OpenAI-o3. Each figure carries a structured dense description covering global layout, module attributes, and topological dependencies. 71K editing triplets are mined by SAM3 element segmentation with OpenAI-o3 written grounded instructions.

656K
Generation Pairs
71K
Editing Triplets
344K
arXiv Papers

Data pipeline: extraction → detection → caption generation → filtering → editing augmentation.

SciFormaBench-2K

2,000 held-out prompts stratified by structural complexity and deliberately weighted toward Medium and Hard, where dense multi-branch diagrams remain the real challenge. Each sample carries an element checklist extracted by GPT-5.4 and corrected by human annotators, then scored on the C/A/T axes with severity-weighted error counting.

500
Simple
900
Medium
600
Hard

Per-axis rubric with critical and moderate error annotations.

Method

SciForma method overview

SciForma training pipeline overview.

Two-Stage SFT
M-DPO
Iterative Refinement
Two-Stage SFT

SciForma fine-tunes FLUX.2-klein-base-9B on SciFormaPile in two stages. Stage 1 trains all 9B parameters on the full 656K generation pairs with resolution bucketing to keep batches spatially homogeneous across widely varying aspect ratios. Stage 2 initializes from Stage-1 EMA weights and co-trains on 244K High-quality pairs together with the 71K editing triplets, so a single checkpoint serves both generation and editing through a concat-then-prune design that adds no new parameters.

656K
Stage 1 Pairs
244K + 71K
Stage 2 Gen + Edit
140K / 90K
Stage 1 / 2 Steps
Multi-Dimensional Direct Preference Optimization

SFT leaves axis-specific artifacts such as missing components, garbled labels, and broken arrows, and these failure modes are near-independent across the three axes (inter-axis ρ < 0.2). MD³PO pairs each winner with dimension-anchored losers, one per axis, and couples them through an InfoNCE-style contrastive objective. The loss only vanishes when the policy prefers the winner on every axis simultaneously, and a softmax over per-axis reward gaps automatically concentrates gradient on whichever dimension is currently weakest.

M-DPO ablation

M-DPO ablation on SciFormaBench-2K. Text axis benefits most (+3.16 pts).

  • Conjunctive objective. Loss only vanishes when the winner beats every axis-anchored loser at once.
  • Auto-focusing gradient. A softmax over per-axis logits sends the largest update to the failing axis.
  • InfoNCE view. Equivalent to InfoNCE with D axis-anchored hard negatives.
  • Standard DPO limit. Reduces exactly to standard DPO when D = 1.
Iterative Refinement via Editing

At inference time a VLM critic inspects the generated diagram against its structural inventory and flags concrete defects, which are compiled into a priority-ranked edit plan. SciForma-9B applies each atomic edit through the same editing branch trained in Stage 2, and the loop repeats for a fixed number of rounds or stops early once the critic reports no remaining structural errors, yielding the closed-loop refinement shown below.

Editing pipeline

Closed-loop editor. Defect detection, priority-ranked edit plan, SciForma-9B targeted inpainting, global verification. The loop terminates when the structural inventory is satisfied.

Leaderboard 🏆

SciFormaBench-2K

GPT-5.4 judge  ·  Click column to sort

Model Avg Simple Medium Hard Component Arrow Text Open
GPT-Image-285.6288.4185.6683.2683.3489.6183.53
NanoBananaPro81.3485.3081.7077.5081.1083.6078.70
SciForma-9B + Edit ★72.4079.8272.5966.0176.7069.9170.14
SciForma-9B (M-DPO) ★69.5177.2069.6962.8674.4966.4667.00
GPT-Image-1.568.9675.5069.6062.6075.7062.5068.20
SciForma-Base (SFT) ★67.5976.0167.4360.8373.5264.6463.84
Wan2.7-Image64.7173.9065.9055.3071.9060.3061.10
SenseNova-U1-8B51.1459.5751.8643.0261.3450.1640.37
FLUX.2-dev-32B48.8157.6049.3040.8062.5038.6044.60
Qwen-Image-251248.7357.5049.9039.6061.2046.6036.80
Z-Image48.5057.2049.7039.4058.3046.9038.50
FLUX.2-klein-base-9B33.8742.8034.2026.0051.5025.2023.60
FLUX.1-dev20.6426.4020.7015.8040.6011.208.50
Bagel-7B16.3118.4016.2014.7035.509.502.20

AIBench

Model Score Component Topology Phase Semantic Aesthetic Open
GPT-Image-280.2790.6981.7784.1890.2454.49
NanoBananaPro77.7787.8074.8182.6788.5455.04
Seedream 5.073.2382.9372.8172.1086.5351.78
SciForma-9B + Edit ★70.6277.8664.4775.4581.2054.10
SciForma-9B (M-DPO) ★70.2977.5364.1774.7180.7954.24
SciForma-Base (SFT) ★68.6176.2661.9573.2678.3353.24
Wan2.665.8468.6056.1172.5680.4351.50
GPT-Image-1.561.6266.2350.8755.9577.5557.50
Seedream 4.559.6867.8952.4248.1474.4755.48

★ = SciForma (ours)

Qualitative Analysis 🖼️

BibTeX 📚

@article{luo2026sciforma,
  title={SciForma: Structure-Faithful Generation of Scientific Diagrams},
  author={Luo, Yuxuan and Zhang, Peng and Zhang, Xinjie and Guo, Xun and Lian, Zhouhui and Lu, Yan},
  journal={arXiv preprint arXiv:2607.18091},
  year={2026}
}