NeurIPS

NeurIPS 2026
Track on Evaluations and Datasets

WordEval

Evaluating Word-Native Operation Fidelity in Document Editing

Jinyoung Park1†* Yupan Huang2† Shimin Zhang2 Jiayu Ding3* Yilin Jia2 Yuzhong Zhao2 Tengchao Lv2 Wenshan Wu2 Shaohan Huang2 Nan Yang2 Xiangyang Zhou2 Dongdong Zhang2 Li Dong2 Lei Cui2 Sun Mao2 Xun Wang2 Tao Ge2 Huitian Jiao2 Si-Qing Chen2 Changick Kim1‡ Furu Wei2‡

1KAIST  2Microsoft  3XJTU

†Equal contribution. ‡Corresponding authors. *Work done during an internship at Microsoft Research.

KAISTKAIST CILabMicrosoftXi’an Jiaotong University
Motivation

One request, two agents

Request Turn these two items into a numbered list.
Agent AWediting in Word
1.

First item

 

New item

inserted
2.

Second item

No number. The old numbers stay fixed.
What Word recognizesOrdinary paragraphs

“1.” and “2.” are typed characters.

Agent BWediting in Word
1.

First item

2.

New item

inserted
2.3.

Second item

Word numbers it and updates the rest.
What Word recognizesA numbered list

Word manages the numbers.

Looking the same is not enough. Whether an edit worked depends on what Word recognizes.
Real example

Both pages show bullets. Word reads no list.

Each “•” is a typed character, so ListString is empty. Line spacing matches, the list does not, so the task fails.
Operation fidelity: the requested effect, realized as a Word state change with the correct target, attribute, and parameters.

Method

How WordEval checks an edit

WordEval judges the saved document, not the steps the agent took.

The evaluation loop
1

Task

Winput.docx“request”
2

Agent

its own tools
steps not judged
3

Saved output

Woutput.docx
missing or invalid = fail
Wreference.docxmade in Word · never shown to the agent
4

Word-based checker

requested changepreservation
5

Verdict

✓✓✕Fail
all must pass

4Word opens the file and reads object properties

The checker compares the requested change and what must stay the same with expected values from the reference document, the input document, or case metadata. Across the 546 single-edit tasks it reads 139 object properties on 65 runtime objects, such as Shape.Rotation or Font.Bold.

Benchmark

The task landscape

Atomic: one edit per task · Compositional: several linked edits in one document

642editing tasks546 Atomic · 96 Compositional
15Word feature areas231 Word commands
319atomic intentseach linked to an Atomic task
Results

Key results

Case pass rate (%) on 642 tasks
AtomicCompositional
87.1%Claude Code · Opus 5, best overall. Three human participants average 82.8%.
44.5–66.7%the other seven agent systems
−3.2 to −19.1points from Atomic to Compositional, in every system
A closer look

Rotate, then align

Rotation and sizes match. A 60 × 120 pt rectangle turned 90° is 60 pt tall on the page, so its visible top edge sits 30 pt lower than the stored frame top says. The same agent passes each edit when asked for it alone.

Analysis

What the results reveal

RQ1

Where do agents succeed?

It depends on the feature area: Tables 83.2% and Paragraphs 81.1% lead, Pictures 44.9% trails.

Tables15 feature areas, mean of 8 systemsPictures
RQ2

Do similar single-edit scores predict combined edits?

No. Systems that look alike on single edits diverge once edits share a document.

Gemini 3.8 Flash
Atomic
57.9
Compositional
49.0
GLM-5.2
Atomic
58.2
Compositional
39.6
RQ3

How do outcomes break down?

Differently: valid files that fail a check, invalid files, or no file at all.

IncorrectInvalidMissing

Per-area results, composition classes, atomic-intent links and failure diagnoses are in the paper.

Conclusion

Whether a Word edit took effect is a question of document state, not appearance alone.

WordEval makes that state the unit of evaluation. Current agents handle many single operations, but every one of them loses accuracy when edits share a document.

More in the paperPer-area resultsComposition classesAtomic-intent linksFailure diagnoses
Thank you

Scope: synthetic documents · English instructions · Windows + Word

← → move · F fullscreen · N notes (▶ = press →) · T reset timer · P progress bar
SPEAKER NOTES