Evaluating Word-Native Operation Fidelity in Document Editing
Jinyoung Park1†* Yupan Huang2† Shimin Zhang2 Jiayu Ding3* Yilin Jia2 Yuzhong Zhao2 Tengchao Lv2 Wenshan Wu2 Shaohan Huang2 Nan Yang2 Xiangyang Zhou2 Dongdong Zhang2 Li Dong2 Lei Cui2 Sun Mao2 Xun Wang2 Tao Ge2 Huitian Jiao2 Si-Qing Chen2 Changick Kim1‡ Furu Wei2‡
1KAIST 2Microsoft 3XJTU
†Equal contribution. ‡Corresponding authors. *Work done during an internship at Microsoft Research.
Motivation
One request, two agents
Request Turn these two items into a numbered list.
Agent AWediting in Word
1.
First item
New item
inserted
2.
Second item
No number. The old numbers stay fixed.
What Word recognizesOrdinary paragraphs
“1.” and “2.” are typed characters.
Agent BWediting in Word
1.
First item
2.
New item
inserted
2.3.
Second item
Word numbers it and updates the rest.
What Word recognizesA numbered list
Word manages the numbers.
Looking the same is not enough. Whether an edit worked depends on what Word recognizes.
Real example
Both pages show bullets. Word reads no list.
Each “•” is a typed character, so ListString is empty. Line spacing matches, the list does not, so the task fails. Operation fidelity: the requested effect, realized as a Word state change with the correct target, attribute, and parameters.
Method
How WordEval checks an edit
WordEval judges the saved document, not the steps the agent took.
The evaluation loop
1
Task
Winput.docx“request”
2
Agent
its own tools
steps not judged
3
Saved output
Woutput.docx
missing or invalid = fail
Wreference.docxmade in Word · never shown to the agent
4
Word-based checker
requested changepreservation
5
Verdict
✓✓✕Fail
all must pass
4Word opens the file and reads object properties
The checker compares the requested change and what must stay the same with expected values from the reference document, the input document, or case metadata. Across the 546 single-edit tasks it reads 139 object properties on 65 runtime objects, such as Shape.Rotation or Font.Bold.
Benchmark
The task landscape
Atomic: one edit per task · Compositional: several linked edits in one document
642editing tasks546 Atomic · 96 Compositional
15Word feature areas231 Word commands
319atomic intentseach linked to an Atomic task
Results
Key results
Case pass rate (%) on 642 tasks
AtomicCompositional
87.1%Claude Code · Opus 5, best overall. Three human participants average 82.8%.
44.5–66.7%the other seven agent systems
−3.2 to −19.1points from Atomic to Compositional, in every system
A closer look
Rotate, then align
Rotation and sizes match. A 60 × 120 pt rectangle turned 90° is 60 pt tall on the page, so its visible top edge sits 30 pt lower than the stored frame top says. The same agent passes each edit when asked for it alone.
Analysis
What the results reveal
RQ1
Where do agents succeed?
It depends on the feature area: Tables 83.2% and Paragraphs 81.1% lead, Pictures 44.9% trails.
Tables15 feature areas, mean of 8 systemsPictures
RQ2
Do similar single-edit scores predict combined edits?
No. Systems that look alike on single edits diverge once edits share a document.
Gemini 3.8 Flash
Atomic
57.9
Compositional
49.0
GLM-5.2
Atomic
58.2
Compositional
39.6
RQ3
How do outcomes break down?
Differently: valid files that fail a check, invalid files, or no file at all.
IncorrectInvalidMissing
Per-area results, composition classes, atomic-intent links and failure diagnoses are in the paper.
Conclusion
Whether a Word edit took effect is a question of document state, not appearance alone.
WordEval makes that state the unit of evaluation. Current agents handle many single operations, but every one of them loses accuracy when edits share a document.
More in the paperPer-area resultsComposition classesAtomic-intent linksFailure diagnoses
Thank you
Scope: synthetic documents · English instructions · Windows + Word
← → move · F fullscreen · N notes (▶ = press →) · T reset timer · P progress bar