Skip to content

NeurIPS 2026

WordEval:Evaluating Word-Native Operation Fidelity in Document Editing

WordEval evaluates agents on Word editing operations, with Word itself as the verifier. Each requested change is checked in the document's objects and properties, across 231 commands, alone and combined with other edits in one document.

Read the blog

Jinyoung Park1†* Yupan Huang2† Shimin Zhang2 Jiayu Ding3* Yilin Jia2 Yuzhong Zhao2 Tengchao Lv2 Wenshan Wu2 Shaohan Huang2 Nan Yang2 Xiangyang Zhou2 Dongdong Zhang2 Li Dong2 Lei Cui2 Sun Mao2 Xun Wang2 Tao Ge2 Huitian Jiao2 Si-Qing Chen2 Changick Kim1‡ Furu Wei2‡

1KAIST2Microsoft3Xi’an Jiaotong University

†Equal contribution.‡Corresponding authors.*Work done during an internship at Microsoft Research.

Video

ExampleSix benchmark tasks, replayed. Word reads the properties each request names from the agent’s output and from the reference.

Documents
output.docx
Document saved by the agent
saved by the agent
reference.docx
Reference document for the task
built for the task
Request the agent was given
WordEval
opens both files in Word,
reads the properties the request names
Verdict
output.jsonwhat Word reads in output.docx
reference.jsonwhat Word reads in reference.docx
Figure 1. Six real cases from the evaluation, replayed. The property values are the ones WordEval read from the saved files; the images are renders of those files. The first case comes from the evaluator-validation cohort, the others from the main experiment. Watch the demo video.
642editing tasks546 single edits · 96 combined requests
15Word feature areas231 Word commands
87.1%best overall pass rateClaude Code / Opus 5

Abstract

Microsoft Word is a central environment for office automation, and the faithful execution of its editing operations is an important capability for document agents. Unlike plain-text editing, editing in Word modifies a structured object model, where the requested effect is realized in object properties. Existing benchmarks, however, evaluate office workflows or desktop automation, treating Word documents as one domain among many and verifying outcomes with keywords, per-task property checks, or screenshots.

We introduce WordEval, an operation-fidelity benchmark that instead centers evaluation on Word editing operations, testing whether each requested edit is realized as intended changes to Word objects and properties. It contains 642 cases across 15 Word feature areas, ranging from text and lists to charts, shape geometry, and SmartArt. 546 atomic cases cover 231 commands, 96 compositional cases combine multiple edits in a single document, and a Word-native oracle checks task-specific object properties and preservation conditions. Cases are constructed from editing-unit specifications, yielding controlled input documents, reference documents, and aligned oracle checks. Each compositional case is linked to corresponding atomic cases, supporting fine-grained diagnosis of editing failures.

We evaluate eight agent systems and find that many struggle on the Atomic track, with performance varying across feature areas. Moreover, every system scores lower on Compositional than on Atomic, with gaps that vary substantially by system. Exact atomic-intent links and recorded failure diagnoses provide complementary views of this variation.

Benchmark

The task landscape

WordEval builds two kinds of task from the same editing units, covering 231 Word commands in 15 feature areas. An Atomic task asks for a single Word command on an input document built for it. A Compositional task combines several editing units in one document; the four composition classes group these tasks by how their units interact. Each unit of a Compositional task is an atomic intent, linked to the Atomic task that applies the same command, so a drop in score reflects combining edits rather than new commands. The benchmark contains 546 Atomic tasks and 96 Compositional tasks with 319 atomic intents.

Select a feature area or a class to see its Word commands or scenarios.

Figure 2. The task landscape. Each Compositional task consists of atomic intents, each linked to an Atomic task; a band’s width is the number of atomic intents from a feature area into a composition class. Hover to trace a flow, select to list commands or scenarios.

Method

How WordEval checks an edit

WordEval judges the saved document, not the steps the agent took. Word opens the output, and an oracle reads the object properties the request should change, together with what should stay the same. A task passes only when every required check passes.

The evaluation loop

  1. reference.docxevaluation only
Figure 3. How a task is evaluated. Select a stage to see what happens there and how it played out for the task in Figure 1.

Coverage

What the oracle reads

Word represents a document as runtime objects with properties, such as Shape.Rotation or Font.Bold. For each Atomic task, the oracle reads the properties that its command should set. Across the 546 Atomic tasks, it reads 139 object properties on 65 runtime objects, shown here by feature area and runtime object.

Figure 4. Object–property coverage of the oracle on the Atomic tasks, the hierarchy of the paper’s appendix table. Rings run outward from feature area to runtime object to object property. Across the 546 Atomic tasks, the rings count 866 task–property reads: each task counts once under every property the oracle reads for its Word command. Each task’s check then focuses on the properties its own request changes: an ObjectRotateGallery rotation counts under Rotation and both flips, and its check compares Rotation. A property read in several feature areas, such as ColorFormat.RGB, appears under each.

Explore

One task from each part of the benchmark

Pick a feature area to see one of its Atomic tasks: the instruction the agent received, the rendered input and reference pages, the oracle’s check, and the object properties the oracle reads for it. Pick a composition class to see one Compositional task assembled from its atomic intents, each numbered to match its phrase in the instruction.

Figure 5. One task per feature area and per composition class, read from the top: a feature area leads to one of its Atomic tasks, and a composition class to the atomic intents of one Compositional task and the task they combine into. In an area’s view, the bottom row shows the runtime objects and object properties the oracle reads for the shown task, then the others it reads in that area; Figure 4 shows all 15 feature areas. Instructions, checks, and intents come from the released case specifications; pages are Word renders of the released documents.

Results

Key results

Eight agent systems run on the same 642 tasks. Claude Code with Opus 5 achieves the highest Overall Case pass rate at 87.1%, while the remaining systems score between 44.5% and 66.7%, and three human participants average 82.8%. All eight systems score lower on Compositional than on Atomic tasks.

Table 1. Main results, pass rate (%).
Agent systemAtomic 546Compositional 96Overall 642
CaseMacroCaseMacroCaseMacro
Claude CodeOpus 587.589.284.485.587.188.8
CodexGPT-5.6, high67.668.261.560.366.767.3
Kimi CodeK3, high63.468.153.155.861.866.8
OpenCodeGemini 3.8 Flash, high57.963.249.051.356.561.9
ZCodeGLM-5.2, high58.263.739.640.455.561.2
OpenCodeDeepSeek V4 Pro54.961.740.640.352.859.5
Qwen CodeQwen3.8-27B50.456.431.232.447.553.9
Mini AgentMiniMax M347.354.929.230.644.552.4
Humanmean of 385.286.769.468.482.884.8
Table 1. Pass rates (%) under the Word-native oracle. Case weights tasks equally, whereas Macro averages within each Atomic command or Compositional scenario and then across them. The best agent result in each column is highlighted. Human reports the mean across three participants.

Analysis

What the results reveal

We first compare Atomic performance across Word feature areas, then test whether success on individual edits carries over to combined requests, and finally break down how tasks fail.

RQ1: Where do agents succeed?

Performance depends strongly on the Word feature area. Tables and Paragraphs achieve the highest mean pass rates, at 83.2% and 81.1%, with strong performance on cell merging, row-height distribution, paragraph alignment, and indentation (Figure 6).

In contrast, Pictures has the lowest mean pass rate at 44.9%, followed by SmartArt (50.7%) and Page Structure (53.8%). These areas include image cropping, picture effects such as glow, bevel, and shadow, SmartArt node and formatting edits, and header and footer configuration. Image and diagram editing and page-level configuration thus remain harder than table and paragraph formatting.

Macro pass rate (%) <2020–4040–6060–8080–95≥95

Select a cell to see its task count.

Figure 6. Atomic command-macro pass rate (%) by feature area, with missing outputs counted as failures. Columns are ordered by the mean across systems; darker cells mark a lower pass rate, using the same bins as the paper.

RQ2: Can agents combine edits they can do alone?

Every system scores lower once edits are combined, but by different amounts. Compositional pass rates fall below Atomic ones by 3.2 to 19.1 points (Table 1). Systems that look alike on individual edits can diverge on combined requests. Gemini 3.8 Flash and GLM-5.2 pass 57.9% and 58.2% of Atomic tasks, but 49.0% and 39.6% of Compositional tasks.

By composition class. Scope-changing tasks have the highest pass rates for each system shown, from 58.3% to 100%. Layout-coupled tasks vary the most across systems, from 23.1% to 84.6% (Figure 7A).

Conditioned on atomic success. Restricting to cases whose linked Atomic tasks all pass raises additive pass rates by 41.6 points for K3 but only 2.3 points for GPT-5.6 (Figure 7B). Other classes are less uniform. On layout-coupled tasks, GPT-5.6 has a conditional pass rate of 66.7% against a raw rate of 76.9%, and GLM-5.2 passes only two of its eight eligible cases.

Passing each edit alone does not ensure passing them together.

Figure 7. Compositional case pass rates by class for five selected systems. (A) Case pass rates over all cases in each class. (B) Conditional case pass rates restricted to cases for which the same system passes all linked Atomic tasks. Rates with fewer than five such cases are omitted, and those cells show the passed and eligible counts. Hover a bar for its count; missing outputs count as failures.

RQ3: How do outcomes break down?

Systems fail in different ways. Some return documents that fail a requested check, while others fail before the requested edit can be evaluated at all (Figure 8A).

Incorrect output
For Opus 5, 59 of 83 non-passing cases are valid documents that fail a check, and none are invalid.
Invalid or missing
For GPT-5.6, invalid documents and missing outputs together account for 18.4% of all cases.
Missing output
Gemini 3.8 Flash ends 41.7% of all cases, 268 of 642, without a submitted document, including runs stopped at the 600-second limit.

Most incorrect outputs contain the requested element but differ in its settings or Word structure. Among the 59 incorrect outputs of Opus 5, 23 are property-value mismatches and 20 are structure mismatches (Figure 8B). A shadow preset or a footnote numbering format may differ, and in the sin x and ln x equations Opus 5 inserts an invisible function-application character (U+2061) that leaves the rendering unchanged, so a screenshot cannot show it.

Figure 8. Failure analysis. (A) Output status of non-passing cases, as a share of all 642 cases per system, with each total at the right. (B) Mismatch types of the 59 incorrect outputs of Opus 5 (bold in A), one type per case.

Explore more

WordEval measures Word-native operation fidelity by checking each requested edit against the document’s objects and properties.

For a broader look at how we evaluate agents on Office tasks, see our papers on two related benchmarks. WordInferBench tests whether agents can infer the intended edits from Word document context, while OfficeEval tests agents on practical tasks across Word, Excel, and PowerPoint.

Citation

@inproceedings{park2026wordeval,
  title     = {WordEval: Evaluating Word-Native Operation Fidelity in Document Editing},
  author    = {Jinyoung Park and Yupan Huang and Shimin Zhang and Jiayu Ding and Yilin Jia and Yuzhong Zhao and Tengchao Lv and Wenshan Wu and Shaohan Huang and Nan Yang and Xiangyang Zhou and Dongdong Zhang and Li Dong and Lei Cui and Sun Mao and Xun Wang and Tao Ge and Huitian Jiao and Si-Qing Chen and Changick Kim and Furu Wei},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026}
}