Skip to content

Microsoft · KAIST · Xi’an Jiaotong University

WordEvalEvaluating Word-Native Operation Fidelity in Document Editing

AI is moving from assisting with writing to operating office software. In Microsoft Word, agents can now format paragraphs, modify tables, apply effects to shapes, and update charts. Evaluating these actions requires more than checking whether the document looks correct.

Consider a request to turn three paragraphs into a bulleted list. An agent may type bullet characters before the text, even though Word still represents the content as ordinary paragraphs. The page looks like a list, but the requested list formatting has not been applied. A document can appear correct while differing from the requested result in its underlying formatting, properties, or structure.

To evaluate this gap, collaborators from Microsoft, KAIST, and Xi’an Jiaotong University developed WordEval. It verifies whether requested edits are applied to the intended document elements with the required settings, and whether properties covered by preservation checks remain unchanged.

The WordEval paper was accepted to the NeurIPS 2026 Track on Evaluations and Datasets.

Verifying document state in Word

Word represents a document as structured elements—including paragraphs, lists, tables, pictures, shapes, and charts—with properties that determine their formatting and behavior. WordEval opens each result file in Microsoft Word and reads the relevant elements and properties through Word’s automation interface. Where needed, it also inspects the saved file to confirm that the requested change persists.

Documents
output.docx
Document saved by the agent
saved by the agent
reference.docx
Reference document for the task
built for the task
Request the agent was given
WordEval
opens both files in Word,
reads the properties the request names
Verdict
output.jsonwhat Word reads in output.docx
reference.jsonwhat Word reads in reference.docx
Figure 1. How WordEval reaches a verdict, replayed on six real cases from the evaluation. The evaluator opens the agent’s saved document and the reference document in Word, reads the properties the request names, and compares them. In the first case both documents show bullets, yet Word reads no list in the agent’s output.

The same distinction applies beyond lists. Permanently cropping and replacing a picture can reproduce the expected appearance while losing the original image that Word’s crop settings preserve. Typing a title in a header is also different from inserting a content control linked to the document’s Title property. WordEval checks whether an edit acts on the correct object, uses the required property values, and preserves the properties covered by its checks.

Constructing tasks and reference documents

Each task is generated from a single specification. An input builder first constructs the starting document, and a reference executor then applies the specified operation through Word’s automation interface to produce the expected document state. The instruction and verification checks are derived from the same specification: which elements and properties to inspect, what values they should take, and which existing properties must be preserved. Agents are free to choose their own tools and steps, and every final result faces the same set of checks. Building tasks this way keeps the instruction, the answer, and the evaluation criteria consistent with one another.

WordEvalfrom an editing specOne spec yields the task, the reference and the checks
1

Write an editing spec

commandtargetvalues

One spec per task

2

Build the input

input.docxinstructionchecks

All derived from the same spec

3

Apply the edit in Word

Word automationreference.docx

Run the operation on the input

4

Combine and link

AtomicCompositional

Each combined edit links to its Atomic task

642tasks
15Word feature areas
546Atomic · 231 commands
96Compositional
Figure 2. How WordEval builds a task. A shared editing specification defines the input document, instruction, and checks. The reference executor applies the specified operation to the input document in Word to produce the reference (green step). Each Compositional edit links to the Atomic task that uses the same command.

Evaluating individual and combined edits

WordEval contains 642 tasks across 15 Word feature areas, including text, paragraphs, lists, tables, pictures, charts, SmartArt, and page structure. Its 546 Atomic tasks cover 231 commands, while 96 Compositional tasks cover 27 scenarios.

WordEval evaluates both the fidelity of individual edits and the additional challenges of combining them in one document. Each Atomic task examines one operation, such as changing paragraph spacing, cropping a picture, or adding an effect to a shape. Compositional tasks place several edits in a single instruction. These edits may have explicit sequential dependencies, or they may affect one another through their targets or the page layout. Both tracks draw on the same operation pool and use the same Word-native evaluation criterion, and each component edit is linked to an Atomic task that uses the same command. Comparing these linked tasks helps reveal whether operations that succeed on their own still succeed when combined in one document.

Some combined requests have explicit dependencies: a chart must be inserted before data labels can be added to it. Other operations, such as positioning and alignment, interact through the page layout. Linking each component edit to an Atomic task makes it possible to distinguish the difficulty of an individual operation from the additional difficulty of combining operations.

What the evaluation reveals

The team evaluated eight agent systems, with overall pass rates between 44.5% and 87.1%. Every system scores lower on Compositional tasks than on Atomic tasks, by 3.2 to 19.1 percentage points. Some systems with similar Atomic scores differ sharply in compositional ability. OpenCode (Gemini 3.8 Flash) and ZCode (GLM-5.2) reach Atomic pass rates of 57.9% and 58.2%, and Compositional pass rates of 49.0% and 39.6%, respectively. On eight requests that involve layout interactions such as positioning and alignment, ZCode (GLM-5.2) passes every corresponding Atomic task yet completes only two of the Compositional tasks. Operations an agent can perform one at a time do not always all come out right once they are placed in the same document.

WordEvalAtomic vs Compositional · pass rate (%), 642 tasks

    Figure 3. Atomic and Compositional case pass rates of eight agent systems and the human baseline. All score lower on Compositional tasks.

    WordEval also separates failures to produce an evaluable document from editing errors inside valid Word documents. Some agents produce no file, or output an invalid or corrupted document. Other files open normally but contain wrong property values, missing elements, edits on the wrong target, or unintended changes to properties that should have been preserved. A closer analysis of the 59 incorrect outputs from Claude Code (Opus 5) shows that even when an agent selects the correct shape, it may still apply the wrong shadow or reflection type. Completing an edit takes more than producing a similar-looking page. Each operation must act precisely on Word’s actual document structure, and the changes must remain compatible with one another.

    Figure 4. Failure analysis. (A) Output status of each system’s non-passing cases, as a share of all 642 tasks. Missing output means that no document was submitted, invalid output means that Word cannot open the document or it fails OOXML validation, and incorrect output means that a valid document fails a requested or preservation check. (B) Mismatch types in the 59 incorrect outputs from Claude Code (Opus 5) (bold in A), one type per case.

    Toward reliable Word-editing agents

    WordEval provides tasks and verification tools for evaluating how accurately agents use Microsoft Word’s editing features. By inspecting the document state in Word, it distinguishes visually similar outputs that differ in whether the requested operation was applied correctly. Its linked Atomic and Compositional tasks also reveal how performance changes when individual operations are combined into a larger request.