Microsoft · KAIST · Xi’an Jiaotong University
WordEvalEvaluating Word-Native Operation Fidelity in Document Editing
AI is moving from assisting with writing to operating office software. In Microsoft Word, agents can now format paragraphs, modify tables, apply effects to shapes, and update charts. Evaluating these actions requires more than checking whether the document looks correct.
Consider a request to turn three paragraphs into a bulleted list. An agent may type bullet characters before the text, even though Word still represents the content as ordinary paragraphs. The page looks like a list, but the requested list formatting has not been applied. A document can appear correct while differing from the requested result in its underlying formatting, properties, or structure.
To evaluate this gap, collaborators from Microsoft, KAIST, and Xi’an Jiaotong University developed WordEval. It verifies whether requested edits are applied to the intended document elements with the required settings, and whether properties covered by preservation checks remain unchanged.
The WordEval paper was accepted to the NeurIPS 2026 Track on Evaluations and Datasets.
Verifying document state in Word
Word represents a document as structured elements—including paragraphs, lists, tables, pictures, shapes, and charts—with properties that determine their formatting and behavior. WordEval opens each result file in Microsoft Word and reads the relevant elements and properties through Word’s automation interface. Where needed, it also inspects the saved file to confirm that the requested change persists.
The same distinction applies beyond lists. Permanently cropping and replacing a picture can reproduce the expected appearance while losing the original image that Word’s crop settings preserve. Typing a title in a header is also different from inserting a content control linked to the document’s Title property. WordEval checks whether an edit acts on the correct object, uses the required property values, and preserves the properties covered by its checks.
Constructing tasks and reference documents
Each task is generated from a single specification. An input builder first constructs the starting document, and a reference executor then applies the specified operation through Word’s automation interface to produce the expected document state. The instruction and verification checks are derived from the same specification: which elements and properties to inspect, what values they should take, and which existing properties must be preserved. Agents are free to choose their own tools and steps, and every final result faces the same set of checks. Building tasks this way keeps the instruction, the answer, and the evaluation criteria consistent with one another.
Write an editing spec
One spec per task
Build the input
All derived from the same spec
Apply the edit in Word
Run the operation on the input
Combine and link
Each combined edit links to its Atomic task
Evaluating individual and combined edits
WordEval contains 642 tasks across 15 Word feature areas, including text, paragraphs, lists, tables, pictures, charts, SmartArt, and page structure. Its 546 Atomic tasks cover 231 commands, while 96 Compositional tasks cover 27 scenarios.
WordEval evaluates both the fidelity of individual edits and the additional challenges of combining them in one document. Each Atomic task examines one operation, such as changing paragraph spacing, cropping a picture, or adding an effect to a shape. Compositional tasks place several edits in a single instruction. These edits may have explicit sequential dependencies, or they may affect one another through their targets or the page layout. Both tracks draw on the same operation pool and use the same Word-native evaluation criterion, and each component edit is linked to an Atomic task that uses the same command. Comparing these linked tasks helps reveal whether operations that succeed on their own still succeed when combined in one document.
Some combined requests have explicit dependencies: a chart must be inserted before data labels can be added to it. Other operations, such as positioning and alignment, interact through the page layout. Linking each component edit to an Atomic task makes it possible to distinguish the difficulty of an individual operation from the additional difficulty of combining operations.
What the evaluation reveals
The team evaluated eight agent systems, with overall pass rates between 44.5% and 87.1%. Every system scores lower on Compositional tasks than on Atomic tasks, by 3.2 to 19.1 percentage points. Some systems with similar Atomic scores differ sharply in compositional ability. OpenCode (Gemini 3.8 Flash) and ZCode (GLM-5.2) reach Atomic pass rates of 57.9% and 58.2%, and Compositional pass rates of 49.0% and 39.6%, respectively. On eight requests that involve layout interactions such as positioning and alignment, ZCode (GLM-5.2) passes every corresponding Atomic task yet completes only two of the Compositional tasks. Operations an agent can perform one at a time do not always all come out right once they are placed in the same document.
WordEvalAtomic vs Compositional · pass rate (%), 642 tasks
WordEval also separates failures to produce an evaluable document from editing errors inside valid Word documents. Some agents produce no file, or output an invalid or corrupted document. Other files open normally but contain wrong property values, missing elements, edits on the wrong target, or unintended changes to properties that should have been preserved. A closer analysis of the 59 incorrect outputs from Claude Code (Opus 5) shows that even when an agent selects the correct shape, it may still apply the wrong shadow or reflection type. Completing an edit takes more than producing a similar-looking page. Each operation must act precisely on Word’s actual document structure, and the changes must remain compatible with one another.
Toward reliable Word-editing agents
WordEval provides tasks and verification tools for evaluating how accurately agents use Microsoft Word’s editing features. By inspecting the document state in Word, it distinguishes visually similar outputs that differ in whether the requested operation was applied correctly. Its linked Atomic and Compositional tasks also reveal how performance changes when individual operations are combined into a larger request.