Video
ExampleSix benchmark tasks, replayed. Word reads the properties each request names from the agent’s output and from the reference.
WordEval evaluates agents on Word editing operations, with Word itself as the verifier. Each requested change is checked in the document's objects and properties, across 231 commands, alone and combined with other edits in one document.
Jinyoung Park1†* Yupan Huang2† Shimin Zhang2 Jiayu Ding3* Yilin Jia2 Yuzhong Zhao2 Tengchao Lv2 Wenshan Wu2 Shaohan Huang2 Nan Yang2 Xiangyang Zhou2 Dongdong Zhang2 Li Dong2 Lei Cui2 Sun Mao2 Xun Wang2 Tao Ge2 Huitian Jiao2 Si-Qing Chen2 Changick Kim1‡ Furu Wei2‡
1KAIST2Microsoft3Xi’an Jiaotong University
†Equal contribution.‡Corresponding authors.*Work done during an internship at Microsoft Research.
Video
ExampleSix benchmark tasks, replayed. Word reads the properties each request names from the agent’s output and from the reference.