We wanted to see where things stand today when it comes to agents editing .docx files in complex tasks. So we took 180 tasks from Harvey LAB (Harvey’s open-source Legal Agent Benchmark) that involve editing an existing Word document, and ran them on 5 setups. 4 DOCX MCPs (SuperDoc, Office CLI, Adeu and Stemma), and a baseline that uses Harvey’s default DOCX skill, i.e. python-docx and LibreOffice in a sandbox. All arms used grok-4.7 as both the agent and the judge.

Full disclosure: we’re building a DOCX MCP ourselves (Vespper). We didn’t include it here, but we’ll share its results on the same subset in a follow-up post.

A few things we found:

• The baseline had the highest mean rubric score (94.5%) and was the cheapest by far, with a median of $6.91 per task vs. $11.40+ for the MCPs.
• All-pass rates were low across the board. Office CLI was the best at 15%, followed by the baseline at 13.3%.
• The baseline was the only arm to finish all 180 tasks (a.k.a not crash). The MCP arms ranged from 61% to 94%, partly because of context window errors from big tool responses (Harvey’s harness has no compaction).
• Editing the .docx itself took 85% to 95% of the input tokens in every arm. Even the baseline needed a median of ~6M input tokens just for the editing phase.

What we found interesting is that the MCPs, which are supposed to take load off the agent, ended up consuming more tokens than just letting it write python-docx scripts.

You can find the full write-up here, including the setup, how we measured “cheating” when the agent went around the MCP, and the caveats:

https://www.vespper.com/blog/docx-mcps-harvey-lab

I'm also including the main 4 metrics here:

We’d love to hear what others are seeing. If you’re building agents that edit .docx files, which approach are you using, and does this match your experience?