Every support agent we have put into a client's operation started with a week of questions. A benchmark now shows what happens when a coding agent skips that week.
Sierra, which sells customer service agents, and Princeton researchers published it on September 4, 2026: a coding agent gets a business's records, a client who holds the requirements, a production API, a codebase and a cost limit, and must deliver a customer service agent scored against held-out simulated users. Across 53 tasks in four domains, the strongest configuration passed 23.9% of evaluation simulations; expert-authored reference agents passed 82.2%. The failures: shallow queries instead of reading the records, almost no communication with the client, and shipping the first design that ran.
IDC, the research firm, quoting its analyst Philip D. Harris, on X: most enterprise AI is already running with more autonomy than it has earned, and most buyers cannot prove otherwise.
What transferred to our line was the client interview as a deliverable: questions, answers and the cases that came out of them, written before any agent is built and owned by a person on the client's side. The coding agent builds against that file. Without it, it builds against its guess.
The 82.2% is not a better model. It is a person who asked.
Sources
- Quan Shi, Keshav Dhandhania, Karthik Narasimhan and Victor Barres (Sierra, Princeton University), hyper-tau-bench: An Environment for End-To-End, Realistic Agent Construction, arXiv 2609.04611, September 4, 2026. https://arxiv.org/abs/2609.04611
- IDC on X, September 14, 2026. https://x.com/IDC/status/2099513631444439188