A third of failed agent runs in AgentMail's test ended at one hidden default

In AgentMail's test, an agent looking for an email lists the inbox, sees nothing and concludes the email never arrived ^[28]. The inbox only looked empty because it was filtered ^[26]. In the run the report shows as typical, Claude Opus 5.5 inside Claude Code checked the inbox eight different ways, spam, trash and junk among them, and never asked for unauthenticated mail, the one view the documented path uses ^[29].

The default list leaves out spam and mail from senders that cannot be authenticated, and the documentation explains how to ask for both ^[27]. The person who answers for that outcome on Friday is whoever shipped the integration. Luca, who writes Refactoring, says many teams underestimate MCP servers and "think about them as simple API wrappers, assign the task to an intern, and call it done" ^[5].

The stakes are distribution. Luca writes that products now appear inside agent hosts such as ChatGPT, Claude and Grok Bot as plugins the model can use, and that agents sometimes act on their own to recommend plugins that fit the current job ^[1]. Those plugins run on MCP servers, which wrap and expose the product's functionality ^[3]. He says agents are becoming important distribution platforms, with many users discovering new products from inside them ^[2].

What the practitioners agree on

Refactoring, Block and Notion give the same advice on tool design. Block, in a playbook at engineering.block.xyz, says it has developed more than 60 MCP servers ^[11]. The playbook tells teams to start from the workflow that needs automating and work backwards to the tools ^[12]. It advises against exposing raw endpoints such as GET /user or GET /file, which it says usually produce verbose, poorly scoped outputs ^[13]. In its example, two tools, get_user and upload_file, become one upload_file(path, owner) call that gets the user, uploads the file and returns a success or error message ^[14].

Luca puts the failure in terms of cost. In his experience the most common mistake is turning each existing endpoint into a tool ^[6]. When a job departs too far from what single endpoints do, the model spends tokens and time finding the path, chains five calls where one would do, and may still fail the workflow ^[7]. His fix is to list the handful of end-to-end workflows people ask agents to do with the product and build one tool, or a tight chain, for each ^[8].

Notion built the mirror first. Its first release was a downloadable open-source server: users created a new Notion API integration, then either copied the API key into MCP headers or built a Docker image ^[17]. Each API endpoint mapped 1:1 to a tool ^[18]. Early adopters said the technical barrier was too high and that the mapping gave agents a poor experience, with heavy token use from hierarchical block data in JSON as one example ^[19]. Notion wrote that it wanted to "move quickly to get something in the hands of our users" ^[20]. It now hosts its own server, with a tailored suite of tools optimized for AI agents and functionality slices reachable only through the MCP server ^[21].

Granularity cuts both ways. Luca's default is tools that finish a job in one or two calls ^[10]. If tools are too fine-grained, the team pays for multi-hop agent loops. If a single mega-tool carries a kitchen-sink schema, arguments get brittle, recovery gets hard and, in his words, "auth gets messy" ^[9].

What agents do with a default

Teams tell themselves that an agent with the documentation in hand will find the missing parameter. In AgentMail's runs, linking the docs lifted success from 76% to 84% ^[31], a gain of 8 points ^[38]. The report says the docs mostly helped agents find the tool, and on one job they made agents worse ^[32].

One caution on the source. AgentMail builds an email API for agents and used it as the test bed, running 25 agents, each a model inside a harness such as Claude Code or Codex, through seven tasks with and without links to the docs ^[23]. Scope ran every agent, and AgentMail wrote the tasks and graded the recordings ^[24]. Every figure in this section comes from that one report.

Agents in the test rarely failed at the work itself. 80% of runs finished, and writing replies, saving files and summarizing was almost never what went wrong ^[25]. The report ties most failures to an inbox that looked empty because it was filtered, an endpoint that did less than its name, or a spam label that contradicted the user, in which case the agent refused a job it had been asked to do ^[26]. An empty list was not fatal on its own: 76% of the runs that hit one kept looking and finished, and the failed runs were the ones that stopped there ^[30]. About a third of failed runs ended at the hidden-list default ^[28]. Counting the 20% of runs that did not finish as the failed ones ^[36], that default sits behind roughly 7% of all runs ^[37].

Block makes a related point about error text. Its Goose file reader raises an error on files over 400kB, because Block finds models work around the limit with shell commands such as sed -n ^[15]. The playbook prefers error messages the agent can act on to vague prompt instructions ^[16].

The host adds its own spread. Claude Sonnet 5 scored 69 in one harness and 38 in another ^[33], a gap of 31 points ^[39], and the report says that for some models the harness matters as much as the model ^[33]. The report concludes that most fixes belong to the tool: say what a list left out, name endpoints for what they do, say what a label means ^[34].

What the sources leave open

None of the four sources reports a host rewarding or penalizing a server for how it was built or how it handles permissions. What they do report is that endpoint mirrors cost agents tokens and calls ^[7], and that early users of one mirror said so ^[19]. The one public penalty in the set is Figma's. According to Luca, Figma drew backlash for hardcoding the list of allowed clients in its MCP server, which disabled tools such as Pi and OpenCode, and one of the protocol's creators said this is not how the protocol should work ^[4]. That case is about which clients may connect. What a connected agent may do once connected is a separate question.

Notion's design is the only permission model in the set. Each user goes through a one-click OAuth flow to connect securely to the same public integration, and Notion writes that users install MCP in their workspace, "granting the permissions they have normally in the app to the MCP integration" ^[22].

In our view, tool breadth and credential scope move together. A tool that does many jobs needs a credential that allows all of them. An integration that takes the user's own permissions gives the agent that user's reach within the app on every job, however narrow the job.

We'd name two owners, who can be the same person. The server owner keeps the list of workflows and the wording of descriptions and errors. The scope owner decides what the credential may touch for each workflow, and can explain it when a customer asks why an agent reached something. The price is a decision per tool before release. That slows a first version. Notion took the fast path first, saying it wanted to "move quickly to get something in the hands of our users" ^[20], and then moved to a hosted server with tools tailored for agents ^[21].

A sort for Monday

A team can sort every tool in its server on two tests. The first is whether the tool finishes a named job in one or two calls ^[10]. The second is whether a named person chose the credential scope it runs with.

Job-shaped with a chosen scope needs no change, and Block's single upload_file call is the shape to copy ^[14]. Endpoint mirror with a chosen scope needs a redesign; the cost shows up as extra tokens and calls ^[7], and Notion's early adopters reported it ^[19]. Job-shaped with an inherited scope works until someone asks why the agent could reach what it reached, and our case for filling that cell rests on the reasoning above. Endpoint mirror with an inherited scope carries both defaults and goes first.

A third test applies to any tool that returns a list or a search result: the response should say what it left out ^[34]. AgentMail's report puts the rule in a line: "Treat an empty result as a question, not an answer." ^[35]