How to Evaluate Your First B2B AI Pilot

An AI pilot can produce good-looking answers and still leave the customer with more work. Someone has to check the output, correct it and take responsibility when it goes wrong. That work belongs in the evaluation.

Before extending your first B2B AI pilot, answer three questions with the customer: does the completed work meet their quality requirements, does the whole workflow improve on their real alternative, and who will decide what happens next?

A fast draft doesn’t settle those questions. Here’s how to make the decision concrete.

Define the job the customer needs completed

Choose one bounded workflow with the customer. For a support tool, that might be preparing replies to routine account questions for human approval. It wouldn’t include sending those replies, changing an account or handling sensitive cases unless that authority was explicitly agreed.

Have the customer select tasks that represent the intended work, including difficult cases. Define unacceptable failures before running them. An invented policy, exposed personal information or an incorrect instruction may matter much more than a stylistic flaw. The customer needs to decide which failures block continuation and what human review must remain.

Keep corrections caught during review separate from errors that escape it. A batch with no observed escaped errors doesn’t prove the system is safe in every future case.

GitLab’s AI evaluation plan separates end-user time savings, technical quality and the economic buyer’s cost model. It’s an enterprise process; the useful adaptation for a startup is to agree on those different requirements for one workflow.

Count the work after generation

Anthropic’s November 25, 2025 productivity research explicitly leaves out work outside Claude conversations, including checking the quality or accuracy of the output. Those are model estimates, not measured returns from your pilot.

Measure the customer’s completed task. Include preparation, review of every relevant output, corrections, rechecking and escalation. Also record setup and ongoing support. If someone is blocked waiting for the model or another system, count that time; don’t count machine waiting as human work when the person can do something else.

Keep calendar turnaround separate. A workflow can use fewer staff minutes while taking longer to deliver the answer. The customer may care about both.

Work through one batch before projecting a year

Everything in this example is hypothetical, including the task counts, times and prices. These are alternative assumptions to test, not observed results or a required pilot sample.

Suppose a customer has 100 routine support replies to prepare. The manual process takes six minutes per reply, including its normal checking and corrections:

100 × 6 = 600 human minutes, or 10 hours.

Now suppose the first AI-assisted batch needs 60 minutes of setup. A person reviews all 100 drafts. Correction time is additional to that initial review, and includes the rechecking within each correction episode.

Here are three possible workloads. Each assumes the same task mix and the same required final quality. The last column values human capacity at $40 an hour and adds a hypothetical $25 of usage per batch, before any vendor fee.

Scenario	Review minutes	Extra correction minutes	Total human minutes	Labor plus usage
Optimistic	100	20	180	$145.00
Planning	300	90	450	$325.00
Cautious	500	200	760	$531.67

In the optimistic case, reviewing each draft takes one minute and five replies need four additional minutes each: 60 + (100 × 1) + (5 × 4) = 180 minutes.

The planning case assumes three minutes of review per draft and 15 correction episodes of six minutes: 60 + (100 × 3) + (15 × 6) = 450 minutes. That leaves 150 minutes of capacity compared with the manual baseline.

The cautious case assumes five minutes of review per draft and 25 correction episodes of eight minutes: 60 + (100 × 5) + (25 × 8) = 760 minutes. The customer spends 160 minutes more than before, even if generating each draft is fast.

These simplified cases assume one correction episode for each affected reply. If the real process needs repeated attempts, count them all. Record which task types cause that work; a slow exception hidden inside the average may be a reason to narrow the product’s scope.

Put the price beside the capacity you might free

At the assumed $40 an hour, the manual batch represents $400 of labor capacity. The planning case uses $300 of capacity plus $25 of usage, leaving $75 before your vendor fee.

Add a hypothetical $100 fee for that batch and the modeled total becomes $425 against the $400 baseline. That’s an illustration of why the fee belongs in the comparison, not a recommendation for what to charge.

The $25 usage assumption is paid separately in this example. If usage is bundled in your fee, count it once. Add integration, security, migration, support and other relevant costs rather than hiding them outside the model.

Capacity value isn’t automatically a cash saving. Freeing two and a half hours doesn’t reduce payroll by itself. Ask what useful work the customer could actually do with that time, or which real expense it would avoid. Neither the arithmetic nor a satisfied user establishes willingness to pay.

The 60 setup minutes are a first-batch cost here. Later batches may differ. Show the first batch and recurring work separately; only remove setup that genuinely won’t recur.

Agree on the decision before extending the pilot

The calculation can change what you do next. If a few task types consume the correction time, test a narrower scope. If the review burden is unavoidable, change the workflow or reconsider the offer. If required quality fails, a lower price doesn’t make the output acceptable.

Put the next evaluation into a short shared agreement. Fill this in with the customer:

• Workflow and authority: We will test [task types] using [permitted data]. The system may [allowed actions]. [Person] retains responsibility for [review, approval and exceptions].
• Representative work and baseline: The customer will select [tasks and relevant variation]. Our comparison is [current process], measured through [the same completed outcome].
• Quality gate: Unacceptable failures are [types and severity]. We will record [caught corrections and escaped errors separately]. [Person] decides whether the required quality is met.
• Work and cost: We will measure [setup, review, correction, escalation, blocking wait and recurring support]. Costs include [usage, vendor fee and other relevant costs].
• Decision: On [date], [customer decision-maker] will review the evidence with [users and other required stakeholders] and choose [continue at an agreed scope and terms, revise a named assumption and retest, or stop].

This makes an extension a test of something specific. It doesn’t turn technical acceptance into a purchase commitment or bypass a customer’s security and buying process. Confirm who can authorize continued work and what approval still needs to happen.

If you haven’t established whose work should improve, return to the customer discovery interview guide. If the work is useful but the commercial next step stays vague, why B2B deals go quiet covers that separate conversation.

For help choosing how to reach potential customers before a pilot, the First 1,000 Customers playbook is available through its signup page. It includes the playbook and The Sauce, my weekly newsletter for founders; unsubscribe anytime.

Before the next extension, replace one claim about what the AI saves with the customer’s observed workload. Then decide together what that evidence earns.