The AI pilot worked. Should we scale it?
The demonstration is convincing. A task that took an hour appears to take minutes. The team wants to extend the pilot across the business, and the board is being asked to support the investment.
Before expanding it, establish what the pilot has demonstrated and what remains untested. Working well in a small, closely supervised trial is useful evidence. It does not answer every question about everyday use.
Test the work that will actually arrive
A pilot may use clean data, familiar tasks and enthusiastic users. Wider use introduces exceptions, incomplete information and people with less time to check the output.
For an illustrative customer-support tool, test difficult complaints, ambiguous requests and cases where the correct answer is to escalate. Examine serious mistakes separately from average performance. A high overall score can conceal failures that matter greatly to an affected customer.
Ask who selected the test cases and whether people outside the pilot team reviewed the results. Record the limitations alongside the successes.
Understand what human review involves
“A person checks the answer” needs more explanation. Who checks it? What knowledge do they need? How much time do they have? Can they reject the output and resolve the task another way?
Measure the work of checking, correcting and escalating. If the tool produces a plausible but wrong answer, the reviewer may need more expertise than the person who previously completed the task.
Management should define responsibility for the service as a whole. Buying a tool or appointing a technical team does not settle who is accountable when a customer receives an incorrect answer.
Rebuild the cost estimate for normal use
Include integration, data preparation, security, training, review time, support and monitoring. Examine how costs change with volume and supplier pricing.
Compare the proposed approach with the existing process using consistent measures. Time released may create capacity; it becomes a financial saving only when the operation changes accordingly.
Check with the relevant specialists that the intended data use, contractual arrangements and legal requirements have been addressed. An approval for one use does not settle the position for another.
Expand within limits that someone can enforce
A sensible next step may be a controlled expansion to a defined group of users or tasks, with clear performance criteria and a review date.
Agree what will be monitored, who can pause use and how work continues if the tool is withdrawn. Include arrangements for assessing supplier or model changes that could affect the results.
The next approval should identify the scope, budget and conditions for continuing. If important evidence is missing, name the test that will provide it. The board can then assess a specific proposal rather than a general promise about AI.
Further reading: NIST's voluntary AI Risk Management Framework Core and AI RMF Playbook cover evaluation, oversight and monitoring across the system lifecycle.