What COOs Should Ask Before Putting AI Agents in the Workflow

Insurance operations leaders are being pitched AI agents from every direction. Some of those agents will genuinely transform how work gets done. Others will produce an impressive demo, a stalled pilot and a sceptical team.

Having worked with dozens of carriers, MGAs and brokers through their first agent deployments, we have seen what separates the two. It rarely comes down to the model. It comes down to the questions the operations team asked before they started.

Here are the ones we think matter most.

1. Which workflow, end to end?

The most common mistake is starting with a task rather than a workflow. "Extract data from ACORD forms" is a task. "Get every new submission from the inbox to an underwriter, complete and prioritised" is a workflow.

Tasks are easy to automate and rarely change the economics of a team, because the person still has to do everything around the task. Workflows are harder, but they are where the capacity is released. Before choosing a vendor, write down the workflow you want to change, every step in it, and who does each step today.

2. What does the agent do when it is unsure?

Every agent will meet documents it cannot read, requests it does not understand, and cases that fall outside the rules. What matters is how it behaves when that happens. Ask to see:

• How the agent signals uncertainty, and at what level of confidence it stops and asks
• Where uncertain cases go, and what context comes with them
• How your team's decisions on those cases feed back into the agent's behaviour

An agent that guesses confidently is far more dangerous than one that escalates too often. The right answer is an agent that knows its limits and makes escalation cheap.

3. Can we see why it did what it did?

In a regulated industry, "the model decided" is not an acceptable answer. For every action an agent takes, you should be able to see what information it used, what rule or guideline it applied, and what it produced. That record should be exportable for internal audit and regulators. If a vendor cannot show you this on day one, it will be very hard to add later.

4. Does it work in our systems, or beside them?

Some agents operate in a separate interface and expect your team to copy results back into the policy admin or claims system. That simply moves the rekeying from one place to another. Ask whether the agent writes directly to your systems of record, using your existing permission model, and what happens when those systems are slow or unavailable.

5. Who owns the agent after go-live?

Workflows change. Guidelines are updated, new forms appear, a broker changes their submission format. Someone has to keep the agent aligned with the business. Ask whether your operations team can adjust instructions, rules and escalation thresholds themselves, or whether every change requires the vendor's engineers. The best deployments we see are owned by operations, not IT.

6. How will we measure success?

Agree on the metrics before the pilot starts, and measure the baseline honestly. For most workflows, the right measures are a mix of:

• Cycle time, from request received to work complete
• Touch rate, the share of cases that need a person at all
• Quality, measured by error rates or rework found in sampling
• Capacity, the volume handled per person on the team

Avoid measuring only cost savings. The biggest wins we see are usually growth: faster quotes, more submissions handled, better broker relationships.

7. What happens to the team?

This is the question COOs most often leave until last, and it should come first. People do their best work with agents when they understand what the agent will take on, what will stay with them, and how their role will grow. The teams that succeed involve their most experienced people in designing the agent's instructions and reviewing its early output. Those people become the agent's most effective supervisors.

A practical next step

If you are evaluating agents for your operation, pick one workflow, answer these seven questions with your team, and use the answers as your scorecard. We have turned this list into a short evaluation worksheet that our team is happy to walk through with you.