Key takeaways
- The resolution rate on a vendor dashboard is not your ROI. Count a conversation as resolved only when the customer got an answer and didn't come back for the same issue.
- Measure a baseline before you switch AI on, or reconstruct one from the last three months. Without it, every number afterwards is a guess.
- Count all the costs, including setup and the hours spent maintaining content, and all the gains, including after-hours coverage and sales the AI helped close.
- Small changes in the real resolution rate swing the result from strong profit to break-even, so improving content is usually the best investment you can make.
- Tidio is our top pick for small and mid-size businesses measuring AI for the first time, because Lyro's resolution rate and the sales-assisted report sit next to the human conversations you're comparing against.
Your AI agent's dashboard says it resolved 45% of conversations last month. Your finance lead asks a simpler question: how much did it save us? With budgets for next year being drafted in late summer, that question will come up in a lot of meetings over the next few weeks, and "45%" isn't an answer.
The gap between the two numbers is where most AI business cases go wrong. Vendors measure what their product touched. You need to measure what changed for your team, your customers and your revenue. This guide shows how to do that, with a worked example you can copy.
Why most AI ROI numbers are wrong
The usual mistakes are in what gets counted:
- Deflection is counted as resolution. A customer who closed the widget in frustration "didn't need a person".
- Silence is counted as success. The customer stopped replying, then emailed the same question the next day.
- Setup and maintenance are ignored. Somebody spends hours each month fixing content and reading transcripts.
- The baseline is missing. Nobody knows the cost per conversation before AI, so there's nothing to compare with.
- Quality isn't measured. Costs fall while satisfaction quietly drops.
- Revenue is left out. AI answering product questions at 10pm helps sell, and that never appears in a support budget.
Fixing these takes a few clear definitions and a spreadsheet.
Step 1: Set a baseline
Before you switch AI on, or by reconstructing the last three months if it's already live, record:
- Conversations per month by channel, and in your busiest month.
- Cost per human conversation: the full monthly cost of your support team (salaries, taxes, tools, management time) divided by the conversations they handle.
- First response time and resolution time by channel.
- Satisfaction (CSAT) and repeat contact rate, meaning how often customers come back about the same issue within a week.
- Conversations outside your support hours, and what happened to them.
- Sales from conversations, if you sell online: orders placed after a chat.
If you don't survey customers yet, start now. Our guide to CSAT, NPS and CES explains which measure to use.
Step 2: Define what "resolved" means
Write a definition before you look at results, and apply it to a sample of transcripts every month.
| What happened | Counts as resolved? |
|---|---|
| Customer confirmed the answer helped, or rated it positively | Yes |
| AI answered, customer didn't reply, and didn't contact you about it again within 7 days | Yes |
| AI answered, customer came back on another channel about the same issue | No |
| Customer asked for a person, or the AI handed over | No (but it's not a failure if the handoff was right) |
| Customer left halfway through without an answer | No |
Measure resolution from the customer's side, not the bot's. If they had to ask again, it wasn't resolved.
Step 3: Count every cost
- AI fees. Per conversation, per resolution or outcome, or an allowance bundled into a plan. Check how your vendor defines a billable conversation.
- Seat or plan changes needed to use the AI.
- Setup time: writing and fixing help content, configuring flows and handoff, testing.
- Ongoing maintenance: reviewing transcripts and updating content, usually a few hours a month.
- Integrations with your store, CRM or order system, if they cost extra.
Step 4: Count the savings
- Human time freed: resolved AI conversations multiplied by the cost of the human time they would have taken. Repeat questions are quicker than average, so use a lower cost for them than your overall average.
- Hiring avoided or delayed, if volume grows and headcount doesn't.
- Coverage gained: questions answered at night and at weekends that previously waited until morning.
Step 5: Check quality hasn't paid the price
Compare AI and human conversations each month on satisfaction, repeat contact rate and escalation rate, and read a sample of 20–30 AI transcripts for accuracy. If satisfaction on AI conversations is well below your human score, your savings are borrowed from customer loyalty. Fix the content first. Our guide on writing help content your AI agent can use shows how.
Step 6: Count the revenue effects
If you sell online, AI that answers sizing, delivery and stock questions can rescue sales, especially out of hours. Track conversations that ended in an order and their value, separately for AI and humans. Keep this line separate from cost savings in your business case, because it depends on attribution choices.
A worked example
The numbers below are illustrative assumptions to show the method. Replace every one with your own figures and your vendor's actual prices.
Assumptions: 4,000 conversations a month. A repeat question costs $3 of human time. The AI costs $0.80 per AI conversation, and every conversation starts with the AI. Content maintenance takes 8 hours a month at $40 an hour. Setup takes 40 hours once.
| Dashboard rate (45%) | Real rate after Step 2 (30%) | |
|---|---|---|
| Conversations resolved by AI | 1,800 | 1,200 |
| Human time saved (× $3) | $5,400 | $3,600 |
| AI fees (4,000 × $0.80) | −$3,200 | −$3,200 |
| Maintenance (8 h × $40) | −$320 | −$320 |
| Net monthly saving | $1,880 | $80 |
The same AI looks like a clear win or a wash, depending on how you define "resolved". In this example the one-off setup ($1,600) pays back within a month at 45%, and barely at all at 30%.
Two more things the table shows:
- Pricing model matters. If the same vendor charged $1.00 per resolution instead, the 30% case would cost $1,200 in AI fees instead of $3,200. A quick rule: divide the per-conversation price by the per-resolution price ($0.80 ÷ $1.00 = 80%). Per-conversation pricing only wins if your real resolution rate is above that.
- Content is the lever. Moving the real rate from 30% to 45% is worth $1,800 a month here, far more than any discount you could negotiate.
Common mistakes
- Using the vendor's definition of resolved without checking a sample of transcripts.
- Judging after one week. Results improve as content is fixed.
- Ignoring busy months. Volume-based AI fees rise in peaks, and so do savings. Model both.
- Cutting staff before the numbers settle. Run AI and people side by side for at least a month.
The best tools for measuring AI customer service ROI
We looked at how clearly each tool reports what its AI resolved, how its AI pricing model affects the maths, and how easy it is to compare AI and human conversations side by side.
Tidio puts Lyro, live chat and the inbox in one place, so AI and human conversations are measured in the same reports. Analytics show Lyro's resolution rate and a sales-assisted report, satisfaction surveys can follow each chat, and billing is based on conversations rather than seats. You can test on the free plan's 50 Lyro conversations, measure, and only then scale up. Larger brands that want the result pinned down can move to the Premium plan, which bills per resolution and guarantees a 50% Lyro resolution rate.
Why we like it
- AI and human conversations in the same analytics
- Lyro resolution rate and sales-assisted reporting built in
- Conversation-based billing makes cost per conversation easy to calculate
- Free plan to run a small pilot before committing
Watch out for
- AI conversations are billed by volume, so model your busiest months
- Less workforce and forecasting reporting than Zendesk for large teams
- Premium's guarantee and per-resolution billing are quote-only and start from 3,000 Lyro conversations a month
Intercom bills Fin per outcome and reports in depth on what it resolved, where it escalated and which content is missing. It suits larger teams and software companies that want to analyse AI performance closely.
Why we like it
- Detailed reporting on AI outcomes and content gaps
- Per-outcome pricing ties cost to results
Watch out for
- Seats plus per-outcome fees are expensive for small teams
- More than a small business needs to get started
Zendesk includes a number of automated resolutions per agent and bills further ones per verified resolution, alongside workforce tools for planning staffing. A good fit when the ROI question is about headcount across shifts and regions.
Why we like it
- AI billed on verified resolutions above the included allowance
- Workforce and reporting tools for large teams
Watch out for
- Needs a dedicated admin to set up and report well
- Advanced AI features sit on higher plans and add-ons
Freshdesk includes AI agent sessions up to a plan allowance and offers Freddy Copilot as a per-agent add-on, with reporting inside the help desk. Practical if you already run Freshdesk and want to measure AI without adding a tool.
Why we like it
- AI sessions included up to a plan allowance
- Reporting inside an existing help desk
Watch out for
- Copilot and extra sessions add cost
- AI answer quality trails the leaders on open questions
A 30-day ROI review you can run
- Week 1: record your baseline and write your resolution definition.
- Week 2: export AI conversations and classify a sample of 100 against the definition.
- Week 3: fill in the cost and savings table with your real numbers, plus a busy-month version.
- Week 4: compare quality, list the content gaps behind unresolved conversations, and fix the top five.
Repeat monthly. If you're still choosing a tool, our guide on how to choose an AI agent for customer service covers how to run a fair trial. For the wider rollout, see how to automate customer support.
Frequently asked questions
What is a good resolution rate for an AI agent?
It depends heavily on your business and your content. Businesses with many repeat questions about orders, shipping and policies resolve far more than those with complex, account-specific issues. Measure your own true resolution rate over a few weeks rather than comparing with vendor averages.
What is the difference between deflection and resolution?
Deflection counts every conversation that didn't reach a person, including customers who gave up. Resolution counts only conversations where the customer got what they needed. Deflection always looks better, which is why it makes a poor ROI measure.
How long does it take for AI customer service to pay off?
Setup costs are usually small for a small business, so the question is less about payback time and more about whether the monthly saving is real. Expect two to four weeks of content improvements before the resolution rate settles, then judge.
Is per-resolution or per-conversation pricing cheaper?
It depends on your resolution rate. Divide the per-conversation price by the per-resolution price. If your real resolution rate is above that number, per-conversation pricing is cheaper; below it, per-resolution pricing wins.