Your CSAT Dashboard Is Lying to You. Here's What To Do About It.

Every CX leader knows the numbers. CSAT, first response time, resolution time, QA scores, contact rate and ticket deflection all have their place on the dashboard.

They can tell you that something has changed. CSAT dropped. Response times increased. One contact reason suddenly spiked. But there is a limit to what those numbers can explain on their own.

A customer can give an interaction a poor score because an agent handled it badly. They can also give exactly the same score because the wrong item arrived, the website created the wrong expectation, a subscription renewed unexpectedly or a policy frustrated them. The number records the outcome. It does not necessarily tell you where the experience actually broke.

That distinction is becoming more important as CX teams gain access to something much richer than another dashboard: the customer conversations themselves.

In this episode of CX After Hours, Naomi Oriol, Director of Customer Experience at Freebird, joined hosts Anya Kelly and Guillaume Luccisano to discuss how her team is using AI to analyse support conversations at scale, uncover problems hidden inside thousands of tickets and bring much stronger evidence back to the rest of the business.

Watch or listen to the full episode:

Apple Podcasts

Spotify

YouTube

Your dashboard is a thermometer, not an MRI

In the episode, Naomi doesn't argue that CX teams should stop measuring performance. Her issue is what happens when those metrics become the entire conversation.

During the episode's 'Vent' segment, she described how CX teams have become conditioned to celebrate lower handle times, faster first responses, higher ticket deflection and stronger CSAT while customers may still be reporting the same underlying problems again and again.

"The scores, they've become something that CX has to defend in a meeting instead of a diagnostic that the business uses."

The problem is not the metric itself but expecting the metric to diagnose the business problem behind it.

Naomi's analogy is a useful one:

"It's like a thermometer. But we're treating it like a MRI type of diagnostic tool."

A thermometer can tell you something is wrong but it cannot tell you what caused it. CSAT works in much the same way. Naomi describes it as a gauge of whether a customer felt good or bad about an interaction, but that score cannot tell you whether the source of the frustration was the product, warehouse, delivery experience or something the support team actually did. In many cases, by the time the score appears, the customer has already been experiencing the problem for some time.

That is not an argument for abandoning the survey. Continuing to monitor customer satisfaction with survey-based feedback is still how you know the temperature has moved at all. The more useful question is what sits underneath the number.

The most valuable data is hiding inside the conversations

Historically, CX teams have tried to answer that question using contact reasons and ticket tags.

Those tags are useful. They can help teams understand broad categories of volume, plan capacity and spot obvious trends. But they are also dependent on a human deciding what a conversation was really about.

That gets messy quickly.

The same ticket might reasonably be described as a shipping issue, delivery problem or warehouse issue. A customer might raise two or three problems in a single conversation, while the agent chooses whichever tag seems most relevant in the moment. Across a large team and thousands of conversations, those small differences start to matter.

"It's hidden in the actual transcripts of the conversations with your customers."

At Freebird, Naomi's team exports its ticket data and uses AI to ask questions across the raw conversations themselves. Instead of being restricted to predefined categories, they can search for specific behaviours, complaints, language and patterns that they did not necessarily know to create a tag for in advance.

That changes the kind of questions CX can ask. Rather than simply reporting that "missing item" tickets increased this week, the team can investigate whether customers are repeatedly missing one particular item. Instead of knowing that cancellation contacts are rising, they can look at where sentiment changes during those conversations and what agents are saying immediately before it happens.

As Naomi explains, this produces "much richer insight and deeper analytics" and allows her team to find problems that traditional tagging either would not uncover or would uncover much later. Turning that raw signal into deep shopper insights the whole business can act on is the part that used to be impossible at any real scale.

CX can finally bring the rest of the business receipts

One example from Freebird shows why that difference matters.

A few support agents noticed customers mentioning that a travel case was missing from their orders. Under a traditional reporting system, those conversations would probably have been tagged as "missing item," added to the weekly numbers and monitored to see whether the category continued to increase.

Instead, Naomi asked AI to scan the ticket transcripts specifically for customers reporting a missing travel case.

It found more than 100 instances in three days.

"I found over 100 instances of customers missing a travel case in the last three days. And I'm like, 'Oh, yeah, that's a big problem.'"

The team then pulled the relevant ticket IDs so the findings could be audited and took the evidence to Operations, Inventory and Planning. Working with the 3PL, they discovered that an SKU had been edited incorrectly and that more than 1,000 orders were potentially affected. Freebird could then proactively contact those customers, tell them the missing travel case was on its way and offer a coupon rather than waiting for each person to discover the problem and open a ticket.

That is a very different outcome from simply reporting that "missing item" contacts went up.

It also changes the conversation CX can have with other departments. Previously, a CX leader might walk into a meeting saying, "We're hearing that customers are confused about this," or, "It feels like people don't like this feature."

As Naomi points out, leadership teams do not always act on feelings. They respond much more readily when CX can quantify the problem, provide customer examples and show how much it is costing the support team to resolve.

"Now with AI, we can have real receipts."

Naomi has already seen that shift internally. When her COO wanted to understand whether customers were becoming confused by a post-purchase upsell, she could search the previous 90 days of conversations for specific examples, request the relevant ticket IDs and customer quotes, and quickly build a report based on what customers were actually experiencing.

The result is that CX becomes less of a department reporting its own performance and more of a source of customer intelligence the wider company can query. It is the same operating principle Naomi described on the Commerce on Autopilot panel at Shopify NYC, where her team handled more than 115,000 tickets in a quarter without letting automation run ahead of the knowledge behind it.

Better analysis creates better coaching too

The opportunity is not limited to Product and Operations. Analysing the conversations underneath the scores also changes how CX teams manage their own agents.

Freebird has built an AI-assisted QA tool connected to its SOPs and internal documentation. AI takes a first pass at grading selected conversations before a human auditor reviews the result and provides feedback where necessary. Over time, that creates a much richer feedback loop than simply giving an agent a score or telling them their first response time is too high.

The important part is being able to identify what actually changed the customer's experience.

"We can actually look at the transcript and pinpoint the moment where the sentiment changes for the customer."

In one case, Naomi's team noticed a pattern in subscription cancellation conversations. Agents were moving too quickly into retention mode, immediately presenting benefits, changing frequencies and offering reasons to stay before properly understanding why the customer wanted to cancel.

The problem was not necessarily the retention offer itself. It was the way the conversation was unfolding. Customers could feel overwhelmed rather than listened to.

In another example, the team found that a specific line was contributing to poor experiences among already frustrated customers who had received an unexpected subscription order. Telling those customers they would also have to pay for the return label was often the moment the interaction deteriorated. Freebird adjusted the policy for those cases and began waiving the return label. Getting ahead of that category at all depends on being able to resolve billing issues and payment failures automatically before the customer has to write in angry.

A traditional dashboard might show a low CSAT score at the end of both interactions. Conversation analysis can show the team what happened immediately before it.

That creates something far more useful for an agent than "your CSAT needs to improve." It creates an actual coachable moment.

CX cannot fix a journey that broke upstream

There is another reason understanding root causes matters. It makes it much harder for the wider organisation to push responsibility for every bad customer outcome onto the support team.

Retention is a good example.

When a customer finally reaches CX saying they want to cancel, that decision may have been shaped by everything that happened before the ticket existed: the ad they saw, the product page, checkout, shipping, unboxing, onboarding, product education and their experience using the product.

By the time they arrive at support, the damage may already have been done.

"CX is not gonna act like a bodyguard at the door preventing people from leaving."

Naomi's team will still try to help. If a Freebird customer is struggling to get a good shave, an agent can ask about technique, share troubleshooting advice or offer another product. If there is a genuine education gap, there may still be an opportunity to save the relationship.

But if the customer clearly wants to leave, adding friction simply to protect a retention metric can make the experience worse.

The more important work is figuring out why they reached that point in the first place and taking that insight back upstream.

"There are so many things that happen before they get to us. So you can't expect CX to, like, save the day when everything else that happened before is broken."

That may ultimately be the biggest opportunity created by AI in CX. Not another automation percentage or another dashboard, but a much better way for AI-powered support teams to connect what customers are saying with the decisions being made throughout the rest of the business.

Start with the raw data and ask better questions

For CX leaders wondering where to begin, Naomi's 10% Fix is deliberately simple.

"Get the raw data and put it into a tool and start to ask questions that you care about, that you're curious about, that other departments might wanna know."

You do not need to abandon CSAT, response time or your existing reporting stack. Those numbers can still tell you whether the operation is healthy and where something might deserve attention.

The change is what you do next.

When the number moves, go into the conversations. Ask what customers are actually experiencing. Look for the repeated language that never made it into a ticket tag. Find the point where sentiment changes. Quantify the problem. Pull examples. Give Product, Operations and leadership something concrete enough to act on.

Your dashboard can tell you that the temperature has changed but the conversations can tell you where to start looking for the fire.

Watch or listen to the full episode:

Apple Podcasts

Spotify

YouTube