Commerce on Autopilot: Shopify x Yuma NYC panel recap

What 100 operators, 115,000 tickets, and a new benchmark taught us at Commerce on Autopilot.

You can automate most of your support volume without hurting CSAT. That was the thesis of our NYC panel, and both the data and the operators in the room backed it up.

On July 14 we co-hosted Commerce on Autopilot with Shopify at their SoHo space, as part of our AI+CX Club. Around 100 CX and ops leaders filled the room over wine and cheese, out of roughly 400 who registered, and Shopify's official numbers are still landing. The crowd skewed toward smaller merchants this time, but the engagement was the real signal. The topic clearly landed, and a handful of conversations are worth following up on.

The panel ranged wider than support, into Shopify's bigger AI push and how bots are starting to read your store, and we'll get to that. But the heart of the night was customer experience, and one question ran underneath all of it: how far can you automate before quality slips? The answer, it turns out, is a lot further than most teams assume.

The headline numbers

Three findings from The State of CX Automation in Ecommerce 2026 framed the night:

• Top merchants sustain up to 87% automation, and that ceiling is universal. It isn't about how big you are or which helpdesk you run.
• Automating more didn't cost brands their CSAT. The ones automating the most kept customers just as happy.
• Across 100+ merchants, the report modeled about 86,000 agent-hours saved. The gap between good and great is setup and consistency, not smarter AI.

The trade-off everyone fears doesn't exist

Start with the myth the whole room came in carrying: automate aggressively and customers get frustrated. The data said the opposite, and Guillaume Luccisano, our CEO, put it bluntly.

The one that genuinely surprised me was that automating more didn't cost anyone their CSAT. Everyone assumes there's a trade-off, automate aggressively and customers get frustrated. The data showed the opposite: the brands automating the most were keeping customers just as happy as the ones barely automating. The trade-off everyone's afraid of basically doesn't exist.

Guillaume Luccisano, CEO, Yuma

Why does it hold? Because speed matters to customers more than a human does. A fast, correct answer beats a slow one from a person, almost every time. The second surprise in the data was how much quiet work this represents.

We're talking 86,000 agent-hours of work that just quietly disappears. When you see it in aggregate, you realize how much of a support team's day is spent on tickets that never needed a human in the first place.

Guillaume Luccisano

What 115,000 tickets a quarter actually feels like

Then Naomi Oriol, Head of CX at Freebird, grounded it in reality. Her team handled more than 115,000 tickets last quarter, automated the bulk of it, and held CSAT at 4.42. That's roughly 1,200 to 1,400 tickets a day, and it doesn't arrive evenly, a product drop or a shipping delay and it doubles overnight.

The before picture will be familiar to anyone in the room: the whole team living in the queue, triaging from the moment they logged on, a backlog that never quite hit zero. It felt like firefighting more than customer experience. What changed after automation wasn't the volume, it was that most of it stopped reaching a person.

The team question came up fast, and Naomi was clear: they didn't cut headcount, they repointed it. First response times went from hours to minutes, and her agents stopped copy-pasting tracking links and started handling the genuinely hard cases, the high-emotion and high-value conversations, plus maintaining the knowledge base and reviewing what the AI was doing. Her best people became CX operators instead of ticket-clearers.

The most useful moment was the honest one. Freebird got greedy early and switched automation on for a category before the knowledge behind it was deep enough.

The AI was confident and wrong on some edge cases, which is worse than being slow. The lesson was blunt: the AI is only as good as the knowledge and the guardrails you give it. Automating fast is easy. Automating well is a knowledge problem, not an AI problem.

Naomi Oriol, Head of CX, Freebird

So how do you hold a 4.42 through that kind of volume? Naomi's answer was mostly unglamorous. Treat the knowledge base as a living product, not a one-time setup. Review AI outputs every week, not just when something goes wrong. Keep humans firmly in the loop on the edge cases and anything high-stakes. And watch CSAT per workflow, not just the top-line number, so a dip shows up in one ticket type before it drags the whole score down.

View Gallery

Your richest data is sitting in the support queue

The point Guillaume kept pushing, and Naomi doubled down on, is that your support tickets are one of the richest piles of data your brand owns, and most of it goes unused. Every question, every complaint, every "will this fit" is a signal about your product, your page, and your roadmap.

AI is finally making that pile usable. Tools like Ask Yuma turn it into real insight instead of letting it sit, which closes a loop that rarely closes on its own, insight comes into CX every day and almost never makes it back to the product or the roadmap.

The conversation went wider: your store is now an AI endpoint

Alongside the support story, the panel got into Shopify's bigger AI push, and the line that stuck was people don't read anymore, bots do. When an agent or an LLM looks at a product page, it reads the structured fields, the title, the description, the variant options, the metafields, which live on the product object and syndicate out to catalogs like Google Merchant Center. If your best information only lives in an image, it's invisible to the systems now doing the discovering.

There's real money in getting it right. One large merchant ran an A/B test on Yuma's Sales AI across 25 million sessions and saw roughly a 17% lift in add-to-cart and around 12% on conversion. At that sample size it's signal, not noise, a real store and a real test, not a demo.

What the room pushed back on

The Q&A is where the honesty came out. Disclosure came up again and again, and the consensus was to tell customers up front they're talking to AI. It's better CX, EU legislation is heading that way, and the frustration almost always comes from finding out late, not from the AI itself.

A few smaller signals were telling too. Several merchants have started naming their AI agents and swear the small touch changes the interaction. Someone reminded the room that around 80% of support isn't chat at all, it's email, SMS, and other channels, so the website widget is only part of the surface. And one sharp question landed on personalization: it depends entirely on the data quality of the customer record, and most merchants admitted they aren't auditing what their support team can actually see at the moment of the conversation.

What to do Monday morning

Naomi's advice for anyone about to scale volume was the one she wished she'd taken sooner: build the loop from support back into your knowledge base and your product data early. Tag your contact reasons properly, feed what you learn into your docs, and start narrow, get the knowledge deep, then widen the automation. Doing it the other way around is what bites you.

The complementary move, from the discovery side, is to read your own product descriptions and make sure they answer the questions actually coming into support, then audit them periodically. Same insight, two directions: your support queue already knows what your customers and their bots are asking.

The through-line

Here's what tied the night together. Automate the volume, and you don't lose CSAT, you gain it back in speed and in the hours your team can finally spend on the conversations that differentiate you. Feed what support learns back into your product and your roadmap. And treat your support documentation as the actual product, because that's what quality runs on.

That's the thesis behind The State of CX Automation in Ecommerce 2026, built on real merchant data, no survey, no vibes. If you want the full picture behind the panel, including how the top brands automate without touching their CSAT, you can read the report there.

Thanks to Shopify for hosting, to Anya Kelly for running the panel and keeping it sharp, and to Amanda Sannella and Naomi Oriol for the conversation. Great crowd, great questions. More of these coming.