Turn support SOPs into a tested AI agent

This playbook was originally published on G2 AI blueprints.

What problem does this solve?

Ecommerce CX teams running AI agents on Shopify stores spend weeks configuring automations intent by intent. Policies keep changing (return windows, shipping cutoffs, peak-season promos), the configuration drifts out of sync, and off-script or multi-request tickets escalate to humans. Maintaining the automation turns into a second job, and BFCM exposes every gap at the worst possible moment.

How does it work?

1. Gather your current support SOPs, macros, return and shipping policies, plus a sample of a few hundred recent tickets from your helpdesk.
2. Connect Yuma to your Shopify store and helpdesk (Gorgias, Zendesk or similar) so the agent can read orders, tracking and customer history, and take backend actions like cancellations, refunds and address edits.
3. Ask Yuma to draft a Handbook from those SOPs and past tickets: one readable document covering brand voice, policies and a playbook per topic. Running several stores or brands? Give each brand its own voice section inside the same Handbook.
4. Connect Claude to Yuma through MCP and run the Handbook audit prompts to check the Handbook against your policy docs. Ask it to flag contradictions, missing edge cases and outdated rules, then propose edits. Review every edit before accepting it.
5. Write tests for each playbook using real tickets and the reply and action you expect. Run them, fix the Handbook wherever a test fails, and rerun until they pass.
6. Publish the Handbook. Nothing goes live until a human publishes it. Roll out on a subset of topics or ticket volume first and read transcripts daily during the first week.
7. Extend the same Handbook to outbound playbooks triggered by order events (shipping delays, delivery issues, abandoned checkouts over email and SMS) and to phone calls where Voice is enabled.
8. When a policy changes, edit the relevant Handbook section, rerun the tests and publish. The agent picks up the change on the next ticket with no reconfiguration.

What's the biggest win?

Maintaining automation becomes maintaining documentation instead of decision trees. Petlibro reaches 79% automation with Yuma, and Yuma is involved in 64% of Tediber's customer queries. One versioned Handbook governs inbound email and chat, outbound messages and phone, so a policy change reaches every channel on the next ticket, and per-playbook tests catch regressions before customers ever see them. Setup that used to take weeks of intent-by-intent configuration compresses into drafting, testing and approving a single document.

What's required to run this?

Run steps 1 to 6 on your own tickets with a 30-day free Yuma trial before committing to a rollout. Copy the audit and testing prompts into a Claude project or skill so the whole team can reuse them. State each policy once in the Handbook and reference it from playbooks, so a single edit never leaves a stale copy behind. Add a test for every new edge case before publishing the fix. Let Claude or ChatGPT read and propose Handbook changes over MCP, but keep human approval on every publish. Include order-status and tracking lookups in your tests, since most where-is-my-order tickets depend on live Shopify data. Before November, replay last year's BFCM peak-week tickets through your tests as a stress rehearsal.

What are the constraints?

Reply quality depends on Handbook quality: vague or contradictory policies produce vague answers. Backend actions require the right Shopify and helpdesk permissions. Tickets outside documented policy still route to human agents by design. Automation rates vary with catalog complexity, ticket mix and how thoroughly policies are documented. Voice and some outbound channels may depend on plan and rollout availability.

The prompts for this playbook

Use these with Claude connected to Yuma through the Yuma MCP. Replace everything in [brackets]. Claude proposes; a human reviews and publishes in Yuma.

Prompt 1: audit the Handbook against your policies

Read our current Yuma Handbook and compare it with these policy documents: [paste or attach return policy, shipping policy, warranty, promo terms, SOPs].

Return:
1. Contradictions: anywhere the Handbook says something different from the policy docs. Quote both sides.
2. Gaps: policies, edge cases or product categories the Handbook doesn't cover.
3. Outdated rules: dates, prices, cutoffs, carriers or promos that look expired.
4. Duplicates: rules stated in more than one place, which will drift apart over time. Suggest where each rule should live once.

For each issue, propose the exact Handbook edit. Do not apply anything.

Prompt 2: find what real customers ask that the Handbook doesn't answer

Search our Yuma tickets from the last [30] days. Find questions or requests that:
- the Handbook doesn't cover, or
- human agents answered differently from what the Handbook says.

Group them by topic, give a count per topic and 2 example ticket IDs each. Rank by volume. For the top 5, propose a Handbook addition.

Prompt 3: write tests for each playbook

For the [playbook name] playbook in our Handbook, write [10] tests using real tickets from Yuma where possible. For each test give:
- the customer message (from a real ticket, with personal data removed)
- the expected reply in our brand voice
- the expected action (refund, cancel, address edit, escalate, none)
- why this case matters (common, edge case, or high risk)

Include at least 2 tricky cases: one multi-request ticket and one that should escalate to a human.

Prompt 4: apply a policy change safely

Our policy is changing: [describe the change, e.g. "return window goes from 30 to 14 days for sale items from 1 November"].
1. List every Handbook section and playbook this affects.
2. Propose the exact edits, stating the new rule once and referencing it elsewhere.
3. Write 3 new tests that would fail under the old rule and pass under the new one.

Do not apply anything. I will review and publish.

Prompt 5: peak season rehearsal

Find our Yuma tickets from [last year's BFCM week, e.g. 24 November to 2 December]. Return the top 10 request types by volume, and for each one tell me whether the current Handbook has a playbook that handles it. Flag any request type with no playbook or no tests.

Rules to keep

• Claude proposes, a human publishes. Never let edits go live without review.
• Add a test for every new edge case before you publish the fix.
• Save these prompts in a Claude project or skill so anyone on the team can rerun them.

Tools in this playbook

• Yuma AI
• Claude
• Shopify
• Gorgias