Now that you have reviewed the performance of AI Agent, you can improve it through coaching. Specifically, by identifying and filling gaps in knowledge (in Help Center articles, Guidance, and Actions), as well as providing feedback to improve the AI algorithm.
When reviewing a ticket handled by AI Agent, you can:
We recommend setting aside at least one hour per week to review AI Agent’s responses — or more if your team has the bandwidth. Coaching is especially impactful when you get started with AI Agent, when you’ll find the most opportunities to create additional knowledge content and provide feedback.
On top of reviewing AI Agent’s conversations and rating the responses, you can improve AI Agent’s performance by refining your knowledge sources (Help Center and Guidance).
Here are some tips on how to approach refining your AI Agent:
Your AI Agent setup is only effective if it matches the questions your unique shoppers are asking. To understand this, you have two great options:
In your Automate Overview, under the your Performance data, look for the section Recommendations. These are the most asked questions in the past 90 days.
You can click View All for even more recommendations.
Cross-check these questions against your Help Center and Guidance knowledge content. If a common question is not covered in your knowledge content, then creating content about that questions is sure to boost AI Agent’s automation performance.
Another way to see what your shoppers are contacting you about is to head to Statistics and, under Ticket Insights, select Intents.
Here, you can also see the reasons shoppers reach out to you:
One great advantage of this dashboard is that you can focus on certain time periods or channels.
For instance, you can apply a filter to only see the top intents among Email and Contact Form messages.
Similarly, you can apply a filter to see Intents data from November-January of the previous year, to understand how your Intents change during Black Friday—Cyber Monday to better prepare for the upcoming holidays.
Just like with your top questions from shoppers, you can cross-check this data with your current Guidance and Help Center content to see if there’s an Intent with a high volume that isn’t currently covered. Fill this gap in your knowledge content to automate those questions with AI Agent.
While the above tip gives you a high-level sense of what customers are asking, it’s also helpful to see real tickets. This gives you visibility into why AI Agent may not be able to resolve a ticket — even if you have Guidance or Help Center content on the ticket’s topic.
The AI Agent Handover view isolates all the tickets AI could not resolve on its own — either because it did not have adequate information, or because it falls within a Handover topic.
Spend an hour each week going through the tickets in this View. Within each ticket, look for the reason the AI Agent handed it over – likely because:
Let’s take a look at this ticket:
The customer wanted to update the address on a recent order. AI Agent correctly identified the customer’s order and was able to tell the customer that because the order is unfulfilled, an address update should be possible. However, after responding, AI Agent was handed over the request.
By looking at the righthand bar, in the Ticket Feedback tab, you can see the resources AI Agent used — in this case, 2 pieces of knowledge but no Action.
This ticket indicates you should set up the Update Shopify Order Shipping Address Action to fully resolve issues like this moving forward.
Let’s look at another example — this time, where the problem is missing information in a Guidance:
In this example, where the customers seemed to ask about upcoming discounts, AI Agent was able to correctly identify the topic but — as you can see in the right-hand Ticket Feedback section — it was not able to find any knowledge to use.
The agent was able to come in and let the customer know there are no current or upcoming discounts to share, and instead direct them to the brand’s Instagram page for future updates.
In this case, you could create a Guidance instructing AI Agent that when customers inquire about discounts when no active discounts are live, instead direct customers to follow the Instagram page.
And when you do offer discounts, you can edit this Guidance to empower AI Agent to share the correct discount when appropriate.
Earlier in this playbook, you set up Tags for AI Agent. Here’s a refresher:
You can easily set up unique Views for AI Agent.
For instance, let’s say your focus for the week is maximizing the number of WISMO tickets AI Agent can resolve.
Since your AI Agent is tagging all tickets “for questions about order status” with the tag “wismo,” you can use this tag — as well as the ai_answered tag that always gets added when AI Agent responds — for a View:
You can also choose to apply the “ai_handover” tag if you’d only like the View to show tickets AI Agent was not able to resolve and instead handed over to your team.
📺 Having trouble improving your AI Agent’s performance? Register for our next AI Agent webinar.
