Today’s consumers expect exceptional brand experiences from initial engagement to checkout (and beyond). So, not only do they demand fast responses and self-help options — they also want continued customer support. But companies with small teams and limited office hours struggle to keep up. With chatbots, businesses can offer sales and customer support to customers around the clock.
However, offering 24–7 customer service this way doesn’t come without flaws and hiccups. Chatbots aren’t perfect and require ongoing reiteration and tuning. Knowing what to do and when requires monitoring the right chatbot analytics. So let’s review how chatbot analytics can enhance your customer experience and which metrics to track.
What’s chatbot analytics?
Chatbot analytics (or conversational AI analytics) consists of KPIs and metrics that guide chatbot improvements needed to enhance the customer experience. For example, customer satisfaction, fallback rate, and customer engagement.
Tracking the right metrics and KPIs also gives insights into your customers’ needs so you can better meet them.
What’s the main objective of chatbots?
Businesses use chatbots to engage prospective buyers and customers without the help of humans. It’s a method to automate customer services and sales while providing a great experience with your brand.
Here’s a look at the top benefits of using chatbots:
We also find businesses using a virtual assistant for a variety of other reasons:
The two types of chatbots (and which is better)
Now, there are two types of chatbots:
- Chatbots with rule-based programming (e.g., when x happens, do y)
- Chatbots with artificial intelligence (AI) and machine learning (ML) (e.g., learns mistakes and improves over time)
An AI chatbot using machine learning is best because it continues to evolve. How does it work? By “feeding” it thousands or even millions of samples to learn patterns and how to adapt. For example, an Intelligent Virtual Assistant uses AI and ML to find relevant information based on conversations. It’s able to “understand” text and spoken words using a technology called Natural Language Processing (NLP). This makes it more enhanced than a typical chatbot because it helps customers resolve issues faster, without needing live operators.
How efficient are chatbots?
As chatbots become smarter, more companies will adopt them. And this is a trend we’re already seeing. We can thank the pandemic for the rapid acceleration in AI adoption. One survey shows 25% of nearly 2,400 business leaders stated they increased AI adoption because of COVID.
And another 61% of high-performing companies reported increasing their AI investments. These are businesses across various sectors, including healthcare, retail, financial services, legal, and tech.
But is AI adoption worth it? Another report shows how businesses use AI technologies to personalize customer engagements:
And these companies are seeing a positive impact from AI investments, such as revenue acceleration. A 2021 report reveals 41% of businesses are getting more actionable insights, and 89% see increased revenue. It’s even helping 80% reduce their marketing costs.
Why is it important to analyze chatbot metrics?
Understanding the virtual assistant performance of your AI chatbots is critical to ensuring your customers are happy. If you’re using conversational AI to answer questions and offer support, and it’s not achieving this, then you need to know immediately to prevent losing customers.
Treating chatbots as a set-it-and-forget-it solution puts you at risk of hurting customer satisfaction and sales. One report shows 41% of consumers use chatbots to get information and help faster than using search. And another 5% said they preferred to engage with a chatbot vs. a human to reduce wait times.
This is why we find consumers using chatbots to do everything from booking appointments and hotel rooms to making purchases.
“I use chatbots in my company to give immediate answers to FAQs presented by regular customers and clients,” says Eliana Levine, HR Manager and Co-Founder of FindPeopleEasy. “To measure the performance of chatbots, I pay attention to the chatbot activity volume, bounce rate, retention rate, chatbot response volume, chatbot conversation length, and so on.
I use these metrics to improve product marketing campaigns and add more questions to the data. It also helps chatbots answer customers fast.”
By monitoring the right metrics, Levine grew sales by 28% and increased website traffic.
The issue with conversational AI you shouldn’t ignore
AI chatbots have improved exponentially over the years, but they’re still far from perfect. They require ongoing tweaking and A/B testing to enhance their ability to answer customer questions and provide help.
When your chatbot can’t do the basics, it can present challenges that hurt your brand’s reputation and revenue. Here’s a look at some consequences of AI chatbots that don’t meet customer expectations:
Don’t let this happen to you. Let’s review the top chatbot metrics you should watch for and analyze to prevent these issues.
What are the key chatbot metrics I should track?
There are dozens of conversational AI KPIs businesses should check based on their business goals. Some help to improve sales and enhance marketing, while others address customer support issues.
Let’s dive into the top metrics that show whether your chatbot is performing well or needs improvement.
- Customer satisfaction
How happy are customers with using your chatbot? One way to know is to look at your customer satisfaction rate.
There are several ways to collect this information, like asking users to complete a survey after their chat.
Max Hauer, CEO of Goflow, asks chatbot users to click thumbs up or thumbs down after an interaction.
“The user satisfaction rate provides an overall picture of our customers’ sentiment after using our chatbot. It’s a valuable metric for improving our customer service, sales, and marketing teams,” says Hauer.
Look at your average chat rating to determine if most customers are happy or dissatisfied. If the latter, dive deeper to determine the why.
- Completion rates
Reducing customer service costs is one of the top goals of implementing conversational AI into your pipeline. But if users aren’t getting what they need from the chatbot, human workers will need to step in.
Low completion (or deflection) rates reveal a problem with the bot’s answers or its issue with understanding user requests. And leads to customers closing the chat without completing the conversation.
Some call this metric Bounce Rate, which looks at how many people leave the chat soon after engaging (but without getting what they came for).
- Fallback FBR
Another way to identify when your chatbot is struggling to understand user requests is by looking at the fallback rate. When this is high, the chatbot is delivering irrelevant solutions to the user — or worse, none.
For instance, it’ll reply, “Sorry, I don’t understand,” showing the chatbot is confused by the request. Most times, the conversation will end early and a human will take over (defeating the chatbot’s purpose). It’s a critical metric to follow, particularly for rule-based chatbots.
Shivanshi Srivastava from PaydayLoansUK looks at a metric called Confusion Rate to identify this issue. According to her, it reveals the limited nature of the bot, which many find frustrating (especially if they thought they were speaking to a human).
When this occurs, she changes the responses to increase customer engagement.
- Goal completion rate
What’s your primary goal for adding a chatbot to your pipeline? Is it to convert visitors into leads or leads into sales? Or maybe it’s to help visitors find resources to answer their questions during the consideration stage?
Whatever it is, you need to track whether your virtual assistant is meeting the goal more often than not.
- Human takeover rate
How often do customers ask for human assistance when engaging with your chatbot? This shows how well (or poor) your virtual assistant performance is.
The human takeover rate measures the percentage of conversations that escalate to a human. Some instances will have the user asking for a human. Others will be the chatbot switching to a human assistant because it couldn’t understand the request.
Alex Wang, Co-Founder of Ember Fund, says to identify when this occurs in a conversation and in what category (sales, support, etc.), so you can optimize the chatbot accordingly.
- Total tickets created
If your CRM is integrated with your chatbot, then tracking the total tickets created is a must. Not only does this signal the number of people running into issues your chatbot can’t resolve. It also sheds light on the problems they have.
Support tickets detail the problem, so use them to categorize issues and find patterns. Then upgrade your chatbot to better address these issues in the future.
- Missed chats
Technical issues can occur — like a chatbot failing to accept a conversation started by a user. Or when it fails to connect the user with a live agent.
These instances are critical because they hurt the user experience and deter customers from engaging with the chatbot again.
- Unrecognized customer queries
Some chatbots have multiple-choice options (e.g., rules-based assistants). Others allow the user to type in their question (AI operated). When a user can frame questions on their own, they may be unrecognized or outside of the chatbot’s capabilities.
If there are too many unrecognized customer queries, you’ll need to either train the chatbot to answer them or use machine learning to understand similar phrases so that you can optimize the training cycles and extend the coverage by integrating more advanced NLU (Natural Language Understanding).
This is another reason to use AI — you can train an AI chatbot in semantics, so it can understand different variations of a question (vs. having to be an exact match to its programming). It makes chatbots more human-like and capable of understanding requests, no matter how it’s posed by the customer.
Map out customer conversations to cover various scenarios and problems they may run into. Then fill in the gaps using the data you collect from the unrecognized queries — semantics training will take care of the rest.
9. Average handling time
How quickly is your chatbot handling customer inquiries? The role of an AI virtual assistant is to speed up customer service. So it should be fast to answer FAQs and common requests it’s programmed to handle.
After integrating your chatbot, check handling times to see if it’s shorter than when humans were handling the messenger.
10. Number of bot sessions initiated
Are site visitors and customers using your chatbot? By looking at bot sessions initiated, you’ll see usage frequency. Compare this with the number of visitors your site receives to determine the percentage that use it.
Now, the goal isn’t to get everyone to use your chatbot. But if no one is using it, maybe it’s not visible enough, or people aren’t happy with the chatbot’s results.
Another similar metric is the “total number of users.” The goal is to see how many people are using it. And if you dig further, you can learn how many users are new vs. returning.
11. New vs. returning users
Why is this metric important? Because it shows how many people use the chatbot. And determines if the chatbot is successful — hence the high rate of returning users.
“I track this metric weekly, which reveals how users value and are happy with our services,” says James Parsons, Founder and CEO of Content Powered. “It improves customer experience and builds trust and loyalty, bringing us more new clients through word-of-mouth and referrals.”
12. Blockers
Where are chatbot users getting stuck during conversations? Is there a common drop-off point? Investigate to see if patterns exist. If so, find the blocker and remove it.
For example, maybe a resource link the chatbot shares is broken, leaving the customer without a solution. Update the link or provide an answer in the chatbot to prevent further issues.
13. Conversation logs
Seeing the chat logs between users and chatbots reveals lots of insights. Look for problem areas and what happens right before the customer bounces away or requests human intervention.
If you have voice calls, you can use a tool like Symbl.ai to transcribe the audio and generate common topics, themes, and questions. It’s an AI-powered conversational analytics platform that analyzes conversations to detect sentiment and understand intents that can be converted into voice or text virtual assistant flows.
Use real-time information from unstructured voice calls to find conversations where customers are angry, frustrated, or showing other negative emotions. Then review what’s happening to find ways to optimize the chatbot to perform better and faster.
14. Conversation duration
How long a user spends on a chatbot is telling. If it’s short, it may be doing a great job of providing answers. Or folks are dropping out because of its ineffectiveness.
“The conversation length between the chatbot and user should be short enough to inform, yet long enough to engage. Too long? Then the user hasn’t found what they’re looking for. Too short? Then the bot failed to engage the visitor, causing a potential customer to head to the competition,” says Chris Gadek, VP of Growth at AdQuick.
“But measuring session duration and conversion rate can determine whether you need to adjust your chatbot content. We found a session containing 5 steps lasting roughly 4 minutes boosted our lead conversions by 40%.”
Baruch Labunski, CEO at Rank Secure, also uses conversation duration as a metric.
“We’ve used this metric in real-life situations to address areas where the chatbot couldn’t resolve an issue or help with a problem. This results in the customer leaving the chat quickly. Looking at the metrics leads us to the transcripts, specifically identifying the problem areas.”
15. Revenue generated
Is growing sales your top priority? Then track revenue generation. Review how many users purchase through the chat or a live agent (via the chat or phone).
It doesn’t matter who closed the deal — your live agents and the chatbot are working together to complete the same goal.
16. Lead generation
Or maybe you’re not heavily focused on sales and want your chatbot to drive more leads. In this case, your chatbot’s goal is to get users to fill out a form, download content, or sign up for a newsletter.
Track how many visitors convert into leads via the chatbot to see how well it’s performing.
17. Sentiment
Are the folks using your chatbot happy or angry during the conversation? Some may turn to your chatbot for support and will start out upset and then become satisfied after receiving help.
This is a great sign your chatbot is doing well. But if the opposite is true, it’s time to step in and adjust the chatbot.
Optimize your chatbot for better performance
Using conversational AI to enhance customer service and sales workflows isn’t the future — it’s the now. If you’re already using chatbots or plan to soon, then it’s critical to monitor the right metrics.
And it’s not something you do once or twice. Keeping a close eye on your KPIs (weekly, bi-weekly, or monthly) prevents your chatbot from causing damage to the customer experience.
So do what you can to learn what works and what doesn’t, including:
- Giving post-chat surveys
- Creating an escalation path
- Writing dialog that feels human
- Adding videos, articles, and other content to improve engagement
- Updating your chatbot with new information and resources
- Using NLP to improve sentiment detection and understanding
- Use other unstructured conversation data like audio calls, transcripts to scale the intents and scope.
“To improve my AI pipeline, I collect the transcripts of the various questions and responses that the tool worked on. I identify common question clusters from the transcripts and refine the pipeline to address them better. I also check on variations in questions and adjust the AI to these variations,” says Gisera Matanda, Co-Fonder of WeLoans.
“In the beginning, the AI tools will mostly not be as good as you would wish. However, the tool becomes better with time, analysis, and improvement of the various metrics. I, therefore, advise that, in the refinement of a conversational AI pipeline, let the user and the tool teach you what needs to be improved.”
If you’d like to process unstructured chatbot conversations and turn them into actionable insights, then explore Symbl.ai today.
