Enabling Support Admins to increase their AI Models' accuracy

Helpshift is a B2B SaaS startup that builds Customer Support CRM software. It is used by leading companies like Microsoft, Square, Brex, Tencent Games etc. to provide support to their users.

Context

One of Helpshift's core AI capabilities is to automatically classify tickets into categories based on the user's query. This allows the company's Support Admins to set up workflows to assign tickets to the right Agents. To be able to classify tickets automatically, Support Admins train the AI Models by providing them with historical tickets data.

Once the AI Models are trained, an easy and effective way to keep improving their accuracy is for Support Agents to provide feedback on the tickets it classifies.

Problem Statement

Agents don't provide enough feedback for Helpshift to improve its automatic ticket classification. Furthermore, Admins are unable to do it as the feedback window gets closed when a ticket gets resolved. This restricts the Support Admins from improving the accuracy of their ticket classification.

Research

This classification feature has a lot of parts and affects different features in Helpshift's CRM.

Concept map of ticket classification feature

Once I got a good undestanding of the feature, I went through some customers' verbatim feedback and interviewed a few stakeholders to get a detailed idea of their problems. I synthesised these observations to identify the focus areas of this project.

Affinity diagram of all the observations

Objective

This has led us to the following objective –

Let Support Admins provide classification feedback for multiple tickets efficiently.

Prototyping

An initial tabular prototype listing a ticket and its corresponding label in each row. Admins can provide feedback by changing any ticket's label that was classified incorrectly.

Another prototype letting the admin focus on a single ticket at a time. This was later modified after some internal usability tests.

In addition to enabling Admins to provide feedback right from the dashboard, the below solutions were included to solve their needs

Extending the feedback window to let Admins and Agents provide feedback well after the ticket is closed.
Introducing a new metric 'Change in % of Issues classified' to indicate how well the model is improving over time. Additionally, to untie the accuracy metric from Agent's non-existent feedback, it will be computed based on the Admin's feedback.
Prioritising certain types of tickets for better improvements in the Model's performance
Enabling the Admin to provide feedback on a specific set of tickets
Reminding Administrators to provide feedback by sending monthly email reminders with the Model's current and historical performance
Enabling the Admin to efficiently do this task by providing keyboard shortcuts for all the feedback actions

Usability Testing

I conducted internal usability testing on the last prototype and identified a problem that most participants faced. Most of the them were puzzled when they were asked to undo an already given feedback. Since I also wanted to provide shortcut keys for the feedback actions, this could potentially become a problem for all users.

To solve this, I decided to show the list of tickets on the left side of the screen. This would help with its discoverability and also makes it faster for the Admins to go back to the previous ticket and undo or override the feedback.

Final Designs