Keep the call visible
Anchor the assessment to the conversation instead of separating review from evidence.
Delivered case study · Conversation assessments
Conversation
assessments
Exploring how customer-centre supervisors could move from manually reviewing a small sample of calls to finding the conversations that need attention.
At a glancePlatform scale
and review coverage
and review coverage
180countries with deployments
+10mdevelopers using the platform
+320kcustomers served by the platform
<5%of calls reviewed in an average week
1 by 1conversation assessment workflow
01
Situation
Supervisors are responsible for conversation quality, compliance and coaching.
Reviewing customer conversations is part of the supervisor and customer-centre manager role. They check whether customer experience professionals follow the expected script, meet regulatory and internal requirements, and receive support or training when needed.
The workflow supports structured assessments, but every review still requires someone to open the call, listen manually or check the transcription and record the outcome.
02
Problem statement
The process made quality review and improvement difficult to scale.
Because conversations were assessed one at a time, supervisors could review less than 5% of calls in an average week. This limited their view of the wider operation and made issues difficult to track and address through ongoing monitoring.
- Missed compliance issuesRegulatory risk and financial loss could go undetected.
- Limited monitoring coverageThe small sample made recurring issues difficult to track and address.
- Coaching gapsSupport depended on which conversations happened to be selected.
- Hidden patternsTrends across customer experience professionals, teams and conversation types were hard to see.
- Manual classificationTagging added time to an already demanding role.
- Missed commercial opportunitiesRelevant cross-sell or upsell moments could remain unseen.
03
Role & team
Discovery and structure preparation within a wider initiative.
I worked with the same core team as the wider customer-centre experience, focusing this phase on discovery, framing and structural foundations.
Product designer (me)Product manager
Engineering managerDeveloper / UI engineer
ArchitectContent designerContent writer
Scope note: later development expanded to external integrations and moved to another team.
04
Key design decisions01 02 03 04
Make quality visible while keeping review focused.
Make criteria reusable
Use consistent language operators and question structures across products.
Show evidence beside the call
Help reviewers connect each judgement to a specific moment in the conversation.
Design for coaching and scale
Support live coaching, historical review and a model that can grow.
05
Current scenario
How it used to work.
A supervisor searched historical reporting for one conversation, opened the call, selected a questionnaire and rated each question. Comments could be added before the assessment was saved.
Historical reporting→
Search for a conversation→
Open conversation→
Select questionnaire
AssessRate each question→Complete assessment
CommentAdd comment (optional)→Save comments
06
UI structure
Keep the conversation visible while the assessment stays focused.
The screen kept the call context and assessment task together. Select a view to explore how the structure supported the review.
Page structure
A focused review surface.
The timeline, conversation context and questionnaire stay together so the supervisor can assess the call without losing orientation.
Scalability
Extend the review pattern across products.
The same assessment structure can support historical review, live coaching and product-specific integrations. Coverage can move from less than 5% of calls reviewed manually to up to 100%* through connected criteria and automation.
*Coverage is configurable by an administrator or developer, depending on the product setup. Research indicated that admins may set specific queues to 100% coverage and others to 0%, based on contract scope, legal requirements and cost.
Channels
Adapt the conversation context.
Voice, chat and digital channels can use the same review structure.
Teams
Reuse the language.
Supervisors, managers and quality teams share a consistent pattern.
Integrations
Keep the model extensible.
External providers can supply signals without changing the assessment frame.
From
→Small, manual sample.
Open one call → listen → tag manually → rate each question.
To
Connected review at scale.
Keep context visible → apply connected criteria → record structured evidence → coach with context.
Conversation context
Keep the call visible.
Caller, queue and timeline anchor the supervisor in the selected conversation.
Assessment frame
Make the criteria explicit.
The questionnaire selector sets the script or criteria being applied.
Rating and coaching
Record consistent evidence.
Shared rating controls support quick review, with comments available when context is needed.
From
→Open one call → listen → tag manually → rate each question.
To
Keep context visible → apply a questionnaire → record structured evidence → coach with context.
Questionnaires
Move from manual setup to connected criteria.
Previously, questionnaires were uploaded manually, one by one or in batches. The new direction supports prompting or API-based access to existing questionnaires, depending on the product.
Questions
New experience
Prompt for a language operator or connect an existing one through an API, depending on the product.
PromptAdd language operatorsSelect reusable signals to build this questionnaire.
Enablement
Add language operators
Select reusable language operators to build the criteria for this assessment.
Operator nameDescriptionParametersContext capabilities
Script AdherenceCustomEvaluates whether the customer experience professional follows the required support script.maxScore—
SentimentService nameEvaluates the sentiment of a conversation.——
SummaryService nameCreates a summary of the conversation.——
Upsell OpportunityCustomIdentifies a relevant cross-sell or upsell opportunity.——
Edit view
Select items for display.
To prevent scrolling, select up to 6 items
← Back to transcripts
Conversation details
Analysed on Aug 7, 2024 at 17:14 BST Voice
✦Participant oneAssistant
AMParticipant twoAlex Morgan
ChannelVoice
SourceAnother company product
More
Find keywords and phrases ⌕
Assistant
00:00:00IntroductionThank you for calling the enrolment centre. I’m a virtual assistant. How can I help today?
Alex Morgan
00:01:32IntroductionI’m looking for help with my account.
Assistant
00:02:00I’m happy to help with that. I’ll ask a few questions first.
Alex Morgan
00:03:10PlanI’d like to understand the available options.
Cross-product variations
Support live coaching as well as historical review.
Supervisors can observe an active conversation while a customer experience professional is working, or return to historical reporting for a completed assessment. The same signals and user context carry across both views.
Channels
Adapt the conversation context.
Voice, chat and digital channels can use the same review structure.
Teams
Reuse the language.
Supervisors, managers and quality teams share a consistent pattern.
Integrations
Keep the model extensible.
External providers can supply signals without changing the assessment frame.
+44 20 7946 0958Alex MorganChristina Smith07
Concept explorations & prototype validation
Make the change testable before it becomes expensive.
I used lightweight concepts and working prototypes to explore how questionnaire setup, question reuse and conversation evidence could fit together. Each iteration focused on a decision the team needed to make before moving into development.
Concept exploration
Keep the review anchored.
Tested how the conversation, questionnaire and assessment actions could share one focused surface.
What to test → orientation while reviewingConcept exploration
Make criteria reusable.
Explored a prompt or API-led path for adding language operators and assembling product-specific questionnaires.
What to test → setup effort and flexibilityPrototype validation
Show evidence beside the call.
Rendered operator results next to the transcript so a supervisor could connect a signal with the conversation context.
What to test → confidence in the evidencePrototype validation
Protect question history.
Validated the prototype against the existing questionnaire model. Titles could change without losing assessment history, but changing a question’s meaning could make reports difficult to interpret. Teams could duplicate a question, then rename and edit the copy without changing other questionnaires or historical analysis.
What to test → safe evolution without changing historical reports08
Design opportunity
Move from isolated assessments to scalable quality insight.
How might we help supervisors understand which conversations need attention, identify patterns across a larger sample, and focus human review where it has the greatest impact?
From
→Open one call → choose a questionnaire → listen → rate every question → add comments.
To explore
Surface meaningful signals → prioritise review outliers → understand patterns → identify root causes and coach with context.
09
Next questions
The next draft needs the solution story.
This first version establishes the current workflow and the scale problem. The next iteration will add the design direction, research evidence, collaboration model and outcome once those details are confirmed.
- What signals or classifications should be automated or suggested?
- Prioritisation was explored elsewhere.A separate initiative addressed how supervisors could prioritise conversations for review. Details are omitted here to protect privacy.
- What views would help teams compare patterns across customer experience professionals or queues?
- What changed after the new experience was introduced?
10
Takeaway
Make quality visible before it becomes a problem.
The opportunity is to move from a small, manually selected sample towards structured signals that help teams see patterns, focus review and coach with context.
Surface meaningful signals→Prioritise review outliers→Understand patterns→Coach with context