Skye is a virtual support assistant on Carmax.com.
This case study follows my redesign of Skye and its transformation from a legacy support chatbot to an AI shopping agent that resulted in a 10% increase in messages sent and a 18% drop in live agent escalations.
Objective
The primary objective for Skye in early 2024 was to increase containment, or the number of chats that do not lead to a live agent escalation.
Baseline: 41% containment
Goal: 50% containment
To ensure that containments were due to satisfactory experiences, I worked closely with Analysts supplement this by measuring Customer Satisfaction.
Research
My Product Manager and I spent weeks analyzing Skye and interviewing people who recently used Skye (recruited through our C-SAT survey). I also recreated Skye’s conversational paths as flow charts to study them more easily.
Top insights in early 2024:
-
Skye was unhelpful - People struggled to get the help they needed and frequently had to escalate.
-
Skye felt bogged down - Rigid and unresponsive, Skye was constantly hindered by obstacles and unable to adapt.
-
AI is powerful but intimidating - While many are excited about AI’s potential, just as many remain skeptical.
Experience vision for Skye in 2025:
-
Empowering - More than just helping, Skye should empower customers to feel confident and succeed.
-
Dynamic - Skye should adapt seamlessly to every need and context, responding with fluid intelligence.
-
Humane - Intuitive, responsible, and human-centered, Skye should feel approachable and trustworthy.
Product Strategy
Approach: Pivot to AI
We knew we would not be able to achieve the experience we wanted without language models, which required re-platforming. After weeks of looking at different LLM providers, it was decided that we would build an in-house architecture. This meant a whole new redesign.
-
Re-platform Skye
-
Re-design Skye
-
Partner with an AI partner
Roadmap
We decided we would gradually build up Skye’s capabilities over time, hopefully buying us time as we searched for an AI provider. with my team to form an approach to gradually enhancement over time:
-
Escalation: Connect users to human support
-
General: Answer general knowledge questions
-
Precise: Answer questions grounded in data like inventory
-
Personal: Answer questions that require authentication
-
Action: Do things for the user; take actions on their behalf
Principles
A few principles that guided my design direction in the following months:
-
Skye should be helpful
-
Skye should be minimal and direct
-
Skye should be clear and guiding
Ideas
Idea wall
It’s difficult not to come up with a lot of ideas when you first join a product team, even before you’ve found focus. That’s why I find it useful to create a messy-desk space for unfiltered ideas. No effort goes into it. Just the shortest path from mind to whiteboard. Ideas may be there indefinitely, or harvested at the right time.
Components
There were basic elements that would need to be built , I sketched then mocked a set of components and shared with my front-end dev. This in combination with our existing design system, enabled them to get a head start building as the design evolved.
RAG Rollout
From this, a few ideas started to emerge:
-
Make escalation as easy as possible rather than containing
-
Reduce latency, unnecessary steps, and
-
Earn containment by being as helpful as possible using generative
-
Use a concise and informational
-
Be transparent and informative during transitional
-
Show users what is available to them
Future Explorations
I visualized concepts to get leaders and stakeholders excited. This included: Generative output, voice-based navigation, AI-assisted live agent chat, and others.
Design Iterations
Iteration One
Challenge: The first version of Skye would not be generative, have intent detection, or conversational pathways. The only capability available to us was semantic search, which turns a message into a search and returns a link with a matching answer.
Features of an iteration one:
-
A contextual header
-
A restart chat
-
FAQ-based suggested messages at the start of the
-
Visible readout of search
-
A link component containing quoted
-
A follow-up question that asked if the response was
-
If selected “no” then start
-
Visible readout of escalation
-
Chat continues after escalation ends
I presented this design internally and received this set of update requests:
-
No quick actions, only text
-
Don’t follow responses with a
-
Reduce likelihood of idle chats
Iteration Two
I quickly followed up with a new version:
-
Text input only
-
Made input instantly active on chat
-
More minimal way give feedback on
-
Chat time-out to reduce idle
-
Button to end-
-
Transitions and motion
I user tested it and had it internally critiqued.
What worked:
-
Intuitive, easy to use
-
Escalation showed relevant information
What didn't work:
- De-escalate button was confusing
Iteration Three
The third version was mainly about resolving details. It resulted in very positive user tests. This was the first version we brought to production.
-
Redesigned header with contextual menu
-
Simplified many design elements
-
Optimistic escalation prompt
-
Improved hierarchy, alignment, and consistency
I organized the design file to ensure it was easy for developers to inspect. The spec file was also wired up as a prototype, which helped communicate interaction design and flows.
Releases
With each release, we hit roadblocks, learned, and made breakthroughs. This section outlines how I worked closely with the developers to ensure quality in production.
Release One: Semantic Search
With a version in QA, we finally tested the fully functioning semantic search.
Immediately we noticed we weren’t getting great responses. We showed users and stakeholders, and the answers were almost always wrong. People liked the UI, but it didn’t matter. The whole team was in a bit of a panic.
In response, my team and I started to reintroduce intent detection (the way the old bot worked).
With our developers busy, I wanted to make sure we kept improving design in production, so I started contributing code to production to make sure this got done. I spent a couple of months fixing visual bugs and pushing PRs alongside my devs.
Release Two: Generative POC
To everyone’s surprise, our lead dev built a version of Skye than ran on GPT-4o mini and started sharing it around.
In a matter of days, we got the go-ahead from all stakeholders and an AI provider was chosen. The results spoke for themselves. Our entire development team shifted towards working on building an agentic AI architecture.
But Skye’s design needed a few updates to handle generative output:
-
Updated legal
-
Message actions
Final Design
