Re-designing the Vehicle Ad Landing Page

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2Fkpd7ys4M2fftaxd8HZlQ5pjfBZj1%2Fskye-chat%2F98446aaf-5b1b-4b52-a8e9-3ce1bbeb0f4d.mov?alt=media&token=41b2e213-6084-4cc4-8c36-7d8fb2cba1fc","id":"98446aaf-5b1b-4b52-a8e9-3ce1bbeb0f4d","width":3456,"height":1944,"filename":"Final-Desktop-4K.mov","type":"video/quicktime","caption":"","border":true}]}
```

Skye is a virtual support assistant on Carmax.com.

This case study follows my redesign of Skye and its transformation from a legacy support chatbot to an AI shopping agent that resulted in a 10% increase in messages sent and a 18% drop in live agent escalations.

---

## Objective

The primary objective for Skye in early 2024 was to increase containment, or the number of chats that do not lead to a live agent escalation.

**Baseline**: 41% containment

**Goal**: 50% containment

To ensure that containments were due to satisfactory experiences, I worked closely with Analysts supplement this by measuring Customer Satisfaction.

---

## Research

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2Fkpd7ys4M2fftaxd8HZlQ5pjfBZj1%2Fskye-chat%2F82fd6906-15f7-4804-82b7-67efd5406c53.png?alt=media&token=87335ee4-d8e0-43a0-88de-5a3f0b18dbd6","id":"82fd6906-15f7-4804-82b7-67efd5406c53","width":3456,"height":2080,"filename":"Flows.png","type":"image/png","caption":"","border":false}]}
```

My Product Manager and I spent weeks analyzing Skye and interviewing people who recently used Skye (recruited through our C-SAT survey). I also recreated Skye’s conversational paths as flow charts to study them more easily.

Top insights in early 2024:

1. **Skye was unhelpful** - People struggled to get the help they needed and frequently had to escalate.

2. **Skye felt bogged down** - Rigid and unresponsive, Skye was constantly hindered by obstacles and unable to adapt.

3. **AI is powerful but intimidating** - While many are excited about AI’s potential, just as many remain skeptical.

Experience vision for Skye in 2025:

1. **Empowering** - More than just helping, Skye should empower customers to feel confident and succeed.

2. **Dynamic** - Skye should adapt seamlessly to every need and context, responding with fluid intelligence.

3. **Humane** - Intuitive, responsible, and human-centered, Skye should feel approachable and trustworthy.

---

## Product Strategy

### Approach: Pivot to AI

We knew we would not be able to achieve the experience we wanted without language models, which required re-platforming. After weeks of looking at different LLM providers, it was decided that we would build an in-house architecture. This meant a whole new redesign.

* Re-platform Skye

* Re-design Skye

* Partner with an AI partner

### Roadmap

We decided we would gradually build up Skye’s capabilities over time, hopefully buying us time as we searched for an AI provider. with my team to form an approach to gradually enhancement over time:

1. **Escalation**: Connect users to human support

2. **General**: Answer general knowledge questions

3. **Precise**: Answer questions grounded in data like inventory

4. **Personal**: Answer questions that require authentication

5. **Action**: Do things for the user; take actions on their behalf

### Principles

A few principles that guided my design direction in the following months:

1. Skye should be helpful

2. Skye should be minimal and direct

3. Skye should be clear and guiding

---

## Ideas

### Idea wall

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2Fkpd7ys4M2fftaxd8HZlQ5pjfBZj1%2Fskye-chat%2F135b2750-3fea-4421-b81c-490ec89224ee.png?alt=media&token=140dfa9a-bdcf-4749-ad8e-9bd09ac0ce3d","id":"135b2750-3fea-4421-b81c-490ec89224ee","width":10876,"height":10808,"filename":"Concepts.png","type":"image/png","caption":"Idea wall","border":true}]}
```

It’s difficult not to come up with a lot of ideas when you first join a product team, even before you’ve found focus. That’s why I find it useful to create a messy-desk space for unfiltered ideas. No effort goes into it. Just the shortest path from mind to whiteboard. Ideas may be there indefinitely, or harvested at the right time.

---

### Components

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2Fkpd7ys4M2fftaxd8HZlQ5pjfBZj1%2Fskye-chat%2Ffcf0f09a-6935-4351-99cc-62b0fd8a02e6.png?alt=media&token=293489f7-9e89-4014-bb8c-7f02fb8b6a82","id":"fcf0f09a-6935-4351-99cc-62b0fd8a02e6","width":3117,"height":2062,"filename":"Sketching-Design-System-Small.png","type":"image/png","caption":"Component sketches","border":true},{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2Fkpd7ys4M2fftaxd8HZlQ5pjfBZj1%2Fskye-chat%2Fbc7cc5a1-1b32-4a52-88e7-a63e9213bb2d.png?alt=media&token=974dfbd3-0b78-4ee5-bf5e-5552412e5196","id":"bc7cc5a1-1b32-4a52-88e7-a63e9213bb2d","width":3518,"height":1618,"filename":"In-chat-Components Small.png","type":"image/png","caption":"Link and image components","border":true}]}
```

There were basic elements that would need to be built , I sketched then mocked a set of components and shared with my front-end dev. This in combination with our existing design system, enabled them to get a head start building as the design evolved.

---

### **RAG Rollout**

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2Fkpd7ys4M2fftaxd8HZlQ5pjfBZj1%2Fskye-chat%2Fe8f25f13-1fdb-4e41-b3d8-29f6d6d87831.png?alt=media&token=0fba3eaf-e159-4976-971c-bcd791fd7ad4","id":"e8f25f13-1fdb-4e41-b3d8-29f6d6d87831","width":4672,"height":1847,"filename":"Skye Chat - Playground.png","type":"image/png","caption":"RAG-search mock","border":false}]}
```

From this, a few ideas started to emerge:

* Make escalation as easy as possible rather than containing

* Reduce latency, unnecessary steps, and

* Earn containment by being as helpful as possible using generative

* Use a concise and informational

* Be transparent and informative during transitional

* Show users what is available to them

---

### **Future Explorations**

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2Fkpd7ys4M2fftaxd8HZlQ5pjfBZj1%2Fskye-chat%2Fb26434a0-2e6b-49d1-8232-5b2844742374.png?alt=media&token=88ee9056-39ae-4339-bdc1-90ba44fcd8cc","id":"b26434a0-2e6b-49d1-8232-5b2844742374","width":1990,"height":1510,"filename":"Conversational Nav - 3:4.png","type":"image/png","caption":"","border":false},{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2Fkpd7ys4M2fftaxd8HZlQ5pjfBZj1%2Fskye-chat%2F5cdef30f-9db6-401c-a411-9efa76ea5a9f.png?alt=media&token=202b1e0b-7d83-49c6-895e-a3fa8f849bc8","id":"5cdef30f-9db6-401c-a411-9efa76ea5a9f","width":3120,"height":2054,"filename":"Gen-Snippet.png","type":"image/png","caption":"","border":false},{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2Fkpd7ys4M2fftaxd8HZlQ5pjfBZj1%2Fskye-chat%2F893f63e0-c24c-4344-a924-eaec3ac94454.png?alt=media&token=bc39b425-e184-4143-a174-31a456460dd5","id":"893f63e0-c24c-4344-a924-eaec3ac94454","width":1336,"height":992,"filename":"Personal Skye 2:7.png","type":"image/png","caption":"","border":false},{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2Fkpd7ys4M2fftaxd8HZlQ5pjfBZj1%2Fskye-chat%2Fec67573b-3702-431c-b101-10c039eb0413.png?alt=media&token=cd0d90ea-13fa-4d7d-b762-68567829205f","id":"ec67573b-3702-431c-b101-10c039eb0413","width":1336,"height":992,"filename":"Personal Skye 3:7.png","type":"image/png","caption":"","border":false},{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2Fkpd7ys4M2fftaxd8HZlQ5pjfBZj1%2Fskye-chat%2F28dfc97a-7fdc-4434-b895-98821b111068.png?alt=media&token=6a98afe2-4de9-40f4-b1e1-675222a3d837","id":"28dfc97a-7fdc-4434-b895-98821b111068","width":1458,"height":992,"filename":"Personal Skye 6:7.png","type":"image/png","caption":"","border":false}]}
```

I visualized concepts to get leaders and stakeholders excited. This included: Generative output, voice-based navigation, AI-assisted live agent chat, and others.

---

## **Design Iterations**

### **Iteration One**

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2Fkpd7ys4M2fftaxd8HZlQ5pjfBZj1%2Fskye-chat%2Fd3e1feb9-04ba-45e6-ae71-f1e573d9d0b8.png?alt=media&token=4535ce47-0d53-42d5-90f4-42fb66526cd7","id":"d3e1feb9-04ba-45e6-ae71-f1e573d9d0b8","width":10316,"height":5046,"filename":"Screen-Flow.png","type":"image/png","caption":"","border":false},{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2Fkpd7ys4M2fftaxd8HZlQ5pjfBZj1%2Fskye-chat%2F4240fdb3-f145-43e1-bd4b-b751368f3c6d.png?alt=media&token=89fb6b3c-b24d-47ce-a2fc-44e3fb698342","id":"4240fdb3-f145-43e1-bd4b-b751368f3c6d","width":4970,"height":6630,"filename":"Flows.png","type":"image/png","caption":"","border":false}]}
```

**Challenge:** The first version of Skye would not be generative, have intent detection, or conversational pathways. The only capability available to us was semantic search, which turns a message into a search and returns a link with a matching answer.

Features of an iteration one:

* A contextual header

* A restart chat

* FAQ-based suggested messages at the start of the

* Visible readout of search

* A link component containing quoted

* A follow-up question that asked if the response was

* If selected “no” then start

* Visible readout of escalation

* Chat continues after escalation ends

I presented this design internally and received this set of update requests:

* No quick actions, only text

* Don’t follow responses with a

* Reduce likelihood of idle chats

---

### **Iteration Two**

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2Fkpd7ys4M2fftaxd8HZlQ5pjfBZj1%2Fskye-chat%2Ff2d41ae8-1722-4e0a-9722-af0351088cd2.mov?alt=media&token=bb503f80-643a-4352-9781-234446b2d8ca","id":"f2d41ae8-1722-4e0a-9722-af0351088cd2","width":3456,"height":2080,"filename":"Screen Recording 2024-11-15 at 9.22.14 PM.mov","type":"video/quicktime","caption":"","border":false}]}
```

I quickly followed up with a new version:

* Text input only

* Made input instantly active on chat

* More minimal way give feedback on

* Chat time-out to reduce idle

* Button to end-

* Transitions and motion

I user tested it and had it internally critiqued.

**What worked**:

* Intuitive, easy to use

* Escalation showed relevant information

**What didn't work**:

* De-escalate button was confusing

---

### **Iteration Three**

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2Fkpd7ys4M2fftaxd8HZlQ5pjfBZj1%2Fskye-chat%2F98446aaf-5b1b-4b52-a8e9-3ce1bbeb0f4d.mov?alt=media&token=41b2e213-6084-4cc4-8c36-7d8fb2cba1fc","id":"98446aaf-5b1b-4b52-a8e9-3ce1bbeb0f4d","width":3456,"height":1944,"filename":"Final-Desktop-4K.mov","type":"video/quicktime","caption":"","border":true}]}
```

The third version was mainly about resolving details. It resulted in very positive user tests. This was the first version we brought to production.

* Redesigned header with contextual menu

* Simplified many design elements

* Optimistic escalation prompt

* Improved hierarchy, alignment, and consistency

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2Fkpd7ys4M2fftaxd8HZlQ5pjfBZj1%2Fskye-chat%2Fe1753649-db00-4b16-83bc-8c669ee5e000.png?alt=media&token=943aa076-1bf0-4182-9430-91196943a73e","id":"e1753649-db00-4b16-83bc-8c669ee5e000","width":3456,"height":2080,"filename":"Screenshot 2024-11-15 at 9.35.08 PM.png","type":"image/png","caption":"","border":false}]}
```

I organized the design file to ensure it was easy for developers to inspect. The spec file was also wired up as a prototype, which helped communicate interaction design and flows.

---

## **Releases**

With each release, we hit roadblocks, learned, and made breakthroughs. This section outlines how I worked closely with the developers to ensure quality in production.

### **Release One: Semantic Search**

With a version in QA, we finally tested the fully functioning semantic search.

Immediately we noticed we weren’t getting great responses. We showed users and stakeholders, and the answers were almost always wrong. People liked the UI, but it didn’t matter. The whole team was in a bit of a panic.

In response, my team and I started to reintroduce intent detection (the way the old bot worked).

With our developers busy, I wanted to make sure we kept improving design in production, so I started contributing code to production to make sure this got done. I spent a couple of months fixing visual bugs and pushing PRs alongside my devs.

---

### **Release Two: Generative POC**

To everyone’s surprise, our lead dev built a version of Skye than ran on GPT-4o mini and started sharing it around.

In a matter of days, we got the go-ahead from all stakeholders and an AI provider was chosen. The results spoke for themselves. Our entire development team shifted towards working on building an agentic AI architecture.

But Skye’s design needed a few updates to handle generative output:

* Updated legal 

* Message actions

---

## **Final Design**

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2Fkpd7ys4M2fftaxd8HZlQ5pjfBZj1%2Fskye-chat%2F95c1670b-4317-4bc3-a237-6deba4608d00.png?alt=media&token=e0f0b48f-e3de-4503-b3b1-56561aa784e3","id":"95c1670b-4317-4bc3-a237-6deba4608d00","width":2128,"height":3468,"filename":"Generative Skye 2.png","type":"image/png","caption":"","border":false}]}
```

Designing Skye

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2Fkpd7ys4M2fftaxd8HZlQ5pjfBZj1%2Ftapas%2F5aae1074-858f-42c1-b2f5-4d98aee10ae7.mov?alt=media&token=31724d26-411a-467d-a28f-cef825e7fe0d","id":"5aae1074-858f-42c1-b2f5-4d98aee10ae7","width":3456,"height":1994,"filename":"Tapas-Final-4K.mov","type":"video/quicktime","caption":"","border":false}]}
```

## **Recap**

After spending a few months trying to help users who click our Google ads to discover more of our inventory, we saw some modest gains in the amount of users who viewed other inventory. But it wasn't where we wanted to be, and our page was beginning to look very similar to another one: the Car Detail Page.

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2Fkpd7ys4M2fftaxd8HZlQ5pjfBZj1%2Ftapas%2F8e42a0ba-9213-42a9-8688-72f759519cf3.png?alt=media&token=10f0000f-0252-446e-965c-c466cac84649","id":"8e42a0ba-9213-42a9-8688-72f759519cf3","width":3456,"height":1994,"filename":"VALP-Comparison.png","type":"image/png","caption":"VALP","border":false},{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2Fkpd7ys4M2fftaxd8HZlQ5pjfBZj1%2Ftapas%2Ff8a907e1-5ab6-47d5-a59f-223a6708521d.png?alt=media&token=fdc92e6c-5af8-4eaa-9c06-bbfa9d544618","id":"f8a907e1-5ab6-47d5-a59f-223a6708521d","width":3456,"height":1994,"filename":"CDP-Comparison.png","type":"image/png","caption":"CDP","border":false}]}
```

The similarity between these two pages led to a desire to AB test them against one another, to see which one led to more sales leads (ship, reserve, etc).

I felt that the Car Details Page would definitely end up on top, as it had a sales CTA. If that happened, I assumed the landing page would disappear.

So in order to solve the problem we faced, we'd to find a way to solve it on another team's page.

## **Problem**

We needed to find a way to make CarMax's inventory more visible on other CarMax pages without changing anything about them.

And we needed to strike a balance between visibility of inventory without getting in the user's way as they explored the car.

## **The Idea**

In real life, when you're looking closely at one car, you don't lose sight of all the other cars around you in a dealer lot. You just look up and around.

This thought gave rise to a design idea: Persistent recommendations that you could easily access at any time. You would just "look up" so to speak.

To get the idea across, I took our existing recommendation strip and placed it right above the page. It showed the first car selected, indicating that you can switch between them like tabs.

I shared the idea with a product leader. Intrigued by the idea, they suggested I make a prototype and test it out.

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2Fkpd7ys4M2fftaxd8HZlQ5pjfBZj1%2Ftapas%2F0c6d95ce-1f28-45b7-87e1-81d043675295.png?alt=media&token=27438601-aebe-4a68-96f3-e86e16fa0fa8","id":"0c6d95ce-1f28-45b7-87e1-81d043675295","width":1920,"height":1853,"filename":"Tapas-Concept.png","type":"image/png","caption":"","border":false}]}
```

## **Prototype and Test**

I created three prototypes:

One and two were the existing Vehicle Ad Landing Page and the Car Details Page—the two pages being AB tested. User testing these two would help explain results that will eventually come from them.

The third prototype was a version of the Car Details Page with this new proposed design idea added. The component started in a minimized state, stayed sticky while scrolled, and expanded when you hover over it. When you clicked between cars, it would keep the current car highlighted to show you what's selected, like tabs.

All three prototypes started users from the same prototyped Google search results page. I also made browser tabs to let users switch between or even close tabs.

We were looking to answer two questions on each test:

1. How successful was each at letting users explore the details of one car?

2. How would users switch between and explore cars, and would users prefer the recommendation strips over Google in any of the tests?

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2Fkpd7ys4M2fftaxd8HZlQ5pjfBZj1%2Ftapas%2Fb13aaaec-e252-45bb-8b20-9fd6fd7a0cb5.mov?alt=media&token=ed23ea9d-32e7-4bd3-8b56-89038155b521","id":"b13aaaec-e252-45bb-8b20-9fd6fd7a0cb5","width":1920,"height":1564,"filename":"Shorter.mov","type":"video/quicktime","caption":"","border":false}]}
```

## **Test Results**

We only tested each prototype with 5 users each (15 total), but we were still astonished with the results. It was a strong start.

In the two control prototypes, most didn't even notice the recommendations (some did), and preferred to go back to Google to look at the other options. The landing page performed worse in letting users explore car details than the car details page, as expected, but did no better in exposing our inventory.

But the test with the recommendations strip above the page performed beyond our wildest expectations. One user even going so far as to excitedly say, ”I don't have to go all the way back to Google!”

Here's what worked:

* Tapas successfully got users to prefer it over Google

* Surprising amount of adoption and comparison engagement

* Tapas grabbed someones attention while they were on their way to click back, and they chose Tapas instead

* Someone happily said, ”I don't have to go all the way back to Google!”

* Blew the other 2 control prototypes out of the water.

There were zero failures, but some ideas for improving the experience:

* Preserve scroll position after clicking a car.

* Side by side comparisons.

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2Fkpd7ys4M2fftaxd8HZlQ5pjfBZj1%2Ftapas%2Fdd159280-a763-4bd8-825c-6857c2526c4a.mov?alt=media&token=fdc7cf75-ccbb-4b94-961f-a14711c32f7e","id":"dd159280-a763-4bd8-825c-6857c2526c4a","width":1920,"height":1108,"filename":"AllTheWayBack1080.mov","type":"video/quicktime","caption":"","border":false},{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2Fkpd7ys4M2fftaxd8HZlQ5pjfBZj1%2Ftapas%2F68a74ecf-25ea-47c7-8c3a-5be760343430.png?alt=media&token=e1c6e616-f604-488d-8901-302c70a23822","id":"68a74ecf-25ea-47c7-8c3a-5be760343430","width":1788,"height":2236,"filename":"Synthesis slice.png","type":"image/png","caption":"","border":false}]}
```

> "So honestly I found it very easy because again, like I said, these little tabs up at the front, up at the top were really nice… just to see the make and model I'm looking at and just be able to click on the next one or on a different one without having to go all the way back to Google."
>
> –LittleBookworm, UserTesting Participant

## **Mobile Prototypes**

With 70% of our users on mobile devices, I had to iterate and test it on mobile as well. This took slightly different strategy. I couldn't just responsively scale it down as the other was reliant on hover.

I first mocked up a simple prototype to feel out the behavior.

Then made three higher fidelity versions:

1. A sticky minimizing one with dynamic tab-like navigation, meaning you could switch between the same set of recommendations and compare them. This was more of a scaled down version of the first test.

2. Another sticky minimizing prototype, but this one statically generated a new set of recommendations on each click. I wanted to test this because that's how our recommendations currently work, and I knew it would be less development effort.

3. A non-sticky, fixed version more integrated into the page that also used the simpler static navigation.

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2Fkpd7ys4M2fftaxd8HZlQ5pjfBZj1%2Ftapas%2F8921bffa-2e13-4dc4-af04-51f8470c67f0.mov?alt=media&token=61721789-027f-4571-bf3b-4ab68a849de6","id":"8921bffa-2e13-4dc4-af04-51f8470c67f0","width":780,"height":1580,"filename":"Design-Iteration2-Prototype-ScrollBehavior-1.mov","type":"video/quicktime","caption":"","border":false},{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2Fkpd7ys4M2fftaxd8HZlQ5pjfBZj1%2Ftapas%2F94b386ae-7eef-40d4-b28b-8f2647db656f.mov?alt=media&token=c43a5da8-f969-49f7-93df-0942264dcfea","id":"94b386ae-7eef-40d4-b28b-8f2647db656f","width":780,"height":1318,"filename":"Design-Iteration2-Test1-Dynamic.mov","type":"video/quicktime","caption":"","border":false},{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2Fkpd7ys4M2fftaxd8HZlQ5pjfBZj1%2Ftapas%2F758daef6-6935-49ea-9245-bf2744f5e0cc.mov?alt=media&token=950ac6c3-fc78-4354-afb6-67204d4f613e","id":"758daef6-6935-49ea-9245-bf2744f5e0cc","width":780,"height":1318,"filename":"Design-Iteration2-Test2-Static.mov","type":"video/quicktime","caption":"","border":false},{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2Fkpd7ys4M2fftaxd8HZlQ5pjfBZj1%2Ftapas%2Fcace013b-2a16-4235-a647-2255e8a4cfaa.mov?alt=media&token=e0035730-edb3-4051-99ff-bc2fbd926421","id":"cace013b-2a16-4235-a647-2255e8a4cfaa","width":780,"height":1318,"filename":"Design-Iteration2-Test3-StaticSticky.mov","type":"video/quicktime","caption":"","border":false}]}
```

## **Mobile test results**

I wasn't entirely satisfied with any of the options. If anything, I preferred the fixed option lower on the page, but maybe with the dynamic navigation. It felt more like a part of the site, and the dynamic navigation made more engaging comparison shopping.

So during our tests, I was surprised to find that users seemed to resonate more with the sticky dynamic one. We observed that it:

* Was highly engaging

* Made comparison easy

* Minimization was appreciated

## **The Pivot**

I shared these test results with our product team at the same time that our AB test results were in. This was intentional.

It showed why CDP outperformed the VALP in terms of sales, as there's more car information to engage with, but also why both were underperforming when it came to engaging with other inventory. People had a strong tendency to return to Google.

But it also showed another way that seemed to engage users more with inventory without engagement with car details. Just what we wanted!

As a result of the AB test, the landing page team was (predictably) dissolved. The engineers would move to other teams.

But to my surprise, and thanks to this design and these tests, my PM and I would go on to continue building this idea out further.

The caveat was that we had to do as much on our own as possible over the next quarter, without engineers, then we would be assigned to a team who would build it for us. We would have to really rely on prototyping and testing ideas to remove as much uncertainty as possible. I would have to mock up impeccable design spec to make the handoff go smoothly.

## **Engineering Conversation**

We met with the engineering team who we would work with, and discussed the technical feasibility of what we wanted to accomplish.

That's when things started to get tough.

First of all, we learned that, because our site was static and not dynamic, it would be difficult to incorporate the kind of dynamic navigation I wanted.

Second, the way the recommender engine worked meant we were indeed going to need to probabilistically generate a new set of recommendations each time.

So the core functionality would have to be closer to that of a normal recommendation strip.

On top of that, we learned we'd only have access to 1 developer for 1 sprint (2 weeks). This meant simplifying the design as much as humanly possible.

I broke down the features to assess the viability of each part independently, and make sure we were setting aside anything we didn't need to focus on.

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2Fkpd7ys4M2fftaxd8HZlQ5pjfBZj1%2Ftapas%2F41f8d4d8-ae4c-4797-acfa-e2410c185a6e.png?alt=media&token=7a7d43fa-071b-41a0-b426-c370b2825891","id":"41f8d4d8-ae4c-4797-acfa-e2410c185a6e","width":3616,"height":4608,"filename":"CanVsShould.png","type":"image/png","caption":"","border":false}]}
```

## Legal and SEO

After meeting with legal, I learned that we had to make sure that, no matter what, the car's year, make, model, price and miles were visible before clicking a page.

Then I met with the Car Detail Page team, where we would be incorporating our component. During this meeting we learned of that our component would shift the page down after loading, and complicate a Google SEO metric called cumulative layout shift, or CLS.

So back to the drawing board, where I had to figure out a way to design something deliverable by 1 developer in 1 sprint, while minimizing layout shift and maintaining a minimum amount of information.

## **Final Iteration**

I made several iterations to reduce the vertical height, and landed on horizontal orientation.

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2Fkpd7ys4M2fftaxd8HZlQ5pjfBZj1%2Ftapas%2F1328e0cf-1871-409c-b7e5-7f89f3ce24ed.mov?alt=media&token=498e84e1-352e-4706-a789-82f05f2f2f08","id":"1328e0cf-1871-409c-b7e5-7f89f3ce24ed","width":3456,"height":1994,"filename":"120BPM.mov","type":"video/quicktime","caption":"","border":false}]}
```

## **Final User Tests**

I then tested this version in one more set of user tests, and this time the results weren't so rosy. Most users didn't notice it was there at all, and one who did said it felt like an ad.

## **Final design exploration**

I made one last effort to find a way to make it feel like less like an ad. But we didn't have the time to take the design in any new directions. So these explorations remained tabled.

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2Fkpd7ys4M2fftaxd8HZlQ5pjfBZj1%2Ftapas%2F0c567b1c-2817-41f0-8ed2-5240fd0d8008.png?alt=media&token=5ffc6ca3-a400-4f39-a1e5-632e4db503a2","id":"0c567b1c-2817-41f0-8ed2-5240fd0d8008","width":2248,"height":2093,"filename":"Ways 1.png","type":"image/png","caption":"","border":false},{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2Fkpd7ys4M2fftaxd8HZlQ5pjfBZj1%2Ftapas%2Fb1886340-b690-424f-a9a0-4a9757e636a5.png?alt=media&token=9490aea4-6c5b-4d7e-b42e-db41d51f8f7e","id":"b1886340-b690-424f-a9a0-4a9757e636a5","width":2737,"height":2093,"filename":"Ways 2.png","type":"image/png","caption":"","border":false},{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2Fkpd7ys4M2fftaxd8HZlQ5pjfBZj1%2Ftapas%2F1f441602-e450-4c58-a6e1-5de5ab789026.png?alt=media&token=a9adeb48-f537-495d-bf6f-13f51d392348","id":"1f441602-e450-4c58-a6e1-5de5ab789026","width":2757,"height":2093,"filename":"Ways 3.png","type":"image/png","caption":"","border":false}]}
```

## **Handoff file**

I put a lot of effort into crafting a highly detailed design spec to make development easy. The developer we worked with did an amazing job, and although we didn't ask them to incorporate the "hide on scroll down" functionality, they went above and beyond and did it anyway.

When it came time for QA, I had one of the most rapid and collaborative all-hands-on-deck week's I've ever experienced. We had two developers from two teams come together and fix issues that my PM and I found almost as soon as we found them.

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2Fkpd7ys4M2fftaxd8HZlQ5pjfBZj1%2Ftapas%2Fd5ae49cb-f6c1-491c-be14-48c28485eecb.png?alt=media&token=ed2b7df4-e058-4f74-91be-024581b86b81","id":"d5ae49cb-f6c1-491c-be14-48c28485eecb","width":2780,"height":2318,"filename":"Components.png","type":"image/png","caption":"","border":false},{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2Fkpd7ys4M2fftaxd8HZlQ5pjfBZj1%2Ftapas%2F63bd9fa1-31e5-49f0-b517-1018417ff4fa.png?alt=media&token=8c86c516-c759-4544-a0ed-f7f25ab502ed","id":"63bd9fa1-31e5-49f0-b517-1018417ff4fa","width":7640,"height":6256,"filename":"Desktop flows.png","type":"image/png","caption":"","border":false},{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2Fkpd7ys4M2fftaxd8HZlQ5pjfBZj1%2Ftapas%2F49e61a6a-245b-4dd8-961e-36fde42e80e6.png?alt=media&token=bd98926b-b878-48c4-992f-3a02113cab5d","id":"49e61a6a-245b-4dd8-961e-36fde42e80e6","width":6762,"height":6345,"filename":"Dev-Spec-Big.png","type":"image/png","caption":"","border":false},{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2Fkpd7ys4M2fftaxd8HZlQ5pjfBZj1%2Ftapas%2F84c9c961-c23a-48cb-9672-6437e84dc1bf.png?alt=media&token=0365a1ba-f7e6-4beb-a6e0-4d5d066c4be8","id":"84c9c961-c23a-48cb-9672-6437e84dc1bf","width":2295,"height":1252,"filename":"Overview.png","type":"image/png","caption":"","border":false}]}
```

## **Final Design**

```cv
gallery
{"layout":"hscroll","images":[{"src":"https://firebasestorage.googleapis.com/v0/b/maitake-project.appspot.com/o/pages%2Fkpd7ys4M2fftaxd8HZlQ5pjfBZj1%2Ftapas%2F1714347a-8977-4f16-bf11-c275809fe06c.mov?alt=media&token=7164d7a0-7616-4de0-8528-078f6eaa67f2","id":"1714347a-8977-4f16-bf11-c275809fe06c","width":3456,"height":1994,"filename":"Tapas-Final-4K.mov","type":"video/quicktime","caption":"","border":false}]}
```

```cv
figma
{"url":"https://www.figma.com/proto/hJgVkvm8Evj8mS3QYZTe7P/Tapas---Archived-3%2F20%2F23?page-id=9%3A8659&node-id=27-54900&node-type=frame&t=ptDVu8oC3TbYTBSF-0&scaling=scale-down&content-scaling=fixed&starting-point-node-id=27%3A54843"}
```

## **Impact**

So after one week of development and another in QA, it was shipped to production to a very small number of users in our test split.

While eagerly awaiting the results of the test results, I heard rumors that our analyst was getting excited. Apparently she said, "This is a million dollar idea."

The following week, I heard that our test didn't reach enough users to be considered statistically significant. There was so much further I wanted to take this design, but it our experiment was done, and it was time to hand it all over the next team.

## **What I learned**

I already knew this in a sense, but this was a validation that designer-dev collaboration is key.

I also learned a little bit about the politics: How the right product demo can take you really far, but broader organizational dynamics means some things are just outside of your control.

Re-imagining Recommendations

Pruduct Designer

Machine Learning

I spent some time on our internal Visual Design team, where I crafted a fully customizable system of sticker-like icons that highlighted various regions of a car. The purpose was to give designers a flexible system to show where blemishes were located on the car, but the result was applicable across many use-cases.

Car Part Sticker System

My goal was to imagine the future of EVs at CarMax. I was tasked with aligning with set company goals, reviewing available research, discovering problems and opportunities, and depicting this future to inspire leadership and create alignment.

* I partnered with designers across the organization to create a large, detailed, blueprint of the end-to-end customer journey, as well as identifying pain points, both for the customer, and for the business, at every turn.

* I used generative AI to rapidly create consistently-styled sketches to help tell the story of a present and future customer experiences.

* I created many design concepts that addressed problems and opportunities identified in this exercise, to help that future feel more real.

The Future of EVs

A one hour class to teach designers how to use LLMs with activities reminiscent of design school. The goal is to build exposure and familiarity among designers through a crash course of different AI tools and capabilities.

Prompt School

Problem: There are many types of damages a used car may have, yet we can only feasibly capture, process, and present to customers a small number of them.\
Research Question: What types and locations mattered most to customers?\
Methods:

* Use card-sorting to first understand customers' terminology and taxonomy of car damages.

* A survey asking customers to identify the severity and importance of various blemishes.

Vehicle Blemish Discovery

Code-named Tapas, a component that allowed users to explore CarMax inventory while also getting deep into car details. This was an experimental approach to engaging users coming from search ads with our inventory.

Heads-up Recommendations

I am currently experimenting with designed and testing a multi-agent LLM prompt architecture to bring text generation to our chatbot in a safe, efficient, customer-friendly, and on-brand way.

Prompt Architecture for Chatbots

During my first week on a new product team, I created a vision board for future of CarMax's chatbot experience, Skye. My goal was to imagine the best chat experience in the world. My task was to visually showcase as many unique and innovative ideas as I could come up with in one week. The images showcase what I came up with.

Concept Wall - Future of Chat

My goal was to test the immediate impact of brand photography across the CarMax site. I was tasked with testing different pre-vetted photos on various pages to find which had greatest emotional impact (sometimes referred to as a "visceral" experience).

My approach was to conduct 5-second tests with candidates recruited on UserTesting.com. Each tester gave feedback one version of a photo from each page, and gave feedback on each.

I used an LLM to aid in the synthesis of my research by generating keywords that summarize the feelings and thoughts expressed by all users for a given photo, as well as a by scoring the emotional valence of each. This was used as a loose proxy for experience alongside my own manual synthesis (affinity clustering, summarizing, etc).

Photos Emotional Impact Test

Led the successful redesign of CarMax's web chat and support bot.

Chatbot Redesign

I redesigned the landing page that people would see when they click on one of CarMax's car ads.

Jason Marsh

2024

2023