Recap
After spending a few months trying to help users who click our Google ads to discover more of our inventory, we saw some modest gains in the amount of users who viewed other inventory. But it wasn't where we wanted to be, and our page was beginning to look very similar to another one: the Car Detail Page.
The similarity between these two pages led to a desire to AB test them against one another, to see which one led to more sales leads (ship, reserve, etc).
I felt that the Car Details Page would definitely end up on top, as it had a sales CTA. If that happened, I assumed the landing page would disappear.
So in order to solve the problem we faced, we'd to find a way to solve it on another team's page.
Problem
We needed to find a way to make CarMax's inventory more visible on other CarMax pages without changing anything about them.
And we needed to strike a balance between visibility of inventory without getting in the user's way as they explored the car.
The Idea
In real life, when you're looking closely at one car, you don't lose sight of all the other cars around you in a dealer lot. You just look up and around.
This thought gave rise to a design idea: Persistent recommendations that you could easily access at any time. You would just "look up" so to speak.
To get the idea across, I took our existing recommendation strip and placed it right above the page. It showed the first car selected, indicating that you can switch between them like tabs.
I shared the idea with a product leader. Intrigued by the idea, they suggested I make a prototype and test it out.
Prototype and Test
I created three prototypes:
One and two were the existing Vehicle Ad Landing Page and the Car Details Page—the two pages being AB tested. User testing these two would help explain results that will eventually come from them.
The third prototype was a version of the Car Details Page with this new proposed design idea added. The component started in a minimized state, stayed sticky while scrolled, and expanded when you hover over it. When you clicked between cars, it would keep the current car highlighted to show you what's selected, like tabs.
All three prototypes started users from the same prototyped Google search results page. I also made browser tabs to let users switch between or even close tabs.
We were looking to answer two questions on each test:
-
How successful was each at letting users explore the details of one car?
-
How would users switch between and explore cars, and would users prefer the recommendation strips over Google in any of the tests?
Test Results
We only tested each prototype with 5 users each (15 total), but we were still astonished with the results. It was a strong start.
In the two control prototypes, most didn't even notice the recommendations (some did), and preferred to go back to Google to look at the other options. The landing page performed worse in letting users explore car details than the car details page, as expected, but did no better in exposing our inventory.
But the test with the recommendations strip above the page performed beyond our wildest expectations. One user even going so far as to excitedly say, ”I don't have to go all the way back to Google!”
Here's what worked:
-
Tapas successfully got users to prefer it over Google
-
Surprising amount of adoption and comparison engagement
-
Tapas grabbed someones attention while they were on their way to click back, and they chose Tapas instead
-
Someone happily said, ”I don't have to go all the way back to Google!”
-
Blew the other 2 control prototypes out of the water.
There were zero failures, but some ideas for improving the experience:
-
Preserve scroll position after clicking a car.
-
Side by side comparisons.
"So honestly I found it very easy because again, like I said, these little tabs up at the front, up at the top were really nice… just to see the make and model I'm looking at and just be able to click on the next one or on a different one without having to go all the way back to Google."
–LittleBookworm, UserTesting Participant
Mobile Prototypes
With 70% of our users on mobile devices, I had to iterate and test it on mobile as well. This took slightly different strategy. I couldn't just responsively scale it down as the other was reliant on hover.
I first mocked up a simple prototype to feel out the behavior.
Then made three higher fidelity versions:
-
A sticky minimizing one with dynamic tab-like navigation, meaning you could switch between the same set of recommendations and compare them. This was more of a scaled down version of the first test.
-
Another sticky minimizing prototype, but this one statically generated a new set of recommendations on each click. I wanted to test this because that's how our recommendations currently work, and I knew it would be less development effort.
-
A non-sticky, fixed version more integrated into the page that also used the simpler static navigation.
Mobile test results
I wasn't entirely satisfied with any of the options. If anything, I preferred the fixed option lower on the page, but maybe with the dynamic navigation. It felt more like a part of the site, and the dynamic navigation made more engaging comparison shopping.
So during our tests, I was surprised to find that users seemed to resonate more with the sticky dynamic one. We observed that it:
-
Was highly engaging
-
Made comparison easy
-
Minimization was appreciated
The Pivot
I shared these test results with our product team at the same time that our AB test results were in. This was intentional.
It showed why CDP outperformed the VALP in terms of sales, as there's more car information to engage with, but also why both were underperforming when it came to engaging with other inventory. People had a strong tendency to return to Google.
But it also showed another way that seemed to engage users more with inventory without engagement with car details. Just what we wanted!
As a result of the AB test, the landing page team was (predictably) dissolved. The engineers would move to other teams.
But to my surprise, and thanks to this design and these tests, my PM and I would go on to continue building this idea out further.
The caveat was that we had to do as much on our own as possible over the next quarter, without engineers, then we would be assigned to a team who would build it for us. We would have to really rely on prototyping and testing ideas to remove as much uncertainty as possible. I would have to mock up impeccable design spec to make the handoff go smoothly.
Engineering Conversation
We met with the engineering team who we would work with, and discussed the technical feasibility of what we wanted to accomplish.
That's when things started to get tough.
First of all, we learned that, because our site was static and not dynamic, it would be difficult to incorporate the kind of dynamic navigation I wanted.
Second, the way the recommender engine worked meant we were indeed going to need to probabilistically generate a new set of recommendations each time.
So the core functionality would have to be closer to that of a normal recommendation strip.
On top of that, we learned we'd only have access to 1 developer for 1 sprint (2 weeks). This meant simplifying the design as much as humanly possible.
I broke down the features to assess the viability of each part independently, and make sure we were setting aside anything we didn't need to focus on.
Legal and SEO
After meeting with legal, I learned that we had to make sure that, no matter what, the car's year, make, model, price and miles were visible before clicking a page.
Then I met with the Car Detail Page team, where we would be incorporating our component. During this meeting we learned of that our component would shift the page down after loading, and complicate a Google SEO metric called cumulative layout shift, or CLS.
So back to the drawing board, where I had to figure out a way to design something deliverable by 1 developer in 1 sprint, while minimizing layout shift and maintaining a minimum amount of information.
Final Iteration
I made several iterations to reduce the vertical height, and landed on horizontal orientation.
Final User Tests
I then tested this version in one more set of user tests, and this time the results weren't so rosy. Most users didn't notice it was there at all, and one who did said it felt like an ad.
Final design exploration
I made one last effort to find a way to make it feel like less like an ad. But we didn't have the time to take the design in any new directions. So these explorations remained tabled.
Handoff file
I put a lot of effort into crafting a highly detailed design spec to make development easy. The developer we worked with did an amazing job, and although we didn't ask them to incorporate the "hide on scroll down" functionality, they went above and beyond and did it anyway.
When it came time for QA, I had one of the most rapid and collaborative all-hands-on-deck week's I've ever experienced. We had two developers from two teams come together and fix issues that my PM and I found almost as soon as we found them.
Final Design
Impact
So after one week of development and another in QA, it was shipped to production to a very small number of users in our test split.
While eagerly awaiting the results of the test results, I heard rumors that our analyst was getting excited. Apparently she said, "This is a million dollar idea."
The following week, I heard that our test didn't reach enough users to be considered statistically significant. There was so much further I wanted to take this design, but it our experiment was done, and it was time to hand it all over the next team.
What I learned
I already knew this in a sense, but this was a validation that designer-dev collaboration is key.
I also learned a little bit about the politics: How the right product demo can take you really far, but broader organizational dynamics means some things are just outside of your control.