Welcome! The Product Craft series on MarketMaker provides concrete walkthroughs of Product Management decisions, frameworks, and real-life examples.

One step back and one step forward

Back when I was a wee little product operations manager, one of the senior PMs on my team asked me and the designer, “what happens when the user goes backwards in the flow?”. I literally experienced a kind of galaxy brain moment - of course your users are going to do something weird and unexpected, you should not assume that they’ll only go one way through a flow. And that kind of behavior is something a good PM (and designer!) needs to spec out.

Today, we’ll cover one very specific type of backwards behavior, which is ping-ponging back and forth on a search results page.

Search results ping-pong

A simplified search and discovery user journey on, say, an e-commerce website may look something like this:

1. Users enter their search term

2. They view the search results page (“SRP”)

3. They view details of one or more items

4. They decide which item they like

5. They buy it (checkout)

However, step 3 contains a lot of tactical messiness. Users will view one item, go back to the SRP, view a second item, etc. Or they may open up several browser windows or tabs at once. Search results are one of the product use cases where PMs should actively expect that normal user behavior is to go back-and-forth a few times, before moving to the next stage of the user journey.

Some examples of bad search result products

What does it look like when a website or app does this badly? Well, Aetna’s search results provide an example of that (Aetna is a large health insurance provider in the US):

View video here (much better than the screenshots I can embed): https://www.awesomescreenshot.com/video/24754600?key=9f32546381dfca7b9bd1d48fd91b8acc

• You literally can’t open the detail page in another window

• When going back, you are booted all the way back to the start of your search and have to type in your search term again

(And if you can’t tell from the comments in the video, this was immensely rage-inducing)

The proper behavior here would be to:

1. Allow the user to open the detail page in a new tab

2. When going back, return the user to the same position in the search results page (i.e. if the user was looking at item 9 out of 23, bring them back to the scroll position of item 9, rather than plunking them on item 1 at the top of the list)

3. Retain the user’s search term and show that as an option in when the user types in search terms

Another example of bad design here is Libby (app that lets you borrow ebooks from public libraries), where pressing “back” from an item in a wishlist takes you… To a completely different part of the app that is not your wishlist. 🤦 The proper behavior here should be to return the user to that same specific wishlist, and also the same position.

How to get a fix prioritized

Let’s say you’ve just taken over as PM for Aetna’s provider search (congrats /s), and ooh boy this is what you’ve found. 

How do you then go about getting this fix prioritized? It’s actually quite tricky - this isn’t obviously a blocker, it’s just a bad user experience, and Aetna doesn’t work like a shopping website where a poor search experience is directly tied to lost revenue. We’ll walk through 5 ways that you can get this fix across the line.

1) It’s a bug

This is actually kind of the best case here. (And I don’t know why more PMs don’t discuss this aspect of our work and sprint planning). You dig up your designer or an engineer who built this, show them the terrible Aetna video, and you ask: “Is this expected, or is this a bug? ಠ_ಠ”

Gloriously, if this is a bug, all you have to do is cheerfully ticket it and plunk it into your backlog. Teams / pods are supposed to pick up production bugs as part of their sprints; you don’t need to go through the whole song and dance of quarterly product planning. This is probably a nice little P2 or low P1 bug fix (to make the arrow icon open the details page in a new tab) (okay, fine, maybe 2 bug tickets and 1 improvement ticket to cover the three fixes I suggested), and your team can simply sweep it into an upcoming sprint. Nice!

2) Make the business case

Sad trombone, you talk to your team and this is Expected or As Designed. Sigh. 

Your next tactic to try is to directly link this change to business metrics. For Aetna, this is actually kind of weird and interesting. For most standard applications, if a user can’t find what they want, the company will not make a sale, and that will be lost revenue, so making a business case is quite straightforward. For Aetna though… Presumably, when you find a medical provider and file a health insurance claim, Aetna loses money. So I did wonder for a few minutes if this was some insane dark pattern designed to purposely prevent you from successfully finding a provider.

Some sleuthing, however, seems to indicate that this is just… Weirdly designed. If you click into a specialty, you land on a TOTALLY DIFFERENT TYPE of search results page, which has all the good behaviors we want:

View video here (much better than the screenshots I can embed): https://www.awesomescreenshot.com/video/24754635?key=ea7dd7648df88b67da93b54dd81c0361 

It looks crazy, but as other practicing PMs will attest… Sometimes the best you could do looks a little zany (boy would I love to know what went on internally at Aetna for the final product to end up looking like this).

Okay, so, what kind of business metric might be valuable to Aetna to improve? The one we should think of here is probably support costs. If someone can’t successfully work our search page, they might instead contact support, which is pretty costly. A chat ticket costs about $3 - $7, and a phone call is about $10 - $20 (source). At 39 million members (source), that can quickly add up. So the job-to-be-done for Aetna’s website is likely to allow insurance members to self-serve and thus avoid these support costs.

To ballpark the approximate dollar value, we might do something like this:

1. Estimate the number of failed search cases per year that result in a support ticket, e.g. use tracking data to count sessions where the user entered the same search terms multiple times (like I did), and see which of those cases also had a “help me find a provider” support ticket created in the next 7 days. That will give you the baseline for “incidents per year”

2. Multiply incidents per year by cost per support ticket (and here you might get fancy and segment these by the support channel, i.e. chat vs phone, as the cost per ticket is different)

3. To get extra extra fancy, apply some haircut (as presumably fixing this won’t solve 100% of these cases) - let’s say we just take a random stab and guess that this will help with 30% of such cases

And that will give you a rough estimate of the dollar value saved per year if the search experience was better.

3) Directly appeal to user experience

This works if you work at a relatively enlightened product company, but may not work elsewhere. You can directly make the case that this is a very poor user experience, and put it into your roadmap as part of a portfolio strategy that has both metric-impacting projects, but also smaller user experience tweaks like this one. (Aside: at one of my favorite PM experiences, I was cheerfully told by my manager “don’t worry about monetizing this, just focus on providing value to the user, and we’ll figure out monetization after”, and I was like “COOL YES OKAY”. Ah, life in the zero interest-rate policy years…).

If you are very, very lucky, just describing the issue should be more than enough. If you need to whip out data, you might want to use datapoints like:

• Percent of searches that display this behavior (i.e. user types same search term multiple times)

• Is it a top user feedback item from surveys? (Hint: This is why you really should have an evergreen “website/product feedback” survey available for your users at all times)

• Likely overkill for this particular example, but you can also do user research studies and see if participants get stuck here or get very frustrated here

Next we’ll cover two… Mmm… Chaotic good approaches, let’s say.

4) Chaotic good: Do it under the table

Sometimes… You should just do it. Just sneak it in. It’s not even that bad, it’s totally kosher to ticket small improvements, and let eng pull them in if a sprint has space. I’ve done this. 

• Users hated it that we only processed a certain payout once a day, so we just… Made it once an hour. Straight rollout. All support tickets about “where is my payment” vanished.

• We had an internal tool showing a table of job statuses  at Wonolo, and in ye olde days I used to sit there and manually count up the number of jobs with each status, and I very quickly Got Mad. So we just… added a summary table. Boom.

• We had an internal tool that stored custom lists of users, and another tool that did things to a list of users. But you couldn’t use the lists from the first tool in the second tool 🤯, and that was dumb (and also made me mad). So yeah we made it so that you could use the list of lists in the second tool.

Now, typically for external user-facing features I would 80% of the time say “no, do it properly”, because a lot of good PMs have been proven wrong about small tweaks, but this might be a valid 20% case where smuggling is fine. (You’ll notice that two of my examples above were internal tools, where the risk was lower because my slack would have exploded with messages if other users hated those changes). This is a super common design pattern, and the change is very two-way (i.e. it can easily be rolled back). If Aetna has some kind of suggestion box / product feedback box already set up, or if the Support team and Product team have a good way to pass feedback to each other, so that you can quickly know if the change was bad, this would be a pretty low risk change in my book, making it a valid “do it under the table” item.

5) Chaotic good: Trojan horse it

Maybe veering into chaotic neutral land, the other option is to trojan horse it. This is when you smuggle improvements into other product features. Let’s say that you, as the Aetna PM, get asked to add something to the search results page, like a “top provider” badge or something. Well, as part of doing that… You could ALSO include the fixes in that product. Patient open on the table and all that.

These again look like fairly small (low eng cost / low eng complexity) changes, so trojan horsing it in is pretty valid. Eng and QA may even be happy that they can just knock out many changes at one go rather than doing it in bits and bobs.

The dark side here, however, is scope creep. If you have to trojan horse in a very big change and it totally blows out the eng cost of what is supposed to be the primary benefit, that’s not okay in my book - you as the PM should be ruthless about what actually needs to be in scope. But, also, as a PM, if you find that you have to trojan horse in all the step-change improvements you want, or have to trojan horse in any little user experience improvement, you should really question whether you want to keep working at a company that affords you so little autonomy and trust as a PM that you literally have to smuggle all your work in under the radar.

🏆 The “most valuable tech tool” shoutout for this post goes to the Awesome Screen Recorder and Capture plugin for Chrome, which is how I made the videos of the Aetna site. Surprisingly easy to use, for someone with zero video editing experience; and I also use it all the time to annotate screenshots when ticketing bugs and questions for eng. Second best PM plugin of all time! (Sorry, ad block is still more important…)

Thanks for reading! Subscribe below to make sure you get each new article from MarketMaker. I also welcome comments, corrections, and feedback - connect directly with me on LinkedIn for that.