When was the last time you used your own product?
Probably the 2nd most embarrassing question to answer, right after “When was the last time you talked to your users?”.
By now, we have established that I work in a central capacity building a Messaging product that is integrated into 17 marketplaces around the world and that this comes with a whole world of drama, complexity, and fun (of course).
A while back, I was having a conversation with a colleague about how the performance of our product according to our success metrics, vary so greatly across the different marketplaces. Some are better than others, while others just have the worst. This we found perplexing because we were providing a full stack product. Apart from local translations and branding customizations, the Messaging experience in the marketplaces should be, more or less, the same. At least, we thought so.
We came up with many hypotheses.
Maybe some markets are just not “chat” friendly. Lack of trust(and everything else that came along with it). They just don’t want to use our Messaging because it lacks the right tools to push for a successful transaction (aka lack of features).
The list was long. And we were pretty convinced that some of them can be right.
We almost launched a full research project. Except that we couldn’t, at that time – for reasons that are too long winded I would need to dedicate a post to that.
We needed something fast. We needed something cheap (preferably free). So I had an idea!
I suggested to re-create a workshop we did a few years back in my previous work, wherein we got the whole team to not just test but to fully “experience” the product we were working on. The idea was pretty much to do the same. I wanted the team to get into a room for a few hours and put their feet in the shoes of our users – To try being both buyers and sellers in our marketplaces.
The point is to observe. To identify the parts that made their buying or selling experience pleasant as well as painful. So we can pinpoint the things that are worth investigating further. And also pinpoint the things that immediately need fixing.
“Oh, so you mean dogfooding?” was the reaction I got.
Right! There’s a term for it? I just like to call it “extreme internal usability testing for when you don’t have a budget/time/resources”. I’m not exactly the savviest when it comes to tech terms and buzzwords of the now. When I first did this workshop years ago, we called it “Wow Workshop” (to be honest, I prefer that name vs dogfooding. I think it has more flair to it and all that’s missing are jazz hands!).
Ok, I digress.
Test devices at a ready!
I sent out the invite to the entire team. Much to my surprise, a couple of people actually showed up, bright and shiny, excited to test, brandishing their phones like first-year students at Hogwarts learning their first spell.
We picked the marketplace that we have integrated with that had the worst metrics and we set up what we needed to do.
- We will pretend to be sellers. We will post an ad, and wait for potential buyers to contact us and engage them in the most natural conversation.
- We will pretend to be buyers. We will reply to a few ads from different categories and wait for sellers to respond back to us and engage them in the most natural conversation.
- We will note down our experience. Noting down everything that made our conversation experience good or bad from the moment it began to the moment it ended.
It was half a day of testing. Most of it spent waiting for the real users to interact with us. And in between, we were oohing and aahing at the cool things we realized were happening, and wincing at the things that didn’t sit well with us.
By the end of the session, everybody in that room had a long list of observations and wanted a minute on the soapbox to share them.
Of course, we learned things!
Because the whole point of this blog was to share learnings, right?
1. Our responsibility to the product we were working on doesn’t just start and end within Messaging.
How buyers can start a conversation, how either user can know where to find their new messages, and how they get outside of the Messaging area – they were all still part of the user journey that we need to care about even though they were technically outside of our product and it would have been so easy to write them off as “not our responsibility”. And that was made pretty obvious when such small things like “users who didn’t have an account in the marketplaces, when they reply to an ad, don’t even have a chance to know that there is a Messaging system in place” have affected our KPIs drastically.
We also started realizing that everything in the marketplaces, although we sit in different offices, we work in different parts of the sites, at the end of the day – they are all shared responsibilities. Everything that we do, or don’t do has an effect on everybody’s KPIs – or if we’re going to be human about it, has an effect on the users.
A boss once told us that “the success of our product can only be measured by the success of our marketplaces”. We are part of a whole.
2. Putting yourself in the shoes of your user – cannot be automated
at least not yet – I don’t presume to be clairvoyant to say what the machines won’t be able to experience
That confusion we felt trying to figure out where the Messaging Inbox is, in the wild world of the marketplaces? That impatience we had while waiting for somebody to respond to our last message sent 2 hours ago? That excitement that was there when we got 10 new message notifications in one go? That annoyance from all the spam messages?
I will bet a ton of jamon that somewhere out there, there is a user who probably goes through that entire spectrum of emotions on a regular basis, that we only felt at that moment we were testing (because we were forced into this workshop by the annoying Product Manager in the team who has run out of decks to make).
Empathy. It’s not just a fancy word. And it’s actually good to practice it. And be reminded that real people actually either benefit or suffer from what you’re getting paid to go to work for every day. There’s a responsibility there that we all have to acknowledge. And by all, I mean not just the people who have to “own the user journey” according to their job descriptions. 😉
3. There’s merit to being scrappy. Working fast and cheap to get the most amount of learnings possible doesn’t just mean running experiments.
It’s easy to be spoiled when you work in a big organization. There’s somebody responsible for every single thing. Usually. But sometimes – well you only have yourselves to rely on.
In our case, we had hypotheses that needed validating. Normally, we’d enlist the help our researchers. But they couldn’t help – at least this time. And getting external help was not an option that was available to us then either.
So we did what anybody would do in our place – we prioritized problem hypotheses that we could validate ourselves. If we managed to prove them true – then YAY! If we didn’t, at least we were a few hypotheses less from where we started and it was still a learning we could put in our pockets.
There were so many things we could do ourselves to get that validation happening. We could’ve done an experiment. We could’ve added more tracking. We could’ve done interviews. My point is – there are about 100M ways to validate hypotheses. Not all of them will always be available at any time to you. But having options means there will always be at least 1 that will be. It’s just a matter of choosing one that will generate the most learnings given whatever circumstance you are facing at that moment.
And that’s how we ended up deciding to just do some testing on steroids.
To close things…
One of my previous bosses once told me that to be a good Product Manager, you have to be both “obsessed with your users” and “obsessed with your data”.
I took that to heart, mind you.
But now, I think I would like to add something to that too. To be an ok-ish Product Manager, you also have to be “obsessed with the problems.” Whether we’re talking about the problems of your users (ideally) or the problems you might have internally (if there are any).
Don’t get me wrong. It doesn’t mean you immediately have to solve that problem – it rarely does. It just means being able to sniff them out, and understanding their impact on the big picture (if it has any). Later, of course, you should start solving them – if they are worth solving, that is.
In the end, we didn’t have to build a fancy new feature to solve the mystery of the falling metrics. It was just a matter of understanding what the problem was in the first place. It was understanding the problem that was a different story. But to do that, you will always have options.
TLDR;
- We saw falling metrics in some marketplaces, but not in the others
- We were confused and wanted to find out more
- But we couldn’t do a full-on research
- So we prioritized the hypotheses that we could validate ourselves
- We did a full on testing of our product as a means of validation
- We validated our problems
- And realized that our ownership of the Messaging experience doesn’t just start and end when the users are within Messaging
- We also validated the value of testing our own product in the wild
- And that validating problems don’t always to be a huge research production
- Sometimes all it takes is just knowing how your product really works
*****************
If you have any feedback, questions, or even violent reactions – please don’t hesitate to leave a comment or send me a message at hello@kaxuson.com
And in the future, if you’d like to receive updates from me, links to interesting articles and conferences about Product Management, and other interesting Technology reads to go with your morning coffee, then please sign up for my newsletter here. Don’t worry, I hate spam too. It’s also GDPR friendly (Mailchimp said so).