How to iterate on product has always been one of the top concerns at startups. But while they previously had to rely solely on customer feedback and instincts, they now also have a trove of data to balance. This dynamic creates new opportunities but also requires a new kind of arbitrage.
We discussed this new context during TechCrunch Disrupt 2021 in a panel conversation with Jean-Denis Grèze of Plaid, Stephanie Mencarelli of InVision and Pete Thompson of eBay. We explored different perspectives on being data driven, segmenting your user base, iteration speed and more.
The role of data
Our conversation started with a provocative question: Can a startup or a tech company ever be too data driven?
According to Thompson, the answer is “absolutely not” — but Mencarelli wasn’t so affirmative.
Thompson:
It’s a matter of how you use the data and how you balance it with other forms of feedback that you get. But I would say that it’s just getting more and more important to uncover things within the organization with data that you can’t do with any other means. It identifies things that human curation or manual processing wouldn’t uncover.
Mencarelli:
I will maybe slightly disagree and say that there is a point where you can get too data driven, where you’re not seeing what I call the edges of the innovation bell curve. So it’s really important to see what smaller cohorts are doing, because they could be early adopters to a new behavior.
However, Thompson himself had added some nuance earlier, noting that “You need to be able to also think about other things that you’re not doing or features that you could build that the data won’t actually tell you.” He gave an example of this at eBay:
We’ve recently been launching something that we call searching through images, so not just text queries. And this is a great example where our data would never have told us that on our site, for instance, for fashion, the younger audiences just want to be able to browse and see other things that look like the same image, not based on a text-based search. And these are things that you have to glean through other forms of feedback loops.
The importance of segmenting well
Following up on Mencarelli’s point about observing smaller cohorts, the panel moved on to an important topic: Segmenting.
Grèze:
When you look at data or product roadmapping, if you have too much data, there’s the danger that it looks like a haze. It’s the average of everybody, and from the average, it is difficult to build great products. So you have to find a way to segment your customers into little buckets. [ … ] I really urge people to first narrow down their set of users, and then look at the data and really go deep into the kind of product signals that they’re getting.
Grèze explained how Plaid does this:
Something that has been really important for us is to identify what we call representative customers, of whom we think that if we solve their problem, we solve the problems of a host of other developers. And the idea is to treat those representative customers like partners.
Plaid found such a partner in Copilot, a personal finance management app that TechCrunch covered last year. There were several reasons this collaboration made sense: Copilot was filing very sophisticated support tickets; a large segment of Plaid’s user base had similar use cases and needs; and the team was willing to work with Plaid to help develop the product roadmap thanks to their feedback.
The need for segmentation is perhaps even more pronounced at eBay: As Thompson noted, it has “around 20 million global sellers across under 190 markets. And it could be high school students with a side hustle or a global brand like Nike. They have obviously very diverse needs. But we are a horizontal marketplace that needs to be able to account for all of them.”
The way eBay addresses this could be applicable to other e-commerce companies, Thompson said:
One of the things that we’ve been working on is to do the segmentation, understand the different needs and build for those needs with a vertical focus, but at the same time with an eye toward horizontal extensibility.
That’s the nuance or the sophistication that I think especially an e-commerce company needs to have. The ability to really understand, for instance, a sneaker seller and what their needs are, but at the same time not have to do that across 400 different categories. Figure out how to find the archetypes and commonalities that you can then take horizontally to meet the scale that you need to.
That type of extensibility is also something that InVision thinks about in the context of its free-to-paid model, Mencarelli explained:
I think of it as a pool that you fish out of; some fish will always stay in the [free tier], and that’s OK, but it’s all about that virality and making sure that the things that you are creating have enough value to punch you into a paid service. So like Pete said, it’s very unique depending on the customer cohort, but how we build things that are extensible is definitely what we’re focused on at InVision.
Relative speed
On another provocative note, I cheekily asked Grèze whether “move fast and break things” resonated with Plaid.
Grèze:
I think for Plaid, first, because we’re in financial services, you can’t really have the “break things” part — that’s just a no-go when people’s mortgages or money moving around depend on you. But moving fast is important; it’s just that moving fast looks different.
Why does it look different? Because when you’re building API solutions, “the timeline is very extended. [ … ] Your development cycle effectively is the length of your development cycle plus the length of the development cycle of your customer. That means you have to have higher conviction upfront, because you can’t just A/B test your way to success. Also, if your customers are using your API, you can’t deprecate it.”
Taking this into account, what matters is “relative speed,” Grèze said. “We’re not moving fast relative to Facebook; we are much slower than Facebook, but we’re moving fast relative to other API companies.” How? Thanks to product conviction, design partners and finding a set of customers with whom to iterate. In Plaid’s case, that’s self-serve customers: “You can ease [them] into a new experience without a lot of lift necessary on both sides.”
Building roadmaps
Mencarelli said that speed is very important at InVision, as is pizza, which inspired them to come up with the concept of “slices”:
These slices are intended to be a grouping of features that bring value to a defined customer cohort. Instead of just shipping feature one, feature two, feature three, we need to tell a story about how they all come together. And that’s how we formulate our roadmap. [ … ] You have these slices that deliver value because they’re a group of features, not a singular thing. And so that is what creates meaning in our roadmap.
Thompson also insisted on the need to go beyond incremental innovation, but at the same time without going too much into step-function innovation. He used the analogy of a staircase:
You have got to work on both sides. You’ve got to go up incrementally [by] asking your existing customers what else that they would like to see and start working up the staircase. But you also have to create a way for you to jump to the top of the staircase and look backward and try to figure that out.
Hearing more voices
By way of coincidence, the conversation landed on Spotify, with which Thompson had collaborated at Sonos and where Mencarelli previously worked. This led her to tell an anecdote about the music company: It was struggling to get useful feedback from Swedish users it brought into its office, because they just loved the service too much. While this problem might be uncommon, the scrappy way Spotify solved it will likely resonate with startup founders.
Mencarelli:
What we did was, we went into hostels all around town, because that got us a more international audience. We could buy them a coffee or give them a month free of Spotify. And that kind of scrappy guerrilla research [is necessary to hear from people].
Earlier in the panel, Mencarelli also shared some thoughts on an important topic: Inclusivity. One of the silver linings of the pandemic at InVision, she explained, was the ability for more people to share feedback. This led her to an optimistic prediction about the future of work and product development:
I think that what is going to be changing our industry is the ability to hear more voices and more ideas to spur that inner innovation, whether it’s because you’re allowed on Zoom or you’re visually collaborating and making [things] together.
You can read the automated transcript here.
