Which is better? Data-driven or data-informed?

创业投资 2017-09-25


Casper Sermsuksan | 2017-09-25T12:30:00+00:00

A month ago I got a chance to speak at Tech in Asia Product Development Conference in Jakarta about data-driven versus data-informed. Post-conference I was approached a few times to discuss more and speak at other events about the topic. Since people attended my talk found it useful, I thought I might as well share it with the global community.

Recently, the term data-driven started to become a buzzword, adding on to the list of other cool, techie jargon : disruptive, pivoting, design thinking, and many more. So, let me first attempt to get us all on the same page.

What’s Data-Driven?

Being data-driven means letting data be the center of a team and company’s decision making process. Data then plays an essential part of your company. Decision makers reply mostly (and sometimes solely) on data. Some decisions can be made without human being involved.

Some of the conversations that came up from a data-driven company are:

  • “Let’s A/B test this, and implement the version with the better result”
  • “Don’t talk to me unless you have have data”
  • “We’ll just follow what the data tells us,”

The truly data-driven organization will implement this process across teams and functions. This means that every team (product, marketing, customer success, operations, etc.) uses data intensively for all their decisions.

So, what are the benefits?

It eliminate a lot of human biases involved in decision making

In fact, a mentor of mine Misha Chellam created flashcards to help you memorize them. Data helps get the team members and stakeholders on the same page with less of their own judgement.

It is a lot less time-consuming because we take human out of the equation

How many of you have been in a 3-hour long meeting that people argue back and forth with no sign of compromise? Even better if you work in at a startup. Sometimes we discuss and argue for the sake of doing it, then the HiPPO made the final decision. (Yep, I know how you feel).

In fact, a study from MIT has proven the benefits of being data-driven. Prof. Erik Brynjolfsson and his colleagues studied 179 large publicly-traded companies and concluded that those companies are 5% more productive and profitable than their competitors.

However, being data-driven also has its drawbacks.

It requires a huge amount of data for the decisions to be accurate

You have to watch out for outlier distortion. Outliers are a data point from which are radically different from your average data. In gaming, positive outliers are considered “ whales .” Those customers whom make either big purchases or make them very frequently, generating a significant amount of revenue for the company.

At my previous company, we even coined the term, “whale hunting.” In other industries, it might be those customers who use your product to solve a different problem than most of your customers. Their behaviors are so different, thus should be excluded from your data to make decisions about your average customers. Few data points together with the outliers can lead your company toward the direction that you don’t want to go.

It requires someone with data science knowledge and a lot of resources to be beneficial

For startups and big organizations that just started Data Science department, it might be hard to be completely data-driven due to lack of capabilities and resources. Current employees might not have enough knowledge to build the infrastructure to support. Some might not even have anyone who has expertise with data. At my previous company, we have 10 data scientist and at Kulina, we have 0.5 person (myself and our head of tech).

There is still a bias in the way we gather data

Most of times, what people say is not what they do . So, if we gather data by asking them implicitly, we risked making decisions on the wrong information. For example, during the scandalous time of Uber and the #deleteuber movement, most of my friends said they would never use Uber again. When Lyft (its main competitor in the US) has a surcharge, they went back and ordered Uber for their regular rides!

Let’s look at an example from my experience

Another great example is when we were trying to figure out which social media platform we should spend more time and money to engage with our users. We sent out a survey asking them, and the result was quite surprising. We saw Facebook on top of the list, which is expected. However, what puzzled us was the fact that Google+ was second on the list (no hard feeling to the team at Google).

A few team members asked if I would agree with letting them spend more time building our Google+ presence. “Maybe we didn’t know our users well,” a team member mentioned. Being data-driven, I would probably greenlight this. But being stubborn as usual, I discussed with another product marketing manager on what could’ve gone wrong.

We came to a conclusion that our users might not understand the difference between Google and Google+. In fact, when we look deeper into the way the question was asked. Instead of using the icon, we used the wording Google+ in the questionnaire. If you didn’t pay close attention to the survey, you probably thought we were asking if you use Google!

So, what is a data-informed decision?

Making data-informed decision takes data only as a factor that can be dismissed from time to time. This kind of decision making allows other factors such as customer experience, gut feeling, brand consistency and the HiPPO to take the lead.

The benefit is that data is put to be challenged

Because we don’t purely base our data, we use it to create a hypothesis. No matter how good data is, it has its limitation. It is just a snapshot of reality that doesn’t paint the full picture of our customer journey and behavior. We are forced to user other factors to help make better decisions, including our own judgement.

One of the example was at my previous company. To give you a little bit of background information, the nature of our gaming business was having great revenue on the weekend because that’s when our users played games. On a Monday, one of the new product directors approached me regarding a low weekend revenue, and he was very worried.

With some experience and guess, I suspected that our revenue was down because we ran sale the prior weekend. The same effect occurred for the past five times we ran sales. If we look at the same cycle before, our average was actually better. In fact, we figured out later that it was because one of the marketing campaigns that were launched to encounter this specific problem.

Facebook’s Newsfeed version that moved the key metrics was actually done without being data-driven

Adam Mosseri and his team back then decided to create the newsfeed without data. In fact, it got a lot of negative reactions that users were angry and even created a Facebook group I AUTOMATICALLY HATE THE NEW FACEBOOK HOMEPAGE (watch his talk here ).

“At the end of the day, we have to have a gut to make bold decisions without data.” Adam Mosseri

This doesn’t mean that data-informed decisions don’t have their drawbacks

It requires a lot of time to discuss and analyze, and conclusions could not be formed easily. But remember at the end of the day:

“All the data in the world won’t fix a fundamentally bad product.” Andy Carvell .

In order to be innovative and build the right product for our customers, we cannot just sit in the room looking at rows of data and charts. Hint: Data doesn’t have to always be big data. Gather qualitative feedback by talking to your customers, observing customers struggles through usability tests and figuring out the jobs customers are hiring you for are key to building products.

So, my recommendations to you is

For smaller companies, you should always aim to be more data-driven. You probably don’t utilize the data you collect (or even collect them at all). The more data you have, the better hypotheses you will generate.

For larger companies, you should be able to make bold decisions that not move the key metrics but also improve customer experience. We should use data to help us make better decisions, but don’t reply on it a hundred percent!

About the Author


Casper Sermsuksan

Casper is currently the COO and Head of Product at Kulina, a food subscription service startup in Indonesia. I spend my time outside of work writing articles, speaking at conferences, mentoring other founders and leaders, and building communities for entrepreneurs and product managers. If you’re traveling to Thailand or Indonesia and are interested to speak, please contact me at casper.sermsuksan@gmail.com. I also welcome an opportunity to speak and share knowledge with your community!


There are currently no comments.

To comment, you must be a member. Become a member today orlog in.

Startup Grind

责编内容by:Startup Grind (源链)。感谢您的支持!


黑莓首款防水手机Motion泄露:大下巴设计,加入神秘按键... 科客点评: 我觉得吧,如果价格适中的话还是可以考虑一下的。不然的话,黑莓Motion单凭防水其实是很难有市场的。 眼看更多厂商的新品都开始支持IP67以上的防尘,黑莓似乎也坐不住了。继此前黑莓官方曾放出的预告片后,爆料大神evleaks也公布了关于这款黑莓防水新机的信息以及渲染图,不知道大家...
Most Unusual Designs from Popular Creative Designe... In order to be a great and creative designer , you need inspiration, and you need to be incredibly literate in the field of design. Art must not be r...
设计的困境与出路 – Weekly NO.34 ElemeUED Weekly:设计类阅读推荐周刊,饿了么UED 陪你一起读 : ) 这是我们第 34 次为你推荐优质文章,往期推荐可点击查看目录。 Airbnb 设计副总裁:设计的困境与出路 设计师们经常会聚在一起聊设计的出路,聊未来的发展,聊团队的建设。Airbn...
15 Skills Every Design Client Expects You to Have 15 Skills Every Design Client Expects You to Have While every design project is a little bit different, almost every client expects you to be a jac...
用户不再通过Facebook获取新闻 转而通过WhatsApp... 新浪科技讯 北京时间6月14日晚间消息,据路透研究所(Reuters Institute)的一项调查结果显示,通过 Facebook 等社交网络获取新闻的用户越来越少,因为许多年轻人转而通过WhatsApp等消息应用来讨论新闻事件。 该研究结果显示,在美国利用Faceboo...