How to design your data stack for curiosity
“I have no particular talent, I am just passionately curious” — Albert Einstein
When was the last time you thought about optimizing your analytics toolset for curiosity? Yet what is the value of all the data and analytics in the world if not paired with human curiosity?
If you ask most organizations what is important to them in terms of the data stack, usually it is things like moving to the cloud, getting clean data, building dashboards and reports to serve the execs running the business, security, and governance. Cultivating curiosity hardly ever makes it to the top of the list. If you take a moment and think about why that is the case, it is usually because no one considers it their job. Clearly, a lot of leaders consider data strategic and one of the most important tools for their business, but why does that not elevate curiosity as a concern? My guess is that curiosity is just assumed to be present, similar to qualities like motivation and ambition.
To some extent, that is true. All of us are born curious. A five-year-old asks around 300 questions a day. But then something happens, and we start asking fewer and fewer questions to the extent that most grown-ups rarely ask questions that could surprise them. Most knowledge workers and leaders aspire to be curious and see themselves as someone open to new data, but their curiosity is poorly supported both culturally and technologically.
Everything that has ever been invented or discovered or improved almost always originated from a curious mind. And while curiosity is hard to quantify on its own, there are many examples of the relationship between curiosity and better business outcomes.
When St. Jude children's research hospital, one of the largest charitable organizations in the US for children's health, allowed their fundraising team to explore their curiosity around their fundraising data, they realized that events within two miles of a Whole Foods store tend to produce much better results than anywhere else.
When one of the leading companies in the Insurance and Banking industry in Australia, adopted self-service analytics for their business users, people started asking a lot of questions they would have never asked. One of these questions around insurance claims data led to the discovery of an anomaly that saved the company 30 million dollars within months of deployment.
When one of the investment funds in Canada enabled their traders to ask their own data questions, within an hour of training one of the traders found they were being overcharged for securities lending by another bank to the tune of millions of dollars.
When one of the largest technology companies in the US enabled their accountants to ask data questions at will, within a few days they discovered misuse of travel policy that was leading to loss of millions of dollars.
While these are just a few anecdotes, I hope you can see how enabling people to be curious about data can result in substantially better business outcomes.
So, what can you do to cultivate curiosity in the context of data and analytics? To quote James Clear, the famous author of “Atomic Habits”, albeit a bit out of context, "You do not rise to the level of your goals. You fall to the level of your systems." That is to say, the biggest lever you have in terms of cultivating curiosity is deploying the right data stack. So, here here’s how you can ensure your data stack is built to cultivate curiosity:
Reward curiosity instead of punishing it
Imagine you are managing the inventory for a very large retail chain in the middle of a pandemic. One week your customers are asking for exercise bikes in hordes, the next they are interested in stockpiling toilet paper, and the week after all the rage is sanitizers. You are curious which product is rising in demand now so you can gather as much inventory as possible for the following week. For most folks, the first step to answering this question is consulting a dashboard. But perhaps you find that to manage the scale of data, your data team has aggregated transactions to daily and broad product category level instead of SKU level or sub-category level. That means if you want granular data, you have to submit a data pull request, which sits in the queue of other requests, and there is a wait time of two weeks or more to get the answer. You are not even sure if your request will yield meaningful insight, so you feel bad piling on one more request when everyone is frantically trying to get their data.
Compare that to an experience where you can drill down into whatever slice of data you need and ask questions in a Google-like interface and you are rewarded with instant insights. Which experience will encourage you to be more curious? Which one will bring better results? For Canadian Tire, the choice was clear.
“During much of the second quarter, Canadian Tire Retail was forced to temporarily close or operate in a limited capacity about 40% of its stores. Still, Canadian Tire Retail’s sales were up 20% from a year earlier”
Grant people permission to be curious
Another reason people often curb their curiosity is because they don’t feel empowered to explore it. “Curiosity killed the cat” is often the refrain from frustrated adults to a curious child who has exhausted the adult’s patience or explored something dangerous. While well-meaning, over time children feel disempowered and stop asking questions. The corporate equivalent of that is, “if I allow everyone to ask their questions, they will reach wrong or biased conclusions”, or “when everyone has their own version of data, no one can trust the data.”
These concerns were absolutely valid in the older generation analytics stack when everyone dumped their own data in excel with different filters and different metric definitions. This even gave rise to the term “excel hell.” When your analytics tools couldn’t handle more than a few million rows, you had to work with Tableau Extracts and PowerBI reports, and it was absolutely important that these things were tightly governed.
But with the modern data stack, you can leave data in granular form in a cloud data warehouse, define your metrics in one central place1, and let everyone do their own analysis without worrying too much about having different versions of the truth. Both granular security and centralized definitions without locking down the grain of aggregation is very much possible with the right stack.
Invoke curiosity through adjacency
We have all gone from watching one clip on YouTube to looking at a cat playing a piano an hour later. Less often, we have also gone through the Wikipedia rabbit hole where you go to read the page of your favorite author (P. G. Wodehouse) and an hour later you are reading about different kinds of coffees (If you must know, this is the path I took Code of Wooseters -> Cow Creamer -> Espresso based drinks). When you are looking at one concept, you are naturally curious about all the adjacent concepts. If the adjacent piece of information is easily accessible in context, it is natural for people to let their curiosity go wild. There are many ways of incorporating this into your analytics stack. Often people hardcode links to related analysis in their data visualization. In ThoughtSpot, you can drill in any direction possible given the data model. Also, the explore feature uses machine learning to collect the next best data question in that context, personalized for the user and made accessible with one click.
Make curiosity a social phenomenon
We are social beings (some more than others.) When your friends or people you respect are looking at something cool, it is hard for you to not be curious about it as well. This is why we saw significant engagement with analytics when we started showing people who else has looked at the answer to a particular question or Liveboard. Having a social feed of activity and commentary is another idea that we have often played around with to evoke curiosity.
Surface the knowledge gap
When our curiosity is piqued by teasing something we don’t know, it usually has immense power over us. That is why it is so hard to resist those damned notifications on the phone or why it is so easy to grab someone’s attention by starting a sentence with, “Did you know…”
At ThoughtSpot, we are working on a feature called automated business monitoring where a machine learning algorithm is constantly watching the metrics on your behalf and invites you in to dig deeper when one of the metrics behaves in unexpected ways along any of the dimensions you care about.
While these ideas largely put the onus on tools and organizations to change behaviors, ultimately it is up to each of us as individuals to keep our own curiosity alive. Staying curious means being fully present, willing to admit when you don’t know something (or when you’re wrong), and having discourse with people we may not agree with. It is not always easy but it is one of the most rewarding things both here and now and in the long term.
Cognitive neuroscientist, Matthias Gruber, in this talk, explains that when they studied brains under fMRI after arousing someone’s curiosity by asking a trivia question, they found it looks very similar to someone anticipating a reward such as a treat. It makes perfect sense, because what is sweeter than the discovery of a new insight from the question you just asked.2
This is very much possible in ThoughtSpot worksheets. This is also the reason why I wholeheartedly support the idea of a central Metric Store.
In case you were curious about the image at the beginning, it is a concept image of the Curiosity Rover waking up on Mars.