Unless you’ve been living under a rock with no internet access, you’ve no-doubt heard examples of how people are using the platform, and prophecies of how ChatGPT is set to change the course of society as we know it. When something this interesting shows up, surely curious minds want to know how it works. Despite having worked on Machine Learning for over a decade and having an active interest in the space, when I tried to read the research papers initially, I found it to be a daunting task. Each paper that I tried to understand referred to concepts that I did not understand, and when I tried to read another paper for those concepts, I found another set of concepts that I did not understand. It took a substantial resolve to keep following the rabbit holes till I understood the papers that describe how ChatGPT and similar technologies work from first principles.

This article is my attempt at simplifying the timeline of technical advancements that led to ChatGPT as we know it today. If you understand basic computer science and probability, you should have a strong enough foundation to follow along. Without getting into any complex math or very technical discussion, you should get some sense of how these things are wired.

Keep in mind, no one really understands how these models are doing what they are doing, in the same way, that no one really knows how they learn to ride a bike or walk. But that doesn’t mean we don’t have some handle on what's going on and we certainly know how we arrived here. 

In this four-part series we’ll cover:

1. A brief history of computational language understanding, starting from the early collaboration of linguists and computer scientists (This article)

2. Transformer architecture – The engine behind ChatGPT, including a brief study of sequence-to-sequence models starting from RNNs ending at Transformers.

3. Emerging properties of ChatGPT and other large language models (LLMs), where we’ll explore how language models are built using very large Transformer models that have surprising properties.

4. The future of AI – trends to watch for in a ChatGPT world, including applications, future research, and implications for our world.

There is a lot to unpack, so let’s get started!

What is ChatGPT and how does it work

GPT stands for Generative Pre-trained Transformer. The chat prefix to GPT was added because of the chat interface added to the GPT models. ChatGPT is an AI language model developed by OpenAI that can generate a natural language response to human input—basically, it’s an advanced chatbot. 

To get started, we must first examine the two interwoven threads of computational language understanding and machine learning using neural networks. Both are fairly deep and technical topics, but my hope here is that at the expense of some accuracy, we can simplify the concepts enough that anyone with basic computer science knowledge can get an understanding. 

Early natural language processing (NLP)

Computer scientists and linguists have been collaborating for at least half a century in an attempt to make computer programs understand language. Most people in the field believed that the path to doing so involved creating taxonomies of words, breaking sentences into parse trees, assigning parts of speech tags, and then using templates and rules to derive meaning. But as early as the 1950s people realized that there was a lot of information in looking at language purely from a statistical standpoint. For example, a word’s significance to the documents it appears in is directly proportional to the number of times it appears in the document and inversely proportional to how frequently it occurs in other documents. This was observed in 1957 and some version of this idea (TF-IDF) is still used in most search engines today. If someone searches for “the indic languages” the most significant word to match would be indic, followed by languages followed by the because indic is the rarest word in that phrase.

Language Models

Quickly fill in the blanks without thinking too much: The dog [blank] so loudly that it was painful. If you are like most people, you guessed barked and not scratched. If you were to assign a probability to every possible word, you would probably give a fairly high probability to barked and a very low probability to anything else. This process of computing probabilities is called Language Modeling. ChatGPT is essentially a language model. It is also a very large neural network with 175 billion programmable connections which is why it is called a Large Language Model (LLM). How large is large? For comparison, the human brain is estimated to have 10^15 or 1000 trillion connections. 

The fun thing about language models is that you don’t have to stop after one prediction. You can append the predicted word to your text and then make the next prediction and make the next prediction and so on until you have a full page of text. You can see this happen in real time if you have predictive text turned on for your phone keyboard. Most phone keyboards give you a ranked suggestion for the next word. If you keep accepting it, usually you have a bizarre but interesting piece of text created entirely by the phone’s language model. These models are nowhere near the sophistication of ChatGPT. 

The simplest language model would just count pairs of words that occur next to each other in a large body of text and then given the first word, look at the histogram of the next word. This is essentially where language models started. For example, after the word “phishing”, the most likely words may be “email” or “attack”. If you look at Google’s auto-completions, it matches that roughly intuition. 

N-grams

Next comes the idea of counting not just pairs of words (bigrams) but triplets (trigrams) and sequences of four and five and so on. These collectively were called N-grams. As the length of a sequence goes up, the number of possibilities goes up exponentially. So all n-grams cannot be stored and used in language modeling. Fortunately, as the length of a sequence goes up the number of times you find a sequence in a body of text also goes down so if you keep a threshold on counts to be above statistically significant occurrences, you can build it. Once you have a collection of N-grams, a simplistic way to predict the next word would be to find the longest matching sequences for the current text and use the distribution of the next word to predict the probabilities. This was more or less the state of art for language modeling for the longest time.

In fact, when I asked a friend at Google Translate in the late 2000s about how it worked, I was really surprised to learn that the core of it was simply matching n-grams in one language to n-grams in another language without much regard for grammar rules or anything else.

Representing concepts as big vectors

The N-gram model’s biggest drawback is that it has no information about the meaning of the word. In my opinion, representing words as big vectors was one of the biggest advancements in the early days of statistical NLP. Before vectors, people thought that the way to figure out synonymy and related words was to put words in a large tree (ontology) and then find how close they are to each other. In fact, there were armies of linguists whose job was to put words into ontologies. The problem with these word trees is that you’re forcing a somewhat arbitrary structure. For example, in a given ontology, Albert Einstein may show under  People->Scientists->Physicists->Born in the 1800s, while the theory of relativity may show under Abstract Concepts->Scientific Concepts->Physics->Modern Physics. From this viewpoint, the two seem very far apart,  yet they are clearly related concepts. 

Imagine another way to organize words, where you are given a very large whiteboard and each word represents a fixed-size circle with a center at a specific point on this 2D plane. And words are arranged on the whiteboard so that words that are related to each other are close to each other and words that are less related are far apart.

You may find organizing the words in this way near impossible because if these circles are packed together, each circle can have only a small number of (six to be precise) close neighbors, but a word can have many different neighbors that are all slightly different in meaning. Now imagine instead of a 2D plane, we are in 3D and words are represented by spheres. This gives you a lot more room to pack related concepts together. This in fact goes up exponentially with the number of dimensions. So by the time you reach 256 or more dimensions, this becomes an excellent way to represent the concepts behind words so that related concepts are close together and unrelated concepts are far. What this means is that each word can be represented by a few hundred real numbers that represent the coordinates of the center of its sphere. These vectors of numbers of often called word embeddings.

This idea is so important that we are going to try and look at it another way. Suppose you limit yourself to the ten thousand most frequent words in the English language and everything else gets ignored in the following analysis. Also, imagine someone coming up with a thousand topics. And each word is scored on a scale of zero to one whether it belongs to that topic. For example, “mango” may belong to the topic “fruit” with a strength of 1.0, but the topic of “yellow objects”  with a strength of 0.5 and “abstract concept” with a strength of 0.0. In this way, each word can be represented by a vector of thousand numbers. Now if we wanted to see if two words are related all we need to do is see how far apart these vectors are. Of course, if the topics were not chosen well, this scheme will perform poorly. So all you have to do is come up with a thousand topics that would do a good job of capturing the essence of all the ten thousand words, and then come up with 1,000 x 10,000 = 10 million weights. This kind of work has been done manually. It takes a lot of effort and the results are usually mediocre.

Latent semantic analysis (LSA)

One of the first successful attempts at solving this problem in an algorithmic way was the idea of Latent Semantic Analysis published in 1988. The idea was that if you add up the vectors of the words in the document, it should do a good enough job of representing the topic of the document. Now, based on the vector representing the document, if you guess the word distribution in the document, you may not get the exact word distribution but you might get something close as long as we had done a good job of picking the topics and association of words to topics.

For a given document, we could measure the difference in the original distribution of words and the re-constructed distribution of words from the topic vectors. Let’s call this difference the error in reconstruction. Now, if we have a large collection of documents, we could add up the error across all the documents. The key intuition of this paper was that the problem of picking topics and their connections to words could be considered a mathematical optimization problem that tries to reduce the total error. 

Furthermore, if you constrain the function taking you from words to topics and topics to words to be linear, the process of finding the best topics and best vectors for words so that error is minimized is identical to another mathematical problem called Principle Component Analysis (PCA). Luckily PCA has a closed-form solution that can be computed efficiently.

The vectors computed in this way were fairly useful in finding the conceptual overlap between words and very useful for things like web searches. One disadvantage of computing vectors this way was that it ignored coherent concepts, instead reducing language to arbitrary mathematical optimization. Sometimes you could see the hidden meaning behind the numbers in the vector, and sometimes you couldn’t. This was our first clue that advanced mathematical modeling of language would be less and less explainable.

Significance of vector representation of words

Word embeddings from the very start were a powerful tool for measuring similarities in the meaning of two words. As the technique for computing these embeddings improved, it turned out that you could do arithmetic with the meaning of the words in surprising ways. For example, if you take the embedding of king, subtract the embedding of man, and then add the embedding of woman, you get something very close to the embedding for queen. What this means is that you can mathematically manipulate concepts. This is a big deal. So far the only way you could solve this problem was through rules and the application of logic – a brittle way of encapsulating real-life complexities. 

With these embeddings, you can map and translate concepts from one language to another, produce summaries, and mix the meaning of words to get another word. For example, jaguar + animal will represent a different concept than jaguar + car. Or you can ask what word represents sad + lonely.  Put into practice, this gives models the ability to suggest whether an email is significant or spam, perform sentiment analysis of an online review, and most importantly run semantic search. 

Going beyond linear functions

The LSA approach above artificially constrained the functions going from word to topic to be linear for mathematical and computational convenience. The only question was, are there better embedding functions to be found if we relax these constraints so that we can discover better representations of concepts for the various applications listed above? 

While multi-layer neural networks were always a very promising class of functions for representing arbitrarily complex functions, they were notoriously difficult to train and for the most part, had fallen out of favor. However, a few research groups strongly believed in its power and kept trying different techniques to make them do useful things for decades. After many different iterations, Geoff Hinton’s group and Yeshua Bengio’s group successfully demonstrated that neural networks could do a great job of learning representations of words and concepts as embeddings. Along with Yann LeCun’s work on Convolution Neural Networks (CNN), this work is considered pivotal in the resurgence of neural networks in their current form as the most powerful tool in machine learning.

AlexNet and GPUs

2012 was another pivotal year for the resurgence of neural networks. Most experts credited it to AlexNet, which used GPUs to train a large CNN for computer vision tasks with a substantially better output than anything else in the field. Using GPUs was already floating around for a while, but the magnitude of AlexNet’s success opened the door to many others who were training larger and larger neural networks with bigger training sets and more and more compute. One big realization for researchers at this time was that the more data and compute you were willing to give your neural network the better the results. This was not the case with most other machine learning methods. Most other approaches saturated after a while, but the neural nets kept going. The two fields that benefited the most from this approach were Computer Vision and NLP. From there, innovation became very rapid and it hasn’t stopped since. 

Adding context to words

One problem with trying to map words to embeddings was determining context. Words can have very different meanings depending on how they’re used. For example, the word bank in either bank balance or river bank has two completely two different meanings and therefore must have two different embeddings.

From 2013 to 2018 a series of research papers improved the quality of embeddings by accounting for other words in the context and using different model architectures. The most influential papers in this period were Word2Vec (2013), Glove (2014), and Elmo (2018) which tried different techniques for capturing the meaning of a word in its context. The techniques behind these results are totally worth studying but are not that relevant to our purpose of understanding ChatGPT. However, the idea of understanding the context itself is very important to understanding ChatGPT.

Machine translation

One particular problem that was fertile ground for applying various neural network advancements to was the problem of translating text from one language to another. The key idea here was computing sentence embeddings that map a full sentence to a vector and capture all the meaning in the word. This vector can then be passed to another network and translated into text in a different language. In that sense, you have an encoder network that takes the sentence and encodes it into an embedding and there is a decoder network that can decode it. Now this combined network can be trained on all the available training data for any language pair where we have human-generated translations.

Even early attempts at using neural networks looked quite promising compared to classical techniques, but it also provided the grounds for a lot of innovation that followed. 

How ChatGPT is able to understand human conversation

So far, we saw how embedding words or sentences into a high-dimensional space is useful for many NLP tasks. We also saw that interpreting words or sentences within the context in which they occur is very important, and also challenging. In the next article, we will look at various sequence-to-sequence models starting from RNNs building toward the Transformer Architecture which was the key innovation that made ChatGPT possible. 

Next Post

Transformer architecture – The engine behind ChatGPT, includes a brief study of sequence-to-sequence models starting from RNNs and ending at Transformers.

In 1895, a German physicist was trying to determine if he could observe cathode rays escaping from a glass tube and noticed an unexpected glow on a fluorescent screen several feet away. On further examination, it turned out to be a different kind of radiation that we now know as X-ray. Fast forward to today and you can’t even imagine diagnosing many medical problems without an X-ray. 

There are many instances where someone was looking to study one thing and accidentally discovered something completely different and revolutionary — like the microwave oven or penicillin.

A similarly accidental discovery is shaking up the modern data stack. I’m talking about a language for modifying data queries. While it doesn’t quite belong in the same class of revolutions as the above list, it is one of the most powerful enablers of self-service analytics. 

What is query modification?

Before we get into what a query modification language is, let’s level set on what query modification is and why it matters.  New data often leads to new questions. For example, if you look at weekly revenue and you see a trend that you did not expect, you may want to dive deeper into an anomalous point. Often the query to dig deeper is a derivative of the old query that was used to get the first piece of data. For example, in the case above all the metrics and filters remain the same, you just add one more filter and change the group-by clause to how you want to drill. Query modification is the act of modifying the original query to a new query to answer the next exploratory question. 

What is a query modification language?

Imagine you or your visualization tool has written a SQL query to compute total revenue. This may look like this:

Now suppose you want to break this number down by product category. The new SQL you need is going to look like this:

We typically rewrite SQL queries to match our intent. 

But imagine a system that could simply take the command “Drill down by product_category” and modify the first query into the second. You can probably think of 10 or so common ways of modifying a typical query. What if there was a language that allowed you to express all of them? Or even better, you were allowed to compose these commands arbitrarily to keep modifying the query? That is the idea of a query modification language.

A query modification language, or QML, provides a set of instructions that can modify a data query in specific ways. 

In the rest of this blog, I won’t talk in terms of a QML for SQL, but a QML for ThoughtSpot’s search language but the same ideas can apply to any data query language.

For example, suppose you have a query that gives you “top 10 products that produced the most revenue in last year”. Now suppose you want to change the query to consider only the products sold in the USA. A QML will have a phrase like “Add Filter Country = USA”. This of course is one of the simplest examples. But once you consider all the different ways in which a user may often want to change a query, it becomes a fairly rich set of operations. A common set of operations we see users doing are drilling down into a number, showing specific rows that make up a specific aggregate number on a data visualization, excluding specific values (for example null), and changing the sorting order of a query or sorting direction.

In the search above, we have filtered our simple online spend search results for region equals California.

What are the advantages of a query modification language?

The advantage of having a query language coupled with a query modification language is that it becomes a way to do unconstrained exploration. From any question, you can ask a related or follow-up question. And when that inevitably sparks another question, you can keep exploring until you actually get to the insights you are looking for. This entire cycle happens nearly instantaneously, with only seconds between successive queries. 

This is the fundamental reason why we say Dashboards are Dead, and modern businesses run on Liveboards. Dashboards serve you answers to questions that were premeditated months ago. There’s little freedom to ask the next question. Liveboards, on the other hand, get you started in the right place but don’t box you in. The entire neighborhood, including any and all exploratory paths, are accessible to the end-user. 

We have seen the majority of data organizations are stuck in a paradigm where every time new requirements show up, the data teams get busy building new data pipelines and new dashboards - after the user has submitted a JIRA ticket. Too often, the requirements have already become dated before these dashboards are ever delivered. For a Fortune 500 company, it is not unusual to be maintaining tens of thousands of dashboards without even knowing which small fraction actually delivers value. As some data leaders have correctly identified, dashboard producers care more about the dashboards themselves than the people who consume them.

Liveboards introduce a new paradigm for building analytics that drastically cuts down on the busy work for data teams while giving business people what they need from their data.

Discovery of QML

Our initial vision for ThoughtSpot was to build a product that empowered anyone, regardless of their technical skills, to ask data questions in the vocabulary they understood and get the correct answer. We originally did not think about users modifying and refining their queries, other than just editing their queries in a manner that someone would edit their Google search. We were very focused on creating an experience that looked and felt like searching on Google, yet behind the scenes, was generating SQL queries and visualizing data. 

However, we very quickly realized that in real life, data questions are fairly complex and hard to express precisely in natural language. There are typically too many ambiguities that can become tedious for someone asking the questions (although it does make a compelling demo!) When you are building an application that will be used to inform business-critical decisions, it’s critical to limit any potential misinterpretation of the user’s intent.

Armed with these insights, we changed our goal to design an interface that is as easy as using Google and does not require any training, yet works in a very deterministic way without making any probabilistic assumptions. This led to an approach where we built a platform that in effect is a factory for generating Domain-specific languages (DSL) based on your business language and entities relevant to your business (product names, customers, categories, etc.). 

Looking back almost 10 years now, this turned out to be one of the most important and fundamental choices behind ThoughtSpot’s wide adoption.

Building in interactions

Two of the most fundamental interactions for a BI tool are filtering and drill-down. Filtering is seeing the same data visualization but restricting input records to an area of focus, such as a specific time period or specific region.

The search above applies filters for both time (last 24 months) and place (North America, APAC).

Drill-down is expanding on a specific statistic. For example, if you are seeing “top 10 products by revenue” you may choose to drill down into the revenue coming from the top product by asking how revenue from the top product breaks down by region. Most BI products require the dashboard author to pre-program the columns on which a consumer of the dashboard is allowed to filter or the paths along which they are allowed to drill down.

In the interaction above, we drill down on our top selling product type this year to uncover how sales for that product type break down by region.

Of course, designing a tool for business users, we did not want to constrain the consumer of a dashboard to be limited to the filters and drill paths configured by the author of the dashboard. We also did not want any additional effort in enabling any new interactions.

Keeping search and answer in sync

One of the product design points we debated heavily was when someone types a query, and then they modify the query through a UI interaction, what happens to the search query? 

There are three options:

1. The user goes into a new experience where the model of interaction is no longer search

2. The query text remains the same (perhaps greyed out to indicate that it does not match the answer)

3. Query changes to match the new answer

The first option would have been the easiest answer, but it would reduce the experience down to any other traditional BI product after the first UI interaction, so we never really considered it.

The advantage of #2 was that it is a much simpler system to build and for an end user they are always looking at a query they typed as opposed to something system generated that now they have to try and understand.

The advantage of #3 was twofold. First, the user can seamlessly go from modifying the query through UI interaction and then modifying the query in the search bar, and back. All the interactions would make sense. With approach #2, there is a danger that you may ask for “Revenue by state”, then drill down into California by category, and then add the filter “last year” and expect the result to be “revenue for California in the last one year by category” But actually it will be “Revenue by state for last one year”. This is avoided by #3. 

The second advantage of this approach is that as the query changes through UI interaction, it’s a way for end users to discover different ways of asking the question in the search bar.

Between #2 and #3, there was a strong debate. Initially, we did not know how to modify the query in every context in a way that didn’t create too much cognitive overhead for the user (reduce the edit distance between old and new query as much as possible while getting to the semantics of the new query). Prototyping and building approach #3 took two years of iteration and refinement to create something that was consistent, usable, and always correct.

When a user did a drill down or changed the filter in the UI, we wanted to keep their search query in sync with the answer so they could subsequently continue modifying the search query.

After the search above, we drill down twice. Each time, the search bar updates to represent the search behind the data we are visualizing.

Enriching interactions

What started out as enabling three interactions eventually expanded to a substantial list of query modifications. 

Below is the subset of the most important interactions we added to our interaction model: 

• Adding a filtering widget and modifying filter values

• Drilldown in any direction

• Exclude/include a certain data point from the data

• Comparing metrics period over period

• Comparing different cohorts

• Show the most granular data that makes up a specific number on the visualization

• Add/remove/edit sorting columns and sort direction (ascending vs descending)

• Adding/removing a having filter (post-aggregation filter)

• Change time bucket granularity (monthly -> daily) for date-time columns

In building these interactions, it became clear a lot of the query change intent could be described as a combination of more primitive operations. 

1. For example, drill-down means you drop all the other measures in the query other than the measure of interest

2. Add filters that represent the specific data point you want to drill down on 

3. Remove all existing grouping in the query 

4. Add grouping for the column on which you want to drill down 

This led to a mechanism of combining a sequence of primitive operations into a macro query modification operation. This allows for much more complex operations, such as comparing two data points on a visualization to see which segments changed the most contributing to the overall change.

Birth of a query modification language

Once we had built a large subset of the above interactions, we realized that what we were building was a true companion language to our query language, which can be called a query modification language, the first of its kind. This language became the foundation of many powerful capabilities.

Note that the QML described here complements our own ThoughtSpot Modeling Language (TML) but is a very different kind of language in the sense that users are never exposed to a programming language (other than when they invoke APIs) and it is exposed purely as a visual interaction language.

Putting it all together in Liveboards

The query modification language is the foundation of many of the powerful capabilities that make Liveboards so different from dashboards. The QML sits outside of the search bar and gives users two ways of interacting with the query. Many times QML can be a way that someone can get to a complex query by breaking the steps into fairly simple modifications. 

Most dashboards are designed based on requirements given by business users and are mostly static in their functionality based on the requirements known at the time of authoring. In contrast, Liveboards use the visualizations only as a starting point for data exploration. When a business user looks at the metrics and KPIs on a Liveboard, they invariably generate new questions. The answer to those questions generates new questions. 

Having a system based on this QML means that the business user can get to all those questions within minutes instead of waiting on the data team for every new question.

While dashboards typically provide users a static of their data, Liveboards are simply a starting point. In the interaction above, we dive into Explore mode, select a filter recommended by ThoughtSpot’s AI, and then drill down one more level to get to an entirely new answer.

As a business user with all the context in your brain, you can get to the right question to ask your data to truly find insights and add value. Many times you arrive, at this question by asking a lot of questions.

With QML as the foundation layer, Liveboards are the perfect tool to answer any data question.

PS: If you want to see this in action with a real user, this Youtube video has our CFO describing how he uses Liveboards 20 minutes into the video:

This title has the potential to offend two very different groups. Before you take out the pitchfork, let me clarify two things:

1. For SQL enthusiasts: Our collective SQL skills are still going to be very valuable for at least the next decade.

2. For AI enthusiasts: Advances in deep learning and in particular the recent larger transformer models (e.g. GPT-3, T5 ) will have a dramatic impact on the industry, just not in the ways most people think.

At ThoughtSpot, we have spent the last decade building an AI-powered analytics engine that sits on top of SQL systems. We have also built one of the fastest SQL processing engines (we built a distributed in-memory database for our customers that are not in the cloud yet) so I am both a huge AI enthusiast and a SQL enthusiast. I have also spent a lot of time figuring out what today’s AI can and cannot do in the data space. This blog is a distillation of learnings from that decade-long quest that should be helpful for the analyst community in understanding what may be coming in the near future and picking what you could prioritize in skill-building.

The summary is that today AI only works in very narrow and well-defined problem spaces. Also, it is very hard to use AI in domains where tolerance for failure is relatively low. As a result, in the data space, AI is primarily useful in either micro-decisions with low stakes (e.g. ranking search results, ranking ads, product recommendations) or with human-in-the-loop products1.

Where AI struggles

Here are some things that are going to be hard for AI to do in the near future:

Adding missing context to data

Data in itself is quite useless until you put it in the real-world context. For example, what do different tables and columns mean, how to interpret data, how different tables are supposed to join is mostly locked inside people’s brain. A column could be named stripe_revenue, but the fact that it represents monthly recurring revenue (MRR) may be missing. There can be 3 different columns called customer in different tables and they may represent a customer in different contexts, but which one to use again requires a lot of contexts. If you try and use AI to extract this kind of knowledge, your best bet would be to use NLP on documents, code comments, or conversations like in Slack channels. But the technology is no way mature enough to come even close to automating this kind of task, and any attempt to do so will produce unreliable information. In the foreseeable future, someone needs to own building usable, well-documented data models for analytics to happen.

Inferring underlying causal structure and processing behind the data

To do a good job of analyzing data, you have to understand the process that generates the data and how it relates. In the absence of knowledge of causal structures in the data we may see:

1. The algorithms miss an important relationship that may be somewhat obvious to a human mind because they understand the causality.

2. The algorithms may pick up on correlations that have no causal link or significance to the end-user, causing noise that drowns other significant insights.

3. The algorithms may not know the significance of a pattern in the data because the relationship between the observable variables in the pattern and the variables that the end user really cares about may not be known. So it becomes hard to rank different insights based on their significance to the end-user and surface the important ones. 

For example, ad impressions and Google searches may cause people to visit your website. Visitors on the website navigate from page to page and may transact. These transactions are dependent on users liking what they see and inventory being present. Without understanding all these underlying processes, an AI algorithm won’t understand that you are losing potential revenue and wasting marketing dollars by advertising for a product that is out of stock. 

Breaking complex problems into smaller problems

As human beings, one of our most powerful problem-solving tools is our ability to break a complex problem into multiple simpler problems. Take software engineering for example. In some ways, all they are doing is dividing and conquering until they get to primitives that the underlying library or hardware can solve for. There is some evidence that AI can do this in controlled settings. The most famous example in a widely-used product is perhaps the Excel Flash-fill feature, but it is limited to fairly simple functions. Recently DeepMind caught many people by surprise with AlphaCode which performed better than roughly 45% of human participants in a coding competition. While this is impressive, it still works in a very narrow domain of problems with lots of training data on those problems. 

In the context of analytics, breaking a problem into pieces may mean breaking cohort analysis into sub-queries, required sub-queries, or first transforming data into an easy-to-analyze data model and then doing the analysis. Even if we look at the most state-of-the-art AI, we won’t be able to automate such things for several years. 

Adding context to data questions

This is an area where I have spent a lot of time exploring. I’ll give a few motivating examples first:

1. We were talking to an airline and they have two important metrics: A0 = “Average arrival delay for a flight segment” and D0 = “Average departure delay for a flight segment.” When someone asks what A0 for DFW means, it means A0 where Arrival_Airport = DFW. Alternately when someone asks What is D0 for DFW, it means D0 where Departure_airport = DFW. This context is very specific to that customer and you cannot learn this from data available in the public domain.

2. One customer that represents a large fraction of global corporate travel asked, How many customers are in New York today? This means that their departure date was before today, arrival date was after today, and New York was the destination of their travel. Their data model contains “New York” in close to 20 different columns.

3. In a database of movies, someone asks what is the longest movie ever? To answer the question, you need to understand that longest in this question means duration. This is something that a model that has a general understanding of the world should be able to figure out. Below is a pretty reasonable answer from GPT-3. GPT-3 is acting more like a search engine here and extracting the right phrases from its training but with some effort, it can be inferred that “longest” means duration.

   

Our experience with this class of problems is that most can be solved with >90% accuracy using ML. But if you try to go from arbitrary natural language to SQL directly, it often requires getting multiple disambiguation problems right, and it is hard to do it today with greater than 80% accuracy. In an analytics product, this kind of accuracy is not acceptable. This is why at ThoughtSpot our search engine avoids any probabilistic inference when interpreting a query.

Where AI succeeds

Now that we have talked about all the difficulties, let’s cover how AI, when used with the right combination of systems and UX, is a really powerful tool in the data space. It is already having a transformative impact on many data teams.

Making low stakes repeated decisions

If each individual decision does not have a large impact and needs to be done millions of times, it is a perfect candidate for AI. This is where most of the enterprise ML sits today whether it's deciding what ads to show, what products to recommend, or which transactions to flag as potential fraud.

At large enterprises, this work cannot be done with a human in the loop.  Instead, it was traditionally done with heuristics, and machine learning models can definitely improve upon these decisions. With enough scale, this can have a tremendous impact. During my five years at Google, my small team of ML engineers contributed close to a billion dollars in incremental revenue through feature engineering and ML improvements. While a lot of this work was generating hypotheses and improving algorithms, a big part of the work was analyzing. Every hypothesis about how the predictions could be improved could be quickly validated or invalidated by running analytics on the prediction errors. In that sense, I feel it is great for analysts to expand their understanding to at least the basics of ML. Because a good analyst can be an extremely valuable part of an ML team.

Democratizing the wisdom of the crowd

Have you ever wondered what makes Google so smart that when you search for pain in the bottom of my foot it gives you results about plantar fasciitis? There are many things at play here, but the primary source of this intelligence is us, the users. Some users somewhere queried “pain in the bottom of my foot” and followed it up in the session by a query on “plantar fasciitis” and then clicked on the result. This allows Google to learn the association and help the rest of the billions of users when they need the right answer.

The same idea works for analytics as well. When you learn from users and use that to give personalized recommendations to each user, it dramatically improves their experience and reduces the level of difficulty for them to get the right answer. There may be three different definitions of revenue in your data model, but usually, there is one that gets the most use by people in similar roles as you. When you ask for “Closed bookings this quarter” Usually This quarter maps to the “close date” column, while when you ask for the “pipeline created this quarter”, quarter maps to the “creation date” column. These kinds of recommendations in an auto-completion setting become fairly easy to make if you have a Machine Learned model helping users.

Exploring a large space looking for patterns and anomalies

Suppose, at the end of the month your leadership team is doing a business review and someone says that compared to last year we made 10% less from electronics sales this year. Why would that be? All of a sudden a lot of eyebrows are raised, hypothesis after hypothesis flying in the room. Maybe it's the pandemic and people facing hardships are not buying as much as before. If that is the case, we should see some zipcodes drop a lot more than others. Maybe it is that people are not buying more high-end gear as much as they used to. If that is the case, some price buckets should have much lower revenue than others. After a lot of stress, and a lot of calls to the analytics team, you figure out that it was a lack of inventory for specific products supplied by a specific supplier. This doesn’t need to be a manual process. After the right data model has been built by an analyst and the ML algorithms have learned enough from the history of users asking questions, AI algorithms can do a much better job of searching through hundreds of auto-generated hypotheses looking for a root cause.

This kind of automation works not just for monitoring and root cause analysis, but for all kinds of interesting insights. I have seen manufacturing companies save millions of dollars by finding price discrepancies between suppliers, banks save millions of dollars by letting automated algorithms search for lost claims guided by machine-learned systems. One of the most amusing examples of this was that one of our customers was able to spot that sales for fidget spinners were growing fast way before most people realized that it was a trend using automated insights back in 2017.

Language modeling over domains with a large amount of data

Large-scale language models are a really powerful tool. The most well-known model in this class is GPT-3, but every few weeks we do significant research advancing the state of the art in this space. Building analytics products out of these models is still too early. Some obvious uses are 1. Auto-completion for a SQL/code editor, 2. natural language to code translation, 3. natural language generation to describe insights in the data 4. Conversational data apps to reduce UX complexity. The biggest barrier I see in this space is that if there is any missing context in the input, then it is hard for language models to fill it. Also, if you are trying to model a language with little training data, you can try transfer learning and it can do non-trivial things but building a usable product here will still require either a lot of hand-crafted systems or a little more maturity in language modeling. 

Progressive disclosure of complexity

Even though AI today is imperfect, one of the best ways to deploy it in products is to support it with UX. One of the key design principles for us here at ThoughtSpot has been less input more output or LIMO. This means we want the user to do as little work as possible while getting the most value. 

Analyzing data, creating visualizations and building data models is all input-intensive work. We try to eliminate as much input (and intellectual burden) as possible by predicting at least part of the input and making an explicit choice for the user. If we get it right, the user can simply move on. If we don’t get it right, the user can edit those decisions. This is one of the big reasons why ThoughtSpot allows thousands of non-technical business users in hundreds of the world's largest enterprises to be more data-driven with every decision.

The impact of AI on a modern data analyst

Traditionally, the analyst job has consisted of preparing and analyzing data to answer relevant business questions and communicating those insights back to stakeholders so they can take action. For the first part, a lot of automation is making analysts 10X more powerful. The intellectual parts are not going away, but the elimination of rote tasks will enable analysts to do much more high-value work. It is also creating more room for the analyst to focus on impact and communication. In my experience, this often leads to faster promotion for analysts. As an analogy, we are not deploying self-driving cars which would eliminate cab driver jobs. We are upgrading horse carriages to automobiles, which means a lot less dealing with manure and going much faster to longer distances.

Based on what I see in the near future, here is my advice to anyone in the analyst role:

1. Say no to repetitive data pull requests: Even though your company may need it today, it is detrimental to your career. The time you spend on these requests is all time you are not spending learning modern data stack tools that you’ll need in the future. Occasional requests as exceptions are okay, but if your employer is not willing to invest in the right toolset it’s a red flag for your career growth.

2. Be the founder of your analytics community: Try and include as many people in analytical decision-making as possible. Educate people on how to use the data models you have built. Educate your organization on data models, interpreting data, and self-service analytics. Educate people on how to make better data-driven decisions. The bigger your community, the more impact you will create. As dbt’s Erica Louie puts it, scaling knowledge > scaling bodies. Things that I have seen work well in this regard are:

   • Holding regular office hours

   • Maintaining well documented and searchable data models

   • Creating tutorials

   • Hosting internal or external meetups for non-data teams to show off their expertise and share problems  

3. Do more engineering: Learn to automate as many things in your job as you can. If you like, learn python or some other programming tool. You can also consider learning low-code tools for automation and powerful abstractions on top of SQL such as TML or LookML for analytics, or dbt for data pipelines.

4. Become the product manager for analytics: Most of us engineers pride ourselves on being great problem solvers. What I have learned over the years is that there is a lot more leverage in asking which problems are worth solving and what the end-user actually needs. In the analytics domain, this largely means three things: 

   • Ruthless prioritization: Prioritize work that will generate long-term value, not the urgent request coming from someone with a title. This requires a mature leadership for data teams.

   • User empathy: The end-user is not always right. If you keep adding new things to your data model based on in-the-moment requests your data model will become completely unusable. Instead, it’s better to understand user needs and synthesize them into a coherent product definition that is usable.

   • Communication: Being good at communicating complex ideas, influencing opinions, and building consensus are some of the most valuable skills in most careers but are especially important in the analyst role for impact.

5. Learn the basics of machine learning: The future of data will contain more and more machine learning and it's a useful adjacent skill. Additionally, a big part of improving machine learning models is analytics. If you have a business-critical ML task, you need to be working on potentially hundreds of iterative improvements to the model. This means you need to analyze the error, build a hypothesis on what are the missing features, test those hypotheses with some more analytics, train new ML models, and then analyze their results to establish if you have improved things in each iteration. Even if your primary job is not ML, your analytics skills can be extremely useful to an ML team. 

Own the outcome: There has been some debate about whether data teams should be thought of as supporting teams or not. I am a big believer in data teams owning outcomes instead of just having a supporting role for the execution team. For example, your OKR could be helping the Customer Success team reduce churn by 50% or help reduce wasted AWS resources by 30%. This is much more impactful than an OKR like delivering self-service analytics to the Customer Success team. But it requires a level of alignment with the operational teams. It also requires data teams to be able to influence the behavior with data. Accountability without authority can be challenging, however, product managers will tell you that this has always been a part of their job. This is one of the best ways to get yourself a seat at the executive table and put yourself on a fast growth trajectory.

The role of data professionals isn’t going anywhere. Data has never been more important to businesses than it is today – and this importance is only growing in the years ahead. What will change, however, are the skills analysts need to help their organizations build and thrive

When was the last time you thought about optimizing your analytics toolset for curiosity? Yet what is the value of all the data and analytics in the world if not paired with human curiosity?

If you ask most organizations what is important to them in terms of the data stack, usually it is things like moving to the cloud, getting clean data, building dashboards and reports to serve the execs running the business, security, and governance. Cultivating curiosity hardly ever makes it to the top of the list. If you take a moment and think about why that is the case, it is usually because no one considers it their job. Clearly, a lot of leaders consider data strategic and one of the most important tools for their business, but why does that not elevate curiosity as a concern? My guess is that curiosity is just assumed to be present, similar to qualities like motivation and ambition. 

To some extent, that is true. All of us are born curious. A five-year-old asks around 300 questions a day. But then something happens, and we start asking fewer and fewer questions to the extent that most grown-ups rarely ask questions that could surprise them. Most knowledge workers and leaders aspire to be curious and see themselves as someone open to new data, but their curiosity is poorly supported both culturally and technologically.

Everything that has ever been invented or discovered or improved almost always originated from a curious mind. And while curiosity is hard to quantify on its own, there are many examples of the relationship between curiosity and better business outcomes.

1. When St. Jude children's research hospital, one of the largest charitable organizations in the US for children's health, allowed their fundraising team to explore their curiosity around their fundraising data, they realized that events within two miles of a Whole Foods store tend to produce much better results than anywhere else.

2. When one of the leading companies in the Insurance and Banking industry in Australia, adopted self-service analytics for their business users, people started asking a lot of questions they would have never asked. One of these questions around insurance claims data led to the discovery of an anomaly that saved the company 30 million dollars within months of deployment.

3. When one of the investment funds in Canada enabled their traders to ask their own data questions, within an hour of training one of the traders found they were being overcharged for securities lending by another bank to the tune of millions of dollars.

4. When one of the largest technology companies in the US enabled their accountants to ask data questions at will, within a few days they discovered misuse of travel policy that was leading to loss of millions of dollars. 

While these are just a few anecdotes, I hope you can see how enabling people to be curious about data can result in substantially better business outcomes.  

So, what can you do to cultivate curiosity in the context of data and analytics? To quote James Clear, the famous author of “Atomic Habits”, albeit a bit out of context,  "You do not rise to the level of your goals. You fall to the level of your systems." That is to say, the biggest lever you have in terms of cultivating curiosity is deploying the right data stack. So, here here’s how you can ensure your data stack is built to cultivate curiosity:

Reward curiosity instead of punishing it

Imagine you are managing the inventory for a very large retail chain in the middle of a pandemic. One week your customers are asking for exercise bikes in hordes, the next they are interested in stockpiling toilet paper, and the week after all the rage is sanitizers. You are curious which product is rising in demand now so you can gather as much inventory as possible for the following week. For most folks, the first step to answering this question is consulting a dashboard. But perhaps you find that to manage the scale of data, your data team has aggregated transactions to daily and broad product category level instead of SKU level or sub-category level. That means if you want granular data, you have to submit a data pull request, which sits in the queue of other requests, and there is a wait time of two weeks or more to get the answer. You are not even sure if your request will yield meaningful insight, so you feel bad piling on one more request when everyone is frantically trying to get their data. 

Compare that to an experience where you can drill down into whatever slice of data you need and ask questions in a Google-like interface and you are rewarded with instant insights. Which experience will encourage you to be more curious? Which one will bring better results? For Canadian Tire, the choice was clear.

“During much of the second quarter, Canadian Tire Retail was forced to temporarily close or operate in a limited capacity about 40% of its stores. Still, Canadian Tire Retail’s sales were up 20% from a year earlier”

Grant people permission to be curious

Another reason people often curb their curiosity is because they don’t feel empowered to explore it. “Curiosity killed the cat” is often the refrain from frustrated adults to a curious child who has exhausted the adult’s patience or explored something dangerous. While well-meaning, over time children feel disempowered and stop asking questions. The corporate equivalent of that is, “if I allow everyone to ask their questions, they will reach wrong or biased conclusions”, or “when everyone has their own version of data, no one can trust the data.” 

These concerns were absolutely valid in the older generation analytics stack when everyone dumped their own data in excel with different filters and different metric definitions. This even gave rise to the term “excel hell.” When your analytics tools couldn’t handle more than a few million rows, you had to work with Tableau Extracts and PowerBI reports, and it was absolutely important that these things were tightly governed.

But with the modern data stack, you can leave data in granular form in a cloud data warehouse, define your metrics in one central place1, and let everyone do their own analysis without worrying too much about having different versions of the truth. Both granular security and centralized definitions without locking down the grain of aggregation is very much possible with the right stack.  

Invoke curiosity through adjacency

We have all gone from watching one clip on YouTube to looking at a cat playing a piano an hour later. Less often, we have also gone through the Wikipedia rabbit hole where you go to read the page of your favorite author (P. G. Wodehouse) and an hour later you are reading about different kinds of coffees (If you must know, this is the path I took Code of Wooseters -> Cow Creamer -> Espresso based drinks). When you are looking at one concept, you are naturally curious about all the adjacent concepts. If the adjacent piece of information is easily accessible in context, it is natural for people to let their curiosity go wild. There are many ways of incorporating this into your analytics stack. Often people hardcode links to related analysis in their data visualization. In ThoughtSpot, you can drill in any direction possible given the data model. Also, the explore feature uses machine learning to collect the next best data question in that context, personalized for the user and made accessible with one click.

Make curiosity a social phenomenon

We are social beings (some more than others.) When your friends or people you respect are looking at something cool, it is hard for you to not be curious about it as well. This is why we saw significant engagement with analytics when we started showing people who else has looked at the answer to a particular question or Liveboard. Having a social feed of activity and commentary is another idea that we have often played around with to evoke curiosity.

Surface the knowledge gap

When our curiosity is piqued by teasing something we don’t know, it usually has immense power over us. That is why it is so hard to resist those damned notifications on the phone or why it is so easy to grab someone’s attention by starting a sentence with, “Did you know…”

At ThoughtSpot, we are working on a feature called automated business monitoring where a machine learning algorithm is constantly watching the metrics on your behalf and invites you in to dig deeper when one of the metrics behaves in unexpected ways along any of the dimensions you care about.

Parting thoughts

While these ideas largely put the onus on tools and organizations to change behaviors, ultimately it is up to each of us as individuals to keep our own curiosity alive. Staying curious means being fully present, willing to admit when you don’t know something (or when you’re wrong), and having discourse with people we may not agree with. It is not always easy but it is one of the most rewarding things both here and now and in the long term.

Cognitive neuroscientist, Matthias Gruber, in this talk, explains that when they studied brains under fMRI after arousing someone’s curiosity by asking a trivia question, they found it looks very similar to someone anticipating a reward such as a treat. It makes perfect sense, because what is sweeter than the discovery of a new insight from the question you just asked.2

Recently there has been a lot of excitement around the idea of a stand-alone metrics layer in the modern data stack.

Traditionally, metrics have been defined in the BI or analytics layer where various dashboards are used to look at business metrics like Revenue, Sales Pipeline, numbers of Claims, or User Activity. Given that most organizations end up with multiple BI/Analytics tools, the idea has a lot of merits. Instead of pulling data into excel sheets and expecting everyone to calculate metrics independently, why not define the metrics in one central place for all to refer to? In software engineering, it is a classic candidate for refactoring the repeated definition of the same logic and following the principle of “Do not Repeat Yourself” DRY.

However, refactorings are not always easy. If you don’t know the full scope of how your code is being used (or going to be used) you can end up with a lot of empty promises and entangled code. When you are refactoring for eliminating duplication, the trick is to make your abstractions clean enough that the common parts can be expressed in one place, but the parts that vary in different places can be supplied as additional details as needed. For design pattern enthusiasts, this is probably closest to the Strategy Pattern. The same can be said for the metrics layer.

Having deployed ThoughtSpot in hundreds of enterprises and startups, we have learned a lot about what different businesses need from their analytics solutions in different use cases. Whether banks reducing fraud, retailers optimizing inventory, manufacturers improving the supply chain, hospitals enhancing outcomes, or cancer researchers understanding their data, the metrics people care about in the real world are quite varied and require careful design. 

The six metrics classes 

1. Simple aggregations: These are things like Sum(Revenue), Average(Price), Count_Distict(Users). They are really easy to define and an isolated metric definition layer will do a good job of defining these. 

2. Aggregation with scalar functions: Similar to simple aggregations above, but with additional mathematical operators. For example, Profit may be defined as sum(Revenue) - Sum(Cost) - Sum (commissions). Alternatively, you may need some transformations at the row-level such as Sum(Revenue_in_local_currency * USD_Conversion_rate).

3. Metrics that require joins: The simplest example can be a version of Revenue calculation where conversion rates change daily and you have another table that stores conversion rate in a dimension table. Now Revenue definition becomes Sum(Revenue_in_local_currency * conversion_rate(local_currency, transaction_date)). Here conversion_rate(local_currency, transaction_date) is a column from another table that needs to be brought in by joining on local_currency and transaction_date from the fact table. You could bypass the join requirement by defining a denormalized view, but views bring in other issues such as requiring all joins to be executed.

4. Metrics with window functions: Things like Moving Averages, Cumulative Sum, or any kind of aggregation in data that represents a time-series or sequence of events that you need to aggregate on a window around current time falls in this group.

5. Metrics with multiple aggregation levels: This is a special class of metrics that often represent ratios. For example, if you want to define the Market Share of a product in its category, you first have to sum up Revenue grouped at the Product level, then you want to sum up Revenue at the Product Category level, combine the results and then compute the ratio. In addition, if you want to observe Market Share across years then you need to include Year(transaction_date) in the grouping column both for numerator and denominator. So the grouping becomes dynamic. These metrics are handled differently in different BI products. For example, this would look like a Group Aggregate Formula in ThoughtSpot, in Tableau, they are called Level of Detail (LoD) Functions.

6. Multi-fact metrics: Sometimes, your metrics span multiple fact tables that may or may not have a direct relationship with each other. For example:

   1. Sales Fact and Bulk Purchase Fact for a retailer to calculate Profit Margin.

   2. Services Provided, and Insurance Coverage for a hospital to calculate the profitability of different services.

   3. Bank Statements, Credit Card transactions, and Credit History for a Bank to figure out default risk.

   Back in 2015 when we started looking at these kinds of metrics, to our surprise we found that they were not handled correctly in any of the BI tools we looked at (and I believe that is the case still). Most BI tools default simply to joining the tables and aggregating the rows of the joined table, which in the case of many-to-many join would cause the same numbers to be double-counted and give incorrect results. Getting these metrics requires being able to aggregate different fact tables to a granularity that joining the results makes sense and then post join, re-aggregate them. Legacy BI tools have two major issues here. They either couldn’t get to the level of granularity of data needed or required customer SQL, which is hard to maintain and limits the level of interactivity that can be built. Nevertheless, they do cover an important segment of metrics that a lot of businesses care about.

Where to define metrics

As you can see, defining metrics is not always just a matter of defining a mathematical formula or stand-alone SQL fragment.

If you look at a typical analytical tool, you can view them as a three-part stack: logical modeling (Semantic Layer), query generation, and interactive data visualization and exploration.

The logical modeling layer is where you start assigning business context to raw data that exists in tables and columns. The purpose of defining a logical data model is so non-technical business users can express business questions in business terms, read charts and tables more easily, and interact with data visualizations.

For example, you may start at “Top 10 products by revenue” and then drill down into the revenue generated by the top products by country. Typically, the kinds of things that you specify in a logical data model are:

1. Business names for column (rev_txn_usd -> Revenue)

2. Whether a column is a Metric (Measure) or Dimension (Attribute). For example, Revenue is a Metric, but Age or Customer Name is a Dimension.

3. Joins: Exactly how to join tables (which pairs, join based on what join conditions, and whether it’s an inner join, outer join, or some other kind of join) and whether it represents a many-to-one, one-to-one, or one-to-many relationship.

The query generation layer is where user intent is transformed into the appropriate SQL or equivalent to generate the data visualization or answer the user’s questions. If you think of BI as just a way of slicing and dicing metrics in a denormalized table, then it can be a pretty simple thing to do. But in reality, this component needs to cover a lot of ground and do a lot of heavy lifting. More on this later.

Finally, the interactive data visualization and exploration layer is where the business user will ultimately spend most of their time. Whether they’re looking at a dashboard, using drill-down or other exploration interaction models to ask deeper questions, or asking a completely new question through a search-like interface.

Making space for a common metrics layer

If we are going to refactor the stack to make space for a common metrics layer, there are a few choices to make about what part of this stack goes into the refactored component. We don’t want the top layer in there otherwise it will be just another BI tool competing with the rest. That leaves us with three choices:

1. Encapsulate the metric definition

2. Encapsulate the entire semantic layer

3. Encapsulate both the semantic and query generation layers

There have been many proposals around all three versions of a metrics layer, each with its own unique tradeoffs. Moving down the list from one to three, there is increased complexity both in implementation and deployment - but there is also potential for an increasingly more powerful combined stack. 

Possibility #1: Encapsulate the metric definition

This is good for metrics that are simple aggregates with scalar functions (type 1 and type 2), but will not be able to handle anything else. In my experience, this is where most of the implementations of the metrics layer are (I have not looked at every implementation, but I’d love to be surprised).

Possibility #2: Encapsulate the entire semantic layer

This gets a lot more interesting. In this case, the metrics layer would not just be aware of metrics, but also, tables, joins, and dimensions. This allows you to have both complete data models and the ability to define expressions across multiple tables. You can also define metrics that may traverse multiple grains of aggregation. This would be a huge step forward in defining metadata in a common layer. 

However, this requires having a standardized common modeling language that can be understood by different consumers that can encapsulate definitions of joins, metrics of all types above, and dimensions. It also leaves the query generation layer to be implemented independently in each of the consumers.

Possibility #3: Encapsulate both the semantic and query generation layers

In my opinion, this is the most powerful iteration of a metrics layer. Most BI tools today differ heavily in what type of metrics they can or cannot support. There are also subtle differences in how they generate queries for different types of metrics. In an ideal scenario, the metrics layer will encapsulate query generation as well. This however creates a new requirement: defining a new standard interface between the interactive visualization layer and the query generation layer.

It’s easy to think the query layer should be SQL, but I don’t think it’s up to the challenge. SQL is designed to be precise in how things are joined, in what order, and where the aggregation happens. What we need is a language that expresses a business user's intent, and combines the intent with predefined data models to generate queries where the same data model could generate very different queries with very different join structures, a different subset of tables, and different ways of aggregating measures. 

The future of the metrics layer done right

I think encapsulating both the semantic and query generation layers is the real deal - and finally possible. That’s what gets me so excited about the future. 

We like what dbt is doing here. Starting in dbt v1.0, you can define some types of metrics in your models alongside the rest of your existing transformations. Later this year, we may have an integration where ThoughtSpot can read metric definitions from dbt and integrate that into our modeling layer which will be a win for our joint customers. But this integration does not describe what I am talking about in this post. A rich, independent metrics layer that can encapsulate both logical modeling and query generation, with an open standard for consumption layer is yet to be designed and built. 

If you are excited about this too, I’d love to hear from you! In fact, if you want to collaborate on an open language for defining a semantic layer and a query language that can capture query intent while decoupling the information already defined in the semantic layer, I would love to buy you coffee. Or at the very least have a stimulating conversation!

If we as an industry are going to refactor this stack, let’s do it right. We don’t get too many shots at asking everyone to keep changing how the stack works.