Now let's make some queries to this table:
```sql
SELECT MAX(id) FROM my_table;
SELECT * FROM my_table WHERE id = 3;
```
To serve these queries, we need to fetch a subset of the data above. We need to fetch all the data for one tablet to serve the SELECT \*, and we need to fetch id data for all tablets to serve the SELECT SUM(id). We'd then expect our SSD cache to look something like this:
```sql
tablet1/
id
name
tablet2/
id
other_old_tablet/
foo
bar
...
```
We continuously track when tablets in the cache are used to ensure we handle eviction properly. We can use the data from our evictor to construct a list of recently used tablets and columns on every node in our cluster. We use this data to push a snapshot for our engine to s3:
We continually push an updated snapshot like this to cloud storage while the engine is running. When it shuts down this data will be persisted for up to two weeks. When we then restart that same engine, it will be able to retrieve that snapshot and know which data to fetch.
### Reloading Cache From Snapshot [#reloading-cache-from-snapshot]
With the snapshot prepared above, reloading our cache is now fairly simple. We simply fetch the snapshot if one exists, then load the tablets in the snapshot from least to most recently used until one of a few conditions is met.
* We've loaded everything: this is our best-case scenario, and means our cache is fully restored
* We hit our timeout: by default we're willing to spend only 30 minutes loading the snapshot, but this can be adjusted as needed.
* Our SSD cache is filling up: if you switch from a large node to a small node, or storage optimized to compute optimized, there might be less disk cache available. We abort the warmup if we see the SSD cache is filling up to ensure there's plenty of room to cache data for user initiated workloads.
Since everything at Firebolt runs on SQL, we use queries under the hood to fetch this data. We restrict the number of threads on these queries to keep the CPU overhead of the cache load around 10%.
## Applications [#applications]
What does this feature mean for workloads in practice? Let's look at a few examples.
### Online Upgrade [#online-upgrade]
One place where auto-warmup is extremely helpful is in [online engine upgrades](https://www.firebolt.io/blog/live-engine-upgrades-zero-downtime-the-firebolt-method). When we upgrade an engine cluster, we create and mirror traffic to a "shadow cluster". Using auto-warmup, shadow clusters can be warmed up much faster, since they proactively fetch a recent snapshot of the main cluster's cache, rather than relying entirely on mirrored requests to greedily warmup.
Let's look at cache on a high traffic engine during our 4.23 release.
The new node starts up at 16:20 (blue line), and as it receives queries it starts caching data. After an hour, the engine is upgraded, but its cache still hasn't fully converged with the old cluster it's replacing (red line).
Let's compare this to the 4.24 release where auto-warmup was enabled.
We see the same engine start an upgrade at 13:50. By 13:58, the snapshot is fully loaded, and all queries seen during the duration of the shadow period hit cache.
There are a couple major benefits of this approach:
* The final cache is much warmer. This minimizes the risk of a performance degradation to the customer workload after the upgrade. This also ensures we warmup data that wasn't touched during the upgrade period.
* The upgrade took less time. The first upgrade timed out after an hour, and probably should have been run for even longer to fully warm the cache. The second upgrade finished after just 35 minutes when we saw performance fully converge, and could have been promoted even sooner. Firebolt pays the bill for shadow clusters, so there's no direct cost implication for customers, but this lowers our cost of operating engines, and lets us keep Firebolt affordable.
### User Workloads [#user-workloads]
Auto-warmup is clearly beneficial in preparing shadows to receive traffic, but what about running this during an intense user workload? To benchmark this, I'm using a modified version of the [Firescale benchmark](https://www.firebolt.io/blog/firescale-benchmarks-a-deeper-dive) dataset. The benchmark consists of 50 concurrent requests, each picking one of 100k random point read queries against a 1 TiB table, repeated indefinitely.With no auto-warmup, we see a logarithmic decrease in data fetched from s3 throughout the benchmark, as a higher and higher percentage of queries hits cache, but even after 30 minutes, we're sometimes pulling data from s3 as we serve certain queries for the first time.After recording a snapshot from a fully warm engine and trying again with auto-warmup enabled, we can see the difference in the s3 read pattern. The engine is still under a heavy query load and is greedily pulling data to answer those queries, but it is also proactively fetching data it had touched in the previous run. This leads to substantially elevated read traffic initially, but means we eliminate cold reads much faster.
Over the course of the 30 minute benchmark, the engine with greedy warmup averaged 91 QPS and the engine with auto-warmup was able to average 194 QPS as faster reads led to increased throughput.
### Auto-Scaling [#auto-scaling]
When engines [auto-scale](https://docs.firebolt.io/guides/operate-engines/understand-autoscaling), new clusters in the engine come up cold. Since the oldest cluster in the engine is maintaining an up-to-date snapshot, the new cluster has a recent snapshot to fetch from when it starts up. Firebolt's load balancer routes to engines based on reported load, so we will already avoid swamping slow clusters, but auto-warmup will much more quickly bring the performance of new nodes to parity with old ones.
## Try It Out! [#try-it-out]
If you think your workload would benefit from auto-warmup, give it a try. You can enable it on an existing engine by running.
```sql
ALTER ENGINE myengine SET AUTO_WARMUP = true;
```
# Beyond Database Optimization with AI (/blog/beyond-database-optimization-with-ai)
In this episode of The Data Engineering Show, the bros welcome the CEO DuckDB Labs and co-creator DuckDB, Hannes Mühleisen. They delve into the groundbreaking journey of DuckDB, an analytical database that processes billions of queries every month. Learn why DuckDB prioritizes broad compatibility over specialized optimizations, how its extension model works and the emerging solutions for database technology in the age of AI.
Listen on [Spotify](https://spoti.fi/41C0wJx) or [Apple Podcasts](https://apple.co/43T0wHA)
### Episode Highlights [#episode-highlights]
##### The Purpose of DuckDB (01:04) [#the-purpose-of-duckdb-0104]
Hannes gives a full description of what DuckDB is as well as what it is designed to do. He describes the tool as one that understands SQL and is specifically designed to simplify complex analytical use cases.
##### SQLite vs DuckDB (02:53) [#sqlite-vs-duckdb-0253]
Hannes compares two different tools stating that SQLite is an amazing system that is not meant for analytical queries but for transactional use cases while DuckDB is specifically designed for that exact purpose - analytical use cases.
##### The Importance of Collaboration (08:14) [#the-importance-of-collaboration-0814]
Hannes states the need for community collaboration as the database engine space seems to have hundreds of brilliant people trying to solve the same problems. He shares his profound admiration for a team in Munich, praising them for their exploits in implementing concepts only described in paper.
##### The Component-Based Architecture of DuckDB (11:25) [#the-component-based-architecture-of-duckdb-1125]
Hannes highlights a special feature in DuckDB, that is, it can be used as a component and he explains that the in-process architecture is a success because of the memory of data sharing that can be achieved.
##### The Parquet Reader Journey (17:51) [#the-parquet-reader-journey-1751]
Hannes explains how he built his Parquet Reader out of necessity, although he would have preferred not to. He shares how a creator named Ove Korn from Germany donated the reader to a project named "The Arrow Project" and managed it to the degree that the entire project depended on the use of the Parquet Reader and it became an issue to use both independently. Hannes adds that a parquet reader that is competent has no choice but to become a database engine which is one of the interesting things about development.
##### The Role of AI in Database Interaction (22:41) [#the-role-of-ai-in-database-interaction-2241]
Hannes states that he doesn't think that AI has a place in a database engine but rather, it is needed for optimization because the researchers who built their careers on optimization are out of jobs. He explains that the role of AI should be for assistance tasks and not for a total execution.
##### SQL - A Defined Interface (29:20) [#sql---a-defined-interface-2920]
Hannes introduces us to a tool that allows us to pro-programmatically build a query called relational API stating that it helps to simplify the tasks of a programmer. Although, Hannes agrees that using a well-defined interface is important for components like databases, he also argues that SQL can provide a relatively defined behavior within a single system.
##### The Golden Age of Database (38:57) [#the-golden-age-of-database-3857]
Hannes concludes the episode by appreciating Firebolt and other engineers for taking on core engine tasks. He shares his excitement for the golden age of databases where there is a showcasing of what is possible.
# Bill Inmon, the Godfather of Data Warehousing (/blog/bill-inmon-the-godfather-of-data-warehousing)
As people in the data industry go, Bill Inmon is among the top, often seen as the godfather of the data warehouse. In this Data Engineering Show episode, Bill Inmon talks about surviving rabbit holes throughout the evolution of data, the data modeling renaissance, and why ChatGPT is not Textual ETL.
Listen on [Spotify](https://open.spotify.com/episode/3IvsspPtiYeU57McqDCB9k) or [Apple Podcasts](https://podcasts.apple.com/us/podcast/bill-inmon-the-godfather-of-data-warehousing/id1561927688?i=1000623790403)
**Benjamin:** All right, cool. Hi, everyone, and welcome back to the Data Engineering Show. It's a super exciting episode because we have a total data celebrity joining in today. Bill Inmon, thanks for being on the show.
**Bill Inmon:** It's my pleasure to be here.
**Benjamin:** Awesome. And we also have a special guest as co-host today. Robert Harmon, kind of veteran data practitioner. He's an SA at Firebolt right now and apparently a recovering race car addict. So good to have you on the show as well, Robert. Cool. Do you want to kind of intro Bill? I mean, he really needs no introduction because I think all the listeners will know him, but let's still do it.
**Robert Harmon:** Yeah, it's almost humbling to try and introduce Bill. As people in this industry go, he's obviously among the top, often seen as the grandfather or the father of the data warehouse. I like to think of him as the data OG. He's really the guy that set the stage for the rest of us to have careers. And obviously for that, I'm grateful.
**Bill Inmon:** Robert, I like to think of myself not as the grandfather of Data Warehouse or father of Data Warehouse, but as the godfather of Data Warehouse.
**Benjamin:** love that. Nah.
**Robert Harmon:** That gives it much more of an almost infamous tinge, and I think it's fitting.
**Bill Inmon:** Yeah.
**Robert Harmon:** So welcome Bill, how you doing?
**Bill Inmon:** I'm fine.
**Robert Harmon:** Well, so in preparing for this, of course, I was completely out of my mind because it's like meeting Taylor Swift. I mean, this is huge for me.
**Benjamin:** I'm going to a Taylor Swift concert next year. So it's like perfect, perfect. You're giving the example
**Robert Harmon:** Oh.
**Benjamin:** because I'm super stoked about the Taylor Swift concert and also about today's episode.
**Robert Harmon:** So I'm preparing for this interview and I'm thinking, well, what the heck do I ask Bill Inmon? Because if I ask any of the normal questions, well, then I look like I don't know what I'm doing for a living because why would I be asking these questions? And then if I ask the questions I wanna ask, well, everybody's gonna be lost because we've been doing this too long. So I'll lose all of the new practitioners. And just serendipitously. So, you know, one of my contacts came up with a question that I saw on social media that was, that kind of triggered every, you know, my mindset on where to go. And he asked, and he stated, being one of them, if you started your data career post 2010, you're at a massive disadvantage. And this is one of those things that I've been thinking about a lot offline is that not so much the technology or the products or any of that. more the social ramifications of our industry. And I think that's one of those statements that kind of brings that out. So, I'm thinking about generational issues and sometimes I'm thinking about inclusivity issues because a lot of the guys in our industry look a lot like you and me, Bill. So, I kind of wanted to explore this idea that people before, that joined the industry prior to 2010 might be missing a few things. And somehow this opened it up to an idea of, well, hey, I've got Bill here. I've been around since longer than then. Let's, let's try and educate some of the younger guys on some of the stuff they've missed prior to them showing up. Does that make sense?
**Bill Inmon:** Yeah, let me kind
**Robert Harmon:** What?
**Bill Inmon:** of back up and give some perspective.
**Robert Harmon:** Sure.
**Bill Inmon:** The way I look at things is, what we are experiencing today and what we
**Robert Harmon:** Mm-hmm.
**Bill Inmon:** experienced yesterday is nothing but a big evolution. The evolution really started about 1960, which is probably before most of you were born. But in 1960, the world became introduced to the computer. And at that point in time, there was no one, not one person that had any background. It was all fresh, fresh material. And so what we've been witnessing for the past 50 years or so is an evolution. And I have to agree with the person that asked the question. Um, have they missed something? Uh, the answer is yes. They've missed many of the evolutionary, uh, rigmaroles we've had to go through. Uh, uh, but, but is there still opportunity out there? There is, we haven't even begun to start to explore the opportunity that's out there. So yes. Uh, you, you who have just joined have missed. a lot of these struggles, a lot of the fairly nasty stuff that has occurred in the past 50 years, but is there opportunity in front of you? The answer is absolutely yes. Now if you were to ask me where is the opportunity, there's really one answer and the answer is business value. the people that go and find true business value for their company, their corporation and themselves are going to be the people that advance the people that are the most valued in the corporation. And is there business value out there to be found? We haven't even started on business value. So I look at it from a larger evolution and yes, I agree that people have missed some things. Quite frankly, a lot of what they have missed, a lot of what they've missed has been worthwhile missing. I can recall some really stupid things that people said years ago and people believed and I remember when we were told that... secretaries are going to be writing code. That's one thing that was said. I remember somebody saying, we need to develop application programs without programmers. Now, I don't know how that person thought that was gonna happen, but that was once what we thought. And there's been a long list of things that we've been told that to be... turned out to be totally false, but our industry's gone down. And so some of the things that people have missed have been very worthwhile missing because you didn't need to go through the pain that we all went through. However, having stated that, Whatever you do, if you want to get ahead, if you want to take advantage of where we're at, two words, business value. Take technology, done. Once upon a time, we looked at technology for technology's sake. We can't do that anymore. We've got to look at technology as a way of enhancing business value. And once you understand that, then, then you've lost nothing by joining our profession when you've joined.
**Benjamin:** I love that as an entry question Rob, because obviously I am someone who joined the data industry post 2010. So Bill, like for someone who's new in this space, right? You said, okay, there's kind of quite a few ideas in the past, which really didn't pan out and technology goes in cycles, right? So protect me from having the same ideas basically again and reviving them. Do you want to tell some maybe war stories from, yeah. things that really didn't pan out, but that people were excited by in the past.
**Bill Inmon:** You don't want me to start to talk about war stories because I've lived through all of them. But yeah, there's been some, one simple thing that happened long ago. When we were learning to program, we were told that programming would all be simple and easy if we didn't use go-to statements. Now, that's humorous today, but- But there was an element of truth there that the code that was being produced was really quite fragmented code. And indeed, not using the go-to statement did improve things. But was it the solution? Was it a silver bullet? And the answer is absolutely not. And that's one of many things. This other bit about... Even secretaries can start the program. That was a ridiculous idea that somebody that said that didn't know what they were talking about. And I have nothing against secretaries, by the way. My family has secretaries in the family. But there's a skill set and a mindset that's needed for coding. that you normally don't find in secretaries. And so that's another one. And every few years, our industry comes up with what they call the silver bullet. And I remember when IBM told us, man, if you just go DB2, that your problems are gonna be solved by using DB2. Well, that... that didn't work out well either. And so, and that's being polite about how poorly that worked out. So our industry goes nuts over these silver bullets that said, gee, if you just do fill in the blank, if you just do whatever, then everything's gonna be okay. And I remember when we were told, man, going big data. You've got to go get hadoop. You've got to go and do big data. And that's going to solve your problem. Well, guess what that didn't. So I think someday I may actually write a book on all of the rabbit holes that our industry's gone down. And it's a wonder we all survived. Now, part of the fallout from that is that the IT organization has lost great amounts of credibility. And if you don't believe me, go into an IT organization and find out who's in charge of the decisions and who's in charge of the budget. Once upon a time, that was the IT organization. Today, you find that it's the end user. It's marketing, it's sales, it's finance. Those are the people. that are because they trusted IT to help them. And IT kept going after these silver bullets and the corporation no longer trust the IT department. I'm sad to say, in fact, it pains me to say that, but it's the truth.
**Robert Harmon:** And, you know, I can only augment that Bill because I've lived a lot of that myself. I was a practitioner working in data teams for what, 25 years. And I've seen all of this. Um, I've also seen what we used to call rogue development where the end users and start running off in their own direction to solve their own problems. And that's all in my eyes. That's always been a strategic threat to an IT organization, because if the end users are busy solving their problems around you, obviously you're not doing your job right.
**Bill Inmon:** Yep.
**Robert Harmon:** which comes back to your first statement. We need to deliver value as an IT organization or we'll just be ignored. And if we're ignored, then why are we here? Now, you did mention something and the quote that I brought you was from Mark Freeman, just so for clarity sake, so everybody knows I'm not stealing his work. I think why he picked 2010 was the big data explosion.
**Bill Inmon:** Yep.
**Robert Harmon:** And... But from my memory about that time, and maybe my memory is going on me, but about that time, we saw a huge employment increase in our data profession. A number of new people came on board. So there was a swell there among a number of companies. So I really think that might be what he's referring to. What I'm seeing lately, though, is more noise about going back to the pre- big data world where we're starting to hear people talking about modeling again and interesting ideas like maybe we should have constraints. When do you think that it's finally coming back?
**Bill Inmon:** Modeling is not something that is associated with any particular discipline. Modeling is something that is useful in lots of places. And let me ask you this, if you were to build a cabin in the mountains, would you need a blueprint? And you either would need a blueprint in your head, or you would need a blueprint that somebody built for you, but you would be dumb to go off into the woods and build the cabin without having a plan, a model. And so a data model has widespread usage. It's not the IT organization that owns data modeling. It's everybody owns data modeling because we need that. And so today, the end user is waking up the whole motivation for doing new systems. Once upon a time, the IT organization built new systems. Today, it's vendors and end users that are building systems. And the advent, the waking up of the data model, the end users are discovering, oh my gosh, I'm building something that's complex here. I need a data model. So I think that the renaissance of data modeling has come from the awakening of the end user that they're the ones that are building something and they need the data model.
**Robert Harmon:** That's really an interesting perspective that I hadn't quite processed yet. I'm not sure I have a response. I'm gonna have to sit and think about that for a day or two to try and work out all the specifics. But no, I think that's a very valuable observation. So my next question, so we talked about the past a little bit. Are there any advancements currently going on that you're... particularly excited about.
**Bill Inmon:** Well, I have to preface this with a disclaimer that I'm involved with this, but yes, I think there, in fact, I've been asked by a number of students graduating college, I wanna start my career, where should I start my career?
**Robert Harmon:** Mm-hmm.
**Bill Inmon:** And let me answer the question that way. I liken the opportunity. for business value in the world of text to be like California in 1848. We are told by the historians that in 1848, you could walk down to the streams of California and pick up gold. You didn't need a shovel, you could just pick it up out of the stream. And it was there waiting to be found. And I think that the... corporation after corporation is letting text go through their hands and doing either nothing or very little with it. And I think that there is tremendous opportunity there. And I could outline some of the opportunities. One of them is in terms of sentiment analysis, of understanding what your customer is saying. Another one is in terms of medical records. That medical records is kind of an interesting case. I don't know if you've ever taken a look at or had the opportunity to look at medical records, but medical records are written in the form of text. Now, the medical records that we have today are designed for and good for one doctor and one patient. So when a patient is being looked at by a doctor, the medical record is there for them. But what, because it's in the form of text, what a medical record isn't designed for is the whole notion of looking at a hundred thousand patients at the same time. So if we have something like COVID come along and we need to look at a hundred thousand patients, you can't do it if the information you have is in the form of text. The only way that I'm aware of that you can do it is if your text has been transformed into a database. And once your text has been transformed into a database, then you can start to ask the question. Interesting questions. Let's take COVID. How does COVID react to people who smoke? How does COVID react to people that take certain medications? How does COVID react to gender? Are men more affected by COVID than women? How does COVID react to age? What role does that play? And answering those questions are very, very important. But as long as your information you have going through your healthcare system is in the form of text, you can't do, not easily, you can't do. that kind of analysis. So whether it's sentiment, I'll tell you another one that is one of my pet peeves. And that one is corporate contracts. When we first started doing what we were doing, I used to talk to groups of executives and I've asked groups of executives, how many people in this room know what's in your corporate contracts? And to date, not one executive has ever raised their hand and said, oh yeah, I know it's in our corporate contracts. And then I say, well, I guess I'm kind of confused because you guys are corporate executives. Aren't you in charge of liability? Oh yeah, Bill, liability, risk management, we've got that down cold. Well, I said, well, I guess I'm really confused then. You tell me that you have risk management and... liability to your corporation taken care of. On the other hand, you tell me you don't have any idea what's in your corporate contracts. And I said, don't you think in your corporate contracts that there's liability? And the truth of the matter is every corporation out there has got liability in their corporate contracts. That's one of the major purpose of a corporate contract. And at this point in time, The executives say, oh, well, we can't do that. I said, well, why can't you do that? And they say, well, Bill, you see, we've got a million contracts. And each one is different. And we can't possibly know what's in our contracts. And I said, oh, no, you can. Let me show you how you can do it. And trying to convince an executive that they should look at their corporate contracts. is like trying to sell caskets to living people. People only buy a casket when they need it when they're being laid in the ground. And so I got tired of talking to executives, but indeed corporations can know what's in their core. And by the way, from a standpoint of business value, do you think there's business value wrapped up in knowing what your corporate contracts is? there is huge amount of business value wrapped up in those corporate contracts and nobody's looking at them. So that's a that's you know you say where's the opportunity? The opportunity is in tech whether it's corporate contracts or medical records or sentiment analysis. By the way there's a lot more than that that's the tip of the iceberg.
**Benjamin:** Like one thing I'm curious, right? Like kind of, it makes kind of perfect sense, uh, kind of in terms, in terms of the problem statement, and I totally buy into that. So I'm actually kind of a guy who builds databases, right? So I kind of build query parsers, kind of execution engines and so on. And one thing I'm curious about now is like, how does a data processing system look? Right? Like kind of how does the data stack look in the world where you have all of those? kind of just text-based records, right, in terms of query language, in terms of the systems, etc.
**Bill Inmon:** Okay, I'm going to warn you, you ask the question, I'm going to give you the answer. This is going to sound to be self-serving, but you're the one that asked the question. There is technology out there called Textual ETL. Textual ETL takes the raw text and turns it into a database. Now, in order to do that, it's taken a long time for myself and my corporation to figure out how to do it. But indeed, it's a very, very difficult task. We can do that today and I'm happy to say there are people out there that they have gotten the message about Text and they are starting to look at it now Confusing matters immensely is this chat GPT stuff People think oh chat GPT. That's text. We now have a handle on text No, you don't and let me tell you why And by the way, I'm a fan of chat GPT. I have nothing against chat GPT, but chat GPT answers a different question. Chat GPT is good for taking language and text and turning language and text into a question, into a query, into your computer. That's what chat GPT does, and it does it well. What ChatGPT doesn't do is look into the language and take the value and data that's in the language that's there. And I know this for a fact because we've looked at it and played around with it. And so, and it sounds the same. It sounds, oh, ChatGPT text, that solves our problems. No, that doesn't solve your problem because ChatGPT does not go into the text and find what's in the text. Then you say, okay, Bill, thanks for telling me about textual ETL. I've never heard of it before, but it's time you hear about it. Textual ETL, the heart of textual ETL is something called an ontology or a taxonomy. and ontologies and taxonomies are how, that's the magic ingredient of how you go into text and start to understand the text to the point where you can turn it into a database. And you probably don't want to get me off on this subject because I've been working the last, oh. 13 years of my life on it. And I'm gonna tell you, it's a very complex subject. Let me give you a couple of examples of why text is so complex. Let's take the word fire. What does the word fire mean? Well, it could mean that your house is burning and you're on fire. That's one meaning of it. It can also mean that your boss doesn't like you anymore and you got fired this morning. Or it could mean that you've got a gun in your hand and you pull the trigger and you fire the gun. So there's lots of meaning. Our language is full of double meanings, triple meanings for all kinds of things. And in order to do a proper analysis on text, you've got to be able to distinguish between what's being said. And that is no, I've been doing this for 13 years now. And I can tell you there's really two components here, text and context. Text is actually fairly easy. It's not, I don't know, it's fairly easy. What's the devil is context. Context is not easy at all. That's where the problems come in. That's where the difficulties come in. So anyway, how do you start to go from text to a database to where the person like yourself can start to do their analytical magic is there is technology out there called Textual ETL that indeed... does what you are asking it to do. And by the way, when we started, okay, when we started on Textual ETL, we looked at something called NLP, Natural Language Processing. And NLP is an academic exercise. It was never designed to be a commercial product. It is complex. It takes a tremendous amount of time. and it requires high price consultants to make it work. When we started off to build Textual ETL, we wanted all of those things to not be. So we've created something that is inexpensive to use, that is not complex, doesn't require an army of technicians and is fast. And so we have a commercialization. really and truly what Textile ETL, you can think of it this way, is as a commercialization of NLP. But the people in NLP, they want to cling to their life raft. They're not about to hear it. But let me tell you, business people. So when we first started, we thought, well, we'll talk to people in the world of NLP. And they don't want to hear from us. I'll tell you who does want to hear from us is people in marketing, people in management, people in sales, people in finance, and they love what they hear.
**Robert Harmon:** It seems to be a recurring theme through this conversation, Bill, because, you know, at the end of the day, it gets all the way back to the beginning, provide value. And if the people out in the, you know, in the business want to hear from you, you're obviously providing value. And that's sadly not always the case with IT teams. The other thing I notice here is if this does catch on, that's going to be a lot of data. So Benjamin and I may have a little. to do what you get going.
**Bill Inmon:** Yeah, I'll be retired or dead by then, one of the two.
**Robert Harmon:** Hahaha
**Bill Inmon:** but you're
**Robert Harmon:** So,
**Bill Inmon:** absolutely right.
**Robert Harmon:** yeah, so swimming
**Benjamin:** Thank you.
**Robert Harmon:** and more data. I'm not sure I appreciate that, but at least I won't be unemployed for a while. So really, Bill, this did not go the direction I expected it to, and that's a really good thing because sometimes I'm a very boring person. I do have some other more personal questions.
**Bill Inmon:** Sure.
**Robert Harmon:** You're a car guy. You're a car
**Bill Inmon:** Yeah.
**Robert Harmon:** guy, yes. How did that happen?
**Bill Inmon:** You don't know it, but you hit a really sore spot. I am not a car guy.
**Robert Harmon:** Oh
**Bill Inmon:** In February, my
**Robert Harmon:** Uh-huh.
**Bill Inmon:** car had its catalytic converter stolen
**Robert Harmon:** Oh no!
**Bill Inmon:** and my car has been in the shop since February. I've been using my wife's car for half a year now. I don't know if you've tried to get a catalytic converter, but it's like gold or platinum or diamonds. Give me a break. And so I don't know when I'm going to get my car back.
**Robert Harmon:** What?
**Bill Inmon:** But I've owned in my lifetime six Porsches, one Ferrari. I can't resist this. The two best days in the life of a person owning a Ferrari. is the day they own the Ferrari and the day they get rid of the Ferrari. And, oh, you, if somebody came to my front door, parked a Ferrari in my, in front of my house, gave me the keys, I would run as fast as I could. I, I'm not about to take it. Now, now having stated that, Porsche, pardon me. Porsche is as well made as Ferrari is a piece of crap. And I love my Porsches and we need to have my company progress a little bit further. But when my company progresses a little bit further, I'm gonna be getting my seventh Porsche.
**Benjamin:** As a German, I appreciate your support of the German automobile industry.
**Bill Inmon:** Oh.
**Robert Harmon:** Honestly, I think it's possible Bill and I paid for the entire German automotive industry over the years.
**Bill Inmon:** You know what car I loved and people didn't think much of it, but I had a Porsche 914 and that was the rear engine and the thing that I loved about it is the way it handles. It handles differently and better than any other car I've had.
**Robert Harmon:** Exactly.
**Bill Inmon:** But when you tell a Porsche fanatic that you like the 914, they automatically think of you as a real wimp. But I love they're my 914.
**Robert Harmon:** wonderful cars.
**Bill Inmon:** It was a wonderful car.
**Robert Harmon:** yeah, I've owned a couple of them and they're absolutely wonderful cars. They're not gonna tear anything up on the, compared to modern cars, but it's a delightful experience. Everyone should do it once.
**Bill Inmon:** You don't know what driving is like until you've driven a 914.
**Robert Harmon:** Awful experience. Well, there are some things to it. The transmission, the shift forks, I swear were designed for a tractor.
**Bill Inmon:** Yep.
**Robert Harmon:** So you do have to get your, you know, you got to know what you're doing to a little bit, to a solid extent. And then they're great little cars once you get them figured out, but they're a little quirky.
**Bill Inmon:** Yep.
**Robert Harmon:** Well, I don't know as I had a whole lot more at this point. Benjamin, you've been very quiet.
**Benjamin:** I've been asking tons of questions.
**Robert Harmon:** Okay. Well, maybe it's time to start winding this down.
**Benjamin:** Sure. So Bill, it was an absolute pleasure having you on the show, getting to know you. Thanks for everything. Thanks again for all of the kind of, yeah, making the data industry what it is today. It was an absolute pleasure having you on. Yeah. I hope you have a great rest of your day.
**Bill Inmon:** Thank you so much, Benjamin. Robert, nice talking with you. We'll talk again.
**Robert Harmon:** All right, we'll talk to you later, Bill.
# Block Bad Data Before the Write with Nike’s Ashok Singamaneni (/blog/block-bad-data-before-the-write-with-nikes-ashok-singamaneni)
Nike's Principal Data Engineer Ashok Singamaneni joins Benjamin and Eldad to discuss his open-source data quality framework, Spark Expectations. Ashok explains how the tool, which was inspired by Databricks DLT Expectations, shifts data quality checks to before the data is written to a final table. This proactive approach uses row-level, aggregation-level, and query data quality checks to fail jobs, drop bad records, or alert teams - ultimately saving huge costs on recompute and engineering effort in mission-critical data pipelines.
Listen on [Spotify](https://tinyurl.com/56jz6tda) or [Apple Podcasts](https://tinyurl.com/bdheke9h)
\[00:00:00] Ashok: DLP expectations gave an idea to the industry that you can do data quality before actually writing the data into your final tables. As the scale of the product increases, it becomes even more difficult for us to find exactly where the issue went wrong, like, even if a production job fails.
\[00:00:18] Benjamin: Ashok is a principal data engineer at Nike, and he worked on a variety of cool, actually, open source projects as part of his work there.
\[00:00:27] Ashok: I think over the time, in my experience, what I learned is this ingestion layer and the transformation layer, you should treat that as a software product, not like a data engineering product.
\[00:00:39] Benjamin: Hi. This is Benjamin. Before we start with today's episode, I wanted to quickly reach out on a personal note. We've just launched Firebolt Core. Firebolt Core is the free self-hosted version of our query engine. You can run Core anywhere you want, from your laptop to your on prem data center to public cloud environments. Core scales out, and you can run it in the multi node configuration. And best of all, it's free forever and has no usage limits. So you can run as many queries as you run and process as much data as you want. Core is great for running either big data ELT jobs on, for example, iceberg tables or powering high concurrency customer facing analytics on big datasets. We'd love for you to give it a spin and send us feedback. You can either join our Discord, enter our GitHub discussions, or you can just shoot me an email at [Benjamin@Firebolt.io](mailto:Benjamin@Firebolt.io). We'd love to hear from you. We added a link to Firebolt course GitHub repository to the show notes. And with that, let's jump straight into today's episode. Hi, everyone. And back on the data engineering show.
Today, Eldad and I are actually in person in Munich together, which is very nice. He's real, same room. We're both real. Exactly. And we're super happy to have Ashok on. Ashok is a principal data engineer at Nike, and he worked on a variety of cool, actually, open source projects as part of his work there. So one is called BrickFlow, and one is called Spark Expectations. Excited to have you on the show today, Ashok. Do you wanna quickly introduce yourself? Tell us about your background, how you got into data.
\[00:02:04] Ashok: Thank you. I am Ashok Singamaneni, and this might be interesting. I'm a mechanical engineering. I've done mechanical engineering basically. And then I switched to data like probably twelve years ago and it's been a long journey and I've done my master's.
\[00:02:21] Eldad: The industrial revolution ends with data, unfortunately.
\[00:02:24] Ashok: Yeah. So I figured probably when I read the MAPRED newspaper, that's when it clicked that I think this is going to be good. And then Cloudera, Hortonworks, and all of that happened, and I write some books. And then I did my masters and did thesis on, big data analytics on vehicle data, and then later moved into banking industry and then worked in health care and then now in retail domain. So it's been interesting journey.
\[00:02:52] Benjamin: And so after twelve years of data, do you ever think back? Are you like, hey. I next kind of, like, in a couple of years, I wanna become a mechanical engineer again, or you're, like, kind of
\[00:03:01] Ashok: No. I think it's a perfect blend right now. Look at Tesla. You can be a mechanical engineer as well as also be in software. And, also, you can be in robotics as well as in software and data.
\[00:03:11] Benjamin: That's awesome. Cool. So at Nike, right, you worked on both these kind of BrickFlow and Spark Expectations framework. Tell us a bit, like, how did those projects start? Kind of why did you decide to actually open source them? What are they all about? We'd love to learn more.
\[00:03:27] Ashok: Sure. I think coming to Spark Expectations, right, being in the industry for so long and have been in lot of production calls and misfires happening in the production because of the data changes, unexpected column changes, etcetera, in upstream or downstream, and having been in lot of recompute of the data. This all happens because of the majority of the times data quality issues, which can be removed upfront while the data is being processed, as well as also some of them are the processing or or planning mistakes that happens regularly, and that's kind of common. But when I looked at Databricks DLT pipelines that was released a while ago, they have introduced a DLT expectations, which kind of is interesting because if you have seen Great Expectations, Great Expectations is a tool that actually gives you data quality report post processing of the data. After you process the data, you will see the data quality report that you can run on the data and then the report is really fantastic. But DLT Expectations gave an idea to the industry that you can do data quality before actually writing the data into your final tables. And then I reached out to Databricks team like if we can work on, something related to Spark as well. Because many, many companies, we know that using Spark for data processing and transformations. If this can be available in Spark, that would be great. But the timelines didn't work out, so I started working on the project called Spark Expectations in the same name of weird expectations, weird expectations as this project is related to Spark. So I named it Spark Expectations. And, it has more functionality than DLD right now, but it is only supported for Spark. I wish that it supports for all the frameworks like Pandas and other libraries as well, but right now, we are supporting Spark. And what this does is before you write the data into your final layers, the data quality checks happen and restricts the data that is not up to the data quality standards what you want so that you'll need to recompute as well as you will have alerts on the data that has gone bad, and then you'll be able to alert your upstream teams that this data is bad. This need to be resend and reprocessed.
\[00:05:44] Benjamin: How do you define these expectations? Is it like a YAML file, a JSON file? Like, take us through that part.
\[00:05:50] Ashok: I think it's up to the teams which are implementing because right now, the rules, the way Spark expectations expertise is part of a data frame. So you can write it as a YAML file, JSON file, or might as well have that in a table as well and load that as a data frame and give it as an input to Spark expectations. And there are also, like, three kinds of rules that you can provide, like, row level data quality checks, table level aggregation level data quality checks, as well as query deque, which we call it as, like, for referential integrity checks or if you want to have, like, some complex rules that you can run on data quality.
\[00:06:27] Benjamin: Okay. And then kind of that runs basically before I do any of my batch ELT jobs? Does an initial pass over to the data, makes through, yep, everything kind of looks great before I start spotting a lot of
\[00:06:39] Eldad: like, is it sampling the data? Is it running the previous snapshot?
\[00:06:44] Ashok: So there are, like, different layers in Spark expectations. Right? So when you load the data let's say, for example, if I'm ingesting the data and transforming it, when I load the data into the data frame, like the initial data frame when I load it from my source table, you can run source level, query dequeue, or aggregation level checks that you want to do. And then you can run data quality or row dequeue checks as in, like, on each column. Let's say the date has to be greater than this level or any other validations that you want to do on a particular column, you'll be able to do. Not a sampling. It runs on the whole dataset. It gathers all the information. And you can also have conditions like ignore, drop, and fail the job. If a rule fails, you can ignore it, but still alert it that this rule is a soft failure. It failed, but you are getting still an alert. Or you can drop the record. Like, let's say a product table doesn't have a product ID or the product ID configuration is wrong, then that is something serious that should not be there in the data. So you drop that record on the whole and put that in an error table and give that alert to the engineering team that there is some error in the error table you can look at. And you can even fail the job. If it's mission critical, then you fail the job, not process the data, and don't put that data into the final table so that you don't need to recompute that again.
\[00:08:04] Benjamin: But then for me to understand this a bit better, like, does this actually run as a hook within the ELT job itself? So that, basically, every time I start scanning a table anyways for my big batch processing, I also run the data quality check. It's not like I have to scan the table twice now. Right? Like, that I will first pass over all the data, which is really expensive, run the kind of spark expectations, throw everything away, and only when that looks good, I run the batch ELT. It basically overlaps. Is that accurate?
\[00:08:35] Ashok: Yes. I think we use decorator pattern in Python for this. So whenever you tag that spark expectations decorator, there is something called with expectations. If you tag that inside your function and your function returns a data frame, on that data frame, whatever rules you define, it runs all of them and generates the report.
\[00:08:54] Benjamin: Very cool. What have you seen in terms of the, like, overhead this actually introduces for, like, bigger batch ELT jobs? Is it noticeable? Is it very fast? Take us through that.
\[00:09:04] Ashok: Yeah. I think the road e q checks that happens are very fast. It should happen as a pretty standard checks that happens on the scale. But, obviously, definitely, there will be an overhead. So you wouldn't want to put this on all the layers and all the jobs. You only want to put it at the final layer or the final right step where you are actually writing the data, and that is, like, the mission critical production data that you want to have it. And for the query DQ, as well as aggregation DQ, you need to be careful and optimize it well so that the time is not heavy. Like, you need to optimize your queries, obviously, like any Spark job, enter it. But that is an overhead for sure. Like, if you're running data quality checks every day on top of, like, huge scale of data, then obviously you'll have. But it's also possible in streaming too. Like, if you're doing, like, a micro batch, then you would be able to do that on micro batch too.
\[00:09:55] Benjamin: Okay. And so let's say I have, like, I don't know, medallion architecture. I go, like, brass layer, silver layer, gold layer, which you're basically saying, like, on the right to the gold layer, I would basically hook in Spark expectations. And at that point, which then connects with the framework and also error out, I could really make sure that, like, no insert into the gold layer even finishes, um, if there's not a certain kind of data quality check passing.
\[00:10:24] Ashok: Yes. Put it. Exactly.
\[00:10:26] Benjamin: Very nice. Now would it
\[00:10:27] Eldad: be fair to kind of compare that to constraints in SQL or kind of that domain assertions constraints?
\[00:10:36] Ashok: Yeah. I think it's something similar to constraints. Right? But it's also the constraints has a limitation that you can only put constraints on that particular column values. Like, this is what this column expects, but spark expectations does something more than that. Like, you can have, like, aggregation DQR. Like, let's say, for example, the whole count or the sale in that table should be greater than 100,000. If, let's say, there is a big sale that is going on and if the sale is, like, less than 10,000, then that doesn't make sense. There is something wrong in the computation, etcetera. Or differential integrity checks, like, if you have, like, multiple tables in the data model and you want to have, like, correlations that are happening properly or not, if the business logic makes sense or not for this data to be there. So those kind of checks also can be done.
\[00:11:25] Eldad: So if someone is you know, for people who own production grade pipelines, specifically kind of at those stages where the data is not fully cleansed. Right? Like, at the end of the pipeline, everything went through cleansing. It's always safe. It's it's there's a value for every product. But as you go closer to where data is being born, it's nastier. Right? It's not clean. There's no formal way to do it. So every data engineer I know of has a set of toolboxes based on intuition, experience. Obviously, depends if they're engineers and they're writing spark or getting deep into the right into how those queries run. What would you recommend us? Like, when should we seriously start looking into those frameworks and kind of what would be your guideline on transitioning to that mindset?
\[00:12:17] Ashok: Yeah. I think over the time, in my experience, what I learned is this ingestion layer and the transformation layer, you should treat that as a software product, not like a data engineering product like from per se. You will have all the data modeling, data governance, data observability, all of those tools that are there. But I think the general mindset should be the ingestion layer or the raw and the bronze layer and the silver layer should be like a software product. You should have all the checks and balances in place, like data quality and unit testing, integration testing. All of those should be in place so that any mishaps that happens, it's easier to find out and debug. Because as the scale of the product increases, it becomes even more difficult for us to find exactly where the issue went wrong. Like, even if a production job fails, it takes time for you to debug and see, like, lot of human effort also involved, not only the recompute that is happening and the compute costs that are happening, but also couple of engineers, a product person has to be involved for a week or week and half to fix that issue that is happening. But the final goal layer where mostly people use SQL, that can be treated as like a pure data engineering. Like, write SQL, do it fast, build dashboards, break them, and then fix them as much as you can. But like the initial two layers are mission critical that has to be treated as like a software product. That's at least my experience, what I have seen.
\[00:13:49] Benjamin: Very nice. Take us through your experience, like open sourcing a data project, right? It's kind of like, did you go to some meetups? Kind of like, do you actually know some of your users? We'd love to learn more about that.
\[00:14:01] Ashok: Yeah. I think it's been a fantastic journey for open source. It's my first project to to do that, but I have really great mentors. A big shout out to Adi Aditya Chaddhivedi, who's distinguished at Nike and also Scott Haines who was an author at O'Reilly and other people who have helped me through the journey. My senior director, Joe Hollow, who helped me through the process. These guys have helped me through the process of, like, open source. And, also, there is an open source community effort that is happening at Nike as well. So getting through the approvals and getting the org set up and and the project set up and publishing it, it has been great. And I also had an opportunity to talk at Databricks AI Summit. Um, so networking happened. Couple of engineers also looked at the project and held through. And I think there is more that can be done from the project side, and I wish there need not be Spark expectations. There should be something native to Spark so that anyone can actually use directly from the native Spark module.
\[00:15:06] Benjamin: Very cool. So how are you now using like Cursor or, like, any of these Gen AI tools to actually accelerate kind of progress within your own open source project? Like, are you vivecoding a good looking UI?
\[00:15:20] Ashok: I think I've been using Cursor and Cloud Code since, like, over one and a half years' time. I mean, time flies. It feels like eternity that I've been using them for a long time now. But wipe coding is really good if you know exactly what you're doing, at least from the data engineering standpoint. Like, for the regular software engineering, building websites, etcetera, that can be very quick. But from the data engineering standpoint, we have to take some guidelines into place. I've seen people using plot code or cursor on production data directly when they're building and drop the datasets. So, so misfires happen a lot if you are using white coding directly on data engineering projects, guidelines that need to be taken. Use service principles or AWS rules, etcetera, so that they have restrictive permissions when you are using plot code or cursor. That is something that I have learned myself as well. I was working on a small pet project, and I was using SQL mesh. And there were some commands which I didn't know that it destroys the whole dataset. And it's a sandbox handler, but it destroyed everything. Right? So having those guidelines in place is really critical from the permission standpoint. And as well as from wipe coding, I think there are different patterns people are using these days on, like, creating a PRD document first, have a GitHub issue, and set up a process for yourself so that there is a journey from what is the problem that you're trying to solve, creating a proper GitHub issue for all the issues that you are working on, and then helping through the journey of, like, breaking that data issue into small tasks as checklists in your plot code or cursor, and then checking through each one of them, writing unit tests as you go, and having integration tests, and human in the loop that you have to review the code that has been written. Ultimately, at the end of the day, you are responsible when you're checking in the code. It's not Claude or Karsar. That will be blamed if something goes wrong.
\[00:17:26] Eldad: Listening to your think, it actually makes perfect sense to have those cleansing and expectation libraries being used for often nowadays when no one actually knows who's behind each and every change. So as it starts, right, forking, modifying, changing, updating production eventually, setting expectations makes more sense. So I think there's a bright future for kind of right away even say, like, focus on the expectations because we can't control the modifications themselves. So at least we can own the expectations. But this will be super interesting to see how this kind of semi automated yet highly expectation pipeline will turn out.
\[00:18:08] Ashok: Yeah. And it's also the trend that I'm seeing these days, is from the data governance teams and data governance standpoint as well across the industry. Data observability and quality is becoming prime because of AI integrations that are happening. There are tools that are coming out where a CEO or CTO from a company can directly ask questions in natural language and hit the production data and get the data for themselves rather than Someone goes,
\[00:18:37] Eldad: codes it, writes it, generates the reports.
\[00:18:40] Ashok: Yeah. So the leadership is directly looking at the data and if there is something wrong in the data, then there can be some serious repercussions happening on the business decisions. So that's one of the reasons why I see the industry moving towards the trend that rather than having bad data in the tables and then recomputing or reclarifying things, let's not put that data first in the first place and then redo whatever needs to be done from the upstream and fix the data.
\[00:19:10] Benjamin: Cool. So if you look ahead, like, year to year, what are you most excited about in the data space?
\[00:19:17] Ashok: I think from the data standpoint, I'm looking at how the white coding improves and the tools that comes for white coding in terms of the data engineering space exclusively. Because right now, it's all generic patterns that are evolving in the industry. Like, if you look at Reddit or any other blogs, there are certain patterns that people are following for UI coding, iOS, Android apps, but there is not specific that I have seen for the data engineering space. This is how you exactly do white coding. I think I'm more excited towards that on, like, how that can be evolved as a tool or a framework. If there can be something as a spec, it, like GitHub has really suspected recently on, like, how to do wipe coding. But if there is something like that for data engineering, that could be awesome. I would like to collab on that. Nice. Awesome.
\[00:20:10] Benjamin: So to all our listeners, if you want to work with Ashok on Vibe coding for data quality and data engineering, reach out to him. It was so great having you on the show. Seriously, kind of thanks for sharing your experiences. It's always exciting to learn about new open source projects in the data space. So thank you for being on.
\[00:20:29] Ashok: Thank you. Thanks for having me. Have a good day.
\[00:20:32] Unknown: The data engineering show is brought to you by Firebolt, the cloud data warehouse for AI apps and low latency analytics. Get your free credits and start your trial at firebolt.io.
# Building Customer Trust: A CISO's Perspective on Security and Privacy at Firebolt (/blog/building-customer-trust-a-cisos-perspective-on-security-and-privacy-at-firebolt)
### Security as a Core Value [#security-as-a-core-value]
At Firebolt, security isn't just a feature of our service. From our co-founders' initial vision to our cutting-edge technology, through tuning business processes to fortifying our cyber defense, every aspect of Firebolt is designed with security in mind, influencing every decision and innovation we make. Firebolt's holistic approach to security ensures that every aspect of our operations is geared towards maintaining the highest standards of protection for our users and their data while underscoring the critical importance of our security practices and the continuous enhancement of our security posture.
With that, our commitment to robust security measures from day one ensures that every customer's data is protected against evolving threats, allowing customers to trust Firebolt as a secure platform for their most sensitive data. We are proud to have achieved key industry certifications, including ISO 27001, ISO 27018, and SOC 2 + HIPAA, which validate our adherence to stringent security and privacy standards.
### Integrating Security Throughout the SDLC [#integrating-security-throughout-the-sdlc]
As a provider of Cloud Data Warehouse as a Service, Firebolt faces unique challenges in securing vast amounts of data, and at the heart of our Security DevOps approach is the seamless integration of security throughout the software development lifecycle. From the outset, secure coding practices are employed, and as development progresses, each phase undergoes various security testing. This proactive scrutiny helps identify and mitigate vulnerabilities early on, ensuring robust defenses from the ground up. We also actively engage in essential practices like penetration testing and [fuzzing](https://www.firebolt.io/blog/fuzzing-firebolt-catching-0-days-as-fast-as-our-query-processor), allowing us to uncover potential weaknesses and fortify our systems against emerging threats.
### Data Protection and Network Security [#data-protection-and-network-security]
Data protection is paramount, and we employ stringent policies coupled with end-to-end encryption to safeguard data at rest and in transit, maintaining confidentiality.
* Data at Rest
* We leverage AWS's built-in features, like KMS, to handle storage encryption at rest for customer data stored within our systems. Our systems are securely integrated with AWS KMS to ensure seamless key management for data encryption through AWS's infrastructure.
* Encryption keys managed by us securely generate, store, rotate, and retire the encryption keys that we manage, particularly those used for encrypting sensitive internal data
* We enforce strict access controls and separation of duties for individuals authorized to manage the encryption keys under our control.
* We safeguard the storage of encryption keys to prevent unauthorized access.
* We enforce robust authentication mechanisms for users and applications accessing the data. This means that every client, whether it's a UI, SDK, JDBC, or another interface, must be associated with a unique login and user. This approach ensures that each client is individually authenticated and authorized to access the data.
* Initial authentication is handled by our Identity Provider, which validates the user's or application's identity.
* Once authenticated, our fine-grained RBAC mechanism further controls access to specific data.
* We've implemented auditing and monitoring mechanisms to track and log access to encrypted data and encryption key management activities.
* Regularly review and analyze the audit logs for any suspicious or unauthorized activities.
* We ensure the secure transmission of encryption keys from the key management system to the encryption/decryption components to prevent interception or tampering.
* We've implemented secure data deletion practices to ensure that data is adequately wiped or destroyed when no longer needed.
* We've implemented data integrity checks to detect unauthorized modifications or tampering with encrypted data.
* Data in Motion Firebolt
* We've implemented TLS to ensure secure communication across all client-server interactions. This applies to external clients, such as a UI accessing a gateway, and internal communications between components within our infrastructure, providing an extra layer of security against man-in-the-middle attacks.
* Our policy exclusively includes strong cipher suites that offer robust encryption and resist known cryptographic attacks. These suites utilize AES (Advanced Encryption Standard) with 128-bit or 256-bit keys and Galois/Counter Mode (GCM) for authenticated encryption, ensuring data confidentiality and integrity. Additionally, the selected cipher suites support Perfect Forward Secrecy (PFS), meaning that even if a server's private key is compromised, past communications remain secure, as session keys are not derived from the server's private key. We intentionally exclude weaker ciphers and deprecated features like RC4, MD5, and SHA-1, which are vulnerable to cryptographic attacks. By default, we negotiate TLS 1.3, with the ability to fall back to TLS 1.2 if necessary.
Our network architecture is carefully designed, with segmentation for production, development, and testing environments. Each segment is fortified with tailored security controls and subjected to regular security audits. In addition a comprehensive multi-layered security strategy that includes Web Application Firewalls (WAF), Denial of Service (DoS) protection, and bot mitigation is implemented.
### Privacy by Design [#privacy-by-design]
Privacy is another core principle in our system design and is embedded to minimize data exposure and proactively mitigate risks with the addition of comprehensive access controls and robust governance frameworks. We protect user and organizational data. Each tenant's data is securely isolated, preventing unauthorized access or data leakage. Our approach ensures that sensitive information is protected through encryption, access controls, and privacy-enhancing technologies.
1. We follow a data minimization strategy, collecting only the essential information for service functionality. Where possible, we apply anonymization and pseudonymization techniques to reduce the exposure of sensitive data during the processing, analytics, and testing phases.
2. We leverage AWS's encryption services, such as AWS Key Management Service (KMS), Secrets Manager, and Vaults, to securely handle encryption keys and sensitive information. Data is encrypted at rest and in transit using industry-standard algorithms like AES-256. Additionally, secrets such as API keys and credentials are securely stored and rotated.
3. Tenant data isolation is enforced using Kubernetes namespaces within our Amazon EKS environment. Each tenant's resources are segregated into dedicated namespaces, ensuring strict boundaries and preventing cross-tenant access. These namespaces are further secured with role-based access controls and network policies to maintain isolation.
4. We implement holistic runtime security using a defense-in-depth approach. All of our binaries are built with Position Independent Executable (PIE) to randomize the address space for code execution, stack protection, and Control Flow Integrity (CFI) to prevent attacks that target both forward as well as backward edge flows by overwriting return addresses on the stack. Next, these hardened binaries are run inside non-root containers with strict namespace isolations and limited capabilities, thus adhering to the Principle of Least Privilege (PoLP). This guarantees that even in the improbable event of an exploit that defeats binary hardening, an attacker still cannot break out of the container and escalate to root. Furthermore, we deploy an advanced runtime protection tool that observes every process for suspicious activities and malware that alerts and blocks anomalous events. Ensuring that our systems are resilient against threats while safeguarding sensitive data.
5. Access to sensitive data is governed by fine-grained RBAC policies, enforcing the principle of least privilege. These controls extend across all system components, ensuring only authorized users and services, including internal employees, can access specific datasets.
6. Logs are explicitly configured for internal use, forensic needs, and compliance, excluding personal identifiers, authentication tokens, and other sensitive attributes. Where logging is essential, sensitive information is hashed, tokenized, or fully redacted. Additionally, we enforce strict retention policies and secure storage for logs containing minimal necessary data.
7. Our platform is designed to support data subject rights, such as data access, rectification, and deletion requests.
### Change Management and Governance [#change-management-and-governance]
Our change management process is designed to ensure the secure delivery of services. Before deployment, changes undergo thorough risk assessment and testing, minimizing disruption and potential security risks. Governance is central to our security strategy, with policies and procedures in place to ensure compliance with industry standards and regulations. Regular audits and assessments verify adherence to these standards, providing transparency and accountability to our customers.
### Securing Customers Trust [#securing-customers-trust]
We believe in a collaborative approach to security and privacy. By aligning our best practices with customer-managed controls like **Single Sign-On (SSO), Multi-Factor Authentication (MFA), Network Policies, and Role-Based Access Control (RBAC)**, we create a synergistic approach to security. This collaborative effort ensures a layered defense that enhances overall robustness and resilience against evolving threats, demonstrating our commitment to our customers' trust and peace of mind. More information is on our [security documentation](https://docs.firebolt.io/Overview/security.html) page.
# Building Data Products For Data Engineers (/blog/building-data-products-for-data-engineers)
How does a tech stack that always needs to be at the forefront of technology look like? [Roy Miara](https://www.linkedin.com/in/roy-miara-73776a56/) from [Explorium](https://www.linkedin.com/in/roy-miara-73776a56/) talks about building data products for the audience that can't be fooled – data engineers.
Listen on [Apple Podcasts](https://podcasts.apple.com/us/podcast/building-data-products-for-data-engineers/id1561927688?i=1000534802627) or [Spotify](https://open.spotify.com/episode/7dAo7XjwjZ44xz68T0J6Ti)
**Boaz:** Hello, everybody. Welcome to another episode of the data engineering show. So happy to see you with us again.
**Eldad:** Yes, I came back from abroad.
**Boaz:** I missed you.
**Eldad:** I missed you so much.
**Boaz:** So, welcome to another episode where we host data practitioners from all around. Today with us Roy Mira. Did I pronounce it correctly, Roy
**Roy:** Yep.
**Boaz:** Great. So, Roy an engineering manager at a super interesting company called Explorium, engineering manager of data and ML who's been in a variety of data engineering and machine learning focused roles, a variety of startups in his career. And typically, we talked to big brand names before, but it's important to also talk to sometimes smaller, interesting companies...
**Eldad:** Emerging brands.
**Boaz:** Like Explorium so companies that actually build stuff for data engineers and data scientists, which makes Explorium interesting. So, Explorium, if you haven't heard about them recently landed 75 million in investment. What they do, they help with data enrichment and help you sort of use external and public data sources to simplify your training data procedures and stuff like that. We'll have Roy expand on that a little bit but before that, actually Roy ran late a little bit. Roy, why did you run late to this webinar? I think you have a story for us.
**Roy:** Changing a flat tire.
**Boaz:** For whom?
**Roy:** For a pregnant woman.
**Boaz:** This is so nice. Sometimes we have people who are also amazing humans outside of the data engineering show. Thank you for helping...
**Eldad:** And your wife is not expecting.
**Roy:** No, it's not my wife. I had a feeling because I was texting Boaz and then I said, oh, it's going to pop. So, I text him I'm going to be late.
**Boaz:** So, take an example from Roy. If you see pregnant people who need to change flat tires in the heat help them out. Okay, so Roy, tell us a bit about Explorium in your own words, please.
**Roy:** Among data scientists today, we hear a lot that it's all about the data. It's no more about the models, this is the era of data centric AI in general. And big data analytics has always been around data, who has the best data, who is the most accurate, the most relevant data to answer some business questions and in Explorium we're looking at this whole world of machine learning and big data analytics and we say it's all about having the right data. So, what Explorium does, it enables organizations; large organizations, small organizations, to have immediate access to the most relevant data for their business problems. And we do it by obviously aggregating a lot of data sources in a lot of fields, modeling the data correctly and enabling this kind of search engine over our dataset with accordance to what the user has and already has as an internal data from his company. But also, in cases where the user doesn't even have data and only has some business question, he wants to know all the companies that have more than 10 employees and more than $5 million in capital in the East Coast, for example.
**Boaz:** So, who typically are the end users? Is it more for data engineers, data scientists? Is it sometimes for business users as well?
**Roy:** Yeah, our users span from data scientists and deep ML engineers on the far hand and on the other hand, you have people from the business and analytics side that use the platform to do all their exploratory data analysis. So, we see everything in between, obviously.
**Boaz:** Okay. So, we're talking about a company who manages a lot of data, makes it easily accessible, both for engineering and for a business. Super interesting. So, tell us a little bit just so we can understand the data challenges you guys have. What data volumes more or less are you guys dealing with?
**Roy:** Data volume here is tricky because we're processing somewhere around the two terabytes a day, but when I'm saying processing, I'm talking about already structured tabular data. So, we're not looking into raw data, we're looking at fine-grained, high quality data from a variety of sources. And I think one of our major challenges is having this variety of sources and different schemas constantly evolving. So, around a couple of terabytes a day this is what we process and in volume, we're talking also a couple of hundreds of terabytes constantly updating.
**Eldad:** And is that one data source? Is that one global schema with many tables in it?
**Roy:** No, there's hundreds of different sources, most of them are structured but we're talking hundreds of data sources, thousands of features or thousands of different schemas because from every source we generate and aggregate data to many, many different use cases.
**Boaz:** Because Explorium is company where the data is the product, essentially. How does the split between engineering, data teams, data engineering sort of look like? So maybe give us an overview of how many people are in engineering, how many people deal with data, how many people have a data related title versus an engineering related title. Do people without a data title still work on data stuff?
**Roy:** That's a good question, actually, because I claimed that everybody here are data engineers, scientists, and analysts, kind of a mix. We have an infrastructure organization, data infrastructure, which is the team that I'm leading. We're about nine data engineers, a data ops engineer, ML engineers so we kind of work in between. We have another team that's building Explorium's feature store to say.
**Eldad:** So, a user can just add a one of many data sources that are completely structured into their schema, into their model, enrich it and query it, basically.
**Roy:** We have these two flows. One flow, we call the auto ML flow, the ML engine. So, in this flow, a user is coming with his data set, internal data that he collects internally inside his organization and he has some target that he wishes to predict like the classical ML flow. What happens at that point; he can connect to our platform and basically the platform will enrich automatically. So, add sources automatically, according to the context and the analysis of the original core data set that the user uploaded. The platform will automatically analyze the data, understand which data sets are the most relevant, connect them, train the model, evaluate the model based on how much gain do we see from those external features. So, it's connecting the data, it's running feature extraction, it's running feature selection, closes the loop with a model and then finds the best features according to the target.
This is what we call the auto ML, ML engine flow. And we have a flow that is more towards general like, you know, analysis. Meaning that you can upload your data, we analyze the data, but we will present our internal catalog of features. So, you'd be able to see the sources, you'll be able to see the coverage, you will be able to see the feature is with all the description and everything, and the user can add features. So, add enrichments, he can run transformations and basically build what we call the recipe. He could build the recipe of the features that he wants to add with all the transformation and then he can query this recipe in production, he can use this recipe to schedule batch jobs that will update this data. And now we're, we're kind of connecting all of these flows together, create one unified flow where a model is just a part of the recipe.
**Boaz:** And what does your data stack look like?
**Roy:** Our data stack is quite varied. We have a couple of internally built tools, our feature stories internally built but what my team is managing in terms of looking at the entire data stack, we're looking at databases, Postgress, DynamoDB, we have Elastic Search. Those are the main databases that we have. We have a cloud data warehouse, we are using Firebolt for a lot of our internal processing and also for exposing some of the data directly to the users. We're using Spark a lot, we're using DBT, we're trying to be very on the edge of the technology because we are facing new use cases all the time. We kind of have this challenge of trying to be good with any data so we going have to be familiar with any type of warehouse, any type of lake architecture you have to like keep track with what's happening. This is a big part of what we do is try to understand where is the data engineering world is going, because this is where also our users and customers are going.
**Eldad:** By the way, did you try the new latest DBT, Spark integration, any feedback on that?
**Roy:** We have tried it. We actually trying it as we speak so this is something we're building.
**Eldad:** Nice
**Roy:** Yeah. Working nicely. We also ran it based on Presto, which worked nicely and I'm hearing integrations are coming also from your end so we will be trying that as well. As feedback, I love this tool, I think this is the way to go. For us it's been very fitting because of the variety and the constant need to create more pipelines and other complexities and orchestrate everything, and to have it in a way that we kind of democratize it among data scientists and analysts both internally and externally in a way. So, you kind of have to have a unified layer where engineers can sleep in their beds quietly at night without having to worry, waking up on pipelines breaking, over schema changing stuff like that.
**Boaz:** Let's do a quick switch to a fun and blitz round, in which we will ask you a variety of questions where you're not supposed to think too much, just answer quickly. There are no wrong answers, only yes and no. There are no wrong answers only except the ones that are wrong.
**Roy:** Okay.
**Boaz:** Okay. So, are you ready?
**Roy:** Yes.
**Boaz:** Commercial or open source?
**Roy:** Open source and commercial.
**Boaz:** Batch or streaming?
**Roy:** Batch. Streaming is mini batching.
**Boaz:** Write your own SQL or use the drag and drop vis tool.
**Roy:** I like the SQL, come on.
**Boaz:** Work from home or from the office?
**Roy:** Office.
**Boaz:** AWS, GCP or Azure.
**Roy:** Oh, wow. Hopefully all of them, but AWS, GCP.
**Boaz:** To DBT or not to DBT? Although, I think you hinted...
**Eldad:** Not to DBT, always DBT.
**Roy:** To DBT.
**Boaz:** To Delta Lake or not to Delta Lake?
**Roy:** Delta Lake.
**Boaz:** Okay. Thank you. I think Roy answer differently than typically.
**Eldad:** Yes.
**Boaz:** I think you're the first one who said from the office, like home or office.
**Eldad:** Yes\*\*.\*\* Everyone was confused.
**Boaz:** People typically say either home or both.
**Eldad:** Yeah.
**Boaz:** Nobody's just says office, except Roy.
**Boaz:** How many kids do you have?
**Roy:** I have one. Right now, we're living in a very small apartment so working from home is super hard, but also, I like the fact that office is where you work and home is where you live.
**Boaz:** They're separate.
**Eldad:** Old school.
**Roy:** Yeah, old school.
**Boaz:** Nice.
**Eldad:** Nice.
**Boaz:** So, what were some of the bigger data challenges or bigger projects you guys had at Explorium in the last year?
**Eldad:** Traumas, huge success, huge surprise.
**Roy:** As I said before, one of the biggest challenges I think that we have is that we kind of have to be good with any data so the platform and the infrastructure that we build has to be kind of generalized from inception which is hard. And most of engineering organizations and I'm guessing that also your engineering organization, is trying to the right thing and not generalize too early, and not try to have the wrong abstractions over things but when we kind of have to, because in a way, if we're not abstracting in the right way, or if we're not generalizing enough, then with any new source that we find, we kind of have to tweak everything around it. So, this is one big challenge I think that we have. One other challenge that I think is interesting is that you have to understand your user when he uploads, if you look at the auto-ML flow, for example. When the user uploads his data, you have the challenge of understanding exactly what he's searching for and exactly what his data means and that's one challenge and also what he's actually searching for.
Because sometimes you'd be surprised it's not the features that bring the most correlation, sometimes it brings more knowledge. Sometimes knowledge is important when you're doing that analysis. We have users coming to us saying, this is an interesting feature. It's not always with the highest correlation to a target, it's not always with the best statistics, but it's interesting because it tells me something about my business that I didn't know. So, this is another challenge and democratizing this data platform that we built. This is also a big challenge because you have to enable data scientists, both in Explorium and outside and data analysts who actually work with high volumes of data, complex pipelines, and be able to build their own processing pipelines and features and you have to enable them and abstract them from the engineering underneath. You asked before about DBT on top of Spark, I think the cool thing is that you're just writing SQL queries and underneath it runs on hundreds of machines on Spark.
**Eldad:** This is amazing. I remember the days it wasn't long ago where people are so excited about Scala and Java and writing those Spark jobs and owning the threads and the machines and the hardware and the wiring. And it was all poof, it was all gone. I think part of that, or maybe a big part of that is cloud native data warehouses like Snowflake and BigQuery that actually kind of taught many of us that it's okay to abstract, to simplify, and it's okay to have that decoupled from your day to day so thank you Snowflake and BigQuery for teaching Databricks that SQL is good. Now we see everyone, many people we talked, they're using SQL over Spark. That's kind of the biggest change we're seeing moving from developing it to actually declaring it using SQL and as you said, you love it and most people do love it, and it's a good change. SQL is back. We were confused for a few years. We had no SQL, we had new SQL, we had side SQL.
**Roy:** And now SQL is back.
**Boaz:** This is the longest secret rant I've heard.
**Roy:** Exactly. And it's important to rant about it.
**Boaz:** Okay, this is all exciting stuff. Let's talk about it from a more, maybe personal perspective. Tell us about something that didn't go well. Tell us about what we call an epic failure that you guys ran into. Maybe an approach that didn't work well, lessons learned and such.
**Roy:** I have a personal failure so I'm going to talk about my personal one, to own it. You asked me about batch versus streaming. When I started Explorium one of the things that was clear to me is that we need to have this kind of stream ingest, complex stream ingests, that we will have to report some events because we're working with such a variety of sources some of them are more dynamic by nature, APIs, for example. We have many of those and we needed some way to dynamically both enrich using those APIs and then propagate the data again to the data lake and re-ingest it and reprocess it. So, I was building this wonderful Kafka based streaming connector for everything with the best abstraction in the world and it flunked essentially.
**Eldad:** Abstraction is slow.
**Roy:** Yeah, but you got to learn, you got to learn the hard way.
**Boaz:** Why didn't it work?
**Roy:** It was a premature obstruction in a way. And it didn't match the way that now we're looking at processing because we have so many processes running offline because we have to do this complex modeling and connect different sources into one knowledge that we do which happens offline in batch jobs. And it's very important to be able to support quality at scale. Data quality is one of our team's priority and challenges that how do you maintain quality over such a variety of data sources. And there isn't really a good way to provide super blazing fast latency along with quality, because there is some processing that you have to do behind the scenes and so that streaming approach was a bit premature. And now when we're looking at streaming, we're looking at streaming as an additional entry point to those periodic jobs that can run or more batch type of jobs that can run, and look at streaming as it's just another way to get data into our lakes, warehouses, and then, from there to processing.
**Eldad:** So, what you're saying, I might stream data in, stream on write, but it's always batch on read because the batchy part of the schema will make all the streaming one's batchy. So streaming is a one-way ticket, and then you start needing to analyze.
**Boaz:** We see it often. Often, we'll see streaming as just the way I put data in my lake or whatever, but it doesn't mean that end users really enjoy that streaming and that low latency of the data coming in. Sometimes we see that, but definitely it's actually rare.
**Roy:** For a lot of cases, we have updated live data. For example, take weather. Whether is something that, first of all, how much history of weather do you keep and how you do it? So, with whether we work with APIs, for example, when users have to have the data for now, or today, or one hour ago, or right now, you have to deliver it through APIs in live in real time. Every recipe and every auto ML pipeline that we build internally also has this real time face where the user wants to consume it because he has his user or his customer waiting in checkout or whatever. So, we have real-time, but the complex processing and understanding exactly what is the data and modeling it, this happens in batch,
**Boaz:** Let's now move to something positive. Tell us about something you're proud of, a big win in your data work.
**Roy:** We have a big win. I think that combining modeling with the right serving infrastructure, it is a tool that we build internally that enables us to have a more quality matching capabilities over variety and complexity of data. I think that the win was when we started onboarding more and more and more data sources and you have this feeling when you take this new product to production and it works.
**Eldad:** A machine, a working machine.
**Roy:** Yeah, so you kind of reflect and you say, okay, so it was worth investing all of this time really understanding modeling. Data is always talking about something in the real world. Always there is something in the real world that generated this data and when you're actually able to model the real world correctly, then somehow the data behind the scenes kind of falls into place when everything kind of makes sense and I think this is what we saw, and this was really exciting.
**Boaz:** This is a project. Initially we started talking about data engineers versus software engineers and we often talk about the boundaries of blurring between the two. So, such a project where you build that super-interesting flow, do you consider this a data engineering challenge, a software engineering challenge, or both? And which skill set did you need internally to deliver that?
**Roy:** Wow. That's a question I actually talk to a lot of people a lot - what is a data engineer and how is it different from being a software engineer or big data developer? So, I look at the developer world, we have software developers and you have software engineers and you have data developers and you have data engineers. I look at development like writing logic versus engineering, which is construction actually, in a way it's construction. Foundations and understanding things in lower levels.
**Eldad:** So, you're saying that in many ways you're dividing engineers into two groups, those who are responsible to generate the data and deliver it and those who build something on top of it.
**Roy:** I'm honestly not sure where the line is because I'm also looking at the line between where is data engineer meeting an ML engineer, or a software engineer, are they the same person? But I think essentially that the problem I was talking about was the data engineering problem, essentially. Because it had this element of data model and schemas, indexing, how do you index data correctly? It has a lot of those elements that I think makes it very data engineering. So having these schemas and model on one hand, but understanding, which is the right infrastructure to kind of hold everything in place, so it could scale natively I think it's a more of a data engineering problem.
**Roy:** You mentioned quality a lot, data quality and the importance of data being at high quality. How do you treat quality internally within your data pipelines and flow and data stack?
**Roy:** When you're thinking about finding the right data, when you kind of try to think, okay, my user wants to use the platform to actually find the right data, sometimes it doesn't even know what is the right data. It has two elements to it, I think. All of them are under quality in some, in some form, but it has kind of the matching problem which is when a user is coming and he's talking about a certain company or a certain place. How do I know to match it? How to find the geography, the place that the user is talking about, or the organization that the user is talking about, or the combination between the two? So matching is one thing that leads, so our approach there was enabling experimentation. Because it's very hard to get like ground truth because almost every aspect of data that you look at has like those amazing challenges and complexities that only when you start getting your hands dirty you understand what it is, what are they.
So, matching really affects quality. We treat matching as an experiment as understanding how do we tune exactly in every case, the system, and this is where enabling a generalized system that works well with metadata and enables the data owners and the domain experts to change and tweak it and play with it to see that it fits with the real world or with their expectations, so relying heavily on domain experts here. So, this is one thing, the other aspects are correctness. Even if I found the right organization, the right place that the user was talking about and I want to get back points of interest, for example, I need to make sure that if I said, there's a coffee shop there, then there is a coffee shop, that the data is correct.
So, I probably found the right place, which is one thing, but now I need to make sure that I retrieved back data that is correct. And then after you did these two, which are super complex, you have to ask yourself, is this relevant? If my user is trying to predict the sales of his products or optimizing routes for shipping or whatever, is having a coffee place something that he needs to know? Maybe if there's a coffee place in Tel Aviv, people are stopping their car to get coffee and there's always you know, cars, tailing, it takes five minutes more or whatever so those small nuances. So, combining all of those, I think this is how we treat data quality.
**Boaz:** Interesting. Thank you for that. What gets on your nerves the most in your daily work with data? What contributed the most to your hair growing whiter?
**Eldad:** Frustration.
**Roy:** Inconsistencies. We work with a lot of data vendors and you can actually see vendors that they own their data and they provide a consistent stream or consistent delivery. Schema is our consistent file format. You know what, I'm writing my answer. File formats. It's 2021, use compressed RK, use something with Schema. It's not that hard. We were working with textual data, delimited with pipes, and weird stuff and when you compressed with weird compression, you ask yourself why?
**Roy:** I think that's a good new corner we need in the podcast.
**Eldad:** Yes, data format.
**Boaz:** Not data formats, just let people let off steam from the stuff they have, because let's face it, engineering has some pretty nasty parts in our day to day.
**Roy:** Data counseling with the Data Bros.
**Boaz:** So, the rant about your stuff you hate corner.
**Eldad:** Do you support XML as a data source?
**Roy:** We have to.
**Eldad:** Wow. That's amazing. Nice.
**Boaz:** Wait, what are you saying? If you hear me then?
**Roy:** Then use a compressed parquet, Snappy or whatever.
**Boaz:** So, data engineers worldwide, let's agree to use only that from this point forward to make our lives better.
**Eldad:** Please use faster decompression with Snappy. That's the ask here.
**Boaz:** Standardized boom.
**Eldad:** Everyone that hears that. No Zip, no nothing else. Just Snappy.
**Boaz:** How do you stay on your toes in terms of being updated in what's going on in the data world? Any sort of tips on who to follow?
**Eldad:** Aside from this podcast.
**Roy:** I'm an avid Googler when it comes to maintaining my knowledge. I'm following several... I'll try to find it, but I'm using Daily Dev, I don't know if you know them. I think they're Israeli as well, I'm not sure. So, I use Daily Dev, which is kind of a Chrome home page that connects me to really interesting sources.
**Boaz:** Daily Dev, okay.
**Roy:** I'm following several. You probably get lots of awesome pages that are awesome data engineering, awesome ML ops awesome ML engineering so I'm following those. And I'm trying to find projects to follow and kind of lead from there in a way. One example, and people make fun of me on that, but I'll say it anyway. Now I have it on record. You know Spark, everybody knows Spark. I was following Spark as a project and then I started following kind of the people behind Spark and the laboratory in Berkeley behind Spark and then they emerged with this new laboratory that brought us eventually Ray which is a project that I've been following for two or three years. No one was talking about it, no one was using it and a while ago, I'm kind of iterating back and I was getting into the project again, and I'm saying they really made progress and they released a general version like the first full production version and that was really exciting and we started talking to the team behind it.
So, I think it's a lot there. So, finding those projects and then trying to find a way to actually talk to the people. Everything is virtual today, but talking to the people is great, using like Slack channels. I think those are the places where you're actually able to, as you said, kind of be on your toes and be on kind of the verge.
**Boaz:** Yeah. I think like the community and people aspect in data engineering is huge.
**Eldad:** Huge. 36
**Boaz:** Much more than even generalized software engineering because its space is growing so fast, changing so quickly and unless you keep track and keep your eyes open to the open-source projects, to people talking about this and that you lose a lot of great things. And I think there's a lot of community power even without that being formalized that impacts this space we're in, which makes it exciting.
**Roy:** I have this WhatsApp group of data engineers and yesterday someone asked a question about how do I process? I want to process this and that data and the data is partitioned this way and so on. And I was tagging Boaz, I think it's a relevant case.
**Boaz:** It's an interesting case let's share with our listeners. Typically, when we say about communities, we talk about all the famous and open and...
**Eldad:** Big broadcasts
**Boaz:** Forums and insights we're on. Here we have something very local to Tel Aviv maybe, but there is sort of a local data engineering group on WhatsApp, which is very popular here for instant messaging, it's a very local, but very effective group or local practitioners get advice on their daily challenges with data. So maybe the tip from us would be localize more communities, it doesn't necessarily have to be worldwide. Sometimes if you're in the valley, if you're on east coast, if you're on the west coast, if you're here or there, talk to people around you, they're always in the same times or to hop on a quick call, it makes a little bit more personal. So, it's an interesting approach, definitely worked well here for us.
**Roy:** I think this makes the difference. I'm not sure it has to be local, but it has to be a group where people feel comfortable enough to ask stupid questions sometimes.
**Boaz:** Exactly.
**Roy:** And get very straightforward StackOverflow, you know, top result answers. Once you have that, you know you have the right group because people are feeling comfortable and they will ask questions and discussion will move from there.
**Boaz:** Yeah. Good point. Thank you. Okay, I think we're almost done any last famous words for our listeners?
**Roy:** Listen to the data. In a way...
**Eldad:** Nice, nice.
**Roy:** Data engineers, it's very obvious with data scientists and data analysts, I came from data. I did a couple of years of data science, pure data science, and I actually got into data engineering by accident twice. So, the beginning of my career, and then I deviated to data science, and I got back to data engineering again, somehow.
**Boaz:** Keeps pulling you in.
**Roy:** Yeah, it found me, I didn't find data engineering, but listen to the data. If you're able to have the understanding of the data scientist as an engineer, I think it your life a lot easier.
**Eldad:** So, listen to the data and if you don't like what you hear cleanse it.
**Boaz:** Listen to the data is great. It's t-shirt bubble.
**Eldad:** Yes.
**Boaz:** Abstract enough to be a debatable for hours, and it's catchy, well-done Roy. Okay, good. So, thank you so much everybody for joining us for another episode, we'll see you next time. Thanks again, Roy. Bye-bye.
**Eldad:** Bye-bye, everyone.
# Building Geospatial Support in Firebolt (Part-I) (/blog/building-geospatial-support-in-firebolt-part-i)
As the demand for location-based insights grows, integrating geospatial capabilities directly into a data warehouse unlocks a range of powerful use cases—from real-time tracking to spatial analysis and mapping. In this series of blog posts, we'll take you behind the scenes of how we developed robust and fast geospatial functionality for our cloud data platform, enabling users to seamlessly store, query, and analyze spatial data alongside their traditional datasets. We'll walk through the key challenges we faced, the technologies we leveraged, and the architectural decisions that made it all possible.
Geospatial support allows Firebolt to store and process spatial data, such as points, lines, and polygons, that represent real-world locations and features. We'll dive into the various components that make this possible, from specialized libraries to performance optimizations and data pruning.
In Part I, we'll begin by exploring the two main types of geospatial support: GEOMETRY and GEOGRAPHY. We'll explain the reasons behind our decision to implement the GEOGRAPHY type and introduce the S2 geometry library, which we use to power geospatial features in Firebolt. We'll also discuss the benefits and challenges of leveraging this library with a brief look at how S2 Cells can help speed up geospatial functions.
In Part II, we'll examine the underlying architecture and storage infrastructure that enable both high-performance geospatial functions and efficient data pruning through spatial indexing. Here, we will discuss the importance of verification and normalization of inputs during ingestion and see how Firebolt persists S2 cell coverings in its storage layer which enables powerful pruning as well as function optimizations.
In Part III, we'll take a closer look at how we achieve fast and reliable geospatial operations with S2. While S2 offers robust tools for geospatial functions, these capabilities can come at a performance cost. We'll dive into how careful engineering helps mitigate these performance trade-offs by making use of fast checks using S2 Cells before doing expensive snap rounding operations that are required for robust functions.
### GEOMETRY vs GEOGRAPHY [#geometry-vs-geography]
When dealing with geospatial data in a database, most systems use either a GEOMETRY or a GEOGRAPHY type. At Firebolt, we decided to implement the GEOGRAPHY type as it is more accurate for geographical data. Let's have a look at some of the differences between the two types:
GEOMETRY models data on a flat plane. Since Earth is, in fact, not flat, this means that distances and other relations between objects, like containment, become inaccurate the further apart and larger they are. You also have to decide on a projection to map your inputs onto a flat plane, adding complexity to your queries.
GEOGRAPHY models data on a sphere, which much more closely resembles the shape of the earth. Distances and other relations between objects remain accurate, even on a global scale. A projection is also unnecessary since coordinates are directly mapped to the sphere according to the WGS 84 coordinate system.
For example, let's have a look at the following query where we test whether the point at longitude -100 and latitude 45.1 is contained in a large area defined as a polygon within the USA.
The corresponding query using the GEOMETRY and GEOGRAPHY types using the PostGIS expansion for PostgreSQL return different results as shown below.
```sql
-- GEOMETRY: Returns false
SELECT ST_Covers(
ST_SetSRID(ST_GeomFromText('POLYGON ((-115 45, -115 35, -90 35, -90 45, -115 45))'), 4326),
ST_SetSRID(ST_GeomFromText('POINT (-100 45.1)'), 4326)
);
-- GEOGRAPHY: Returns true
SELECT ST_Covers(
ST_GeogFromText('POLYGON ((-115 45, -115 35, -90 35, -90 45, -115 45))'),
ST_GeogFromText('POINT (-100 45.1)')
);
```
### Implementing GEOGRAPHY using the S2 geometry library [#implementing-geography-using-the-s2-geometry-library]
Implementing GEOMETRY or GEOGRAPHY brings many challenges. Consider for example the common problem of determining if a point x is contained in a polygon. We can do this by choosing a point y that we know is outside of the polygon, and counting how often a line between x and y crosses a polygon boundary while making sure to account for edge cases like crossing through a vertex of the polygon, where we might falsely determine that we cross two or zero edges of the polygon.
Of course, implementing GEOGRAPHY comes with its own set of problems: We need to implement all geometric primitives in a way that accounts for earth's curvature and make sure that objects crossing the antimeridian (180 degrees east or west) or the poles are handled correctly.
Solving these problems is hard and takes a lot of time. Fortunately, the S2 Geometry library that powers Firebolt's GEOGRAPHY type and functions can do a lot of the heavy lifting. It provides abstractions for shapes like points, polygons, and line strings and can do many useful computations on them like testing whether one shape contains another shape. It even provides the tools to build powerful spatial indexes using space filling curves (more about this in a later blog post). Thank you to Google and Eric Veach for open sourcing and maintaining the [S2 Geometry library.](http://s2geometry.io/)
S2 also provides powerful tools for spatial indexing by dividing the Earth into a hierarchy of approximately square-shaped cells. This recursive division allows us to approximate complex shapes or collections of shapes through what are called coverings—sets of S2 cells that together represent the shape. One of the key advantages of using S2 cells is the efficiency of intersection tests. Since each cell is represented by a single integer, determining whether one cell intersects with another can be done using simple integer comparisons, which are extremely fast.
This efficiency is especially valuable for spatial queries with selective filters on spatial relations. For example, when querying a table to retrieve all points within a specific polygon, we can significantly reduce the number of points we need to examine by testing whether the covering of the polygon intersects with the covering of the point set. This allows us to quickly rule out large portions of the dataset that don't meet the query criteria, improving both the speed and scalability of spatial queries.
We will go into more detail of how spatial indexing and pruning works in Firebolt in a future blog post.
Of course, these are not the only interesting problems we have to solve and optimizations Firebolt can do. Stay tuned for future blog posts where we talk about some of the most interesting problems we had to tackle as well as how we tuned our in-memory and on-disk representation of GEOGRAPHY to enable fast geospatial functions and powerful pruning.
Try Geospatial today by [signing up for Firebolt for free](https://go.firebolt.io/signup) and look into our [GitHub](https://github.com/firebolt-db/firebolt-demo/tree/main/geospatial) for a cool [demo](https://demo.docs.firebolt.io/geospatial/).
# Building Uber's AI Assistant: How Genie Revolutionizes On-Call Support with Paarth Chothani from Uber (/blog/building-ubers-ai-assistant-how-genie-revolutionizes-on-call-support-with-paarth-chothani-from-uber)
In this episode of The Data Engineering Show, the bros speak with Paarth, a Staff Engineer at Uber, about his work on Genie - an innovative AI assistant that revolutionizes on-call support by combining RAG (Retrieval Augmented Generation) with agent-based automation to help engineers find solutions faster.
Listen on [Spotify](https://bit.ly/4nZd6wE) or [Apple Podcasts](https://bit.ly/415Xrll)
**\[00:00:05] Benjamin:** Hi. This is Benjamin. Before we start with today's episode, I wanted to quickly reach out on a personal note. We've just launched Firewall Core. FireVault core is the free self hosted version of our query engine. You can run core anywhere you want, from your laptop to your on prem data center to public cloud environments. Core scales out, and you can run it in a multi-node configuration. And best of all, it's free forever and has no usage limits. So you can run as many queries as you want and process as much data as you want. Core is great for running either big data ELT jobs on, for example, iceberg tables or powering high-concurrency customer-facing analytics on big datasets. We'd love for you to give it a spin and send us feedback. You can either join our Discord, enter our GitHub discussions, or you can just shoot me an email at [Benjamin@Firebolt.io](mailto:Benjamin@Firebolt.io). We'd love to hear from you. We added a link to Firebolt course GitHub repository to the show notes. And with that, let's jump straight into today's episode.
Hi, everyone, and welcome back to the data engineering show. Today, it's our pleasure to have Parth joining from Uber. He's a staff engineer there. Welcome to the show. It's really great to have you. Do you wanna tell us a bit about yourself, about your role at Uber, what you're working on?
**\[00:01:13] Paarth:** Thank you, Benjamin. Thank you, Eldad, for having me. And, it's my pleasure here. Yeah. So I've been at Uber last four years working on Michelangelo, which is our, like, SageMaker, like, ML platform at Uber. I've been working on feature store, you know, online serving at scale. Basically, we have millions of requests that we need to scale for for with with many models. And then last couple of years, I dove into Gen AI, working on rag, vector search, and building apps like Genie that we are going to talk about, where basically we had on-call productivity, you can say, pains that we wanted to solve out on our own. And that's how Genie organically just came out of a hackathon, as you're talking about. And then it just literally just started growing, last couple of years. And, Arnab and team have been one of our partner teams, and we have been working with many partner teams to really uplevel the bot because we realized that just the bot, the plain rack, doesn't work. So that's how we started this journey of accuracy improvements. Yeah.
**\[00:02:17] Benjamin:** Okay. Very cool. So for those of a kind of listeners who never heard of Genie, I hadn't heard of Genie before. Like, you wanna get the bit more context of what role it fits within Uber, kind of what workloads it serves, basically?
**\[00:02:30] Paarth:** Yeah. Yeah. For folks who don't don't know Genie, basically, Genie is, like think of it as, like, your on call assistant. Right? So and Uber is big into Slack usage. So different infra teams have their Slack channels, whatnot. I think different companies use Teams also maybe. But, basically, you go to a Slack channel. Let's say you have a problem with Spark or you have a problem with Fling or any of those open source technologies. You go to a channel where you have your infra team engineers helping you. And because these technologies are widely used, you have to wait a lot. Sometimes, engineers are dealing with, high severity issues, and they don't have time to help you with every small issue that you're running into. And documentation, as you know, engineering is not easy to maintain. So that's where, you know, the pain started coming that, okay, we are waiting on calls to respond, and then you're feeling frustrated. And that's where it felt like almost like a bot assistant would be very helpful, which can search for you from different services, different documentation, and give you the answer so you do not waiting on on calls as much.
**\[00:03:31] Benjamin:** Nice. Super cool. Can you take us through, like, that data pipe that powers Genie? Like, where are you getting data from and to what systems does it feed? How much data are you handling? Tell us more about it.
**\[00:03:41] Paarth:** Yeah. So, I mean, data sources, what we realized is for our engineers to really get help, data sources really should be internal only because we customize lot of these open source engines for, making it work at Uber scale. So we have data sources like Wiki, Jiras, Stack Overflow. We have our own version of Stack Overflow. We have Google Docs, and then we have people even storing data, custom, you can say, policies, custom information in PDFs. So there is variety of these data sources, and, obviously, source code is another one now. So we listen to and ingest all these different data sources in our own hosted vector DB solutions. And then on top of it, we basically want to do, look up semantic search. And then we're obviously what we're going to talk about, like, how we are customizing search and retrieval for each particular use case to really make it better.
**\[00:04:37] Benjamin:** Okay. Are you able to share how much data Genie is handling overall?
**\[00:04:41] Paarth:** Maybe not the specific numbers, but at this point, close to three fifty plus channels. That's quite a lot. And every channel basically has some flavor of their data that is we are ingesting. So what we have tried to do is instead of building because we are a very small team, instead of building a mega scale pipeline that just ingest all data sources and then keeps a central data source solution, we instead are giving users the flexibility to ingest what data sources they want. Right? And then what we found also is that works better in terms of search. So at least for every team, they are looking at, like, let's say, hundreds of thousands of Wiki pages, hundreds of Google Docs, and then we have design docs that we also, you know, ingesting that. So I would say per use case, you're looking at with a compressed ratio, at least few 100 megabytes to gigabyte or several gigabytes depending upon how much the user is wanting to ingest right here.
**\[00:05:37] Benjamin:** Okay. Super interesting. And as the core vector store, is that also open source technology you're customizing at Uber? Is that like an out of the box existing piece of data infrastructure? Tell us more about that maybe.
**\[00:05:50] Paarth:** Our team doesn't manage the core technology itself, but, yeah, we use OpenSearch as one of the vector DB solutions. Um, we also have our own ingrown vector DB solution as well. So which is what our sister team in the search org manages for us. But, yeah. I mean, it's it's one of the those things that has become popular in general. We're talking about Lance TV is another alternative. And you guys obviously do Firebolt as well, but, yeah, OpenSearch seems to at least our team search team seem to feel like OpenSearch would be a good solution for us. Yeah.
**\[00:06:23] Benjamin:** Right. Makes sense. Was that the first time in your life you built these, like, kind of rag style pipelines at scale? Like, it's new tech for everyone. Right? Kind of how did you even onboard into that? Like, how did you figure out how to make good technology choices? Where did you learn these things from?
**\[00:06:40] Eldad:** And does it feel like doing the Hadoop days at the beginning again? So it's kind of like, okay. Basically, blank sheet. We need to come up with the whole stack from scratch. There is some open source spread at some places, engineers going back to building new infrastructure to serve new workloads. That must be, like, three times in a lifetime kind of exciting thing. Right? Doesn't happen every day.
**\[00:07:06] Paarth:** Yeah. Yeah. 100%. And it this funny thing that while I was at Amazon, I was part of this AWS chatbot team, and we were exactly building same same thing there. And that was pre-LLM era 2018, and it almost seemed like I was like, okay. And it feels nostalgic to literally rebuild what I was building at Amazon before.
**\[00:07:24] Eldad:** So you know how everything ends up. Right? Everything ends up as a cloud native data warehouse. That, that's what happened to Hadoop eventually. Right?
**\[00:07:32] Paarth:** Absolutely. Yeah. Yeah.
**\[00:07:34] Eldad:** But, like like, looking at the tech, looking at the workloads and and those new use cases, like, it feels like innovation at scale is re-happening from scratch again. And this is exciting. To us, it's super exciting, and it must be even more exciting to you given the access to the data that you the team has. So tell us more. Like like, how did you stitch it together? What are you specifically proud of? How do you see that evolving in the next few months? Let's not even think twelve months.
**\[00:08:02] Paarth:** Absolutely. Just to even connect the journey. Right? Like, when we started, it it almost everything just happened very organically. I don't think we planned everything. But, so there was, um, your question of how do we stitch everything together? It yeah. It almost felt like they're doing what EMR was doing. , you know, you have your Hadoop and big data technology, and we needed these pipelines to basically process all this data quickly. And then that's where we started betting on Spark to really help us. And Uber, uses Spark a lot. So we were like, okay. Let's let's go with what is proven well at Uber. So we bet on Spark. We use Spark a lot for, you know, data processing, data parallelly, having to chunk it, shuffle it really quickly, and make sure we can create embeddings at scale because that's really the main two bottlenecks. You chunk your data, you have your all data ready, and then parallelly create the embeddings at scale. Right? So we had to basically scale our you can say the, um, whole infrared layer to chunk data faster to be able to create embedding set scale. And so that that meant also we had to scale our, what we call as gateway engine that, really helps us create those embedding set scales. So we had to scale all these layers. And where I see going with this, I think that's maybe, a very hard question given how fast this technology changes and the expectations of customers is growing so much. But I do see, like I mean, I think, in general, what we are seeing definitely is people want more customization, so which means more Uber internal data sources needing to come on the platform. And then, definitely, everybody is going, going with the agentic route now at this point. And, few use cases that that we have really, really nailed down and worked well, including the one that we published in the blog, you know, we definitely see agents as the way to interact, and especially MCP servers have come now. So agents, MCP servers, and your data custom data sources, stitching it all this together is really how you can make, at least in my opinion, very good, Genya app now.
**\[00:10:08] Benjamin:** The innovation, the space has been, like, super crazy. Like, how quickly there is new frameworks popping up, kind of gaining tractions, etcetera. Like, we're working on some agents now that, for example, like, optimize your SQL queries and kind of these types of things and are deeply built into the product. For me as well, like, kind of, like, having, like, this system c plus plus programming background, it's been fun learning about this, like, completely other part of tech ecosystem. And then one thing I'm personally very curious about is, like, the intersection between, like, LLM and kind of SQL query optimizers because I have this, like, more traditional compiler query optimizer background. Right? And, like, now we're kind of fusing it with LLMs and, like, figuring out, okay, like, when are LLMs great, kind of when are more traditional query optimizers that, like, reason about correctness great. Crazy how in, like, so many pieces of technologies, like, you're now, like, infusing these LLMs and kind of realizing, oh, damn. Like, you can build mind blowing things that were impossible before.
**\[00:11:06] Paarth:** Yeah. Yeah. Totally. And that you hit the point really well that this whole new suite of ways to connect with the databases have come. Right? Like, it is not there. I was reading some blogs about how Google has, basically had this agentic thing which can connect you to all Google Cloud technologies. And, I mean, we at Uber also have done I believe there's another blog by a sister team done basically query Copilot with what they call it, which is your query optimization using LLM and, you know, making sure you can write quick queries. Right? And that's a new suite of things that is not even you know, nobody even thought about that. But, yeah, you can pretty much write all your queries using LLM and build that framework.
**\[00:11:47] Eldad:** Let's not get into predictions on how painful that's gonna be on the market. But if we just focus on the optimizer within the database, it being owning that brain for the last forty years, being responsible to make all smart decisions for any user, any query, any architecture. Like, the one thing that hasn't changed is the way the optimizer feels about the users. It needs to make most decisions for that. Now it's changing. Now the optimizer has part of its brain being outsourced through an LLM, gets back the recommendation. It still needs to do optimizations at lightning speed because, right, you're getting tens of thousands of queries in. You need to optimize all queries. So you need to get feedback from the optimizer, and then your database now needs to think differently. It needs to allow to infuse those hints, that context, that insight that the new LLM optimizer went through into any query. And that's changing how database engineers and databases are thinking about optimization, but it what it also does, which is fairly excites Benjamin and us and anyone who deals with efficiency, it changes the way databases are being released and what the focus is. If my LLM can go for thirty minutes and go over 50 dimensions of optimizations and figure things out that it would take me maybe a week or two people in my company on Slack giving an advice how to change the granularity of a block or how to change the threshold of RAM or whatever optimizers do. Now it's all about how much can we expose, how much technology, how much variation, and, like, how specific can we get. Right? There are 100 ways to implement a join algorithm. Now we're gonna expose all of them because now DLLM can actually make sense out of it. Like, that was unheard of. So it's all about obstruction. Right? Like, you always have to be straight off. Every database engineer will tell you. There's a trade-off between being fast and how fast you get to being fast. It's hard to get fat to be fast so that that, right, that snowflake, abstraction that we like to say comes in, but now it's changing. So now everyone is mama's genius with an LLM. Every agent can just spend this thirty minutes on running queries and getting actual results, actual telemetry no human being has ever looked at. There are levels of telemetry that no human being should ever look at, but agents love looking at it.
**\[00:14:16] Benjamin:** Even I have certain levels of telemetry I don't wanna take a look at when building the database where agent will happily do it for me, and he'll thank me after saying kind of for the opportunity to look into it. Exactly.
**\[00:14:29] Eldad:** So, so, yeah, so it's exciting. And then instead of show tables and show statistics or explain syntax that you build for users, you're now, okay. So how do I actually provide granular, summarized, relevant profiling information for agents so they can learn fast sending less data. So this is very exciting. Um, and this is gonna absolutely change how we build software, and we're not even getting into infrastructure. So it's really exciting that you guys like, the way Uber has always been flexible and open-minded. Right? Like, it's for the last, I don't know, more than ten years, going to the Uber blog, you go there, and there's always engineers experimenting at scale. So meeting you now is exciting and really, like, hearing about it. So tell us, how do users react when you're telling them you use Genie and then you feed it with your completely random Slack chatter, and then it gives you documentation you could have drained off? How do they react to that?
**\[00:15:29] Paarth:** Yeah. And I think we've even evolved from, okay, just being like, okay. Hey. We will give you the right documentation to, like, okay. It was starting to evolve into a situation where you're like, okay. We'll also start taking actions on your behalf. Right? And that's really where I see, I think, to our conversation about the databases being smart and, you know, doing so much preprocessing before you even come to the database. I think it's getting similar situation where the bot can do a lot more now. Right? And I almost connected back to my AWS journey where we're literally trying to do the same thing in natural language where you could manage all your AWS resources using natural language. And and that's really where I think we are going with this. Any problem, be it like, okay, I have a permission issue. Okay. I I need to, let's say, update my spark, resources. I need to get more capacity. Pretty much all of that can be done now but behind the scenes not only bought looking at your documentation through MCP servers now coming and pre previously, obviously, we had the tools that we are all the internal tools that we built to have the LLM be able to leverage it. All of them can pretty much now take actions on your behalf. Right? And that's where agents become so powerful that you can have different sub agents which can take specific action for you, and then you have your this intent agent supervisor which does preprocessing of the query and takes and figures out, okay, which particular sub agent is really the right one for me to, you know, resolve this question? And then that sub agent goes, takes actions, look at the documentation, can do a lot more things, because of this whole agentic framework that has come with LangGraph and other technologies, obviously.
**\[00:17:08] Benjamin:** Nice. Super cool. So if you contrast this to your time building similar looking technology for the end user at AWS, right, like, it is a completely different technology stack, right, in the sense that it's like you solve the same problem, but it's like man or humankind figured out, like, a smarter abstraction to actually solve these problems in a better way. Like, maybe take us through that also. Like, do you feel like the things that you did then at Amazon helped you become a better engineer in this new age, or do you actually think, okay. It's like going back to kind of zero and kind of rebuilding all knowledge you have in this space?
**\[00:17:44] Eldad:** Just my instinct that I've build over my career, and I can't explain that. Like, yeah. I think I mean,
**\[00:17:50] Paarth:** I feel like as you're saying, right, Piel, that right? Like, I mean, no instinct that you derive from goes based. Right? So at Amazon, whatever we built, it was pre-LLM era. So, definitely, I I would say the technology was maybe not as robust mature before. , but that intuition that, you know, you comes from, okay, building this kind of bot, I feel like that intuition came again as we were starting to see this technology come, and we're like, hey. This looks like, okay. Where you can pretty much fit all these pieces together. So I almost felt like, that experience in Amazon was like a starting experience of, okay, how the chatbots can really do lot more. And then this was like a stepping stone to say, okay. Now the technology is finally there. Now you can stitch everything together. It almost feels like going from level zero to level one as building the same similar experiences again. Yeah.
**\[00:18:39] Benjamin:** Yeah. It's funny. Like, we have these conversations every now and then with, like, data engineers. Right? Like, similar to what Eldad said. It was like, okay. Like, back in the Hadoop days and then, like, kind of, like, modern cloud data warehouses and then, like, next generation of data warehouses. And it feels like there's these cycles in every kind of area of technology where just you take that leap, but then some things kind of stay the same. So looking ahead, like, what are you guys working on right now? What are you particularly excited about at the moment? Kind of what are new challenges you want to solve that you might not be solving perfectly right now? Kind of take us through the next couple of months.
**\[00:19:16] Paarth:** I think where we have landed last six months, including the blog we published. Right? Like, so I think we found out that how we can do this agentic solutions, and I think what I call is agentic Genie now. Basically, Genie was our you can say traditional plain React rag style board. Now genie has become agentic genie. Right? So what we have seen with several use cases is agentic genie works well when designed well. When you've analyzed the problem of which type of subproblems the bot should resolve per channel, per use case, and then you go go with, solving each subproblem with each sub-agent. If you do that way, it works really well. Right? And that's what we have seen as delivered good success. I think where our challenge right now, which is where I'm still thinking and it's not something we have designed or thought too well, but it is funny that how or funny or good in a way that how cursor and other IDs have up leveled what agentic experience can mean for code. And I I kind of want to take that same cursor like experience for agentic genie, where you just come and without you having to do anything, figures out all the right sub problems for you as a channel owner, as a use case owner. We decide underneath which particular sub agents to call and, which particular MCP server tools to interact with, and it just figures out everything for you. That's what we'll where I would like this North Star to be. That way, it becomes agentic genie is like a perfect on call assistant what Cursor and other IDs have done for coding.
**\[00:20:52] Benjamin:** Nice. Okay. Super cool. So are you guys using Cursor to build Genie?
**\[00:20:57] Paarth:** I mean, every I think every developer is is likely using Cursor and any other of those, forms of ID experience and CLI experience right now. I mean, yeah, everybody uses that. Absolutely. It makes your life better for sure.
**\[00:21:11] Benjamin:** Yeah. Same here at Fireball. Very cool. So maybe zooming out from Uber a bit. Right? Like, what are you excited about in this space in general? Is there, like, kind of, like, how do you actually stay up to date when building these things? How do you learn about other how other companies are kind of tackling similar problems? Are there meetups you're going to in the Bay Area? Are you just reading blogs and watching YouTube videos all day? Like, take us through kind of that.
**\[00:21:36] Paarth:** Yeah. No. Absolutely. Obviously, meetups is a great way to meet. I think I've been to some of those recent, like, um, meetups. One of them was by Meta, which is called Scale at AI. It was really, really very well done meetup, and I was impressed with how Meta has optimized every single layer of infra, and they have done amazing. They have developed something called MetaMate, which is super cool. And when you look at those experiences, you're like, wow, there is so much more to do here. So I definitely deriving inspiration from all of the, you know, smart co peers in, different industries, different companies, and that's definitely one day. And then I think I I do see still, like, you know, when you apply those inspirations to your use case, there is still a quite a bit of steep learning, trying out different things. So nothing just works what work for other companies just like that. , there is, you know, dollar challenges, core cost challenges, scale challenges that you have to go back and redraw and figure out how it works. But, definitely, I think meetups, blogs, podcasts, like, what we are doing together. I mean, yes. I think looking at all of this is definitely one way to learn. Yeah. And this I think the Bay Area is really good in that sense that there's a lot happening here. So people are really excited to build things. So you find those exciting builders from startups to all the way companies that I've opened yet. Just trying so many things. So it's very humbling to be in Bay Area, and, you know, you just know that you're never done learning because there is new things coming out here in the market all the time, every single day right now almost.
**\[00:23:11] Benjamin:** Yeah. I mean, you're at the epicenter of, like, this next generation of software, basically. Very cool. So this was super interesting, Parth. Any closing thoughts from your side? Like, something you wanted to really kind of chat about, something you wanted to bring up on the data engineering show today?
**\[00:23:27] Paarth:** Basically, I would just say that I think people who are trying out, all these technologies, I would recommend everybody try as many things as you can. And, obviously, keep a problem in mind because just trying in itself, you can just keep endlessly experimenting right now, and you will not go anywhere. I think having a problem in mind always helps. That way, this it's the, energy is little bit focused and directed. I would say that is definitely my learnings, and I think keeping eye for experimentation open. I do see one little bit of a challenge building things. Right? That you build something, you take a bet on one technology, you build something, and then underneath, new things have already surfaced that keep you outdated. So there is I would say it's also a little bit of a pressure situation that whatever you're building is not enough because the expectation has already gone to the next level. So, um, I think it is a little bit hard to, you can say the pace is too fast right now. So as a developer, keeping your customer happy is hard, but then also setting good expectations with your customers is very, critical more critical than before. Because of what we shipped before in software, you know, the pace has completely changed now. So having that expectation and, obviously, keeping your, um, eye for getting feedback from customers is equally important. Um, being humble enough that you know that what you developed is not perfect, and you need to keep iterating and keeping making better. I would just say those would be my quick takeaways as we experiment and go with the next flow of technologies that are coming in here.
**\[00:24:59] Eldad:** Amazing. Nice.
**\[00:25:00] Benjamin:** Couldn't agree more. Thank you for having joined us on the data engineering show today, and we look forward to meeting in person when we're in the Bay Area next time.
**\[00:25:08] Paarth:** Absolutely. Thank you, Benjamin. Thank you, Eldad. It was great to to you guys, and I I really looking forward to meeting you guys also in person very soon, hopefully.
**\[00:25:16] Outro:** Sounds great. The data engineering show is brought to you by FireVolt, the cloud data warehouse for AI apps and low-latency analytics. Get your free credits and start your trial at firebolt.io.
# Caching & Reuse of Subresults across Queries (/blog/caching-reuse-of-subresults-across-queries)
### TL;DR [#tldr]
Ever wonder how you can speed up complex queries without additional resources? In this post, we will show you how Firebolt optimizes query performance through caching and reusing results of parts of the query plan (= subresults) across consecutive queries.
### Introduction [#introduction]
At [Firebolt](http://www.firebolt.io) we are building a data warehouse enabling highly concurrent & very low latency analytics. Another way to frame this is that we strive to be faster than the blink of an eye:
Firebolt's main use cases are "data intensive applications", such as interactive analytics across various industries. These workloads typically consist of high volume, sub-second queries stemming from a mix of tens or even hundreds of patterns.
Such repetitive workloads can benefit tremendously from reuse / caching. In analytics systems, caching as a concept is ubiquitous: from buffer pools over full result caching to materialized views. Here, we will present our findings in a surprisingly little-used approach: caching subresults of operators. The idea itself is not new, with first publications appearing in the '80s, e.g., [\[Fin82\]](https://dl.acm.org/doi/10.1145/582353.582400), [\[Sel88\]](https://www.sciencedirect.com/science/article/abs/pii/0306437988900142). More recent results include [\[IKNG10\]](https://dl.acm.org/doi/10.1145/1559845.1559879), [\[Nag10\]](https://homepages.cwi.nl/~boncz/msc/2010-FabianNagel.pdf), [\[HBBGN12\]](https://vldb.org/pvldb/vol5/p1436_alexanderhall_vldb2012.pdf), [\[DBCK17\]](https://cs.brown.edu/~kayhan/papers/hashstash.pdf), and [\[RHPV et al. 24\]](https://www.amazon.science/publications/why-tpc-is-not-enough-an-analysis-of-the-amazon-redshift-fleet).
In this blog post, you will discover how we implemented a cache for subresults of arbitrary operators in the query plan, including hash tables of hash-joins, and how this can significantly enhance the system's performance. By sharing the impact we have seen in our production environment, you will gain insight into how caching can help reduce query times and improve efficiency for your own workloads. Additionally, we will explore a variation of the eviction strategy which we benchmarked and tuned on real-world data, showing how it outperforms traditional LRU, ultimately saving you time and resources.
Since our cache for now is purely in main memory, we ideally store the subresults in compact data structures which have a small RAM footprint. We therefore devised a custom, space-optimized hash table for our novel "FireHashJoin". With it, we achieve more than 5x memory savings (as measured on production data) compared to the hash table implementation we used before. It thus enables us to cache significantly more of such subresults in the same amount of RAM. We will present this in a follow-up post.
### What do we Mean by "Subresult"? [#what-do-we-mean-by-subresult]
Let us have a look at the following made up query based on the [TPC-H](https://www.tpc.org/tpch/) benchmark schema (see, e.g., page 13 [here](https://www.tpc.org/TPC_Documents_Current_Versions/pdf/TPC-H_v3.0.1.pdf)).
Below we show a simplified query plan for this query. This is a graph which has "operators" (such as "*Aggregate*") as nodes. In our figure the data flows from the bottom to the top and each operator takes care of one processing step, such as aggregating the incoming o\_totalprice & row-counts and grouping them by n\_name. For a more detailed explanation, see, e.g., this [Wiki page](https://en.wikipedia.org/wiki/Query_plan).
So here is the query plan for the example query above:
For each operator, we call the entire data streamed out of this operator a **subresult** – it is what the operator and the subplan below it have computed. Example: all combinations of c*custkey, n\_name streamed out of the lower-right \_Join* together form the subresult of that *Join* operator.
We also refer to the subresult streamed out of the *Aggregate* operator at the top as the **full result**.
For the final artifact which we consider a subresult, let us add a **quick side note about our *Join* operator**. We implemented it as a so-called hash join algorithm. This is perhaps the most common join implementation used in databases. Briefly summarized, the algorithm proceeds in these two phases (for more details, see, e.g., this [Wiki page](https://en.wikipedia.org/wiki/Hash_join)):
1. In the *build phase* it computes a **hash table** for the "build side" of the join. In our case this is the right side of each of the joins depicted above. For the lower join, the computed hash table maps n\_nationkey to n\_name as read from the nation table.
2. In the *probe phase* the algorithm iterates over all rows of the "probe side" of the join (in our figure, the left sides of the two joins are the probe sides). For each row it probes the hash table, i.e., looks up the corresponding key coming from the left. In our example for the lower join, the algorithm would lookup n\_name given the c\_nationkey coming from the left side. The operator then outputs the combination c\_custkey, n\_name for each of the probe rows.
We also consider the **hash tables** computed by Join operators as "special" subresults of the query plan. In our example, the lower join builds a hash table n\_nationkey => n\_name and the upper join a hash table c\_custkey => n\_name.
So to summarize, each operator produces a subresult (the data it streams out) and *Join* operators also create hash tables as "special" subresults.
### Why and How to Cache & Reuse Subresults [#why-and-how-to-cache--reuse-subresults]
Let us now have a look at *why* one would cache and then later reuse subresults of consecutive queries. For similar approaches and in-depth motivation & impact, see, e.g., [\[IKNG10\]](https://dl.acm.org/doi/10.1145/1559845.1559879), [\[Nag10\]](https://homepages.cwi.nl/~boncz/msc/2010-FabianNagel.pdf), [\[HBBGN12\]](https://vldb.org/pvldb/vol5/p1436_alexanderhall_vldb2012.pdf), and the more recent papers [\[DBCK17\]](https://cs.brown.edu/~kayhan/papers/hashstash.pdf) and [\[RHPV et al. 24\]](https://www.amazon.science/publications/why-tpc-is-not-enough-an-analysis-of-the-amazon-redshift-fleet). [Below](#caching-hash-table-subresults) we give more details about the actual impact of our implementation in production.
For full results the motivation for caching is pretty obvious: why recompute if the exact same query comes in a second (or third or …) time. If the full result is small enough, we can keep it in a cache and reuse it as long as the data scanned did not change. Workloads with many consecutive, exactly identical queries are actually more common than one might think. The recent VLDB paper analyzing Amazon Redshift's workload [\[RHPV et al. 24\]](https://www.amazon.science/publications/why-tpc-is-not-enough-an-analysis-of-the-amazon-redshift-fleet) impressively shows this somewhat surprising fact. [Below](https://www.firebolt.io/blog/caching-reuse-of-subresults-across-queries#caching-hash-table-subresults) we also share some stats on our production workload.
Many query engines have a full result cache. Often it is simply based on a hashmap of query-strings to full results. We do this in a somewhat different manner which has advantages given later in this section. But we actually go much further, our system supports caching (and reusing) results of any subplan. These subresults are placed in our in-memory **FireCache** which can use up to 20% of the available RAM. Here is how we determine which subresults to cache:
* On the one hand, our optimizer may add so-called *MaybeCache* operators above any node in the plan. As the name suggests, this operator may cache a subresult – if it is not too large. It may later fetch from the cache and reuse a subresult if the exact same subplan (with the same data being scanned) is evaluated again. Currently, the optimizer places a *MaybeCache* operator 1) at the top of the plan, for a full result cache and 2) at nodes where "sideways information passing" is happening to speed up joins (where the probe-side has an index on the key that is being joined on). For 2) let us only point out that this can be very beneficial for the case described in the next bullet. We will not go into more details here for simplicity. The *MaybeCache* operator is versatile, it can be placed anywhere in the plan. In the future, we may investigate adding it above other, selected operators, e.g., pipeline breakers such as *Aggregate*.
* On the other hand, our system places any subresult hash-tables built for *Join* operators in the FireCache if they are not too large. The reasoning behind this is that these hash tables are relatively expensive to compute and it is very advantageous to reuse them in case similar queries come in consecutively.
So this is what the simplified plan for our example query would look like and what would be placed in the FireCache on the first evaluation:
A *MaybeCache* has been placed at the top of the plan and it places the subresult collected on the first run into the FireCache, as do both *Join* operators with their respective hash tables.
On a subsequent run of exactly the same query (over unchanged data), the *MaybeCache* fetches the subresult from the cache and the entire evaluation can be skipped. This means the latency drops to very low milliseconds. In our toy example this corresponds to a **speed boost of more than 100x** (on a single node, medium engine running over TPC-H with scale factor 100).
But what if the user adds a filter which may be changed across each of the subsequent queries? This is where caching *actual subresults* kicks in. Say the user wants to restrict the date range, e.g., by adding ... AND o\_orderdate >= '1998-01-01' ... (for changing dates) to the query:
The following figure shows how the query plan and the corresponding cache interactions would look like if the query without the restriction was run previously:
Notice that the added filter has been pushed down by the optimizer to be right above the scan of orders. The plan as a whole has changed, therefore the subresult stored by the *MaybeCache* previously cannot be reused. On the other hand, the **right** **subplan** below the upper *Join* (surrounded by a dashed box), has not changed! Therefore the previously computed and cached hash table can be reused in that *Join* operator. This saves us from evaluating the subplan and building the hash table again.
In the toy example, this leads to a **speed boost of > 5x** on subsequent queries (even when each query has a different date restriction).
Note that we would still profit from the FireCache in case restrictions on the customer table would be added – here we would be able to reuse the hash table for the lower *Join*. To further extend the utility of the cache, the optimizer could decide to lift up such filters to have larger common subplans across consecutive queries. This is something we plan to look into in the future.
### Avoiding Thrashing [#avoiding-thrashing]
The example also already shows one potential (and common) challenge with such caches: how to handle possibly many useless items being added to the cache. Say hundreds of such queries come in per second, but each with different date restrictions. All of the final results added to the cache may never be used again. There is a danger of the cache being thrashed and useful items being evicted by these useless ones. In [this section](https://www.firebolt.io/blog/caching-reuse-of-subresults-across-queries#better-eviction-strategy) below we show how we improved on the well known LRU eviction strategy to make such thrashing less likely.
### "Semantic" Cache-ids and their Impact on the Full-Result Cache [#semantic-cache-ids-and-their-impact-on-the-full-result-cache]
In order to add and retrieve items from a cache, one needs a way of deriving keys / ids to identify them. In our case, for each *MaybeCache* and *Join* operator the corresponding cache-ids are determined from the subplan below it. This is described in detail in the next section.
Before we dive into this, let us have a look why this approach can be beneficial already for a full-result cache alone – compared to simply deriving cache-ids from the query text as is done commonly for such caches: since we are operating on the query plan, the cache-ids are "semantic" as opposed to merely text-based. Different query strings may be semantically equivalent and result in the same plan and hence same cache-ids. E.g., all comments have been removed. Ignoring comments is important for BI tools such as Looker which add metadata in comments to queries sent out.
Subsequent queries may be exactly the same, except for the metadata added in comments. The query string changes on each run, but for the FireCache this still results in the same cache-ids and therefore cache hits.
Moreover, the cache-ids are computed after the plan was fully optimized. Rewrites such as constant folding may normalize "different looking queries" to the same plan.
We can even go a step further (as of this writing this is work-in-progress) and push the *MaybeCache* operator below any operators at the top of the plan which do not reduce the cardinality. Note that since we store only relatively small subresults, there is very little (or even no) noticeable overhead when evaluating that final operator on a small subresult fetched from the cache. Why is this useful? Let us look once more at our toy example query. After running it once, we may decide that we are actually interested in the average instead of the total price, i.e.:
This results in the following plan. The final *Projection* preserves the subresult cardinality, therefore the *MaybeCache* can be pushed below it.
Even though the query has changed significantly, we still get a cache-hit and the bulk of the plan is not executed. Say, we then notice that we actually wanted to order by the average cost. The *Order* operator is cardinality preserving as well, so the *MaybeCache* is pushed below it (and below the *Projection*). Therefore, we again get a cache hit!
### Caching in Distributed Engines [#caching-in-distributed-engines]
In distributed engines with many servers, each server has its own, independent FireCache. For system simplicity, there is **no cross-server synchronization** on whether or not to use the FireCache for individual subplans. Individual servers may or may not have cached subresults available, thus some servers may answer a suplan from the cache while others may compute from scratch. Such distributed settings bring interesting challenges, described in [this section](https://www.firebolt.io/blog/caching-reuse-of-subresults-across-queries#consistent-subresults-matter-in-distributed-settings) below.
### Cache-ids for Arbitrary Parts of a Query Plan [#cache-ids-for-arbitrary-parts-of-a-query-plan]
As a well known [joke](https://martinfowler.com/bliki/TwoHardThings.html) goes, there are only 2 hard problems in computer science: cache invalidation, naming things, and off-by-1 errors. Let us now look into the first problem in the context of the FireCache. We tackle it by deriving "cache-ids" for subplans which capture the "full details" of a represented subplan, including which data is scanned over. By construction of these cache-ids, we never fetch invalidated data from the cache.
To compute a cache-id which represents a subplan (say the one below the *MaybeCache* in the previous figure above), we start at the leaves (here the three *Scan* nodes) and then propagate "subplan fingerprints" up through the plan.
A subplan fingerprint needs to represent exactly the data streamed out of that subplan. The slightest change to subplan or the underlying data, e.g., a new row added to the orders table, a new or changed filter, a different aggregation etc. must result in a different fingerprint.
For each operator we implemented a specific rule of how to combine the incoming fingerprints from its children with any parameters / configuration the operator may have (e.g., a complex, nested expression to be evaluated for a *Filter*).
We will skip over the details here and will only briefly touch on how this is implemented for the *Scan* operator. In Firebolt's managed storage, data scanned is split into immutable tablets. If data within an existing tablet is modified, a new tablet is created, reflecting the change (note that the change may be a minimal extension, e.g., reusing all files from the existing tablet, simply adding a delete vector – for our purposes here it is important though that each tablet is immutable). The *Scan* operator is thus configured with exactly which tablets it should scan on which server. Usually, this is a subset of the tablets a table is composed of (e.g., narrowed down by primary-key filters or distribution of tablets across servers). So to compute the fingerprint of a *Scan* operator, we simply hash all tablet-ids and combine these hashes with hashes of the scanned variables.
By construction **we therefore avoid ever serving "stale" cached results for a query!** To illustrate this, let us look at the example plan above once more. Say, from one run to the next nothing changed except that a single row has been added to the orders table. This results in a different set of tablets to be scanned in the leftmost *Scan* node – which in turn gives a different fingerprint for that node. Since the fingerprint is then propagated up through the plan, the fingerprint (i.e., cache-id) used for the *MaybeCache* at the top also changes => We do not use the previously cached, now stale subresult. Instead a new one is computed and stored in the cache.
Note that since nothing else in the plan has changed, both *Join*s actually still receive the same cache-ids for their hash-table subresults as before => These can still be reused. We hope this gives an idea of how the propagation of fingerprints nicely leads to cache-ids which enable reuse in unchanged parts of the plan while avoiding to ever serve stale data.
To make hash-collisions of the fingerprints extremely unlikely, we use the 128 bit [SipHash](https://en.wikipedia.org/wiki/SipHash) function. To date, there are no known collisions of such SipHashes.
### Consistent Subresults Matter in Distributed Settings [#consistent-subresults-matter-in-distributed-settings]
After so-called *Shuffle* operators (which distribute data across servers), one needs to be careful to not cache & reuse possibly inconsistent subresults. This is relevant for SQL queries which may give non-deterministic results on subsequent runs, even if the underlying data did not change. As an example, the following query returns 3 nations from the underlying table, but it is not defined which nations.
Similarly, SUM(o\_totalprice) due to floating-point rounding may give (slightly) different results depending on the order in which the underlying double values are summed up.
Let us look at a made up example to give an idea why it can be problematic to cache non-deterministic subresults on different servers. Say we execute the following query on an engine with two servers.
Let us assume that the small 3-nations-subquery is executed on one server and the subresult is sent to both servers with a so-called "broadcast" *Shuffle*. Also, say that we split the evaluation of the join with customer across these two servers (each server taking care of roughly half of the customer table). We may pre-aggregate the count on these servers independently as well. In a final step the pre-aggregated subresults would be shuffled to one server and merged there to compute the final result: **the count per nation for three arbitrarily selected nations.**
If we are not careful, the subresults of the 3-nations-subquery may be cached independently on the two servers (note, as mentioned above we do not have any cross-server synchronization of the FireCaches). It could happen that we end up with two inconsistent copies of that subresult on the two servers – for instance, due to different evictions happening on these servers. E.g., we may end up with the three nations "GERMANY", "ALGERIA", and "CANADA" being cached on one server and "FRANCE", "INDIA", and "BRAZIL" on the other server. If we would use these cached subresults for the two parts of the join being evaluated, **the final result of the query would have 6 nations** instead of only 3 as expected & the reported counts would likely be off by \~50%! This is of course incorrect and needs to be avoided.
We therefore need to avoid caching possibly inconsistent subresults. To this end, we disable caching on nodes in the plan if these two conditions hold: 1) "separate copies" of a subresults may exist, i.e., post-shuffle and 2) these subresults may be non-deterministic even on unchanged data. We compute this via depth first decent in the plan. In parts of the plan where there are parallel paths, we only allow caching of deterministic subresults.
Note that this does not mean we need to disable caching entirely above a *Shuffle*: if the "separate copies" are combined again into a single copy further up in the plan, from then on even caching non-deterministic subresults is OK again. In particular, the important case of caching full results is always OK, since these full results are always on a single server and there are no separate copies.
### Better Eviction Strategy [#better-eviction-strategy]
As always with caches, there is a risk of "thrashing": if many subresults are added to the cache in quick succession, important ones may be evicted by unimportant ones. I.e., subresults that are not reused in the future may cause others that are still useful to be thrown out of the cache if the memory limit has been exceeded. The FireCache is relatively large, we reserve 20% of RAM for it currently and the subresults stored are in most cases rather small (different thresholds, e.g., 1 MB for the full results). Nevertheless it is important to choose a good eviction strategy to **maximize reuse and thus optimize the processing time saved by the cache**.
The most common strategy is the [least-recently used](https://en.wikipedia.org/wiki/Cache_replacement_policies#LRU) (= LRU) eviction policy. It is easy to implement, fast, and intuitive, since the item not accessed for the longest period of time may be the one least likely to be accessed again in the near future.
In our case we have additional information readily available which could be, but is not used by LRU: we know how long it took to compute a subresult and we know the size in RAM of the subresult. Also, it would be nice to track how often an item was used – this might also indicate how useful it is in the future.
### The "NEW" Strategy [#the-new-strategy]
We thus combined these in the following manner to form our NEW strategy. This is essentially the same as the policy described in the section "Benefit-Based Result Selection" in [\[NBV13\]](https://ir.cwi.nl/pub/21352/21352A.pdf).
* As base-weight, we combine the processing time needed to compute a subresult (and to potentially save when reusing it later) with its size as:
time in milliseconds / size in bytes
In a sense this gives us "how much bang we get per buck".
* For the frequency component we essentially sum up the (decayed, see next bullet) base-weights per access. E.g., if there were 3 consecutive accesses close in time, the final weight of an item would be \~3x its base-weight.
* Finally, to keep an LRU-like component, we decay the weight exponentially with respect to the current vs. the last access time. For this we take a "half life time" of 5 minutes. I.e., after 5 minutes the weight of an item is decayed by half. We experimented with different half-times and the differences were small, but 5 minutes seemed like a good tradeoff for different workloads.
When the FireCache runs out of memory and needs to evict a subresult, it chooses the one with the smallest weight. Computing this is relatively expensive, but with some tuning, we got this to run in 500 to 1000 nanoseconds per eviction which is OK for such "coarse grained" caches.
### Performance in Simulations with Different Cache Sizes [#performance-in-simulations-with-different-cache-sizes]
We benchmarked NEW vs. LRU on real-world data of several customers and compared the total processing time saved by the two strategies. Below is the result for the workload of a selected customer over multiple days. In our simulation we varied the cache size and show the percentage of time saved by the cache:
The results for very small and very large caches are not so interesting: for the former nothing can be cached and for the latter essentially everything may be cached and then reused many times by these very regular workloads – very few exact duplicates, but a lot of reuse potential due to \~100 query patterns, each with large subplans occurring many times in consecutive queries.
For the interesting mid-range of cache-sizes (4GB to 18GB) we get up to **2x more processing-time savings with NEW compared to LRU**, see also this figure which shows the ratio:
## Impact [#impact]
### Caching Hash-Table Subresults [#caching-hash-table-subresults]
For the interesting case of customers with mostly unique queries (i.e., few exact duplicates), stemming from tens / hundreds of repetitive query patterns, we often see very nice speed boosts of 10-100x for individual queries. This is, e.g., the case for the customer mentioned [above](https://www.firebolt.io/blog/caching-reuse-of-subresults-across-queries#performance-in-simulations-with-different-cache-sizes) when discussing our NEW eviction strategy. The customer's SQL queries are often > 100 lines long and contain many joins. But, consecutive queries from the same pattern change only in one literal of a crucial WHERE restriction. Note that in this case the savings come from reusing hash tables in *Join* operators previously stored in the FireCache.
The chart below shows the overall hash-table *build-time saved vs. build-time spent*. This is across all customers in production over several weeks this year. The thick blue line gives the ratio of saved / spent. It shows that **for hash-table build times we overall reap 10x savings** on many days with some spikes being even higher\*\*.\*\*
### Full Result Cache [#full-result-cache]
The *MaybeCache* added to the top of the plan has obvious advantages. As mentioned many other systems add a similar cache (although to the best of our knowledge without the full benefits of the "semantic cache-ids", see [above](https://www.firebolt.io/blog/caching-reuse-of-subresults-across-queries#semantic-cache-ids-and-their-impact-on-the-full-result-cache)).
It is interesting to see for individual customers how much they may profit from such a full result cache. The answer is some customers can profit tremendously: here is a chart with the simulated time savings of the top 15 customers (anonymized), ordered by the potential savings across \~5 days of production queries. I.e., customer "A" may save > 95% of their total compute time if queries would be answered by a full result cache. Note: we will redo this measurement to report actual time savings when the full result cache has been in production for a longer period of time (this was launched only recently).
## Conclusion [#conclusion]
This blog post gave a deep dive on how at Firebolt we give our customer's queries a speed boost by enabling caching and reuse of subresults. For some of the interactive use cases we serve, this is of crucial importance. Without caching results of complex subplans certain query patterns would not achieve the low milliseconds latencies required.
As shown in the "Impact" section above, this also in general yields tremendous savings for our customers, e.g., by saving up to 10x in processing time spent on building hash tables for joins.
We hope we were able to give an idea on how with propagation of fingerprints & derivation of cache-ids we can determine in fine-granular manner which parts of a plan can be reused and which should be recomputed. We dove into the challenges posed by distributed settings. Finally, we gave a quick overview of a modified eviction policy to improve the usefulness of the FireCache under heavy loads when many items are being added (and conversely need to be evicted).
Since our cache is purely in memory for now, compact representations of data structures to store subresults are very beneficial. In a follow up blog post, we will describe in depth a custom, space-optimized hash table which we devised for our novel "FireHashJoin". With it, we achieve more than 5x memory savings (as measured on production data) compared to the hash table implementation we used before. It thus enables us to cache significantly more of such subresults in the same amount of RAM.
# Data engineering from the early 2000s till today - BlackRock (/blog/data-engineering-from-the-early-2000s-till-today-blackrock)
When it comes to data management, have we come a long way since the early 2000s? Or has it simply taken us 20 years to finally realize that you can't scale properly without data modeling. With over 20 years of experience in the data space, leading engineering teams at Cisco, Oracle, Greenplum, and now as Sr. Director of Engineering at BlackRock, Krishnan Viswanathan talks about the data engineering challenges that existed two decades ago and still exist today.
Listen on [Spotify](https://open.spotify.com/show/6hMdnrFKlPbia2k6MkFs8U) or [Apple podcasts](https://podcasts.apple.com/us/podcast/the-data-engineering-show/id1561927688?uo=4)
**Benjamin:** and post. Cool. All right. So welcome back everyone, kind of to the data engineering show. Good to have you. So it's the second episode in a row now without the other data bro, Eldan, because he has out today, but he'll be back in the next episode. But anyway, so let's jump right in. It's a total pleasure having Krish Nan here today. He's a senior director of data engineering at BlackRock. And like he's done... He's done everything, really. It's so cool having you on. He started out at Cisco, spent 10 years there, then went more to the vendor side as a principal PM at Oracle and later as a director of product management at Greenplum. And then most recently, he's actually been as a kind of senior director of data engineering at BlackRock. So great to have you on today, Krishnan. I look forward to talking about kind of the data challenges you're facing, kind of your background and everything. Yeah, do you just wanna kind of give a quick intro of yourself?
**Krishnan Viswanathan:** Awesome. Thanks, Benjamin, for having me. This is a pleasure. And like you said, I've been in the industry for almost a quarter of a century. So now that I think back, it's been a long time coming. But I'm really excited to be part of this conversation. I think the data space and the data explosion in the last couple of decades and all the modernization has put a deja vu into the data. organization, uh, every day seems like a new day, but also seems like a groundhog day in some ways. So I would love to share some of the things that I've learned. And back to my background, I have been, I started off as a data engineer. Actually, it was more of a sensitivity. I did my first initial project as a, as an executive information system for Cisco. Uh, for the on the web based application and then realized at the back of it that we actually needed a data warehouse and a data model and a data cleansing and all of the things that goes with data pipelines to get that numbers right. So that was my marijuana moment.
**Benjamin:** What year was that roughly?
**Krishnan Viswanathan:** That was 1996.
**Benjamin:** Wow, so take us through the stack back then. Like, kind of how did you have a lot of options? What types of systems kind of were out there at the time?
**Krishnan Viswanathan:** Yeah.
**Benjamin:** And I guess it was all on-prem as well, right?
**Krishnan Viswanathan:** This was all, well, think about this. 1993 was when internet became kind of prevalent. Cisco was the main child, main vendor pushing that pipeline and the routers for internet. And before, so when I joined in 96 at Cisco, the executives used to get their reporting in Excel. Every week they would get, and it would be a high level number, pretty much that's it. And no other information. And then they would make phone calls to all the different departments, different groups. And then, you know, by the time they get to a problem statement, the problems are either already solved or it has escalated to an extent that it was too late to solve, right? Whichever way it went. So the premise in 96 was, hey, we are the pioneers in internet. We should build an internet technology that can help showcase what we can actually present to the executives. Out of it was born the executive information system, we call it EIS, the first generation of it. And it was homegrown. I was one of the first, one of the four engineers and eventually you are the lead engineer there. We built it with, we got based out of an Oracle ERP extract for, and then we had an Oracle a large scale data warehouse, scale to single instance Oracle data warehouse. And it had its own challenges extending, but it was that. And then the front end was a purely web-based application. And web-based application, it was pre-Java, pre-EJV, it was all pure HTML. And what we did at the back end was we used to run this extraction process and build this data staging layer, for lack of better word. because he was still, I was not a data engineer at that time and I would do so many things different, but we did that. And then we would run a process between that and the web server to create all these different HTML pages because HTML was limited, right? Web content, web configurations were very limited. There wasn't a real dynamic XML parsing. I think XML was just coming up to speed, but not even that. But anyway, the Java EJB was not even part of the picture. No servlets, right? If you think about those things. So it was all pure HTML, PHP-based query process. So that's that. And the expectation, since it's executive reporting, was they were looking at the overall high number, and they would drill down all the way to, and by region or by product, eventually to a sales rep to a transaction. to see where, if there was an issue, if there was a question, they could go all the way down. So the process itself in the beginning took about six hours for us. Interesting, and again, reliability was a whole different problem. I will talk about that later. And then the data accuracy was another level of challenges that we solved in subsequent releases. But the first version of it, for all the good things that we did, it had its own challenges. And part of it was technology and the immature technologies and the kind of band-aiding them together to make it work. And then the other part of it is, of course, we also could have gone and done a better data design process
**Benjamin:** Right?
**Krishnan Viswanathan:** and data alerting process. So that was fun days. But it was a hugely successful endeavor. If I were to just talk about the outcome of that, the appreciation we got out of the data something like that, it was actually showcased by Cisco as one of the pioneering projects that helped. In those days, Cisco used to call it the virtual close. What it basically means is it used to take like a month for companies to close their book of records. Using this application, this was one of the key applications, I won't want to call it, this is the only application. Using this application, Cisco said they could actually close their books within a day. So.
**Benjamin:** Wow.
**Krishnan Viswanathan:** from the vision to what we could accomplish, notwithstanding all the other disclaimers I put in for it, it was a success. And they were, we actually got funding to make it better. And we did do better. And the second iteration of that was a whole lot better, a whole better concept. And I'll talk about that a little bit later, but that was the genesis of my data engineering role. Actually, let me talk about the second version of it, because I led the second version of that project, which we call E-Exec. And part of that project was to find a better capability, a better tool. And this came around in 2001. So if you are all aware, and if you have followed the internet bubble, and the burst of the internet bubble in the early 2000s, I don't know Benjamin, if you were there, but
I was writing the whole Cisco bubble and I still get nightmares about that because one of my roles in this project as the exec was we would wait for the SVP of Cisco to go in and provide a confirmation on forecasting and judgment, which is basically saying, yeah, I agree with this forecast and this number is going to look good. I was the one who would say, OK, it's done. Click the process, right? Data engineer, making sure that everything is done. The financial analyst would tell me that, and then I would make sure that the process gets itself. But I would validate that everything is working. I still remember that day in October of 2000, some early October of 2000, or maybe October of 2000, when there was a huge downward revision of the forecast from what I have seen in the past. The forecast was always going up, up, up. And then it was like, Well, that was like a fairly deep decline. I was like, something doesn't look right. But I was still an engineer. I had to get the job done. I just did it and went off. Three months later, you saw the whole internet crash and Cisco going from its huge bubble. So I lived that downtown. And I lived it very, very vividly, both in my financial terms and at Cisco. So remember I said, we... Assume that this application will help executives give an early warning and give the executives a good indicator of how the business is performing and if you wanted to do a virtual close-up. Within a day they can close. This application kind of didn't give that exact, it gave them good information, not good at all. It didn't give them the exact alertness that they were looking for, which is something is wrong proactively telling them that, hey, something is wrong, you're gonna have to, you're gonna miss this. And there were a lot of other business reasons for that from a technology point of view. It didn't do that. And that was the next version of this. So the next two years from 2001, 2000, 2001 to 2002. two years, I spent a whole lot of time digging into the data to understand how we missed all of these indicators because it was a big impact on Cisco. We had layoffs, our stock prices never recovered from that. Even today, it hasn't recovered to the pre-internet days. So I did a lot of work on that. And we came to a conclusion that, of course, our engine was not a real engine. It was just a reporting engine. It was not an alerting engine. So we kind of thought that was important. And we went in and we needed a more dynamic and scalable model to improve this, and so that we can easily change some of these parameters. Toward that end, we chose a technology which eventually got called a Siebel Analytics. I don't know if you're aware of that term. But Siebel Analytics was one of the products that Siebel had brought in. And it's supposed to provide this automated dynamic dashboard creation, fast and easy, and it had alerting capabilities, it could multi-device. And this is right at the time when mobile phones are not the smartphones, just the mobile phones were getting more prevalent. Pages, mobile phones, dynamic content, PDAs, I think BlackBerry's and some of those things were prevalent at that time. I'm dating myself here. And we did an analysis of that. I brought that product in. we delivered on that application, and that became another huge success. So that was the genesis. So out of the background work and data validation, we had a second version. And that application actually, if I'm not mistaken, Intelligent Enterprise, if I'm not mistaken, gave that application one of the best BI application in that time. I think it was 2004, if I'm dating myself correctly. And so we got that between with Cisco and Cibo. At that time, we got that. Eventually, the stable got acquired by Oracle. And hence, my transition from data engineer at Cisco to product manager at Oracle, because I was one of those guys who understood the ins and outs of that particular product that became Oracle BI.
**Benjamin:** So, I mean, you, you told us about that kind of stack, right? In the very early days. So having like this kind of like RLQ database generating static HTML, loading that in the browser, like take us through data volumes here. Right? Like if you, if you still remember like what was, cause this was big data back in the day, right? Like kind
**Krishnan Viswanathan:** Yeah.
**Benjamin:** of what, what was big data actually then in the late nineties and early two thousands.
**Krishnan Viswanathan:** Cisco was actually on the pioneering of that. So big data in those days. So to think about Oracle applications, SMP infrastructure was, you can only scale so much. I think it was one single box, 64 cores if my memory's, 32 cores or 64 cores, I don't remember now. And it probably had about 256 GB or maybe even, memory, I have no idea. I think it might be less than that. And it was expensive machines. of a million dollars for that one single box, right?
**Benjamin:** in the like just availability of like that type of hardware. Nowadays, of course, like that's not considered a big box, right? But like, sure, like more than 20 years ago, that's absolutely insane.
**Krishnan Viswanathan:** That was insane. Yeah. And, and on top of it, it was, it was shared everything. And so our process from midnight to like 6 AM, we would take up a lot of processing problems and we would ask nobody else on it, but then there was still conflicts. And when the conflict occurs, there wasn't an easy way to troubleshoot, right? Because everything's there. So we had to start killing other applications in sometimes they would kill our application too. But.
**Benjamin:** Gotcha.
**Krishnan Viswanathan:** I was the HOV lane. I was like the ER lane, right? We would get high priority, high visibility. So that was good to have. But it was also bad because if something happens, I would be on call the next day morning with the VPs and the directors in that office just trying to figure out why I did what I did and sleep
**Benjamin:** Okay.
**Krishnan Viswanathan:** at nights to fix all that. So in terms of data volume, going back to your question, it was internal Oracle applications. There wasn't a lot. Again, we have a lot. So if you expand our bill of materials, there was a lot of transactional level details. So I'm probably thinking, I don't have the exact number, but it's probably closer to like 20, 30,000, maybe not. Maybe about 100,000 records per day.
**Benjamin:** Okay?
**Krishnan Viswanathan:** And we used to do this three times, for three different time zones, Asia Pac, EMEA, and the US. And the US was big. So about 100,000 records per day. The challenge was not the number of records. The challenge was trying to get to an accuracy and a lot of things. So people are entering data on multiple different terminals, orders come in and vendors will load in that information. Somebody fat fingers and instead of one zero, they put two zeros or they put a different number. Executives don't wanna see that and because it would throw off their numbers. One thing I realized in this whole conversation was, at the time of BI and even today it's true. people who really know the data, who really are the business, who are engaged with the business, they intuitively know that data better than us engineers. So why I say that? If there was a fat finger, if there was a number that was incorrect, I would get a call from the SVP of sales of the particular region and say, this number is wrong. I know this cannot be the truth because I have not seen this. I would have known this. So any which way, they were almost always correct. So we would have to... Our challenge was to make sure that those kind of numbers in the upstream system, impacting the upstream system, does not automatically flow through. So we had to build some triggers and metrics. So our processing and data accuracy with lack of actual tuning, all these things were just getting started in pre-internet days. How do you build a reliable data pipeline? How do you build data accuracy and at every stage of your data transformation, how do you put in... controls and alerts in place. I think that was the biggest challenge and that was what took us longer. But in the end, I am a big proponent of data controls, data governance and notification and QC for data as a separate entity. But we did a lot of work on that internally, but yeah, that was my biggest learnings in this whole process.
**Benjamin:** Okay, gotcha. It's so crazy to me because like so many of the things you're talking about from kind of back, back then in the early 2000s, right? Like they're just as relevant today. And if you talk to data engineers today, they're struggling with similar things like data quality, data observability, kind of all of those things, just of course, at a different than scale in many cases. Do you think we've come a long way? Like if you look back, or do you think the problems are actually just the same in many cases?
**Krishnan Viswanathan:** I'm still struggling doing the same things that I used to do back in 20, 25 years ago. Now in my new role at BlackRock. So no, what has happened? I think it is, it's interesting to look at the shift that happened in 2010. So before 2010, 2005, 2010, data warehousing was a big thing. Data modeling was a big thing. Bringing data in. and making sure that the data goes to a certain rigorous data modeling data by accuracy data validation was big to the extent that it actually stifled some of the data volume movement. So there was a data explosion happening on the periphery with internet and the logs, but then there was this use case about decision support systems as they were called in the past. They were going through a certain changes, they were immune to that other world. So you had certain things, like people like us who are building this by hand and taking our sweet time. So building a data warehouse would be a two year effort. And it was granted, like, hey, two years effort. If you want to make a change to a dimension or bring in a dimension, that's a six month effort. And everybody lived with it. And then suddenly this whole data explosion came along in 2009, 2010. And tangentially, MPP became a huge deal, massively parallel processing, analytical databases. So back to 2000, 2005, there was one database for all, like the one thing for all. You either choose Oracle or you choose Sybase or you choose Microsoft, they all did this OLTP extremely well. And then as an afterthought, they give you this big hunk of a hardware, single, SMP single instance hardware that you could just Take the data from your OLTP, do your ODS, and do your data warehousing, and scale. That was your data warehouse, and that was your scale. All the magic was in data modeling. MPP came along, and they broke that paradigm. And they said, you don't really need this big infrastructure. You just load the data and let it scale. And so if you are doing trending, and if you're bringing data from logs, or if you're bringing data from machine-generated, And you're doing trending data. Great. That works awesome. And so it became a huge success. So you could do things that you could never do before. You could never get clickstream applications or log analytics. Great applications, but you couldn't do that within a reasonable size and within a small budget, you couldn't do that. But what it also brought in along with that was this whole concept of data lake, which eventually turned into data swamp. for many people, but that I think we are now about 10 years after we are now starting to realize that. We are starting to realize that, no, you can't just scale your data and your infrastructure good as it may be. You also still need data models. You also still need data quality controls. You also need data governance and right. Those, and those are now, there are vendors who are now doing it at scale for cloud which in the old world mostly done by Informatica on a single node machines. And that's why I think we're seeing a lot of differences now. So to your original leading question, going back to that, I think this next few years, you will see, you're already seeing a lot of that hype. You're talking about data observability, which is pretty much this, right? Data observability goes across the board, not just on availability, but also on a lot of notification and processing time and QC, governance, data maturity. I think those will come into play. I think we are ready for that, but we are not, we don't have a good technology today in my mind that solves for all of that. I also think because of the volume of data that are potential AI ML use cases, right? Which can look at historical data. observability being one of those cases where you can actually bring in a good quality data QC or a data governance model that can give you early warning so you can build on top of it. You can actually be much more effective now than you would have been 10 years ago. So I'm very optimistic about the future.
**Benjamin:** Thank you.
**Krishnan Viswanathan:** We are trying to do some of those things in my current application, because if you see my past, I've been doing some of the data pipelines for supply chain and marketing companies. Clorox was a marketing company for the most part, marketing and finance. But now that I am in BlackRock and it's a financial company, that is again, our challenges are back to data accuracy. providing the right data at the right time with good quality controls. So
**Benjamin:** Gotcha.
**Krishnan Viswanathan:** we're seeing
**Benjamin:** Yeah.
**Krishnan Viswanathan:** that again.
**Benjamin:** I mean, kind of take, take, take me a bit through that. Right. So, right. You have to super long history kind of you've been in BI since the kind of very, very early days at Cisco. So now you're moving kind of into like this kind of more like financial space. And at least in my head as an outsider, right. Can you always like, when you think of finance, you think of like legacy systems kind of, and then very, very high bar in terms of like data quality. Right. Cause okay, like kind of like there's a lot of money at stake in this space and so on. So. take us maybe through the types of data challenges you have in that space and how it's unique.
**Krishnan Viswanathan:** I'm still trying to figure out the uniqueness of our scope. I think a lot of times I've heard about applications and legacy applications in finance and the very hard and well-structured processes and both of them are very true. And most of the financial application, at least in where I'm working, the challenges are... that we have very strict and well-built processes that have survived for multiple decades, and those code has not been touched for multiple decades. People have moved on, but the code's not been changed. We are bringing in modernization into this industry. So now I'm straddling between legacy code and newer infrastructure with data modernization. The challenge is we don't quite understand an existing legacy code which has a ton of processes, which has manual input at every stage and move it to a new technology without breaking something. So that's the challenge. I don't want to get too deep into that, but at the same time, yes, what you are saying is absolutely accurate. If we break something, the unintended consequences are very high. How do we make sure that we minimize? I don't think there will be zero unintended consequences, but how do we make sure that we can minimize those to the extent that the risk is well captured, the testing time can be extended. But that's our challenge. That's also our opportunity in terms of how we can scale because technology is not a challenge anymore when it comes to data processing, infrastructure, architecture. These are all pluggable, quickly adjustable components. I think technology has governed. and moved quite fast. It's now bringing the rest of the organization to validate and use this technology absolutely correctly to make this happen. I think that's going to be the challenge for the next generation.
**Benjamin:** But from a quality perspective, that sounds super interesting because we were talking about this data quality aspect. Did someone enter an extra zero here? Did the distribution of my data change? Should I be firing an alert? I guess here you have this second dimension to it. So you're touching legacy systems, which maybe don't have a lot of observability in place and you might not be able to retrofit that onto them. You're building something new and trying to figure out, does it? Does it do the job? Like what starts going wrong when I turn off the old thing and so on. So that's interesting.
**Krishnan Viswanathan:** Yeah, I think so. Again, I think for what we do, a lot of data comes from external third parties. So we don't control anything here. Uh, but we, and that's what I was talking about using AI and ML to, to fingerprint for like a better word, to look at how we used to get the data in the last few years. And if you see a large enough standard deviation, can we predict and at least quarantine those data so that you can actually have somebody look at it. validate it before you accept it. So those are the kind of processes that we are talking about. I think the key for that is you need to have a proper metadata. You need to have a proper governance model because yeah, I can put a process in place today to do AIML and look at my historical trend. I can come up with an area where I can see some exception but to act on that and act on it real time, I think that's a bigger piece of puzzle. Getting the right infrastructure because people who have been there more siloed in their operation. So I think it is definitely a shift in what we are doing, but I think it is exciting in the sense that it's new. And that's the technological evolution that we're going through.
**Benjamin:** Okay, super, super interesting. So how big is your team, if I may ask?
**Krishnan Viswanathan:** So we are a newly formed group called data platforms and services. And my team, we manage reference data, which is kind of a fairly large amount and it's used across the enterprise. And I have about 25 people in this intervention. And this is just engineers. And then we have product managers and data stewards and other operational analysts, operational engineers and things like that. So yeah.
**Benjamin:** Nice, super, super cool. And also cool that you have like the kind of organizational buy-in to tackle these, these types of challenges. So that's, that's awesome. Um, cool. All right. So in, in the intro, you were kind of mentioning this like data deja vu, right? Uh, kind of seeing, seeing things, uh, over, over and over again. And we talked about that data quality aspect of it, like in, in what other areas of like data engineering as a whole, are you kind of having this like data, data deja vu nowadays that you had in the past?
**Krishnan Viswanathan:** Yeah, I think so. So we talked of quite a few things, right? So what I faced in 2000, 2001 for a company financial metric and seeing that again today.
**Benjamin:** Right.
**Krishnan Viswanathan:** So I moved organizations, but I see the same thing happening here. And there's a lot of things that I am noticing that are consistent in those days here. Some of them is because we operate as a startup and so there are some challenges there. But The other thing is it's also a financial industry and there is a very strict review. So they're very conservative in how they work on this platform. Also remember from 2006 approximately to like recently, I've gone towards the vendor side and I've also worked in other companies which were more open to buying vendor products. We are back here to a point where we most of it is. in-built and in-house. So that's another shift. So I know from a vendor perspective, there is a lot of availability in terms of tools and technologies that we could easily incorporate and put in. But that's not how we build at BlackPak. We invent. We do it here. And a lot of reasons is because we eventually end up sharing it with our clients. So if I were to go and bring a data in, I don't have to only look at. how much money I spend in buying that. I also have to license it for other users. So to make it profitable for us, we have to be able to build it and scale it store-wise. So... The engineering world that I'm in recently, it is interesting because what I'm trying to do is given all the experience that I have had, can I take my team? And my team is fairly young. Average age is probably 30. And I'm bringing the age up pretty high, right? But 30, 32, maybe. So how do I take them through this when they have... One, I think part of them is mostly software engineers, not data engineers.
**Benjamin:** Right.
**Krishnan Viswanathan:** They are, I have to transition them. So I have to go back to dig into my old days of how I reacted to this, get them data savvy. So that's that, right? The whole people transformation, because these are great people. So I need to translate. And then how do I also translate that into what we can objectively achieve year over year and trend better. And I've been only here for a very short time. So it's a long process. I'm still learning a lot of things and a lot of challenges, but I believe that is a, like I said, the next three, four years is going to be massive in terms of how the whole industry changes to solve for this, but also how our company is gonna make a big difference. And I see a lot of potential there.
**Benjamin:** Okay, awesome. So another thing, right, we talked about it earlier in terms of the like, okay,.com, right? And kind of the predictions you were seeing and so on. Like now as well, right? That's kind of like at the macro level at the moment, like not a great time and especially for tech. So one thing which is coming up more and more kind of in conversations I'm having with people in this data space is this question of kind of, right, proving that all this data you're collecting is actually worth it, right? That at the end of the day, it's kind of contributing. at 2D Bottom Line of the business. What are your thoughts on that?
**Krishnan Viswanathan:** Uh... I have dealt with two sides of that coin. So remember my pre-green plum days and my post-green plum and post-green plum days. So when it came to, so the pre-green plum days, I was like, why are we collecting data that we cannot even process? What's the value of that? Why are we creating a data swamp? So I have that, that's part of my brainchild. But when I was part of green plum and I was the product manager for green plum, I had to put that on a really bad burner and not bring that up. I had to talk about all the cool things you can do. But reality is, and this is why I think as a technical and technology industry, we didn't do a good job of educating our clients. You can't just continue to collect data and not process them if you don't even know what exists. And the Hadoop days and the data breaks days, those days... are great for technology and great for infrastructure spend and glad that we had easy money at that time. But now I don't think that's going to fly. Because even at a company like Clorox, then I did a couple, I joined Clorox. I didn't talk enough about Clorox, but I joined Clorox to first upgrade, modernize that data platform. First on-prem through Oracle data warehouses and Exadata and stuff like that. And eventually I did it to migrate to. Google Cloud and Azure. So even there, because we are CPG and margins are really small, there was always this underlying question. Why do we need to collect so much data? How can we optimize our recovery process? So what I ended up doing was not only doing the data transformation, doing the data pipelines, and bringing the data in, I actually ended up building a couple of applications on top of it to showcase what data means to the company like Clorox. So this ties in very well with your question because executives are not sending any black check at that company. We use data to try and predict forecasting accuracy for products. Can I predict how much can I sell it? So we couldn't do it really good, so that was shut down. We tried using NLP and voice interaction to see if we can. predict and call out any product concerns. So that kind of worked out OK. I built a new version of the Executive Information System for Clorox. We called it the Daily Briefing, which was a mobile
**Benjamin:** Thanks
**Krishnan Viswanathan:** application,
**Benjamin:** for watching!
**Krishnan Viswanathan:** but pretty much following the same standard. And it was on a mobile phone. And that became a huge success. That was my last Clorox. And those are all built on the cloud. All of these things are built on the cloud. So I always worked in industries where we had a very narrow path between how much data we collect and what's the value of that data. Nobody gave me a blank check except in my Greenplum days. Even at Greenplum, we were telling customers, nobody gave me a blank check to go in and load as much data as you can. We did one project called the, which eventually became called the CDP, the Consumer Data Platform. And we brought in cookie information and cookie data. And I did it for about one year. And we were collecting about a billion records per day when we explored that marketing data. But in a year, we could not find any valuable metric out of that. At least the marketing team couldn't find any valuable metric, too. And we have brick and mortar. We're not really online. So it may have been better
**Krishnan Viswanathan:** online than us. But nevertheless, that was shut down fairly quickly. And it was all on-prem. We had a Hadoop cluster. I set it up. But again, that goes to show there were some people who were not in the technology world who didn't buy into this whole hype on collect as much data as you want and then run your algorithm on top of it and you'll get the results. I think that it worked. So that was always a challenge.
**Benjamin:** super, super interesting. Yeah, especially that like consumer goods, goods perspective is interesting there. Super, super cool. Awesome, cool. So yeah, do you have any kind of closing things you wanna say, right? So maybe one thing for people getting into this space, cause you said, okay, you're taking software engineers now kind of at BlackRock, right? And getting closer to like the, to this data engineering world. If I was starting today, right, kind of out of high school, kind of going to college to get into this space, like any, any advice for...
**Krishnan Viswanathan:** Again, I think when I grew up, there was no Google, there was no cloud, there was no YouTube, there was no much, I think that's what it's called, online learning, online training, call out a course at us and the, so all those are available today. So all I have to do, which makes my job a little bit easier is point my team to the right direction and tell them go and learn and get better. The biggest shift in from software engineering to data engineering in my mind is the data domain knowledge. I mostly intuitively do that because of my past experience, but understanding how data is processed on top of the code is such a big thing, which software engineers don't care. Software engineers think of data as an afterthought, whereas data engineers think of data as the first thought. So how do I scale? How do I make this consistent, secure? So... That would be my first thing, but in terms of learning, I think there are already so many tools and technologies in place. It's easier. It's a whole lot better today than it was in my days. Now, you might think of me as going back to the old day, but that's true. We didn't have the right tools and technologies and I can get the job done much faster today. In some cases, you're stuck with legacy code. Well, too bad. We've got to get that out. But I see a great potential for people who are... coming into the data world. And I wouldn't have said this earlier this year, which I'm saying now, right? The AI winter is possibly near and here, but it's gonna pass. That is so many opportunities with AI, with data, and with the whole data industry that we don't even know what the future applications are gonna look like. Future applications... are going to be driven by data. Making sure that you have the right data in the right place, processed properly, will make your company more effective, more worthwhile. I've seen this again and again and again. And like I said, I did an application in 2000 without understanding what the implications are. I did the same application at Clorox with the intention of nobody gave me the requirement. I came up with that model. I had my, I pitched that idea to my CIO,
**Benjamin:** and that's just awesome.
**Krishnan Viswanathan:** made that success. And the whole reason was I had that background in data. I had the background in business knowledge to say from an executive point of view. This is what they would look at. And of course I had help. I'm not going to say that I didn't get, but trying to fine tune that I could do that on a different technology on a different platform, because I understood the data and the value of data and how people process that data. So
**Benjamin:** Thanks for watching!
**Krishnan Viswanathan:** data itself is abstract data is difficult to comprehend. Making it easier. That's the job of a data engineer, data scientist. However you want to label it, data analyst, data scientist, data engineer. Those are all different flavors of the same thing. Make sure when you are in front of an audience and trying to pitch, you understand the data. Have a good data representation, valid, not biased, valid data representation that can help others see the picture you want them to see. And... That itself is a big hurdle to pass. And then once you have done that, the third aspect of it is all the cool technologies that are in place. You can build AI and you can build ML, but if you don't do these two things and give them what can be the truth, what are the future of the possibility, I think data will just remain unprocessed, understand.
**Benjamin:** Is this something you can kind of learn or you just learn kind of over time as you're doing it, right? Because especially if you're a kind of data team catering to different parts of an organization, you will also have so many different requirements and kind of different teams trying to do different things with the data and so on. That actually catering to your audience might be really, really hard because it's just, there is no one audience.
**Krishnan Viswanathan:** This is probably the biggest learning I have. I come out and say, like, I know all these things. I would tell you how many times I've gone into meetings. So remember, and I can remember very vividly, I was at Clorox. And this was my first day at Clorox, first few days at Clorox. And I was pretty confident because I knew data, I know what it is, and I could talk data, right? I've done some work on it. And I went into this marketing meeting, and I was... kind of referencing certain things with a different terminology than finance or marketing people would reference. And these are pure marketing guys. And they're like, what the heck are you talking about? I have no idea what you're saying. It's just like we're talking two different languages. So I had to go back. I had to spend time with the marketing team to understand, go back and understand that use cases, understand exactly what they're trying to do. So I had to bite the bullet and spend time, bad as it may sound for me now, to acknowledge. I have to relearn data. That is no easy answer. I think that was my earliest mistake. And now I'm a little bit more careful. And every once in a while, I get overconfident, and I try to think like I know everything, till I get knocked down and said, nope, you don't know. And that's always the case. Data will always have a part of it that's different than what everybody will perceive it to be. And that's the challenge. You are closest to the data. But you still don't have a grasp, I believe, and this is all coming from, this is not my opinion. Just to be clear. You'll still have a grasp of the data, but it will be contrary to many of the biases other people have built in. People have looked at, it's like the, I don't know if you know, and it might be an Indian term, but seven blind men touching an elephant. That's the story of data. Somebody's touching the trunk, somebody's touching the tail, somebody's touching the legs. That's your data. The only person who actually has a full view of the data is potentially a data guy, a data engineer, a data scientist. And how do you present it to the people who are primarily biased and biases is the blindfold and have a view of that bias and they're higher up in the organization? So ego-sacrifice. So how do you do that? And there are areas where you don't confront, and there are areas that you go in and present in open ended points of conversation and eventually bring them along. And that's going to be the challenge. I think today the world is in a place where everybody understands that data is big. Data is huge. You know, LLMs are proving that if you have large learning models, you can actually do a lot of fun stuff, but again, that are going to be skeptics. So making sure that you can wrap your use case wrap, whatever you're trying to do within a right. information providing both the views of it right so you don't want to only provide views that are validating your point of view you also want to provide areas where it says oh here are the area that i found that are contrary to my views but these are potentially having explanation of why you think they are Something that builds to your narrative, something that builds to your storyline is important. So end of the day, telling a story is critical. And data engineers need to get better at storytelling. Data scientists and data visualization engineers need to get better at understanding the data. I think it's a two-way street there. So together, I think we have a very strong team.
**Benjamin:** Awesome, perfect. I think those were great closing words, Krishnan. It was great having you on the podcast, really kind of very, very awesome. I hope you have a great day. Yeah.
**Krishnan Viswanathan:** Thank you very much, Benjamin. I appreciate your time. Thanks for giving me this opportunity. I'm excited to be sharing my information. Take care.
# Data Observability with Millions of Users - Barr Moses (/blog/data-observability-with-millions-of-users-barr-moses)
Barr Moses, CEO of Monte Carlo explains the difference between data quality and data observability, and how to make sure your data is accurate in a world where so many different teams are accessing it.
Listen on [Apple Podcasts](https://podcasts.apple.com/us/podcast/the-data-engineering-show/id1561927688?uo=4) or [Spotify](https://open.spotify.com/show/6hMdnrFKlPbia2k6MkFs8U)
Benjamin: Welcome everyone to another episode of the Data Engineering Show. Today, we are joined by, Barr Moses, who is the Co-Founder and CEO of [Monte Carlo](https://www.montecarlodata.com/). For everyone who has not heard about Monte Carlo before, basically it's a data observability company or a data observability platform and I'm sure Barr will go into much more detail. Some maybe high-level facts about Barr, about Monte Carlo. They have recently raised $135 million series D in May of last year, 2022. Barr is the Co-Founder. Before that, kind of spent time at a bunch of different places and I'm sure she'll tell us more about that in just a minute.
Eldad, of course, thanks for joining as well, as always good to have you.
Eldad: Thanks for having me, as always. Perfect introduction. You're getting better with each episode.
Benjamin: It's amazing All the prep work and kind of practicing in front of the mirror paying off.
Barr: Can I get compliments too on my progress throughout podcast? Is that possible?
Eldad: You're amazing.
Barr Moses: Thank you so much.
Benjamin: Our sample size is one, so I'm not sure if we can. Perfect!
Barr, do you want to basically give us the high-level story of what Monte Carlo does? What data observability is all about?
Eldad: Let's start with yourself as well.
Barr: Yeah, for sure. I'm happy to. It's a big question.
My name is Barr. I started Monte Carlo three and a half years ago. Monte Carlo's mission is to help organizations actually use data by reducing what we call "data downtime." We've coined the term data downtime. In brief, data downtime is periods of time when your data is wrong, erroneous, inaccurate, or just unusable or untrustable for some reason.
That is a reality that data teams everywhere experience. So, we're fortunate to work with hundreds of customers ranging from folks Vimeo, Drata, Gusto, CNN, New York Times, Roche, and many others. All of whom have data teams who want to deliver trusted data and that's really hard to do if you're not thinking about data observability.
Take a step back and thinking about my background, I was born and raised in Israel. Started my career in the Israeli Air Force, and throughout my career, worked with data in different formats. It was actually sort of present in what I would call this acceleration of data.
I guess a decade ago, we said we were data-driven, no one was really using data. We maybe collected a little bit data and thought we were cool and moved on with our lives.
Then, maybe I want to say 3-5 years ago, people realize that you can actually make decisions based on data, and in fact, your decisions might be better if you use data. I think that was a big change for companies. I think we're still in the period of time when we're trying to figure out how to do that. We haven't figured out how to do it quite yet.
One of the most interesting trends, I think that's the backdrop to that, is what people call today data products. By that, I basically mean people using data, whether it's for internal reports, for data teams to make decisions based on data, or actually customers who are using data. Maybe that can be a dashboard that your customers use or your website. There's lots of examples for how customers actually use data and once we start putting that data out there, it'd better be accurate.
My personal story - Prior to Monte Carlo, I was at a company called Gainsight. Fortunate to be part of the category creation story at Gainsight. At Gainsight, we helped organizations basically reduce churn, and increase renewal rates and upsell. Basically, increase customer happiness based on data. I was responsible for the team that was using data, both to make decisions internally but also to surface with our customers. And the problem was that data was wrong all the time, literally just all the time.
Fast forward to today, there are some examples of where data being wrong is really painful. One of the most examples from that time, 2016, was that Netflix was down for 45 minutes because of duplicate data. Netflix was down for 45 minutes is really long time.
Eldad: I remember that moment.
Barr: Do you remember? Where were you in that moment?
Eldad: I was too young. I couldn't enter Netflix.
Benjamin: And also where are you like now?
Eldad: You mention Benjamin?
Barr: Eldad, you actually flew out of the picture at some point. I didn't know if…
Eldad: Oh, really?
Benjamin: You became invisible.
Barr: You literally flew out. You exited on the left. It was very dramatic. There you go. Okay, now you're back.
Benajamin: Welcome back.
Barr: Oh, now you flew out again.
Eldad: Okay, I'm going to fix it in a second. Give me a moment.
Barr: No problem.
Eldad: Please go on.
Barr: I thought it was by design. I was like, "oh, that's a cool feature."
Eldad: Okay, we have no electricity or equipment doesn't work. That's the reason.
Barr: Yeah, no problem. Where were you?
Benjamin: Going, right? Netflix being down for you 45 minutes.
Barr: Yeah. Everybody should remember that day. but guess what? They were down because of duplicate data. How crazy is that? In this world where actually data being down is the main culprit or even worse than applications being down and in that world, you need to make sure that your data's accurate, otherwise your applications and infrastructure are going to be down.
And that is a big change that has happened over my career in the last decade or so. I remember this was in 2016, I was responsible for a data team that was living data. And as I mentioned, the data was wrong all the time and I tried to fix it and I was like, this is so freaking hard. I remember going into a room with a whiteboard and trying to draw things and I'm not an engineer, and I'm like, this is so terrible. Why is this so freaking hard? And I remember asking our customers too, and they were like, "yeah, there's just no way to do this. We just have 6 eyes on every report." And I was like, really? We need 3 or 4 different people to read every single report. That's where we were at.
Eldad: That's a BI mindset.
Barr: Yeah, exactly. Is that normal? At some point, you don't know if you're crazy or the world is crazier or both are like, what's actually happening?
Basically, that actually inspired me and my team to try to build something. Me, and the person on my team, his name is Will Robbins. He's actually at Monte Carlo today, leading work on product and customer success. So, it's pretty cool to continue that path.
The bottom line is we hacked something together which was pretty crap, to be honest, but it worked well enough and improved like nothing that we had today. And then we implemented it with some of our customers and it worked well for them too. And I was like, Hey, why don't we get someone who's is actually an engineer to build this and let's see what happens. Can we build something?"
And then when we look at our software counterparts, they have solutions like New Relic and AppDynamics back in the day, and then Datadog now, and you have to be really irrational to build an engineering team without some observability platform or some observability measure. Why are data teams releasing data out in the wild without, something like observability or ways to make sure that data is accurate?
That basically kind of inspired starting Monte Carlo, spoke with hundreds of data teams, asking them what's keeping you up at night? And kept coming back to this problem that folks had, literally people sweating on Monday morning because they're going to be sharing a report and they're not sure if the data is right. I don't know if you have that, but you can hear your heart beating and you're like, "oh man, is this going to be right? I don't know?" Who's going to call out?
Eldad: We talked to the same people. They couldn't sleep at night. We took the data warehouse part.
Barr: Okay!
Eldad: You're saying so many things and resonate so well. Data was serving consensus and BI and it was all about having multiple versions of the truth and we couldn't get out of it. If you look at companies over the last four or five years, they have actually started to sell their data, utilize their data, build products on top of their data, and we're moving from having three people looking at a dashboard in the best case to having tens of thousands and hundreds of thousands of people, and you can't apply the same mindset anymore.
Also, you compare the Monte Carlo to Datadog and others. I don't think so. I think whenever I hear Monte Carlo, first, it's on a whole different ball game, so it's kind of less observability, more really it's all about data observability, data quality. Data quality needs to come first before it gets out to the customer. Observability goes after. So, it serves different needs. This is why I love Monte Carlo because to me, this is the first company that looks at data, looks how users use data, and rethinks observability from scratch. And it's also nice that you brought real engineer eventually to build something serious out of it, which is amazing, and we hear you all the time.
So, tell us, you quit the second and why you realize that nobody understands you when you drew that on the whiteboard, including the customers. How did that feel?
Barr: I'm not sure was that dramatic, but I think the experience of working with customers and the data being wrong all the time was terrible, really bad. I think that was for me personally. I would get emails from people, like why is the data wrong?
As a data professional, you're like, "I had one job, I was just get the data right." When you can't get that right, I think that's very frustrating. I think when talking to other people and they're like, "yeah, that's kind of like what happens?" We sort of came to accept that as sort of normal. Like yeah, it is wrong and that's okay. I remember living in that period and I was like, "there's just no way that I can accept that." And I don't think that makes sense. We have to get over that.
The days where it's fine that the data's wrong, it's just not cool anymore. Because data is used, for example, I'm reporting numbers to the street. So companies that almost reported wrong numbers to the street. That happens or companies that actually lose millions of dollars because the data is wrong. I'll give you an example. The most notable examples for the last few months. Unity is a gaming company, that basically made one mistake with their ad data and that one mistake cost them a hundred million dollars. One mistake! That's like a big freaking deal. It's not one mistake over a long period of time, multiple issues. One issue - a hundred million dollars. That's crazy! And that was made public. Think about all the issues that are not made public. They're way worse.
Then to give you an example that it's more on the social side. Equifax for folks who aren't familiar is a credit score company. So, basically a assigns credit scores to users and allows those users to take loans, take mortgages basically to live their lives. And it was made public that Equifax issued millions of wrong credit scores to users based on wrong data. That means millions of users that have the wrong credit scores today and literally…
Eldad: Inflation.
Barr: Yeah. They can blame inflation exactly or whatever it is. You think about the impact of bad data has on personal life. Even one person that's impacted in that way is terrible. Think about millions of people who have that is horrible.
I think there's like these trends, one is we're using data more and more. To your point, it's not just three people in some team looking at data once a year, maybe once a quarter, it's actually millions of people using the data, and actually, the stakes are way higher. They're not just looking at the dashboard. They're actually making decisions based on it, or they're trying to get a mortgage based on it. In all those instances, the data being wrong is just way, way worse.
I think there's just this point that we've crossed in the last few years. We can't look back. We can't unsee what we've seen with data now.
Eldad: We went all in on data and it's too late to go back.
Barr: Exactly.
Eldad: 20 years? From a product perspective, data is moving around so much, and there are so many hands being involved, apps, scripts, processes, steps, and in each step, data gets changed, gets expanded, and enriched. Where do you fit in, and if I'm a data warehouse or an engineer, can I use you as a data source to make sure that my data is kind of flow, no matter where it's coming from, I want to use you as my formal data source. Does that connect to my jdbc driver? How does it work from a product integration perspective?
Barr: Yeah, for sure. A few things, I would say, first of all, just to use this as an opportunity to explain the difference between data quality and data observability. I think a few years ago when data teams were thinking about data quality, they were really pulling a report from SAP or something where they had all the data in one place, you just had to dump the data once and then you would use it once a quarter or something like that.
In those instances, making sure that I think the concept of "garbage in garbage out" was really popularized at the time and that made it really important. I think that was the rise of data quality. During that time, it was very important for data teams or for analysts to make sure that the data's accurate, single point of time and that's it!
But the world in which we're today, which is a pretty cool world by the way, in terms of how we use data, is one in which there are so many people using data. So there are engineering teams, upstream, influencing, schema changes making adaptations or changes to code that have downstream implications. There are data engineers. There are data scientists, analytics engineers, and machine learning engineers. The list is long and each of these people actually want to use data and each of them needs to be able to use data.
Lots of things that are kind of a common trend or common thing that customers ask us is, how do I democratize data health for data quality? How do I make sure that the ability to know that data is accurate is not just for one person on the data team, but for everyone who's working with data? I think that has to do with your question around where does data live, and the thing is because there are so many people working with data, data also lives in different places. In order to make sure that your data is actually accurate and trusted, it's no longer sufficient that you're just looking at data in one place. You actually need to make sure that the data is accurate wherever it is. That may be your data lake, your data warehouse. You want to use some of your eTL orchestration solutions to help make sure that the changes that you're making there are not influencing data quality as well as your BI. At any given point in time, you have to be aware of the changes made to data and making sure that data is trusted, remains accurate and trusted.
I'll tell you a little bit about the history and evolution from my perspective. Before starting Monte Carlo, when I personally experienced this, noticed that all of my customers experiences, decided to leave Monte Carlo. I decided to start a company and I in fact started 3 companies in parallel. So, worked on two different ideas that are totally different, to kind of see what kind of pull looks.
In those early days, in order to test this idea of data being wrong, I actually reached out to lots of people and asked them does this ever happen to you? It was really hard to explain what does happen and I was looking recently at some of the wording that I used. It was like "reports breaking" or "dashboard is wrong" and that reality has not changed. Those words still hit a cord with data teams. You still get dashboards breaking. You still have reports going wrong. The thing is it's not just because of one source, it's because we now have data coming from your data warehouse, from your data lake, going through so many different transformations, it's actually hard to trace why the report broke or why the dashboard is wrong.
But that core problem actually hasn't changed, and I think we can trace that back to years ago when people are starting to use data and started to ask themselves, can I actually trust this data?
Eldad: God bless, copy and paste. The source of all evil!
Barr: Exactly. That's right. Multiple sources everywhere.
Benjamin: Awesome! Can you give some concrete examples, say I integrate with Monte Carlo today? What are the actual types of insights you can provide as a product?
Barr: Yeah, totally. I'm trying not to pitch Monte Carlo here. But basically, 'll give specific…
Eldad: We're getting great, existing and future product features from Monte Carlo. So, if there are startups out there trying to compete with Monte Carlo, now would be a time to listen carefully.
Barr: Great sound bite. Not going to happen. Just kidding!
Here's how to think about what data observability actually means and I think we touched on a couple of those already in very tactical sentences.
One is it has to have coverage end to end. By that we mean wherever your data is, it has to cover it. For some data teams that might mean, Firebolt or other competitors, not to be named, but others like AWS, Snowflake, Databricks, different types of places to store your data, aggregate, and analyze it, as well as your orchestrator. So you might have dbt as well, just use that an example and let's choose a BI. Let's say you have Looker. For many modern data teams, be a modern data stack. Actually having a solution that can cover all of those three categories is very important, data warehouse, data lake, ETL orchestration and BI. Again, because data is each of those things, and so actually having integrations that work which each of these is very important for your observability platform, whether that be Monte Carlo or not.
Second important thing is what does it actually do? Or how easy is it to get started with?
Oftentimes in data quality solutions in the past, everything was manual, so you had to manually specify the thresholds that you want for your rules, and you had to manually specify what lineage looks like. That doesn't cut it anymore. Not even close. Solutions need to be automated, and so within 24 hours, you need to have a view of your lineage, both upstream and downstream, both table and field level and you need to have a baseline for what healthy tables look like.
For example, if you have particular distribution of a field, let's say there is some specific null rate. It is possible to learn automatically what is the acceptable null rate, and let users know if that's being violated, without actually any end user specifying that as an example. Having this kind of aspect within 24 to 48 hours, getting started with those sort of machine learning out-of-the-box elements is super important.
Third thing that I would say will be called, the "5 Pillars of Data observability."
Those are:
* Freshness.
* Volume.
* Schema.
* Data Quality, and
* Lineage.
I touched on each of them, but just to kind of explain the history and why they've come together. I mentioned we spoke to hundreds of data leaders before we even started the company. By now, we've spoken to thousands of them. We've worked with thousands of data engineering users and hundreds of data teams, and actually ask them, what are the main reasons for why data goes wrong? And what does it look like when you're trying to troubleshoot it and resolve it?
In reality, there's a lot more commonality than people think. I think one of the main objections that folks have for observability is that they think that everyone is a Snowflake and your data is different. And yeah, that's true. You are a Snowflake. You're very special and your data's different, or actually, there are some patterns along with other folks, in how they use data. And so there is a certain amount of help that automation can introduce.
By codifying these 5 pillars, we are able to capture what the best data teams look for when automating observability.
The first round, "freshness," pretty straightforward, but basical is your data up to date? is easiest way to summarize that.
The second is "volume," which is basically like, is the volume of the data that you have in line with historical patterns are what you'd expect it to be. So, literally in terms of the size of the file or number of rows.
The third is "schema changes." Schema changes is a big deal for folks who know. It's also kind of tied to different trend called data contracts. We might touch on that later. Schema changes cause problems. It is the bane of existence for many data teams and so actually tracking schema changes in an automatic way is a huge help. Fun fact - It's one of the very first thing that we did in the product because it was so impactful on data teams to know what are some of the Schema changes that folks have.
The fourth is "data quality," which is basically you can think kind of from the worlds of profiling and actually making sure that the data itself at the field level is accurate. So, values that you expect. I talked about null rates, unique IDs, etc.
Then the fifth, which is "Lineage," which kind of brings it all together. Talked a little bit about lineage, but power of lineage is that when data goes wrong, the first thing that people ask themself is, where and who cares about this?
We talked about everybody hoarding data and using a lot of data. If you're dumping data and nobody cares about, no one's using, maybe you like only Eldad is looking at it at 6:00 AM, that's it! No one other than Eldad. So, maybe it doesn't matter, maybe it doesn't have to be accurate. But if Ben looking at the data or if Bard's looking at the data, then you really want to make sure the data is accurate, or maybe your customers are using it. And so being able to answer that question, Hey, is there anyone downstream who is using this data? Who should care about that? And if so, this should be a high priority. And maybe you can start thinking about actually announcing automatic severity to your data incidents and saying, Hey, there's specific conditions under which this data needs to be, there's a higher standard for it to be accurate and trusted.
Bringing it all together, I think that's sort of what makes a strong data observability platform, and we see amazing data stories of customers. I just use JetBlue for an example, for folks who've been flying a lot recently for the holidays and with the snowstorm and everything, there's a lot going on. For a company JetBlue, they manage tons of data. So, they use data both to drive their operations, whether it is like flight time or where's your luggage and what's your connecting flight? but also to manage their support and so the data team at JetBlue is actually like a great story of a team that's very, very thoughtful about making sure that data's accurate, so your flight is on time, so you get your luggage, etc.
Working with the JetBlue data team has been really cool. They, for example, manage 100% status rate incident every single week. Where literally every single week they go through each and every incident in Monte Carlo in particular and then triage that to make sure that the data quality issue is resolved and we're actually able to reduce significantly the number of data downtime incidents as a result.
Benjamin: Awesome! especially looking at data lineage, how does, especially in big companies, the adoption of a data observability tool look? Is it you need buy-in from everyone and then you need to integrate it across your entire stack because that's how you get the most value? Or is it like here's a single data team in your big company and you can integrate it with your local partner already get value?
Barr: Yeah. Every big journey starts with small steps towards that big journey. For sure I'm definitely in the camp of like, starts small and grow from there. Mostly because in any organization you want to show value, and have a win story really quickly. I think that that's true for anyone, not just observability, but I think in data in particular, if you put yourself in the shoes of the data teams in large organizations, they're in a tricky spot and I'll explain why. They invested a ton in the last few years in the best data infrastructure. World-class data warehouse, data lake, world-class infrastructure and also they invested a ton in hiring lots of people to use that data. We've seen the rise of data scientists. All of those roles have been growing a ton in the last few years.
Here's a tricky spot! Now, you need to deliver. You need to show that you can use the data, and that's new reality, which I don't think many organizations have proved. You're on the hook to see the ROI of all that investment. Oftentimes, the ROI doesn't exist because people don't trust the data or can't use it.
I would say in order to avoid being in that situation, starting to think early as you're rolling out your data infrastructure, how do I actually make sure that people also trust the data is really important. And for larger organizations, yes, I do think that starts with identifying a particular use case or a particular team. For example, it can be a team that's supporting financial data in particular, that's very important. If you're reporting on revenue growth or customer growth, you want to make sure that there's no questions about that data. Maybe starting with that. Sometimes there's a marketing team that you might want to start with. If they have hundreds of millions or tens of millions dollars in budget, you want to make sure that that money is deployed, that resources are deployed.
Another kind of examples of companies in healthcare, for example, potentially you have production data and you want to make sure that clinical data is accurate. In all of those instances, starting small, making sure that folks have that technology, but also have the mindset. Think about it, going back Eldad's point from before, maybe the difference is that for an organization like Datadog, the concept of observability and engineering is something that lots of folks have done before, but for data teams, it's a new motion. And so understanding who's responsible for what and what you actually do is a totally new ballgame for most people.
So, actually most of the work that we do is helping organizations think through what is a great data observability practice looks like? And oftentimes that has nothing to do with the technology. It's more around the practices and the types of culture that we have in the data team.
Eldad: Love it.
Benjamin: Definitely! Take us through the action part. I'm a data engineer now I get kind of my alert in my Slack channel. I don't know how it works. Kind of this upstream data pipeline is looking weird right now. What actually happens? Is it about spotting failure early to make sure you can kind of keep surface area small? What's usually the immediate action you take when you see these types of alerts?
Barr: Yeah, great question.
Yes, oftentimes, teams actually love getting that information to them directly and so that might be in Slack or Teams, or even via email if you'd like. Large majority, I think use Slack today. You get an alert in Slack, and that could be something, "Hey, this table that gets updated typically every hour, stopped updating for the last day and hasn't gotten an update for the last, let's say seven hours."
There are a couple of things that you can do. First of all, is an automatically assigned severity, maybe you see that it's a high sever severity might be a situation where you drop everything and take a look at it immediate. On the contrary, maybe this is a data set that you know is not being used and not looked at, then maybe you can ignore it for now.
Maybe you can snooze it actually and snooze it until tomorrow because today you're working on a production issue and you need to come back to it later. So assuming you didn't snooze it, you determine that this is actually high severity based on the assigned severity, and the impact of assets that it looks at. What you can take a look at is look at impact radius and start seeing who are the folks who are impacted by this type of issue.
For this particular table, there are three reports in your BI that are actually using this and the particular team that using that as a marketing team, maybe what you could do is you could tag the person on that team and say, "Hey there's an issue here, FYI, I'm investigating."
Then what you could start doing is click into that alert and start investigating and say, "what else is happening around that time? What else is happening to that table that might give me clues as to why this issue is happening?" And then maybe you can see, okay, well the job is running, but actually, no data is arriving, so maybe there's a problem with the low that's feeding data, just as an example, or maybe someone upstream, you noticed made a scheme of change and that scheme of change made it such that the implications are that now data's not arriving anymore. Maybe they change the field type and that messed up the scheduling of the data into that particular table.
On the other hand, maybe when you look at the downstream implications, you recognize that you need to reload all the data to fix that and to make the report accurate now. There's actually a lot of work that might be, involved in both understanding why this issue has happened and also making sure that the data is sort of back to normal or back to its schedule.
It's a combination of looking at both the data itself, looking at metadata and oftentimes looking at the code that's driving all of this to make sure that the data pipeline itself is healthy. So, you might be moving between those different systems and where you're at, and Monte Carlo in particular, we try to bring all of those different feeds into one place. So you actually get both your alerts from Monte Carlo as well as your alerts from dbt in one place, for example. If you're using dbt in this example, you might be not notified about that same issue in the same place.
Bringing all of that together, maybe upon investigation, you realized you understand the root cause of the problem. You understand how to resolve it and now the problem is that you're not responsible for the table itself. So, you can't actually fix it. So, you need to find who is the person who owns this. You can see that information as well and then ping them and work with that person to fix the issue.
At a high level, there are three different things that we're trying to help users do.
One is to know about the problems almost in real time. Just to give you insight into this, most teams are not even aware of these issues and so getting that alert in Slack is a big deal.
The second thing that we help with is resolving these faster. Oftentimes, data teams won't know all this information about, like schema changes or freshness or volume, all the examples that I mentioned, and it would be harder for them to pinpoint what exactly is happening and why?
The third thing is by increasing the communication collaboration on this, you're making it easier for everyone who's touching data to know about data downtime issues and eventually actually preventing these from happening.
Eldad: Would you say that in a few years from now, we will use less classic black box mindset observability plus Jira and more of new context-driven, lineage-driven observability, which listening to you, it's so different. It's so, so different from looking at CPU or disc utilization while someone altered the table but that table affects a hundred Looker dashboard and that's not important. What's important is one of those dashboards serves a customer. There's no way to know that without spending three weeks involving six people, at least using tons of tech and companies try to do it on their own. They try to stitch it. They try to build it, and it's really sad.
But listening to you makes me happy because it is a real thing, and as engineering organizations are trying to become data organizations, I think they have a lot to learn from that. From data engineering mindset where it's taking very seriously. So when someone is modifying something for those scheme, that's like code, that's like a product release. And having data observability that has the schema, that has the data warehouse, it's aware, it knows what an information schema is. It's not like just raw JSON pushed into some time series database, which is meaningless for most people. I love it! It's fascinating!
Barr: Yeah, I think that's spot on. I think as a data industry, if we're successful, it's because we adopted what we need from engineering practices and built that into data engineering. It's going to be impossible for us to actually really, truly become data driven if we don't do that. So, yes, and I think our approach is that starting with the observability is the right place because it's the most imminent problem for engineers.
If you look at data engineering today, the most imminent problem that they have is, "hey, the data is wrong all the time." That's the number one thing that they have as an issue and as a result, data consumers can't trust the data and can't use the data.
I think there's a lot more that we can learn from engineering and that we should over time bring in a lot of those concepts into data, but starting with observability is what we think is the thing that will bring the most value to customers today.
Eldad: Nice.
Barr: And I'm curious to hear more about how you all think about it at Firebolt, but so feel free to let me know if you all are upright.
Eldad: We feel the pain and need to make sure that data quality is right. We've seen some really, scary stories. You've mentioned some of them, but data is so actionable today that customers use data-driven products, their business run on those products. If I'm subscribing to a sales optimization product and that sales optimization product has given me a recommendation how to do my ad spending, and I'm going and just doing this ad spend based on that recommendation, and it's wrong. That affects my business, and it it's real. So, it's much more serious than just looking at two different types of sales numbers, one coming from Salesforce and one from Excel, which is also important. We've been dealing with that for 20 years. But the beauty is that data is really driving the business now versus just being used to understand the business.
So, having data quality should be kind of trivial in core stack of any data driven team. They are listening to you and looking at kind of what you're doing there. We really want and wish Monte Carlo to succeed big time. And having said that maybe we can wrap with that.
If you can share some insights, some recommendations, something with our fellow startups, with our fellow users, how do you see this year coming? What would you recommend for startups going forward? Anything valuable would be highly appreciated.
Barr: I can share only non-valuable advice.
Eldad: Perfect!
Barr: Actually, on that note, I will say, for startups or listeners, I will start by saying that you should not listen to advice. That's my biggest takeaway. I think for any question that you have, both in data in startups and otherwise, you will always get, 50% of people will tell you something and 50% of people will tell you the otherwise, the opposite. So, really I think the answer actually lies in the data. There's no one else but you to take a look at the data and listen to your customers and see what it tells you and actually draw conclusions based on that. Folks who are asking themselves like what does this mean for me and for my team and for the industry, you have the data, you are closest to the customer.
Pick up the phone, talk to the customer, ask them what they think. That is the data that will help you get to the answer, whatever that may be. That's my take on advice.
With that in mind, looking ahead to this year, I think this is the year that's sort of a natural progression of the importance of data teams. I think obviously, like it's no secret, that the market has changed difficult times for lots of folks across the economy and yet I continue to see data teams growing stronger and stronger in organizations. And that means companies doubling down on their data strategy, building more and more data products, and investing in data teams because recognizing that data is a foundation to making strong business decisions, gaining competitive advantages, and generally furthering business outcomes for their customers. And so I think when we zoom out and think about where are data teams today? I think they're on the hook to deliver ROI from all of their investments in the last few years and this is the time when folks are going to be more scrutinized. Yes, I do think that folks will be asking themselves, is there actually ROI on this? And it's a great time for data teams to be able to articulate that and say, yes, we have awesome data, we have best in class infrastructure, and we can also trust the data so we can actually use it to your point, to drive the business. And I think that's an inflection point that I'm really excited for data teams to bring. I hope 2023 will be the year that we see that. I don't think that we can see it soon enough, but I do think it's an inflection point that the data industry needs to drive. And I think there's no better time than this year to do this because data teams are more important than ever. Data is more important than ever and time, the onus on us to prove that is today. So, I'm excited to see us do that.
Eldad: Boom!
Benjamin: Awesome! That were amazing closing remarks. Barr, thank you so much for your time. We really appreciate it. Have a great rest of the day and see you next time on the Data Engineering Show.
Barr: Thanks, great podcast
Eldad: Thank you everyone, looking forward.
Benjamin: Thanks.
# Data Rewind: Conversation Highlights from Zach, Matthew, Joe, and Krishnan (/blog/data-rewind-conversation-highlights-from-zach-matthew-joe-and-krishnan)
In this special roundup episode of *The Data Engineering Show*, the Bros revisits some of the best bits from episodes with data thought leaders Zach Wilson, Matthew Housley, Joe Reis, and Krishnan Viswanathan, spotlighting essential trends and lessons learned across the evolving data engineering landscape.
Listen on [Spotify](https://spoti.fi/40qWYLa) or [Apple Podcasts](https://apple.co/4f2I9CX)
The Data Engineering Show is brought to you by Firebolt, the cloud data warehouse for low-latency analytics. Get $200 credits and start your free trial at firebolt.io.
Benjamin - 00:00:10: So on my end, I'm just kind of like C++ database nerd. I care about how to build concurrent index structures. I don't know, like how to build a fast multi-threaded join and so on. And whenever I look at data engineering, right, and kind of like I interface with a lot of data engineers, Firebolt kind of always seems kind of really daunting just because like the kind of breadth of the field in terms of like just amount of different technologies you have just seems so crazy, right? So like how do you decide on kind of what to like focus on in terms of actually kind of, yeah, like teaching people kind of skills that are valuable for their career?
Zach - 00:00:52: Like for my bootcamp, for example, right, there's six weeks. Only two of those weeks are actually tech specific. The other four weeks are not tech specific. They're tech agnostic. So where, because I think there's a couple things and a couple philosophies that are really important in data engineering that are actually like, they apply regardless of if you're using Spark or Snowflake or Databricks or Presto or Flink or like whatever, you know, tech you want to use for it. And there's a couple of them. Like one is like around like data modeling, how to do proper data modeling for dimensions and facts and how to really get those things like compacted down. And there's a lot of trade-offs in that space that is very art, very, it's not as science and it's a lot more art and you have to understand. Like your consumers. Yeah. And they have to have that empathy. And that part is powerful. Like for example, for me, like when I was working at Airbnb, like I would say 80 to 90% of the impact I had was going to be in two buckets, right? It was in the leadership bucket of inspiring other people and helping them grow. And the other one is data modeling and getting, making robust data models that can then be used by a large number of people downstream. What wasn't as important was like, how good I was at Spark. Like, I mean, I look at Spark as more of like a means of accomplishing something or like as like a, it's just one kind of path forward and that like you just, it solves the problem and maybe it can be a little bit faster. Maybe it can solve those things. But really, this is the fundamental thing that I think a lot of data engineers need to remember is your product that you sell is data. It's not a pipeline. A pipeline, it can help, right? In term of maintenance and like pain and suffering. Like if your pipeline sucks, then like the data is going to be annoying. But like generally speaking, the value you're providing is in the data sets that you provide. And that is like, and if those data sets are not modeled properly, then that's where you can have a lot of unnecessary cost, right? And like, in big tech companies, these mistakes actually cost them millions and millions and millions of dollars a year. Because of like, if you don't model things the right way, then downstream, the compression doesn't work the same way. And then it can blow the data up again, right? And there's a lot of interesting, like tricky things that I've noticed with like how data modeling works. So that's one. I'd say another kind of tech agnostic thing is around data quality, right? And understanding, again, there's technologies here like Amazon DQ and great expectations. And there's going to be 10 trillion more coming. And but like, it's more again around like how to test data like of like, is this quality? Is it not quality? How to validate it, right? That's very like agnostic of like the tech that you're using. And you should definitely be able to do that. And then the last bucket of things that are tech agnostic is storytelling, right? You tell a compelling story, can you like make some cool charts? Can you persuade people to give you the time to make this data? And other things like that, because there's like the story, there's like the before data story. And then there's also the after you have data story. And both of those stories matter. And being able to construct those narratives in a compelling way. Very important persuasion. And then the tech is the last one. And like, and like, in some ways, I think the last one, but also not as important and- But it's tricky because and this is the thing I hate about industry in some regards is that like, If you go into an interview, right? And you go into the interview, like 80% of the questions are going to be on like Spark or Flink and be like, oh, do you know this very specific minor detail about Spark? And it's like, dude, this doesn't matter that much actually in the end. But that's how things are tested, right? And I hope that industry changes in that way.
Benjamin - 00:04:49: Gotcha. So, I mean, looking at this bootcamp, because you're framing it in that context, kind of you said in the beginning, it was like the goal is kind of from good to great. So usually like those will be people who already know Spark, right? Kind of know how to maybe like write Scala code for their Spark stuff, like those types of things. On the other end, and it's hard to say from good to great, is like, okay, you have someone, right? Maybe that, okay, not the influencer with 25,000 followers talking about data engineering, but just someone wrapping up high school who wants to get into data engineering. And if you have to pick up some technology, right? And like there, it just seems kind of like daunting to in a sense, like make that choice, right? Kind of what horses do you bet on? Kind of with what do you get started to actually start with that career?
Zach - 00:05:33: I mean, I totally agree. I think it's similar to, like, so I'm a pretty like athletic, sporty guy, right? And like one of the things that I remember as a kid growing up, my parents were always like, you got to do sports, right? And then I was like, okay. And like, then I tried a bunch of, I tried like soccer, I tried basketball, I tried baseball, I tried like all these different sports that like were all interesting and different. And like, one of the things that I learned about it was, especially going through that process was like, yeah, you just got to pick one and be like, I'm going to get good at this one. And for me, that was basketball. And I mean, I got lucky, I'm tall or I'm 60. So basketball was the easy, obvious choice. And that's one of the things that's tricky about tech sometimes is that the choices aren't so obvious, right? They're not like, oh yeah, this one is 6'7'' and this one's 4'2". So we should go with a taller one or whatever, right? It's not that like obvious a lot of the time. And so I think there's kind of a couple pieces there, like on like how to pick like technologies is one is going to be like, okay, what do you see on social media? Like, I know that that's like, I'm not going to say all social media because I still don't trust TikTok here. But on LinkedIn, like if you have enough of a network on LinkedIn, do a poll. Polls are broken on LinkedIn. Like if you do a poll on LinkedIn, like even if you have no followers, it's going to be seen by like 10,000 people because polls are broken and they're very good. And the reach they get is too good. And so you can learn, you can ask, right? And I found that there's like two or three really high fidelity sources of like, where to get like good information. You have LinkedIn. LinkedIn doesn't give you... The thing about LinkedIn though, is it doesn't give you very... It's not as good about negative feedback. Like... So if you're like asking someone to be like, hey, say why my video sucks. Most people aren't going to do that in the comments section on LinkedIn because like, they're like, I don't want to look like an asshole. And so that's one. Reddit. Reddit's better for that. Reddit's almost too good for that. If you want people to tear you down, go to Reddit. Like Reddit or Blinds even better if you really want to get... But those places, you can also get the more like guidance on like, okay, should I learn these texts or these texts, right? And like, I think like really the big things are going to be just like learning the languages first. SQL, Python, right? If you really want to break into DE, SQL, Python, learn the languages first and then you can learn the tech after that actually. Because you can do... If you just do like Postgres to learn SQL, just like a very basic database, then you can build into the more complicated ones like Snowflake or Firebolt or Spark or whatever you want to use, right? And then like, you can kind of go into the cloud. And like, I found that kind of building more locally first and like kind of learning languages that way. I've had more success with some students kind of teaching them that way as opposed to like being like, okay, here's how you set up an EC2 instance on AWS and now you have a computer in the cloud and it's going to crunch the data for you. And like a lot of that feels like a lot more complexity that they don't really need yet until they have more confidence in their own skills.
Benjamin - 00:08:45: Looking at SQL databases, I guess kind of one of the good things is looking at the space right now. It's like so many, like so many systems are actually kind of converging around the Postgres dialect. There's not like you need to kind of learn like seven different kind of flavors of like SQL or like window function syntaxes, whatever. Actually, like a lot of systems tend to be at least more similar to date than maybe they used to some time ago. So that's super cool. So thanks for the insights there. I mean, like I'm sure a lot of listeners will appreciate that. Cool. Zooming out a bit, right, from the kind of specific technology, like, do you see any kind of big kind of trends at the moment? Like, when I look at LinkedIn, for example, like, one thing that keeps coming up is, like, data observability, kind of data monitoring, data quality. Like, those seem to be what some of the things kind of generating a lot of buzz. Kind of, yeah, what else is out there?
Zach - 00:09:39: Yeah, like, data monitoring, MLOps, data versioning. There's all sorts of, like, interesting things. And then there's, like, a couple of them that come back around sometimes, like, data mesh. Like, I hear data mesh, like, once every three months or something like that. And I'm like, hey, it's there. It's a thing, right? But, like, I think a couple of things that I really am seeing, yeah, is definitely data observability of, like, yo, like, how is this data changing over time? And, like, it's very closely linked with data quality. And honestly, they should be closely linked because if you aren't aware of, like... How your data, the shape of your data, what it looks like over time, then you really don't have good data quality checks because you haven't done your due diligence on looking at what is normal and what is abnormal. I mean, there's data quality checks out there that are very easy to know or what is normal and not normal. Is there any data? No data is abnormal, right? Or this column's null when it should never be null. That's abnormal. Very easy check. But then things, what you define as normal versus abnormal, it gets more and more complicated as you look at more and more different data points together. And that's where if you look at week-over-week row counts, that's going to be one that, what is abnormal versus normal? It depends. Because a lot of times those week-over-week row counts, on Christmas Day, they fail because there's not as much data or there's too much data. And it's actually not... You're looking at the wrong period. Instead of week-over-week, you really should be looking year-over-year and looking at it on zooming out to find the actual pattern that matters the most. And that's why people do week-over-week instead of day-over-day because of the Sunday-Monday phenomena. And Sunday-Monday is super annoying as well. That one's very common to trip people up. That's why week-over-week is better. But it still misses the holiday patterns. So I think that those kind of observability, things are super important because it's linked to quality and that is linked to trust. And because it's like without quality, you don't have trust, right? And definitely, I think that that, I would say, is the big thing that I've definitely been seeing. I've also been seeing a little bit more of a push towards streaming and trying to get more people involved. I've been hearing about ClickHouse so much recently. Everyone's like, you got to try ClickHouse. You got to try ClickHouse. I have not tried ClickHouse yet, but I need to. Just because it's been... I've seen it in every single comment section of all my posts. So yeah.
Benjamin - 00:12:19: One thing I'm curious about in general, right, kind of like coming out of this is we're saying, hey, our tools got much better, right? But we still have many of the same problems we used to have. We still keep cycling around using kind of topics. And both you, Joe and Matt, kind of, right, you're teaching a lot. Kind of you're doing thought leadership. Kind of you're writing blogs. You kind of wrote that super well-known book. You're affiliated with the University of Utah. You're consulting. Kind of like arguably you could say, okay, if we have all of those amazing tools now and we're still cycling around the same kind of types of problems, right? Maybe we're just not teaching it well enough. So what does that mean for your approach to kind of delivering these things to A, professionals, students, those types of things?
Rob - 00:12:58: We failed.
Joe - 00:13:00: The kid comes out swinging.
Rob - 00:13:07: That's a good question.
Joe - 00:13:08: I mean, I think part of the problem, and this is not to trash vendors too much, I think vendors build fantastic products.
Rob - 00:13:14: Horrible products.
Joe - 00:13:14: Yeah, yeah, yeah. But I mean, if I'm in sales for a vendor, I'm not necessarily focused on how I use the tool. I just want to get the tool out there and get people using it, right? And that's where there is more need for people on kind of the meta level to come in and say, all right, you've decided on X, Y, and Z tools. How can we actually use these to help the company? And do that training all along. I mean, I think Joe and I have complained a lot about the lack of training for undergraduates and data specifically. And part of that training as we build it out needs to be, obviously, they need to learn data fundamentals like data modeling, thin ops, cost management, but also what it's like to work inside a business and the kinds of things that businesses care about and how they can communicate better. I mean, communications are notoriously difficult to teach, right? Because how do you teach someone out of a textbook how to communicate with someone? But we need to keep thinking about these problems and figure out how to give students practical, concrete experience with communicating with businesses and stakeholders.
Benjamin - 00:14:16: So how do you do that? Because that was also a very abstract answer.
Joe - 00:14:20: Fair, fair. I mean, I think from our perspective, a lot of this comes down to building better collaboration between undergrad and master's programs and businesses. Shockingly, often we see that you've sort of got this MBA world, that operates almost in a vacuum separate from the business world. And that's not ideal, right?
Rob - 00:14:40: Well, the academic world operates separate from the business world too.
Joe - 00:14:42: Yeah, absolutely.
Rob - 00:14:43: In some cases, that's good. In a lot of cases, I think it's pretty bad. It does a disservice to students. So that's one thing I'd like to see change, right? So you talk about concrete stuff. I would also like to see more apprenticeship-type programs. I think the notion of a university being a necessity, I think is absolutely the wrong way to go. So I think more people could be trained on this from practical things apprenticeships... I'm creating a new MOOC class right now, a course for a really big MOOC. One of the things I'm doing is... It's a simulator. It's your first day on the job as a data engineer. You get to go do business requirement gathering. You get to go find out what stakeholders want. And part of it is identifying, okay, so you're given this list of requirements. What are people actually asking for? So I think that's the other part. We spent too much time teaching tools and not enough time teaching the techniques. So I think those are concrete ways that we could address it. Because it's easy to do PySpark tutorials and stuff. But that's the wrong way to teach data. I think the way we teach it is absolutely backwards. Know the techniques and then learn the tools. That's why we wrote the book the way we did it. It's technology agnostic, for example. And pretty much every company in the universe is using it for their data teams right now. Almost every university that we know is increasingly being used as a default textbook for data engineering. So to me, that's part of the process. But it's not going to be an overnight thing. But I think the way we approached our book is similar to how Martin Kleppmann approached his book. It's agnostic. It stands the test of time. And that's kind of where we need to get to. We are making an effort. It is slow, especially universities are slow. They're so slow. And that's part of the problem with them.
Benjamin - 00:16:20: In the intro, you were kind of mentioning this like data deja vu, right? Kind of seeing things over and over again. And we talked about that data quality aspect of it. Like in what other areas of like data engineering as a whole are you kind of having this like data deja vu nowadays that you had in the past?
Krishnan - 00:16:39: So we talked of quite a few things, right? What I faced in 2000, 2001 for a company financial metrics and seeing that again today. So I moved organizations, but I see the same thing happening here. And there's a lot of... Thing that I'm noticing that are consistent in those days here. Some of them is because we operate as a startup. And so there are some challenges there. But the other thing is, it's also a financial industry. And there is a very strict review. So they are very conservative in how they work on this platform. Also remember, from 2006 approximately to like... Recently, I've gone towards the vendor side and I've also worked in other companies which were more open to buying vendor products. We are back here to a point where most of it is in-built and in-house. So that's another shift. So I know from a vendor perspective, there are a lot of... Availability in terms of tools and technologies that we could easily incorporate and put in, but that's not how we built at Blackhawk. We invent, we do it here. And a lot of reasons is because we eventually end up spending, and sharing it with our clients. So if I were to go and bring a data in, I don't have to only look at how much money is spent in buying that. I also have to license it for other users. So to make it profitable for us, we have to be able to build it and scale it to our clients. So the engineering world that I'm in, recently. It is interesting because what I'm trying to do is... Given all the experience that I have had, can I take my team? And my team is fairly young. Average age is probably 30. I'm bringing the age up pretty high, right? But 30, 32. How do I take them through this? When they have one, I think part of them is mostly software engineers, not data engineers. I have to transition them. So I have to go back to dig into my old days of how I reacted to this. Get them data savvy. So that's that, right? The whole people transformation, because these are great people. So I need to translate. And then how do I also translate that into what we can objectively achieve year over year and turn better? And I've been only here for a very short time. So it's a long process. I'm still learning a lot of things and a lot of challenges, but I believe that is, like I said, the next three, four years are going to be massive in terms of how the whole industry changes to solve for this, but also how our companies are going to make a big difference. And I see a lot of potential there.
Benjamin - 00:19:26: Another thing, right? We talked about it earlier in terms of the dot com, right? And kind of the predictions you were seeing and so on. Like now as well, right? That's kind of like at the macro level at the moment, like not a great time and especially for tech. So one thing which is coming up more and more kind of in conversations I'm having with people in this data space is this question of kind of proving that you're... All this data you're collecting is actually worth, right? That at the end of the day, it's kind of contributing to the bottom line of the business. What are your thoughts on that?
Krishnan - 00:19:56: I have... I have... Dealt with two sides of that coin. So remember my pre-greencrumb date and my post-greencrumb date. So when it came to, so the pre-Green Plum days, I was like, why are we collecting data that we cannot even process? What's the value of that? Why are we creating a data swamp, right? So I have that. That's part of my branch, right? But when I was part of Green Plum and I was the product manager for Green Plum, I had to put that on a really back burner and not bring that up. I had to talk about all the cool things you can do. But reality is, and this is why I think as a technical and technology industry, we didn't do a good job of educating our clients. You can't just continue to collect data and not process them if you don't even know what exists. The hard-op days and the data breaks days, those days are great for technology and great for infrastructure spend. I'm glad that we had easy money at that time. But now I don't think that's going to fly. Because even at a company like Clorox, then I did a couple, I joined Clorox. I didn't talk enough about Clorox, but I joined Clorox to... To first upgrade that, modernize that data platform. First on-prem to Oracle Web Data Warehouses and Exadata and stuff like that. And eventually I did it to migrate to Google Cloud and Azure. So even there, because we are CPG and margins are really small, there was always this underlying question. Why do we need to collect so much data? How can we optimize our data process? So what I ended up doing was not only doing the data transformation, doing the data inference and bringing the data in, I actually ended up building a couple of applications on top of it to showcase what data means to the company like Dropbox. So this ties in very well with your question because executives are not sending any black check at that company. We use data to try and predict forecasting accuracy for products. Can I predict how much can I sell it? So we couldn't do it really good. So that's what we did. We tried using NLP and voice interaction to see if we can. Predict and call out any product concerns. So that kind of worked out okay. I built a new version of executive information system for Clorox. We called it the daily briefing, which was a mobile application, but pretty much following the same standard. And it was on a mobile phone. That became a huge success. That was like my last run. And those are all built on the cloud, right? All of these things are built on the cloud. So I always worked in industries where we had a very narrow path between how much data we collect and what's the value of that data. Nobody gave me a blank check, except in the green firm days. Even in the green firm, we were telling customers. Nobody gave me a blank check to go in and load as much data as you can. We did one project which eventually became called the CDP, the Consumer Data Platform. And we brought in cookie information and cookie data. And I did it for about... One year, and we were collecting about a billion records per day, right, when you explore that marketing data. But in a year, we could not find any valuable metric out of that. At least the marketing team couldn't find any valuable metric. And we are brick and mortar. We're not weak. So it may have been better for online than us, but nevertheless, that was shut down fairly quickly. And it was all on-prem. We had a Hadoop cluster. I set it up. That goes to show. There were some people who were not in the technology world who didn't buy into this whole hype on collect as much data as you want and then run your algorithms on top of it and just get the results. I think that did work.
Benjamin - 00:23:49: So, yeah. Do you have any kind of closing things you want to say? So maybe one thing for people getting into this space, because you said, okay, you're taking software engineers now kind of at BlackRock and getting closer to this data engineering world. If I was starting today, out of high school, going to college to get into this space, any advice from your end?
Krishnan - 00:24:14: I think when I grew up, there was no Google. There was no cloud. There was no YouTube. There was no, I think that's what it's called, online learning, online training, course errors. So all those are available today. So all I have to do, which makes my job a little bit easier, is point my team in the right direction and have them go and learn and get better. The biggest shift from software engineering to data engineering, in my mind, is the data domain knowledge. I possibly intuitively do that because of my past experience, but understanding how data is processed on top of the code is such a big thing, which software engineers don't care. Software engineers think of data as an afterthought. Whereas data engineers think of data as this first step, right? So how do I scale? How do I make this consistent, secure? That would be my first thing. It talks of the... Learning, I think there are already so many tools and technologies in place. It's easier. It's a whole lot better today than it was in my days. You might think of me as going back to the old days, but that's true. We didn't have the right tools and technologies. I can get the job done much faster today. In some cases, you're stuck with legacy code. That's too bad. We've got to get that out. I see a great potential for people who are coming into the data world. And I wouldn't have said this. Earlier this year, which I'm saying now, right? The AI winter is possibly near and here, but it's going to pass that is so many. Opportunities with AI, with data, and with the whole data industry that We don't even know what the Future applications are going to look like. Future applications. Are going to be driven by data. Making sure that you have the right data, the right place, processed properly, will make your company more effective, more worthwhile. I've seen this again and again and again. And. Like I said, I did an application in 2000 without understanding what the implications are. I did the same application at Cora. With the intention of, nobody gave me the requirement. I came up with that model. I pitched that idea to my CIO, made that success. And the whole reason was I had that background in data. I had the background in business knowledge to say, from an executive point of view, this is what they would look at. And of course, I had help. I'm not going to say that I didn't get. Trying to fine tune that. I could do that on a different technology, on a different platform, because I understood the data and the value of data and how people process that data. So data itself is abstract. Data is difficult to comprehend. Making it easier, that's the job of a data engineer, data scientist. However you want to label it, data analyst, data scientist, data engineer. Those are all different flavors of the same thing. Make sure when you are in front of an audience and trying to pitch, you understand the data. Have a good data representation, valid, not biased, valid data representation that can help others see the picture you want them to see. That itself is a big hurdle to pass. And then once you have done that, the third aspect of it is... All the cool technologies that are in place. You can build AI and you can build ML, but if you don't do these two things and give them what can be the truth, the future of the possibility, I think data will just remain unprocessed.
Outro - 00:27:52: The Data Engineering Show is brought to you by Firebolt, the cloud data warehouse for low latency analytics. Get $200 credits and start your free trial at firebolt.io.
# Decomposing Firebolt transactions (/blog/decomposing-firebolt-transactions)
This week, Alex Miller wrote a blog post titled [Decomposing Transaction Systems](https://transactional.blog/blog/2025-decomposing-transactional-systems). He approached explaining transactions from a new fresh angle, but one which immediately felt intuitive. Alex decomposed transaction into four steps:
Every transactional system does four things:
* It executes transactions.
* It orders transactions.
* It validates transactions.
* It persists transactions.
Alex gives examples how these steps map to both generic OCC and PCC systems, as well as examples of real-world systems like FoundationDB and Spanner.
Then, Mark Brooker, followed up with a post of his own about how these steps map to [Aurora DSQL](https://brooker.co.za/blog/2025/04/17/decomposing.html).
So I naturally thought to map them to Firebolt. We haven't published in-depth technical articles about Firebolt transaction manager, but there is [10 minute video](https://www.youtube.com/watch?v=i6yz3Xm3eOg), where we describe how it works to CMU's Database class.
So let's go step by step
> *Executing* a transaction means evaluating the body of the transaction to produce the intended reads and writes.
Firebolt execution happens in parallel across the nodes of Firebolt engine cluster. There is no coordination between nodes or between clusters at that stage. We use MVCC, so all the writes create new version of data which is invisible to other transactions.
> *Validating* a transaction means enforcing concurrency control, or more rarely, domain-specific semantics.
Firebolt employs OCC and therefore validation happens after execution, when transaction is ready to commit. Default isolation level in Firebolt is [Snapshot Isolation](https://jepsen.io/consistency/models/snapshot-isolation), which means that transaction manager needs to check for Write/Write conflicts, i.e. to check if the current transaction (the one about to commit) did not write to the same rows as transactions which were active when the current transaction started, but committed before. If such conflicts are detected, the current transaction is aborted. Note, that validation step happens independently without coordination with other transactions trying to commit.
> *Ordering* a transaction means assigning the transaction some notion of a time at which it occurred.
> *Persisting* a transaction makes making it durable, generally to disk.
This is where Firebolt differs from classic OCC sequence of Execute -> Order -> Validate -> Persist. In Firebolt, Order and Validate are swapped, and Order happens together with Persist. After Validation step, Firebolt's transaction manager will atomically pick the Log Sequence Number (LSN) to correspond to current transaction's commit, and persist it in Write Ahead Log (WAL). But it also checks whether some other transaction managed to commit since validation was completed, because if it did - the validation step needs to be repeated. Think about it as [Compare and Swap](https://en.wikipedia.org/wiki/Compare-and-swap) operation - transaction manager atomically assigns and writes new highest LSN, but only if it didn't changed since the validation step.
Firebolt uses [FoundationDB](https://www.foundationdb.org/) to store all of its metadata including the WAL. So the Order + Persist step is implemented as FoundationDB transaction to ensure atomicity. But also, once persisted, the WAL is automatically replicated just like any write into FoundationDB would.
In addition to that replication which is there to ensure durability, Firebolt also replicates WAL's content into Kafka topics. This happens asynchronously, and is not part of transaction protocol, i.e. it is not needed for correctness, but it is a performance optimization technique - all of Firebolt compute engines subscribe to that Kafka topic, and see all changes, which allows them to do some prefetching and prepare for the new state of the system.
#### Additional note [#additional-note]
Alex has the following note in his blog
> MVCC databases may assign two versions: an initial read version, and a final commit version. In this case, we're mainly focused on the specific point at which the commit version is chosen — the time at which the database claims all reads and writes occurred atomically.
This is exactly what's happening in Firebolt's implementation of MVCC. When transaction starts, it obtains from transaction manager LSN to be used for reads - which define the *snapshot* in Snapshot Isolation.
For read-only transactions, there is nothing else, as their commit is no-op. So read-only transactions only get Execute step - there is no Validate, Order or Persist steps.
The read-write transactions get LSN of commit during Order + Persist step described above.
# Diving Into GitHub's Data Stack (/blog/diving-into-githubs-data-stack)
It's the mother of all development projects. You use it daily. And so do 65M developers around the world. This time on the Data Engineering Show – A deep dive into GitHub's data stack. Arfon Smith KimYen (Truong) Ladia shared GitHub's data engineering challenges and solutions and explained why every developer should know and adopt the ADR protocol.
Listen on [Apple Podcasts](https://podcasts.apple.com/us/podcast/diving-into-githubs-data-stack/id1561927688?i=1000539285425) or [Spotify](https://open.spotify.com/episode/0J4HkBtksS6VIXTqRwczVG)
Boaz: Welcome everybody.
Boaz: Hi, Eldad.
Eldad: Hey, Boaz.
Boaz: We are here with another podcast of the Data Engineering Show. That's the name of our show? Right?
Eldad: Yes.
Boaz: This is it. And with us today, we have two great guests. We have Kim Lydia and Arfon Smith, both from GitHub. So, thank you, Kim and Arfon, for joining us.
Boaz: Welcome our guests.
Eldad: Welcome, welcome everyone.
Boaz: So actually, you know, we are very excited to have you on the show when we were sort of discussing in advance, we wonder how GitHub is dealing with data from an engineering perspective, and you know, how all the way we work with software has tickled into the way we work with data. That is an interesting topic in itself; but for starters, how about you two introduce yourselves? Kim, let's go with you first and tell us a little bit about yourself. What do you do at GitHub? And then we will switch to Arfon.
Kim: Hi everybody. I am Kim. I am a software engineer at GitHub, and I have been at GitHub for almost two years now and I am working on the data platform team. Before joining GitHub, I was helping to build a small data warehouse for a nonprofit.
Boaz: Arfon, now it is your turn.
Arfon: Yes. Hi, my name is Arfon Smith. I am a product manager for data at GitHub, and I support data engineering, data science teams, a whole bunch of, sort of, data-focused teams internally. I have a background in astronomy, so actually, previously, I was running a data archive on behalf of NASA for the Hubble space telescope in Baltimore. Before that, I was actually at GitHub. So, I seem to have liked working at GitHub. This is my second time in the company. I was back at GitHub in 2013 for about three years as well.Boaz: Awesome, Awesome. So, we want to dive into, you know, the data world. What do you guys do? So let us start with sort of, just tell us about your data stack and the kind of challenges it has intended to serve.
Kim: Okay. So, our data stack, at a storage layer, we use Azure Dialect storage, and then we use Azure Hadoop and distributors service or Insight. For KeyCloak, you have metastore for transform data. For ingesting data, we use Airflow. We have people using AML for the machine learning pipeline. For compute, we have Presto, Snap, Azure data explorer. And, for BI, we have some custom tools for the individual customer, like Product 360 that you will hear more about later; and we have a more general tool, like Looker and Power BI as well. So, that's a lot.
Boaz: Yeah.
Kim: We have different things going on.
Boaz: Maybe I should have asked what you don't have.
Eldad: S3, no S3 buckets.
Boaz: I think you're the first guests and we have done like, I think, six episodes or so, so far; all have been on AWS. I wonder, you know, if we can talk a little bit about, sort of, Azure. Were you always on Azure? Was there like it was a transition at some point? Was AWS on the table, if you could share some thought processes there.
Kim: Yeah, we used to be on AWS and we made them move to Archer recently. And during that move, we were not just moving from one cloud to another cloud, we were actually moving the kinds of services that we use as well. On AWS, we were operating hundreds of nodes on the Hadoop cluster and it cost us tons of money. When we moved to Archer, we actually used Archer Hadoop and distributors Insights, which manage all of that infrastructure for Hadoop for us and we pay for service rather than paying for the machine first, and so, it is a lot better in terms of how much we pay the bill for it. So, yeah.
Boaz: And was it a complete transition as in Azure only at this point or do you still keep both in, sort of, a multi-cloud strategy?
Kim: The whole data warehouse is fully operations at Azure, right now.
Arfon: That was a big lift for the team.
Eldad: Who was the person who deleted the S3 bucket, kind of when all the tests are passed and then like someone must have pressed the button, right? "drop bucket," which probably still runs.
Boaz: Terrifying moments, probably. How long did the transitional process take?
Kim: There is a lot of transition that carries out. So, we think we have a lot of things like we may take a quarter to transition one thing and then keep everything else the same. And then, every quarter, we move another product over. I was not involved in the beginning, but I was involved in moving Airflow at the very end. So, yeah, we took 3 months at least to move Airflow from operating on AWS to Azure.
Arfon: Almost it took over a year to move. I think it was a long time. It was a big lift.
Boaz: Are all the mental scars healed by now from that transition? You know, people typically dread these things, like it sounds tough. It typically is tough, but you have done it and it worked well, so kudos! Awesome! Let's talk about all the people involved with data. If you could tell us how the data teams are structured? You know, which software teams deal with data? Are they the data engineering teams? Or they are, you know, data science, and everything in between? We love to understand how that works at GitHub.
Arfon: Yeah. So we have, a fairly new structure actually. So, I am going to talk about what we have just reorged to and maybe about why we have made that change. So, we have a data platform team, which encompasses, sort of, a collection of data engineers; but coming the way, we think about that, they sort of run the common fabric of the data warehouse. So if you are a team that wants to provision storage or, you know, some kind of infrastructure for your service, then they sort of build that paved path for provisioning, compute, storage, that kind of thing, role-based access control, that kind of stuff. That team also owns this BI experience. So, the tools that you would use to interrogate the warehouse go around, and you are on self-service kind of questions as a member of staff. And then, we have a collection of what we call "verticals." So, these are teams that are, sort of, close to full stack in terms of the skills as analysts, data scientists, data engineers, and their managers; and they are focused typically on product areas. So, we have GitHub internally, the way that we build the product that sort of strategy around the service that we are building is broken into a few different verticals. And so we have, effectively, data teams aligned with those. And then, there are other data teams, that are not within this core unit, they are sort of a centralized data team for the company. There are other verticals that are outside. So, they are, sort of, more like satellite data teams, maybe in revenue, sales, that kind of thing. So, they are specifically serving their customers, who have sort of different types of questions. So, this is sort of, I think, an idea that has been pretty hot recently, as a sort of data mesh idea. So, building products for a particular set of customers, really owning that relationship and having sort of long-term relationship with those teams that you support. So, we have been doing this without the reorg for about a year now; and it is working pretty well. And so, we sort of fully embraced this model, actually, just this late last week, we have sort of finally kind of crossed the eyes.
Boaz: I mean, you did a lot of changes recently moving to Azure, reorging the data teams. I wonder what is next in terms of change.
Arfon: Wow. It is stability, I hope. So, you know, I think part of what we have been doing as kind of growing up, like reorging the structure to be ready for sort of next level of growth of the business. I think so there's a lot, you know, the demand for data is very high and I think we have sort of found it a little bit hard to support some areas of the business traditionally. And so, you know, especially with the centralized data team, it could be hard to prioritize across the whole business. So now, we are sort of saying, this is a team that is focused on this area of the business or the product; and if it needs more capacity, then it should go and get a headcount and fund it that way, so it sort of gives us as a data team a little bit more of a logical sort of scaling unit.
Boaz: From a headcount perspective, all the teams we now covered, all the people involved with data, how many people are we talking about?
Arfon: So, in terms of sort of data engineering, data science, I think, we are probably at about 35 or 40, not very big. I think compared to other companies that exclude all of the people who store git on file servers and that is all completely separate. So of course, there is a whole sort of data infrastructure team, a set of teams around, running large-scale distributed git, all the application servers, all of that stuff. And in fact, actually, system observability is also separate from that. So. If you think about that without those headcounts, where they are sort of, I think, it's about 35 maybe, maybe 40 now.
Boaz: So let's get back to the two of you. Now, that you have, actually, mapped out the structure of all the people involved with data, back to you guys. How do you fit in with that? You know, both of you have different sort of roles; Kim, you are an engineer, and Arfon you bring sort of a product hat. So, if you could elaborate on what you do within that.
Kim: Yeah, so for me, I have been on the data platform team from the beginning so this reorg doesn't actually affect me.
Boaz: Got it. Okay. Can you share one of, sort of the recent projects that were, sort of, more memorable or exciting for you? Beyond transitioning Airflow to Azure.
Kim: It is more of transitioning Airflow from one version to another version. So, Airflow is an open-source product. It is continually being developed on, and there are new features right now, and we do want to keep on top of all the bug fixes. So, the most recent project for me was just the mixture that we have, AirFlow 2.0 running.
Boaz: Within, you know, the data platform team, what is the distinction between what a software engineer would do and what a data engineer would do?
Kim: I think all of my team members are software engineers and we consider ourselves as data engineers. We solve any problem that our user have or come to us with from operating the platform for them, helping them integrate from one technology to another technology, or even build new features too. For example, Airflow, I have been building the backfill tools that open source do not have, to solve the needs that we do have.
Boaz: And Arfon. Let's get back to your responsibilities.
Arfon: I have sort of, a bit of, a weird product manager role, I would say. So, we have lots of customers for data internally.
Boaz: Arfon! you're not weird. You're super cool. Come on!
Arfon: You're very kind. It is a bit of a mixed bag, honestly. We do have some products that we serve customers with. Some product teams we have this service product called Product 360, which is a thing we built with software engineers and data engineers, and that is sort of a data product that teams use internally to understand common engagement metrics, acquisition, retention, and churn. That kind of thing around how people are using different parts of GitHub, the product? So, sometimes this is a quite traditional sort of product work where it is kind of figuring out what we should do, managing that backlog, understanding what that product should do? what the customer is? But then, there is also a much more sort of general kind of workaround with all these potential customers we could serve internally, where is the lowest hanging fruit? where is the highest return on investment? So, we have quite a lot of autonomy about how we spend our resources. SWe have got 15 data scientists. What should they work on? You know, I think that is a really good question. And so sometimes I spend my time developing ideas with other teams, sort of getting work to the point where it's ready for a data scientist to go invest some time, where you are sort of doing speculative work. We think there is some potential here. We should spend some time doing some R & D to mature an idea to then go to a product team and say, "Hey, look, we think we could, you know, use data to like, you know, supercharge our product in this way or something." And so, it is a mixed bag. It is pretty varied and I enjoy that.
Boaz: It is very interesting. How many pure product manager or product people are there that have a data-specific road?
Arfon: It is just me. So, we are hiring. We just hired a second person. I am delighted to say because there is a lot. One PM to 40 engineers is way off a good ratio from my perspective. So just add two.
Eldad: So now you have high availability at least.
Arfon: Yes! Yes!
Eldad: What about metadata, by the way? Like, what is the most amazing metadata you have been playing with? Like I am thinking GitHub. I am thinking it is best semi-structured. How does that turn into a Looker dashboard? Or if you can share on the metadata side.
Arfon: Yeah. We have a sort of collection of data streams that we put in the warehouse. We have a thing we called hydro, which is, I think just Kafka, you know, that they streamed a collection of services, write events there. This is the sort of paved path if you are a product team. For this particular part of the product, you want to instrument, measure some behavior in the product, then, you write an event in the product. There is a sort of pretty good abstraction as a developer, in Ruby for writing there or in Go, you know, there are mature client libraries that developers use. And, anything that is in hydro in this Kafka stream ends up in the warehouse automatically, and then, we have got, you know, query tools on top of that, Presto or something else. And then, we have a lot of bags that run and turn those into things that are more consumable. So, yeah, I think probably a good example of where we have to invest a lot. If you think about something like security scanning as a product, I have a vulnerability in my project or dependable, that kind of thing, where there is some dependency that I have in my project and I want to be alerted if, you know, there is some vulnerabilities. So, we care a lot about how successful that product is and whether people are responding to those alerts and whether they saw them and whether they did anything, whether they resolve the alert by pushing new code. And so actually, building a set of dashboards that the product team could use to make data-driven insights about how well the product is performing. There was a lot of data engineering work to design those tables, to collect the right data, and build a performance data model that they could query easily. Because there is a lot of dependable alerts and there is a lot of vulnerabilities that get exposed and people get alerted to, so there is actually a lot of work to make a single part of the product understandable for a product team. So, yeah, it probably will be a good example of how uniquely you get a problem. Yeah.
Boaz: What are sort of the key architectural lessons learned? I mean, you know, when you did the shift to Azure, probably huge architecture redo in a lot of places. Can you talk about Hadoop and how you change your approach there? What are the kinds of things that you do not mess with the old architecture and you are smarter about now?
Kim: One of the bad things that I do not mess is, operating ECG instances. I do not like having to check to see if they are dead or alive and they need to restart or how healthy they are. When we moved to Azure, we also adopted Kubernetes and Terraform. So, we use Terraform to deploy our infrastructure, we use Kubernetes to run our containers. So, it is a lot easier to bring up a new container or deployment. It is less time for a user. So yeah, those are the two big things that I am really happy that we are laying on.
Boaz: Which data volumes were we talking about at GitHub?
Kim: About 40 storage account and they range from a couple of megabytes to a couple of petabytes, and they are different data sets. So, it really depends on the topic or the kind of data that is in the storage that determines how big they are; but yeah, we have a wide range of the data sectors.
Arfon: One good example is the hydro topics. I think they are the most voluminous. It is about 4 billion events per day that get written to the warehouse. So, just to give you an idea. I think that is our biggest the hydro topic, pretty sure.
Boaz: What are some of the use cases that are more real time in nature that are running today?
Arfon: This is a satellite data team. So in the sense like this isn't us, but platform health would be a good example. So spam protection. Those teams use these hydro events, you know, so, very low latency sort of system events, and they are running, their own ML algorithms on the fly to detect spammers and quickly remove them. So, I think one of the things that I kind of think is pretty awesome about the GitHub experience is, it is really rare to see spam, and so that's amazing, so that team is incredible and they are just laser-focused on keeping spammers off the platform. So that is a real-time thing. I think also all the stuff that, sort of, other platforms teams do around availability and GitHub receives a lot of denial of service attacks still and stuff like that, but that is not our world, but I think those are some real-time stuff that are pretty important.
Boaz: It does sound like, you know, at GitHub, there is a culture around being data-driven. I mean, there are so many initiatives going around data, data teams are embedded within different services. Is there any standardization of how to approach becoming data-driven within, sort of, new services being launched or a new initiative or is it all very different across different teams with their own analysts and data people?
Arfon: I think this is part of the challenge of the work that we do. I think I am correct in saying that product operations as a concept is quite new at GitHub. This idea of what should you do when you launch a new product, right? Like there is a playbook, like sort of due diligence. I mean, I think we are good at making sure the docs are in a good state, but what sort of instrumentation should you do in the product to measure success? Do you have success metrics? All that kind of stuff I think actually, GitHub is getting much better at that, but that is not a historical strength, I think. So, I think, there are playbooks now. We built services; it is Product 360 service that we built. Really all you have to do as a product team is change a configuration file and then we effectively ingest that as part of the dag, when it runs at Airflow. We ask product teams to define engagement with their product, so if a customer is active in these areas of the product, then that counts as engagement. This counts as a contribution. This counts as acquisition, that kind of thing. And then if they do that, they have got all these sorts of free dashboards that are built every night. And so I think our general tactic is to try and encourage people to do the right thing by building them good tools that if they just follow the playbook, then they are going to get, you know, they get carrots. But more and more, I think the expectation is that a product manager can talk about their product, can talk about how many users they have, can make intelligent comments about, you know, users that are being particularly successful and those less so. And so I think, sort of the culture of being able to reason with data is much, much stronger at GitHub today than it was, let's say, five years ago. So I think you are being very nice when you are saying it sounds like we have a very strong data culture. I think we are growing a stronger data culture. I think traditionally observability has been amazing at GitHub, like the whole kind of ChatOps stuff and the ability to really manage a service from a chat channel was, you know, foundational in GitHubs' engineering culture. And that's a decade old. GitHub has been doing that forever.
Boaz: Let us do spend a couple of minutes on the observability. Even though it's a decade old. Tell us a little bit more about that.
Arfon: I don't know. I can tell you some. We don't own that, but I know we use services like Centrify, Splunk, that kind of thing. Those are services I know about. I know we are moving towards Open Telemetry for logs and metrics. That kind of thing.
Boaz: I actually, yeah, I caught that on your engineering blog.
Arfon: Yeah. There was a nice post about that recently. I think one thing I would say though, is that we have a divide in the way that we think about that data, which is not good, which is observability, is like an engineering problem and like, you know, metrics and instrumentation is a product thing and in some sense, those data, it doesn't make sense to think of those as separate data streams, you know, product managers can build a dashboard from logs if they wanted to, but they traditionally haven't.
Eldad: This is a great point actually. We see that a lot, like, engineering needs data to figure out how to build something and product needs data to figure out what to build; and when they collide, it usually sparks, you know, in there because each side just looks on its own need and it is really hard to understand the other side. So engineering will say, we cannot wait for a full-fledged data warehouse and we cannot wait for this data pipeline and we need it now, now, now. And the product will say, Well! We have figured out most of the stuff on how to do it. Let us figure out what to do. I think we are entering this kind of era where, as you said, those two parts will eventually converge into one big stream where everyone can take their part. But it is interesting to see that. We see it all the time. Even internally at Firebolt, like, there is a race to space around engineering versus product who can build a better system.
Boaz: We managed a big list of things that usually go through the most horrible grunt work. So what is the worst grunt work in your day-to-day job that you wish could magically disappear?
Kim: That would be Python dependency again, not waking up in the morning and figuring out something broke because of a transient, transient, transient dependency that we did not pin devotion on.
Eldad: Say no to Python, just kidding.
Kim: I have been using Python for a while and it has been the single thing that I have been complaining about Python over and over again.
Boaz: It is a love-hate relationship, I guess.
Kim: Yeah.
Boaz: Tell us about, you know, a great, win from a sort of the last year or something that you are proud of or had a good experience with.
Kim: I would say the backfill tools for Airflow. Airflow is very great with managing a workflow, but they don't backfill at an operation that people do day-to-day, and at GitHub, we have so many people writing deck, writing workflow, shipping day data and the data can change like schema change or the metric needs to be updated, the calculation needs to be updated. They are more factor that they learn about and once more accurate model, things like that, that we try backfill where you are running the whole workflow over a long period of time, could be years of data in. Therefore, does not have a good story around that. So we have to build in our own tooling to let user to use backfill or to run backfill on Airflow easily. Reaching out to the open-source projects and hopefully that in the new versions of Airflow, I would engage more in that and help share our learning and share our use-case and hopefully share the new backfill tools in Airflow.
Boaz: Awesome, super interesting, but stop bragging. Now tell us about a failure. What didn't work so well? what can you warn others about or sort of a lesson learned?
Arfon: I was going to say one thing. I will give you success and then I will give you a quick failure because I do want to tell you about one thing. We did miss 4 billion events per day stream that we have. One thing I think we have done that is really good is we have managed to use this common data source. So that 4 billion per day event stream is actually a log of every request to the Rails application that GitHub like monolith, so it is effectively the Rails log, so every request, it was about 4 and a bit billion per day, and we have managed to sort of figure out how to drive the majority of our company core metrics. So things like your monthly engaged users, your top-level business things that attract by the leadership team and Microsoft and using this event stream and the majority of all the individual product areas. So they use a sort of common data source, which felt really, really good to have this common way of approaching metric definitions. That felt like it was very fragmented beforehand. And so, that felt particularly good. It was very satisfying to sort of be able to, sort of, see them rolling up to some sort of total numbers so that I would say that was a big success in terms of trying to take a sort of rational approach to thinking about metrics across the business.
Boaz: Any special tips, I mean, as the experts in everything Git or is there any unique way in which you approach all the data projects from a Git's perspective, something all the regular Git users can learn from?
Arfon: I have one suggestion. It is not really about Git. It is more about the process, which is I think, I appreciate the ADR format. I do not know if you know, that is an Architectural Decision Record. Is that what it stands for?
Kim: Right.
Arfon: Yeah. So one of the things I do enjoy is that GitHub uses the product extensively still. So the idea is you can just go browse any team, what they are working on. But this folder with all the ADRs and now, which is really lovely, so if you are trying to understand, like, why is the service built this way? Like what decision-making process did we go through? There is usually an ADR. If it was a big decision for a team, there is a thing you can go and read and its reasons and what else did we consider? I like that a lot.
Boaz: Who owns the ADR? Like who in engineering?
Arfon: Usually they are sort of principal engineers responsible for building, you know, the decision-maker, but the fact it is in the repository, means it has been through an extensive review. You can go and look at the pull request associated with that review, you know!
Boaz: Awesome!.
Eldad: Is that somehow related to the change in how kind of, you know, how we are working together in a distributed way where time zones are different, people are moving across places. So getting to those basic disciplines suddenly becomes necessary because you just can't hop into a room and just wrap it up quickly. I think, you know, it's huge and small companies, if you want to work with a distributed workforce with engineers, especially, those advisors are gold.
Boaz: I think we should definitely look into it ourselves. I mean, we've been scaling and starting with remote culture and just, you know, collaborative decision-making in a remote way, it is not as easy as it sounds, so super! Yeah! Thanks. Okay! Now. It's, the advice corner, anything sort of inspirational or who to follow, what can you sort of give back to the community in terms of what to look out for, who to follow things like that? And any less famous words from the data people at GitHub.
Arfon: I guess a company that is well in set technologies that really inspires me is the Jupyter project. So this is Jupyter notebooks, you know, one of the things that they do. I think that this is a project that came out of the IPython originally, this was scientific Python ecosystem. I just think they have built a really compelling set of technologies, cross-platform, cross-language, sort of the de facto way to sort of work as a data scientist today, and the way that the project runs is really incredible, as well as they've got really, really sort of enlightened approach to governance and participation and stuff. So yeah, I think Jupyter is, you know, sort of an exemplar for how I think really good open-source projects can be run. And of course, you know, that sort of toolchain is adopted by all the cloud vendors, you know, SageMaker, Azure notebooks, there's Google collaboratory that are all using the same thing, right? That just seems amazing to me. And I think, yeah, that's a great project. I think they're doing great work.
Boaz: Okay. Now for a quick blitz round:
Eldad: Open source or commercial?
Kim: Open source or commercial? Depends. You have me, I'm a little bit skeptical on either one in terms of whether or not I have used it yet. So I want to get my hand on it before I recommend.
Boaz: AWS or Azure.
Kim: Azure.
Arfon: Azure.
Boaz: Work from home or from the office?
Kim: Work from home.
Arfon: A bit of both.
Boaz: Yeah. Okay. That has been awesome. Arfon and Kim, thank you so much for joining us. It has been super interesting and we will see you around.
Eldad: Thank You.
Arfon: That is cool. Thanks for the time.
Kim: Thank you.
Boaz: Our pleasure.
# Eliminating Redundant Joins in Firebolt for Faster SQL (/blog/eliminating-redundant-joins-in-firebolt-for-faster-sql)
## Tl;dr [#tldr]
In this blog post, we provide an explanation of how our query optimizer is able to detect and remove redundant joins. Eliminating redundant joins is critical for real-world SQL queries, especially those generated by query generators which often contain a significant amount of redundancy. It is also essential for Firebolt's ability to optimize complex correlated subqueries. We'll start by explaining what we call a redundant join operation, and then we'll introduce the data structures we use for detecting them in a complex query plan.
## Introduction [#introduction]
In Firebolt, we are obsessed with removing redundancy from complex SQL queries. We recently presented a [paper at BTW 2025](https://www.firebolt.io/content/optimizing-correlated-aggregate-subqueries-in-firebolt) about how we are able to optimize correlated aggregated subqueries in a way no other system can. In the paper, we explain our approach for optimizing the following query pattern with aggregated correlated subqueries in the projection of the query scan the same data:
```sql
SELECT c.*,
(SELECT min(price) FROM orders WHERE status='O' AND c.key=ckey) "min",
(SELECT max(price) FROM orders WHERE status='O' AND c.key=ckey) "max"
FROM customers c;
```
Our optimizer is able to rewrite the query in such a way that the orders table is only scanned once. That is achieved through a combination of query rewrites, all of which are explained in the paper. One such rewrite, and the focus of this blog, is our ability to remove redundant join operations. The usefulness of this query transformation is not limited to the query pattern presented above, but also in other scenarios as well, for example, the result of inlining views accessing the same set of tables may lead to this type of redundancy in the queries.
Eliminating redundant joins is crucial for several reasons in real-world SQL query optimization. Firstly, we have observed that many complex SQL queries, especially those generated by automatic tools, often contain a significant amount of redundancy. Identifying and removing redundant joins in such queries lead to faster query execution and improved overall system efficiency. Secondly, redundant join removal is an essential step in Firebolt's de-correlation process. Our ability to optimize complex correlated aggregate subqueries, as detailed in our BTW 2025 paper, heavily relies on simplifying the query plan by eliminating redundant operations. This not only keeps the final execution plan concise and manageable, but also enables further optimizations that would otherwise be impossible, ensuring highly efficient query processing.
## Redundant inner joins [#redundant-inner-joins]
A redundant join is a join that could be removed from the query plan without affecting the query results. Eliminating redundant joins reduces the amount of data the engine needs to process, which leads to faster query execution. In other words, you get the same correct results with less computational effort – improving overall performance. Additionally, removing redundant operations can simplify the query plan in ways that enable further optimizations that wouldn't have been possible otherwise.
Let's consider the following example:
```sql
-- joining t with itself
SELECT x.*, y.* FROM t AS x INNER JOIN t AS y ON x.a = y.a;
```
Any non-null value of x.a is guaranteed to have at least one matching partner from y, since x.a and y.a have the same provenance, t.a. If a was a unique key in t, then the following query would produce the same results without performing any join:
```sql
-- self join of t is removed
SELECT x.*, x.* FROM t AS x WHERE x.a IS NOT NULL;
```
See how the entire join has been replaced with just a filter removing the null values from a, as the former join condition used to do. Also, notice that all the columns selected from y have been replaced with the corresponding columns from x. In our terminology, we call y the redundant join operand, as it is the join operand being removed, and we call x the replacing operand, since we are replacing all the appearances of columns from y with the corresponding columns from x.
This self-join on a unique key is the most basic example of a removable join. Let's consider now the following query where column a is no longer required to be a unique key of t:
```sql
SELECT x.*,
y.a1
FROM t AS x
INNER JOIN (SELECT DISTINCT a AS a1
FROM t) AS y
ON x.a = y.a1;
```
Similarly, y can be removed from the query because every tuple from x with a non-null value in column a is guaranteed to have a unique matching partner from y. x and y are equi-joined on columns with the same provenance and the joining columns form a unique key in the redundant operand y. The equivalent query without the redundant join is:
```sql
SELECT x.*, x.a AS a1 FROM t AS x WHERE x.a IS NOT NULL;
```
The join needs to be on columns that form a unique key in the redundant join operand. Otherwise, the output cardinality of the rewritten plan is not guaranteed to be the same as with the original plan with the join.
Another required condition is that the replacing join operand should be able to provide all the required columns from the redundant join operand. Consider the following example where y cannot be replaced with x even though they are joined on a column with the same provenance that forms a unique key on y, because x cannot provide the value for y.max after removing the join.
```sql
SELECT x.*,
y.*
FROM t AS x
INNER JOIN (SELECT a AS a1,
Max(b) AS max
FROM t
GROUP BY a) AS y
ON x.a = y.a1;
```
Finally, the common source between the replacing and the redundant join operands must be unfiltered in the redundant side. For example, the replacement in the following case wouldn't be valid, even though the first two requirements are satisfied, because the rows from t1 are filtered in y.
```sql
SELECT x.*
FROM t AS x
INNER JOIN (SELECT DISTINCT a AS a1
FROM t
WHERE b > 10) AS y
ON x.a = y.a1;
```
The query above is effectively a semi-join filtering the rows from x where there is at least one row in t1 with the same value for with a value for b greater than 10. Rewriting the filter to filter on x.b, as in the following query, would not be valid unless b is a functional dependency of a, where the value of column b is fully determined by the value of column a.
```sql
-- if b is functionally dependent on a
SELECT x.*, x.* FROM t AS x WHERE x.a IS NOT NULL AND x.b > 10;
```
## Redundant outer joins [#redundant-outer-joins]
Query optimizer people know that the real fun always starts with outer joins. There are two types of redundant outer joins we can identify and remove. The first type is shown by the following example where a is again required to be a unique key of t:
```sql
SELECT x.*, y.* FROM t AS x LEFT JOIN t AS y ON x.a = y.a;
```
This query could be rewritten as the following:
```sql
-- if a is a unique key of t, remove the left join
SELECT x.*,
CASE WHEN x.a IS NOT NULL THEN a ELSE NULL END,
CASE WHEN x.a IS NOT NULL THEN b ELSE NULL END,
...
FROM t AS x;
```
Note that the filter "x.a IS NOT NULL" is no longer added since left outer joins must always preserve the rows from its left input. Instead, we need to replace all columns from y with columns from x, but making them NULL when column a is NULL, using CASE WHEN operator, since that is the only case where a row from the left-hand side won't find any matching partner in the right-hand side of the left join.
For an equi-join whose join condition does not reject nulls, for example, a join using IS NOT DISTINCT FROM comparison, there is no need to wrap the columns from x within these CASE WHEN expressions when replacing the columns from y. Implementation-wise though, we always add this wrapping CASE WHEN expressions, use the original join condition as their condition, and let the optimizer simplify them. This way, the code has less special paths to maintain. The resulting optimization steps for these two queries are presented below.
```sql
-- example 1
SELECT x.*, y.* FROM t AS x LEFT JOIN t AS y ON x.a = y.a;
-> (redundant join removal)
SELECT x.*,
CASE WHEN x.a = x.a THEN x.a ELSE NULL END,
CASE WHEN x.a = x.a THEN x.b ELSE NULL END,
...
FROM t AS x;
-> (expression reduction)
SELECT x.*,
CASE WHEN x.a IS NOT NULL THEN x.a ELSE NULL END,
CASE WHEN x.a IS NOT NULL THEN x.b ELSE NULL END,
...
FROM t AS x;
-> (expression reduction, if a is not nullable)
SELECT x.*,
x.a,
x.b
...
FROM t AS x;
```
```sql
-- example 2
SELECT x.*, y.* FROM t AS x LEFT JOIN t AS y ON x.a IS NOT DISTINCT FROM y.a;
-> (redundant join removal)
SELECT x.*,
CASE WHEN x.a IS NOT DISTINCT FROM x.a THEN x.a ELSE NULL END,
CASE WHEN x.a IS NOT DISTINCT FROM x.a THEN x.b ELSE NULL END,
...
FROM t AS x;
-> (expression reduction)
SELECT x.*,
x.a,
x.b
...
FROM t AS x;
```
What if there are more conditions on the join? As long as the unique key from the redundant operand is equi-joined with columns from the replacing operand through columns with the same provenance, we can simply use these wrapping CASE expressions, using the entire join condition as their condition. The following example shows this.
```sql
SELECT x.*, y.* FROM t AS x LEFT JOIN t AS y ON x.a = y.a AND x.b > 10;
-> (redundant join removal)
SELECT x.*,
CASE WHEN x.a = x.a AND x.b > 10 THEN x.a ELSE NULL END,
CASE WHEN x.a = x.a AND x.b > 10 THEN x.b ELSE NULL END,
...
FROM t AS x;
-> (expression reduction)
SELECT x.*,
CASE WHEN x.a IS NOT NULL AND x.b > 10 THEN x.a ELSE NULL END,
CASE WHEN x.a IS NOT NULL AND x.b > 10 THEN x.b ELSE NULL END,
...
FROM t AS x;
-> (expression reduction, if a is not nullable)
SELECT x.*,
CASE x.b > 10 THEN x.a ELSE NULL END,
CASE x.b > 10 THEN x.b ELSE NULL END,
...
FROM t AS x;
```
Let's now consider the second type of redundant outer join shown in the query below, which is the one that we need to cover for optimizing the queries in our BTW 2025 paper.
```sql
SELECT t2.*,
x.*,
y.*
FROM t2
LEFT JOIN t1 AS x
ON t2.i = x.a
LEFT JOIN t1 AS y
ON t2.i = y.a;
```
In this case, we can remove y and replace it with x (or vice versa) as long as column a forms a unique key of t1 even though x and y are not directly joined but indirectly through t2. The resulting rewritten query is:
```sql
-- remove redundant join operand y
SELECT t2.*, x.*,
CASE t2.i = x.a THEN x.a ELSE NULL END,
CASE t2.i = x.a THEN x.b ELSE NULL END,
...
FROM t2 LEFT JOIN t1 AS x ON t2.i = x.a
```
Which can be further simplified to:
```sql
SELECT t2.*, x.*,
x.a,
x.b,
...
FROM t2 LEFT JOIN t1 AS x ON t2.i = x.a;
```
Since the columns from x are already guaranteed to be NULL when the predicate doesn't evaluate to true.
In the example above, both the replacing and the redundant join operands are in the non-preserving side of an outer join. In order for the redundant join removal to be valid, the outer join predicates of the replacing operand must be a subset of the outer join predicates of the redundant operand. In the following example, the join with y can still be removed from the query, replacing its referenced columns with columns from x, as long as column a forms a unique key in t1. However, x cannot be replaced with y due to this new requirement.
```sql
-- cannot remove the join to x due to the extra join condition on y
SELECT t2.*,
x.*,
y.*
FROM t2
LEFT JOIN t1 AS x
ON t2.i = x.a
LEFT JOIN t1 AS y
ON t2.i = y.a
AND t2.j > 10; -- extra join condition
```
Again, the predicates of the removed operand become the condition of the wrapping CASE expressions:
```sql
SELECT t2.*, x.*,
CASE t2.i = x.a AND t2.j > 10 THEN x.a ELSE NULL END,
CASE t2.i = x.a AND t2.j > 10 THEN x.b ELSE NULL END,
...
FROM t2 LEFT JOIN t1 AS x ON t2.i = x.a
```
In order for these rewrites to be valid, we've mentioned earlier that there needs to be a join directly or indirectly joining columns with the same provenance from both sides covering a unique key from the redundant join operand. And that alone is not sufficient. We need to make sure the common source between the redundant and the replacing operands must remain unfiltered in the redundant side. Consider the following example:
```sql
SELECT *
FROM t3
LEFT JOIN (t1
INNER JOIN t2
ON t1.a = t2.i) AS x
ON t3.m = t1.a
LEFT JOIN t1 AS y
ON t3.m = y.a
```
In this case, the inner join between t1 and t2 is the non-preserving side of the outer join with t3. That inner join may filter rows from t1. As a result, we cannot remove any outer join as a redundant join.
## Other types of joins [#other-types-of-joins]
Performing redundant join removal for semi joins is trivial since they are just a special type of inner joins. There is no extra consideration needed for them. In fact, for any equi-semi-join, the columns from the semi-side involved in the join conditions always form a unique key. In the following example, removing the semi-join is possible even if a is not a unique key of t, thanks to the semi-join semantics.
```sql
SELECT * FROM t AS x WHERE x.a IN (SELECT y.a FROM t AS y);
->
SELECT * FROM t AS x WHERE x.a IS NOT NULL;
```
On the other hand, anti joins are not simple. If fun starts with outer joins, pain starts with anti joins. One reason is that there are effectively two types of anti-joins with different semantics on nulls. The following example shows how the two most basic types of redundant anti-joins could be rewritten.
```sql
-- NOT IN
SELECT * FROM t AS x WHERE x.a NOT IN (SELECT y.a FROM t AS y);
->
SELECT * FROM t AS x WHERE FALSE;
-- NOT EXISTS
SELECT * FROM t AS x WHERE NOT EXISTS (SELECT * FROM t AS y WHERE x.y = y.a)
->
SELECT * FROM t AS x WHERE x.a IS NULL;
```
Although we all have rewritten some IN predicate to EXISTS or vice versa in SQL queries, the negated version the two predicates have a subtle difference in null semantic, which is often overlooked.
A simple example is shown below. x and y are indirectly joined. It is obvious that one of the joins is redundant and can be removed.
```sql
SELECT *
FROM t2
WHERE t2.i NOT IN (SELECT x.a FROM t1 AS x)
AND t2.i NOT IN (SELECT y.a FROM t1 AS y); -- redundant (either x or y)
->
SELECT *
FROM t2
WHERE t2.i NOT IN (SELECT x.a FROM t1 AS x);
```
In the following example, y can still be removed since it doesn't filter out any row from t2 that x isn't already filtering.
```sql
SELECT *
FROM t2
WHERE t2.i NOT IN (SELECT x.a FROM t1 AS x) -- cannot remove
AND t2.i NOT IN (SELECT y.a FROM t1 AS y WHERE y.b > 10); -- still redundant
->
SELECT *
FROM t2
WHERE t2.i NOT IN (SELECT x.a FROM t1 AS x);
```
For NOT IN anti-joins, the third condition is a bit different as it is the replacing operand, instead of the redundant operand, where the common source of the data must remain unfiltered.
## Implementation details [#implementation-details]
Our redundant join removal implementation searches for redundant join operands by analyzing the join graph. Within a join graph, the process of removing a redundant join corresponds to removing a vertex (which represents a plan) and an associated edge (which represents the join) from the graph.
Below is a trace from our optimizer after removing the redundant join from one of the examples we saw before.
```sql
SELECT x.*,
y.a1
FROM t AS x
INNER JOIN (SELECT DISTINCT a AS a1
FROM t1) AS y
ON x.a = y.a1;
Sub-plan before rule: RedundantJoinRemovalRule
[0] [Projection] t.a, t.b, a1
| [JoinGraph]
| - Projection: class_0, vertex_1.t.b, class_0
| - Vertices:
| Vertex 0 -> Node 3, provided classes: 0
| Vertex 1 -> Node 2, provided classes: 0
| - Edges:
| type: Inner, predicate: [equals(class_0, class_0)]
| - Equivalence classes:
| Class 0: vertex_0.a1, vertex_1.t.a
\_[1] [Join] Mode: Inner [equals(t.a, a1)]
\_[2] [StoredTable] Name: "t"
\_[3] [Aggregate] GroupBy: [a1: t.a] Aggregates: []
| [Unique Keys]: [a1]
\_[4] [StoredTable] Name: "t"
Sub-plan after rule: RedundantJoinRemovalRule
[6] [Projection] t.a, t.b, a1: t.a
\_[7] [Filter] equals(t.a, t.a)
\_Recurring Node --> [2]
```
Our plan representation uses a DAG representation with binary join operators. However, every node that represents the root node of a join tree is annotated on demand with a simple join graph representing the join graph under it. In the example above, the join graph contains two vertices, which correspond to nodes 2 and 3 in the plan, and one edge, joining the two vertices.
This join graph representation is also used for other optimizations, such as join reordering or aggregation push down.
Equality predicates among expressions from different vertices are represented as equivalence classes. This way we can represent the relation among several vertices with a single edge. The following snippet is the result of adding an extra join operand to the previous example, which is joined to the other two through the same equivalence class, leading to a single edge.
```sql
SELECT x.*,
y.a1
FROM t AS x
INNER JOIN (SELECT DISTINCT a AS a1
FROM t) AS y
ON x.a = y.a1
INNER JOIN t2
ON t2.i = x.a;
Sub-plan before rule: RedundantJoinRemovalRule
[0] [Projection] t.a, t.b, a1
| [JoinGraph]
| - Projection: class_0, vertex_0.t.b, class_0
| - Vertices:
| Vertex 0 -> Node 3, provided classes: 0
| Vertex 1 -> Node 4, provided classes: 0
| Vertex 2 -> Node 7, provided classes: 0
| - Edges:
| type: Inner, predicate: [equals(class_0, class_0)]
| - Equivalence classes:
| Class 0: vertex_0.t.a, vertex_1.a1, vertex_2.t2.i
\_[1] [Join] Mode: Inner [equals(t2.i, t.a)]
\_[2] [Join] Mode: Inner [equals(t.a, a1)]
| \_[3] [StoredTable] Name: "t"
| \_[4] [Aggregate] GroupBy: [a1] Aggregates: []
| | [Unique Keys]: [a1]
| \_[5] [Projection] a1: t.a
| \_[6] [StoredTable] Name: "t"
\_[7] [StoredTable] Name: "t2"
Sub-plan after rule: RedundantJoinRemovalRule
[8] [Projection] t.a, t.b, a1: t.a
| [JoinGraph]
| - Projection: class_0, vertex_0.t.b, class_0
| - Vertices:
| Vertex 0 -> Node 10, provided classes: 0
| Vertex 1 -> Node 11, provided classes: 0
| - Edges:
| type: Inner, predicate: [equals(class_0, class_0)]
| - Equivalence classes:
| Class 0: vertex_0.t.a, vertex_1.t2.i
\_[9] [Join] Mode: Inner [equals(t.a, t2.i)]
\_[10] [Projection] t.a, t.b
| \_Recurring Node --> [3]
\_[11] [Projection] t2.i
\_Recurring Node --> [7]
```
If the extra vertex is joined to only one the other two vertices, i.e. through a different column, our simple join graph will contain two inner join edges corresponding to the two join equivalence classes, as shown in the example below.
```sql
SELECT x.*,
y.a1
FROM t AS x
INNER JOIN (SELECT DISTINCT a AS a1
FROM t) AS y
ON x.a = y.a1
INNER JOIN t2
ON t2.i = x.b;
Sub-plan before rule: RedundantJoinRemovalRule
[0] [Projection] t.a, t.b, a1
| [JoinGraph]
| - Projection: class_0, class_1, class_0
| - Vertices:
| Vertex 0 -> Node 3, provided classes: 0, 1
| Vertex 1 -> Node 4, provided classes: 0
| Vertex 2 -> Node 7, provided classes: 1
| - Edges:
| type: Inner, predicate: [equals(class_0, class_0)]
| type: Inner, predicate: [equals(class_1, class_1)]
| - Equivalence classes:
| Class 0: vertex_0.t.a, vertex_1.a1
| Class 1: vertex_0.t.b, vertex_2.t2.i
\_[1] [Join] Mode: Inner [equals(t2.i, t.b)]
\_[2] [Join] Mode: Inner [equals(t.a, a1)]
| \_[3] [StoredTable] Name: "t"
| \_[4] [Aggregate] GroupBy: [a1] Aggregates: []
| | [Unique Keys]: [a1]
| \_[5] [Projection] a1: t.a
| \_[6] [StoredTable] Name: "t"
\_[7] [StoredTable] Name: "t2"
Sub-plan after rule: RedundantJoinRemovalRule
[8] [Projection] t.a, t.b, a1: t.a
| [JoinGraph]
| - Projection: vertex_0.t.a, class_0, vertex_0.t.a
| - Vertices:
| Vertex 0 -> Node 10, provided classes: 0
| Vertex 1 -> Node 12, provided classes: 0
| - Edges:
| type: Inner, predicate: [equals(class_0, class_0)]
| - Equivalence classes:
| Class 0: vertex_0.t.b, vertex_1.t2.i
\_[9] [Join] Mode: Inner [equals(t.b, t2.i)]
\_[10] [Projection] t.a, t.b
| \_[11] [Filter] equals(t.a, t.a)
| \_Recurring Node --> [3]
\_[12] [Projection] t2.i
\_Recurring Node --> [7]
```
As explained before, in order for this optimization to happen we need to check that the redundant vertex is joined to the replacing vertex through one of its unique keys. As shown in the examples above, we keep track of the unique keys of each node in the plan as computed information that gets attached to the query plan.
The last remaining bit is how to check whether two columns joined in the join graph have the same source. For that, we use another computed property that, for every node, keeps track of the steps its projected columns went through. The following snippet shows our first example again with this extra annotation.
```sql
SELECT x.*,
y.a1
FROM t AS x
INNER JOIN (SELECT DISTINCT a AS a1
FROM t) AS y
ON x.a = y.a1;
Sub-plan before rule: RedundantJoinRemovalRule
[0] [Projection] t.a, t.b, a1
| [JoinGraph]
| - Projection: class_0, vertex_1.t.b, class_0
| - Vertices:
| Vertex 0 -> Node 3, provided classes: 0
| Vertex 1 -> Node 2, provided classes: 0
| - Edges:
| type: Inner, predicate: [equals(class_0, class_0)]
| - Equivalence classes:
| Class 0: vertex_0.a1, vertex_1.t.a
\_[1] [Join] Mode: Inner [equals(t.a, a1)]
\_[2] [StoredTable] Name: "t"
| [Column Provenance]:
| - source: [2]
| column expressions: t.a, t.b
| inverse path:
| - source: t1
| column expressions: t.a, t.b
| inverse path:
\_[3] [Aggregate] GroupBy: [a1] Aggregates: []
| [Column Provenance]:
| - source: [3]
| column expressions: a1
| inverse path:
| - source: [4]
| column expressions: a1
| inverse path: 0
| - source: [5]
| column expressions: t.a
| inverse path: 0, 0
| - source: t
| column expressions: t.a
| inverse path: 0, 0
| [Unique Keys]: [0]
\_[4] [Projection] a1: t.a
\_[5] [StoredTable] Name: "t"
```
By using this property, we can quickly check that vertex\_0.a1 and vertex\_1.t.a have a common source, which is the t.a column.
This column provenance property is effectively a flattened version of the subtree under each node, which makes it a memory intensive property which is also costly to compute. However, it is quite useful for several other optimizations.
# Engines: Online Scaling and Upgrades (/blog/engines-online-scaling-and-upgrades)
Firebolt is a next-gen data warehouse platform designed for data-intensive analytics. It provides high efficiency and low latency, allowing data engineers to deliver scalable analytics faster, more cost-effectively, and with greater simplicity. In Firebolt, an Engine is the compute resource that customers will use to ingest data into Firebolt and to run queries on the ingested data. An Engine comprises one or more clusters, where each cluster is a collection of nodes of a certain type that provides a certain amount of CPU, RAM, and storage. Among the critical capabilities that engines provide to meet the needs of modern analytic workloads are: **1/** Multi-dimensional Elasticity, which allows users to scale their engines across multiple dimensions. **2/** Granular Scaling allows users to incrementally add (or remove) nodes to their engines, and **3/** Seamless Online Upgrades, enabling users to get the latest security updates and performance enhancements without interrupting their workloads. This blog post will discuss how Firebolt implemented zero-downtime upgrades under the hood.
## Firebolt Architecture [#firebolt-architecture]
Firebolt implements a "disaggregated *shared-everything architecture*" with "compute and storage separation." This architecture allows multiple compute resources running different workloads to be fully isolated from each other while sharing the same data. The visual below demonstrates the logical architecture of Firebolt.
**The key components of the architecture are:**
* Firebolt Gateway - Responsible for routing customer requests to the correct Engine.
* Engines Compute - Clusters of machines running in Kubernetes and service customer queries.
* Object Storage - Stores customer data
* Engines Control Plane - Manages the lifecycle of engines and is responsible for provisioning, de-provisioning, scaling, and upgrading engines.
Separating compute from storage allows the compute and storage layers to scale independently, providing customers with flexibility in scaling their engines to achieve optimal price/performance for their workloads. In designing the architecture for engines, we made important design choices around the following:
* **Fast Infrastructure Provisioning**: Enable customers to create and scale their engines quickly.
* **Managing Compute:** Running engines on Kubernetes vs. EC2 VMs directly
* **Multi-cluster Management:** Whether to use Service Mesh for managing multiple EKS clusters
### Fast Infrastructure Provisioning [#fast-infrastructure-provisioning]
To enable customers to quickly create and scale their existing engines, Firebolt pre-provisions a fleet of compute nodes called Warm Pools. When a customer requests a new engine or wants to scale their existing engine, Firebolt leverages the Warm Pool to satisfy these requests rather than going to the underlying cloud provider, thus providing fast engine operations.
### Managing Compute [#managing-compute]
To efficiently manage the compute resources needed for engines, we decided to use AWS Elastic Kubernetes Service (EKS) and AWS Karpenter, an open-source high-performance autoscaler for Kubernetes. The built-in scaling and scheduling capabilities provided by Kubernetes and the autoscaling capabilities of Karpenter gave us a great degree of flexibility in how we dynamically manage our compute infrastructure at scale without the need to maintain raw VM fleets.
### Multi-cluster Management [#multi-cluster-management]
As noted earlier, Firebolt is designed to meet the performance demands of today's low-latency, high-concurrency workloads. To meet these performance requirements, which can be in the tens of milliseconds, we needed a way to support multiple EKS clusters communicating with each other in a fast and secure manner. To manage these multiple EKS clusters, we used sidecar-less multi-cluster mesh to enable direct pod-to-pod [OSI L4](https://en.wikipedia.org/wiki/OSI_model#Definitions) communication between engine nodes without introducing extra hops in network communication.
### Engines Control Plane [#engines-control-plane]
Engines Control Plane is a set of micro-services responsible for Engine lifecycle management. The Control plane is responsible for:
* Provisioning, scaling, and upgrading engines
* Providing engine routing information
* Providing consumption & auditing information
The control plane provides a reliable workflow to manage the lifecycle of engines, interacting with Kubernetes clusters in order to provision and deprovision actual hardware resources. We use [Temporal](https://temporal.io) as a backend for workflow orchestration to satisfy our requirements:
* Provide reliable at-least-once execution for every workflow step.
* Provide capabilities to serialize operations on the objects of interest.
### Engine Architecture [#engine-architecture]
When designing Engines, one of our key goals was to ensure we provide the latest software upgrades without any downtime for our customers. We want to deliver these upgrades to our customers at a regular cadence without requiring them to stop their engines. To deliver these online upgrades without interrupting customer workloads, we considered the two options below:
1. Make changes to the engine nodes in-place while they are running (or)
2. Create an entirely new set of nodes that have the latest upgrades
Although making changes in-place to an engine while it is still running helps deliver the upgrades without customers having to stop their engines, any issues arising from the upgrades could impact the performance of the running workloads. Hence, we decided to pursue the second option above, for which we leveraged engine clusters. As noted earlier, an engine cluster is a collection of nodes. We designed clusters to be immutable objects to avoid disrupting customer workloads during the online upgrades. Making clusters immutable allowed us to guarantee that we won't push configuration changes to already running clusters, which could potentially introduce unexpected and undesirable changes to customer workloads. Before we look at how Firebolt uses immutable clusters to deliver online upgrades seamlessly, let's understand how dynamic engine scaling works in Firebolt. The upgrade process is built on top of dynamic scaling.
### Request Routing [#request-routing]
Firebolt Gateway fleet is a multi-tenant service responsible for routing requests to the correct engine and load balancing across engine clusters. Each engine cluster and each node within a cluster are individually routable. To support that, the Gateway fleet maintains an up-to-date state for each engine cluster, including:
* Cluster state (stopped, starting, running, or draining)
* Routing information for each node of the cluster
* Cluster routing strategy
Once a request lands on one of the Gateway instances, the Gateway service performs authentication engine discovery and then routes the request to the corresponding engine. Each request carries information about the engine it is targeting. At a high level, the flow for the request to reach the engine node is as follows:
1. The Gateway node receives the request
2. Gateway performs authentication
3. Discover requested engine state and routing information
4. If the engine is stopped and auto-start is enabled, start the engine
5. Select the running engine cluster
6. Select engine node within a cluster
7. Route request to selected engine node
The logical engine layout as Gateway fleet sees it is represented below:
The gateway fleet is crucial for load distribution across engine clusters and unlocks online scaling capabilities. Since the Gateway fleet is on the critical query execution path, having the minimum possible request routing overhead times is paramount.
Our goal was to have sub-1ms Gateway latency overhead at p50.
To achieve that, the Gateway fleet implements the:
* In-memory engine routing information on each of the Gateway instances.
* Subscribes to notifications event stream delivered from Engines control plane for any engine cluster state changes. This increases cache hit rates by proactively populating the in-memory cache with up-to-date engine states.
In the unlikely case of a cache miss, Gateway will explicitly fetch fresh Engine routing information from the Engine's control plane via a synchronous request.
## Online Scaling [#online-scaling]
The Control Plane for Engines drives the process of online scaling of any engine.
The process to scale an engine:
* Create a new engine cluster and start it
* Switch traffic over to a new cluster
* Drain the old cluster and stop it
* Delete old cluster
To ensure the scaling process does not cause downtime, the Gateway fleet and Engines control plane work together to avoid traffic disruption. Gateway fleet receives up-to-date state from the Engines control plane regarding the state of each engine cluster. Using this information, Gateway selects the appropriate cluster for sending traffic and avoids disrupting the queries running on the old clusters. The Engine's control plane tracks whether an engine cluster has any active queries before shutting it down during the drain phase.
We'll demonstrate the scaling process in more detail by scaling from a 2-node engine cluster to a 3-node engine cluster. The scaling process along other dimensions (changing the node Type or the Number of Clusters) is conceptually the same.
All operations in Firebolt can be performed via SQL or UI. When the user first creates an engine with the SQL command below, Firebolt will start with a 2-node engine cluster receiving customer traffic:
```sql
CREATE ENGINE MyEngine IF NOT EXISTS WITH TYPE = S NODES = 2;
```
As the workload continues running, let us say that the user wants to scale out their engine from a 2-node cluster to a 3-node cluster to meet their performance demands. They can easily do so using the following SQL command:
```sql
ALTER ENGINE MyEngine SET NODES = 3;
```
The below visual shows what this scaling process looks like in Firebolt.
Now, we will go under the look and look into the details of this scaling process:
### Create and start a new engine cluster [#create-and-start-a-new-engine-cluster]
1. A new 3-node engine cluster is created.
2. Control Plane starts the cluster and updates its state to RUNNING once it's up and running.
### Traffic switchover [#traffic-switchover]
1. Gateway fleet receives a notification from the control plane stating that the Engine state has changed.
2. The Gateway fleet starts routing traffic to the new cluster.
3. The Engines control plane switches the old cluster to a draining state
4. The gateway fleet receives a notification stating that the engine state has changed and detects that the old cluster is in a DRAINING state. The Gateway fleet stops sending traffic to the old cluster.
### Drain traffic from the old cluster [#drain-traffic-from-the-old-cluster]
The Engine control plane tracks the number of queries running on the old cluster. Once the number of active queries reaches zero, it invokes engine cluster shutdown.
These operations allow Firebolt to perform "blue-green style" deployment when changing engine shape to the specification requested by the customer.
### Graceful drain [#graceful-drain]
As mentioned above, old clusters are gracefully drained during online scaling before shutting down. This is necessary to ensure queries running on an engine when the engine was scaled to a different configuration can be completed successfully.
Steps to execute graceful drain:
1. The Engines control plane marks the Engine Cluster as DRAINING. It sends notifications about the state change.
2. The gateway fleet receives an engine state notification, detects a draining cluster, and removes it from the list of clusters eligible to receive traffic.
3. Any new query that reaches the Gateway fleet will be routed to engine clusters that are still RUNNING.
4. The control plane watches the number of running queries on the cluster that are being drained. Once no running queries remain, the control plane invokes the engine cluster destroy sequence that removes underlying Kubernetes resources and engines cluster metadata.
We had several options for how graceful drain would check for running queries:
* Waiting for each engine node to notify the control plane when it has no active queries
* Rely on Kubernetes graceful drain mechanism to self-shutdown each node once no running queries remain
* Making control plane poll engine cluster for the number of running queries
Each engine cluster performs distributed query processing. Engine cluster nodes have to be running to execute distributed queries. Because of that, individual nodes could not make a shutdown decision, and the control plane had to coordinate the process. We opted for the third option of the control plane polling the cluster. It coordinates the shutdown sequence and reduces the amount of traffic flowing from each engine cluster to the control plane during normal system operations.
## Online Upgrade [#online-upgrade]
Firebolt leverages both online scaling and graceful drain to perform engine upgrades.
### Single engine upgrade [#single-engine-upgrade]
The Control Plane is driving the engine upgrade process. It takes a sequence of steps:
1. For each engine cluster, the Control plane creates a new shadow engine cluster with the same specifications as the old one, except a new Firebolt database version is used. The control plane starts a shadow cluster.
2. Control Plane changes engine traffic configuration to mirror mode. Any request to an engine cluster is mirrored to the corresponding shadow cluster. Routing is performed to ensure the same load distribution on primary and shadow clusters.
3. Gateway fleet receives new traffic configuration and starts traffic mirroring.
4. The upgrade verification process starts.
5. The upgrade process waits for verification to finish.
6. If the verification result is a success, the upgrade process performs promotion.
7. The promotion process makes the shadow a new primary and starts a graceful drain process for the old cluster.
8. Once the graceful drain is complete, the control plane shuts down an old cluster and removes it from the system.
9. The upgrade process stores rollback information for the engine in case a regression is detected later, and the fleet needs to be rolled back.
### Upgrade verification process [#upgrade-verification-process]
The upgrade verification process tracks various metrics from old and new engine clusters. Tracked metrics include query latencies and success rates.
The verification process uses a sliding window to compute latency distributions for the queries that ran within verification windows. It detects whether there are any latency regressions caused by the rollout of the new database software version.
When tracked metrics converge within an acceptable margin or error, the verification process returns control to the single-engine upgrade process.
If the error rate for the queries running on the new version of the database software increases, the verification process aborts, signaling the engine upgrade process to rollback.
### Fleet wide upgrade [#fleet-wide-upgrade]
At the fleet-level, the engine upgrade process updates engines based on the segment - a group of customers at a time, grouped by the risk profile in order of:
* Firebolt Internal
* Preview
* Stable
For a single segment upgrade process is:
* Upgrade running engines in the segment
* Update the default version for new engines in the segment
* Update the version for the stopped engine in the segment
If a regression is detected during the upgrade, rollback procedures are performed promptly to mitigate the impact on customers.
The above process continues until all engines are running using the new version.
## Summary [#summary]
Firebolt engines provide multi-dimensional elasticity to our customers, allowing them to achieve the desired price-performance without causing downtime for customers.
Learn more about [engines](https://www.firebolt.io/elasticity), or [get started for free](https://go.firebolt.io) with Firebolt now.
For more information about Firebolt Engines, read our whitepaper [here](https://www.firebolt.io/resources/firebolt-elasticity-technical-whitepaper).
# Exploring your data lake in Firebolt using just TVFs (/blog/exploring-your-data-lake-in-firebolt-using-just-tvfs)
**TL;DR:** Firebolt supports `read_parquet()` and `read_csv()` functions that make it easy to explore data in your data lake or object storage. This blog post describes how these table-valued functions (TVFs) work under the hood and how to use them effectively.
With read TVFs, Firebolt takes care of everything: locating your data objects, inferring the schema, querying them efficiently, and returning structured table values. The `read_parquet()` and `read_csv()` functions are straightforward to use:
1. Provide either:
* A URL to your data, with credentials if required, or
* A Firebolt Location object.
2. Firebolt infers the schema from the most recently modified file in the provided location.
3. The function returns structured rows of data.
4. Profit.
The simplicity makes these functions useful for ad-hoc data exploration and incremental data loading. There is no overhead of defining a persistent schema object (as required by external tables) or ingesting the data (as with `COPY FROM`).
### Show Me What To Do [#show-me-what-to-do]
Here are some examples. All of these work on public S3 buckets. You can provision a Firebolt engine in us-east-1 on AWS and run them right away:
**1. Query public data directly. Use glob patterns to query multiple files at once.**
```sql
SELECT * FROM
read_parquet('s3://firebolt-publishing-public/help_center_assets/firebolt_sample_dataset/rankings/*.parquet')
LIMIT 10;
```
It is often helpful to use the `list_objects()` TVF to discover your file name and directory structure.
```sql
SELECT * FROM
list_objects('s3://firebolt-publishing-public/help_center_assets/firebolt_sample_dataset/');
```
**2. Create permanent tables from query results while customizing CSV parsing options:**
```sql
CREATE TABLE levels AS
SELECT * FROM read_csv('s3://firebolt-publishing-public/help_center_assets/firebolt_sample_dataset/levels.csv',
header => TRUE,
empty_field_as_null => FALSE);
```
**3. To access non-public data, specify the credentials using a location object. This approach is recommended for both ease of use and security.**
```sql
CREATE LOCATION my_location WITH
SOURCE = AMAZON_S3
CREDENTIALS = (AWS_ROLE_ARN = '
### Why Do We Need a Metadata Indirection Layer? [#why-do-we-need-a-metadata-indirection-layer]
Why can't we just interact with the objects on S3 directly? Couldn't we periodically list objects in a bucket to determine what data belongs to a table?
This approach introduces significant challenges:
1. **Latency**: Listing S3 objects is relatively slow, taking tens to hundreds of milliseconds. This delay would impact query performance if applied to every operation.
2. **Consistency**: Periodic listing would make it difficult to handle concurrent writes. Data modifications by other writers wouldn't be visible immediately, and ensuring proper conflict resolution between concurrent writers would become increasingly complex.
The metadata indirection layer addresses these issues by efficiently managing collections of objects like tablets. It ensures low latency and supports multi-writer consistency, enabling Firebolt to maintain performance and reliability.
Additionally, this metadata architecture supports advanced data warehouse features, such as table cloning, which we will explore further in this post.
### Tablets metadata [#tablets-metadata]
Tables in Firebolt are composed of multiple tablets—potentially tens of thousands. Each tablet is represented in metadata by an identifier and an address, which serves as a pointer to its corresponding objects on S3.
This tablet metadata is lightweight, while tablets on S3 can be as large as 20 gigabytes, the tablet metadata in Firebolt is only about 100 bytes.
This makes metadata operations to be executed very quickly. This efficiency is a fundamental aspect of Firebolt's metadata layer: working with compact metadata handles is significantly faster and more efficient than directly handling huge S3 objects.
Before zero-copy cloning, Firebolt maintained a strict one-to-one mapping between metadata handles and data objects on S3. With the introduction of zero-copy cloning, this mapping has been relaxed, allowing multiple metadata handles to reference the same S3 data objects.
### What Happens During a Clone Operation? [#what-happens-during-a-clone-operation]
When a clone operation is executed, a new metadata entry is created that points to the same tablets as the original source. This means that the physical data itself is not duplicated. The tablet doesn't contain any information about its table or database, so by duplicating only the table metadata, we effectively store the same logical reference multiple times.
Since the actual tablet is shared between the tables, the underlying data isn't replicated, only the metadata is.
As noted earlier, metadata is much lighter than the actual data. This makes copying tablet metadata far more efficient and lightweight than copying the physical data itself.
### Tables are independent after cloning [#tables-are-independent-after-cloning]
After a table is cloned, the original and the clone evolve independently. Any changes, such as updates, inserts, or deletes, to one table do not impact the other. This ensures that each table maintains its own state, even for the data that was initially shared.
When new data is inserted into either table, a new tablet is created in S3 along with a corresponding metadata entry. This metadata entry is exclusively associated with the table where the insert operation was performed.
For deletes or updates, directly modifying the shared tablet would impact other tables that reference it. To prevent this, tablets are designed to be immutable. This also plays nicely with S3 which doesn't allow in-place modification of objects. Instead, Firebolt maintains a change log for each tablet. This log allows each table's metadata to reference a different unique deletion log, ensuring that every table preserves its own distinct state, even when sharing the same underlying tablet.
### Cross-Database Clone [#cross-database-clone]
Tablet data files are not tied to specific logical objects like tables or databases. This allows the references of a tablet to be copied from one database to another, while the metadata still points to the same shared storage.
Zero-copy cloning is Firebolt's first cross-database operation, enabling data transfer across databases without the need for exporting and re-importing. Cross-database cloning also simplifies setting up dedicated databases for testing or staging environments, which we plan to support in an even simpler way through a CLONE DATABASE command.
Implementing cross-database cloning is significantly simpler than supporting full cross-database queries, as it involves only metadata operations without the complexities of query planning.
### When to Delete Objects on S3? [#when-to-delete-objects-on-s3]
When two tables reference the same tablet in storage, what happens if one table is dropped? Rest assured, your data is safe.
Tablets in storage are not immediately deleted when a table is dropped or data is removed. Instead, they are retained for future cleanup by the storage garbage collector.
For zero-copy cloning, while data isn't kept indefinitely, the garbage collector periodically checks for active metadata references against the S3 storage. Any tablet referenced even once is preserved, preventing accidental deletions.
In summary, dropping a table does not trigger storage cleanup that could affect another active clone of the same data.
### Performance [#performance]
We already covered that Zero-copy cloning is extremely fast because only table metadata is copied. The operation's complexity depends not on the table's size but on the number of tablets it contains at the time of cloning.
For perspective, tables can consist of thousands to tens of thousands of tablets, each with metadata of about 100 bytes. [In our FireEdge demo](https://youtu.be/m_PgdHLc65s?si=g37EFRt93hScrMTW), a 60TB table with \~8,000 tablets generated metadata totaling less than one megabyte.
In our first development version, tablet metadata was written one at a time, requiring multiple round trips to our metadata server. While faster than re-ingesting a table, it wasn't ideal. The first version of zero-copy cloning we shipped batches metadata writes, significantly improving latency. Though there's still room for optimization, the current speed allows us to clone huge tables very quickly. In the demo linked above, cloning the 60TB takes about four seconds.
This figure demonstrates that the time required for a clone operation scales linearly with the number of tablets in the table. It also highlights the significant performance improvement achieved by batching metadata writes, reducing the overall cloning time.
### Clone Storage Costs [#clone-storage-costs]
Zero-copy cloning eliminates the need for physical data duplication, allowing tables to share the same underlying storage without consuming additional space. This approach significantly reduces storage costs.
Firebolt's pricing model separates compute and storage costs, meaning multiple tables sharing the same storage directly lowers storage expenses. For instance, as of today, the storage price in the us-east-1 region is $23/TB per month. Cloning a 60TB table instead of re-ingesting it can save approximately $46 per day. Check out [Firebolt's Pricing Calculator](https://www.firebolt.io/pricing#pricing-calc)
Storage billing and usage details can be monitored through Firebolt's information\_schema views, such as [storage\_billing](https://docs.firebolt.io/sql_reference/information-schema/storage-billing.html), [storage\_metering\_history](https://docs.firebolt.io/sql_reference/information-schema/storage-metering-history.html), and [storage\_history](https://docs.firebolt.io/sql_reference/information-schema/storage-history.html).
### Conclusion [#conclusion]
Firebolt's zero-copy clone feature offers a cost-efficient and fast solution for cloning tables or databases without incurring additional storage costs.
This blog explores the technical design of the metadata and storage layers, which form the core infrastructure enabling zero-copy cloning.
Additionally, we discuss performance benchmarks with practical examples and highlight tools designed to enhance visibility into consumption and storage costs.
# Firing Up Firebolt’s Client Ecosystem (/blog/firing-up-firebolts-client-ecosystem)
## Asynchronous query execution [#asynchronous-query-execution]
Asynchronous query execution refers to starting a long running query without holding up the connection in client, whilst receiving a token to reference said query to be able to see it's status in time. This features enables users to not use precious resources on just maintaining a connection when in fact their client is not doing anything.
Applications can't always wait for a submitted query to finish running before moving on. Imagine performing a bulk insert that will take minutes, and your entire logic flow grinds to a halt while waiting for that to complete. Not a good pattern, right? Asynchronous query execution allows you to avoid this problem, submit a query, leave the execution wholly to Firebolt, and move on. This has landed in all of Firebolt's client drivers, ensuring that your client, application, or hardware can get back to tackling tasks while you put Firebolt to work.
How does it work? In the example below, we assume that the Firebolt driver runs in your VPC on an EC2 instance on AWS (it can be on any other cloud or on prem):
Without async execution:
During the time the Firebolt executes the query, in this instance we assume it takes 1 hour to finish, your resource (EC2 instance) will stay up and running, costing you money even though it is not doing anything useful: it is just waiting for the query to finish.
With async execution:
When executing a query asynchronously, the client will get back a tokenId that can be then used to:
* check the status of the initial query
* cancel the initial query (only if the query has not finished by then)
Notice that you can stop your EC2 instance for the duration of the time it takes Firebolt to actually run the query, and then you can bring it up to check the status and continue whatever work you were planning to do.
This approach not only frees up your application's resources, but also introduces real infrastructure cost savings. By decoupling query execution from your client's uptime, you avoid paying for idle compute — which is particularly impactful when running large-scale insertions or transformations that may take tens of minutes or more. You can integrate this flow into an automated lifecycle where your EC2 instance submits the query, shuts down, and then is reactivated via a scheduled job or event trigger to resume downstream processing once Firebolt has completed the work, or once it's time to submit another query.
The diagrams above illustrate both approaches: the first shows how your EC2 instance stays active (and incurring cost) throughout a synchronous query; the second highlights how asynchronous execution lets you pause client-side activity while Firebolt completes the task independently.
Implementing it with our clients isn't very difficult, either:
```java
// Statically typed drivers require the user to cast/unwrap classes to the Firebolt implementation
FireboltConnection connection = DriverManager.getConnection(url, clientId, clientSecret).unwrap(FireboltConnection.class);
try (FireboltStatement statement = connection.createStatement().unwrap(FireboltStatement.class)) {
statement.executeAsync("INSERT INTO table SELECT checksum(*) FROM GENERATE_SERIES(1, 2500000000)"); //long running query
String token = statement.getAsyncToken(); //token from which you can identify the query by
//checking query running status
boolean isRunning = connection.isAsyncQueryRunning(token);
boolean isSuccessful = connection.isAsyncQuerySuccessful(token)
//you can also cancel the query execution if it is still running
connection.cancelAsyncQuery(token); //this should return true if the query was cancelled
}
```
```javascript
// But dynamically typed drivers do not have this issue
const statement = await connection.executeAsync(
"INSERT INTO table SELECT checksum(*) FROM GENERATE_SERIES(1, 2500000000)",
executeQueryOptions,
);
const token = statement.asyncQueryToken; // used to check query status and cancel it and can only be fetched for async query
//checking query running status
const isRunning = await connection.isAsyncQueryRunning(token);
const isSuccessful = await connection.isAsyncQuerySuccessful(token);
//you can also cancel the query execution if it is still running
await connection.cancelAsyncQuery(token);
```
## Streaming query results [#streaming-query-results]
If you're querying massive datasets, sometimes you're also working with large result sets. Not every query has a LIMIT 10 at the end — and if you're reading a million rows or more, this can pose a very real challenge when it comes time to actually *work* with the results. A naive implementation might simply return the entire dataset to the client all at once. That could work for smaller queries, but with large outputs, it's a fast track to running out of memory, triggering crashes, or severely degrading performance.
But with streaming, Firebolt handles things differently. Instead of delivering the full result set in a single payload, it streams rows in manageable chunks. Your application doesn't need to hold the entire dataset in memory — it can start processing as soon as the first rows arrive, consuming them incrementally in a forward-only fashion. This makes it not only safer (no memory overflows), but also faster in practice, since you can begin working with the data before the entire query completes.
This approach is ideal for ETL pipelines, large data exports, or any kind of batch process where the size of the result isn't known ahead of time — and where robustness is just as important as speed. Firebolt drivers and SDKs provide this functionality in a simple manner, keeping the driver-intended way of parsing results, but not actually storing the entire result at once.
Streaming becomes especially important in production environments where reliability and scalability matter. For example, if you're building an analytics API that returns results to end users or pushing query results into a downstream system like S3, Kafka, or a data lake — you're likely dealing with large, unpredictable volumes of data. Without streaming, you'd be forced to manually implement workarounds like pagination or break your queries into smaller chunks, adding complexity and latency. With streaming, Firebolt handles that complexity for you behind the scenes, so your application can simply iterate over rows as they arrive, no special logic required.
What's more, streaming isn't just about protecting against crashes — it's also about *responsiveness*. Because data arrives as it's ready, your application can begin transforming, displaying, or transferring the output without waiting for the full query to complete. That improves time-to-first-byte and makes your system feel faster and more responsive to users — especially valuable in dashboarding tools or real-time reporting interfaces. In short, streaming transforms how your app consumes data: from waiting and hoping, to processing and scaling.
On the topic of error handling, you do lose the ability to know before receiving the results if there were any errors encountered during processing (i.e.: select 1/(i-100000) as a from generate\_series(1,100000) as i), but each driver provides mechanisms for not generating exception vulnerable code even though an error may be encountered in the middle of the result set.
JDBC supports result streaming by default, so the statically typed driver presented here will be .NET:
```csharp
FireboltCommand command = (FireboltCommand)conn.CreateCommand();
command.CommandText = "SELECT * FROM large_table";
// Execute the query without storing the whole result in memory
using var reader = command.ExecuteStreamedQuery();
// or use the asynchronous version
using var reader = await command.ExecuteStreamedQueryAsync();
// Iterate over the streamed results in the same way as with a regular DbDataReader
while (await reader.ReadAsync())
{
//process results with reader.GetValue(x) for example
}
```
```javascript
// JavaScript's implementation leverages its streaming template
const statement = await connection.executeStream(`select 1 from generate_series(1, 2500000)`);
const { data } = await statement.streamResult();
data
.on("meta", (meta) => {
//store/process meta
})
.on("data", (row) => {
//process data
})
.on("error", (err) => {
//capture error
});
```
## Server-side prepared statements [#server-side-prepared-statements]
If your application accepts user input — whether it's a search box, a filter in a dashboard, or a dynamically generated report — chances are you're constructing SQL queries on the fly. And when you're doing that, you're also opening the door to potential vulnerabilities like SQL injection. A common approach to mitigate this is to use parameterized queries, where user values are passed separately from the SQL structure. But even then, *where* and *how* those parameters are handled makes a difference — and that's where server-side prepared statements come in.
With server-side preparation, the Firebolt engine — not your application — is responsible for securely binding parameters. You send the SQL statement with placeholders, and the actual parameter values are passed separately. Firebolt compiles the query and injects the values safely on the server, eliminating the risk of those values being interpreted as part of the SQL logic. This setup adds a layer of security by removing the burden from your application code and protecting against common injection attack vectors.
It's also worth noting that server-side prepared statements don't just improve security — they can also improve performance. Since the query structure is compiled and cached by Firebolt, repeated executions of the same statement with different parameters can reuse the same execution plan. This is especially valuable for multi-tenant apps, dashboards, or reporting tools that generate the same query with slightly different inputs. With fewer parsing and planning steps, your queries run faster and more predictably.
Firebolt also uses this feature to help standardize how parameters are defined across all supported drivers. While most languages and frameworks have their own syntax for prepared statements — like ?, :name, or %s — Firebolt's server-side prepared statements use a numbered placeholder format like $1, $2, and so on. This consistent format helps reduce confusion across languages and environments, making it easier to maintain query templates in a shared codebase. It also improves readability and makes the logic behind parameter ordering explicit.
That said, Firebolt still supports the native client-side style of prepared statements if you prefer — and in fact, it's the default for many drivers. You can continue using the syntax that matches your language of choice, and the driver will substitute parameters before sending the final query to Firebolt. But if you want to take advantage of server-side preparation — for the added security, performance, and standardization — you'll need to update your queries to use the $1, $2, etc. format and enable the appropriate mode in your client.
In short, server-side prepared statements help you build safer, more efficient applications — while encouraging best practices and consistency across teams, drivers, and use cases.
Two different ways of implementing them:
```java
// Besides GO, all languages need one more element in the connection string to make the connection server-side prepared statement ready. In this case we have prepared_statement_param_style=fb_numeric (the native option is the default, but you could specify it in the connection string if you want)
Connection connection = DriverManager.getConnection("jdbc:firebolt:my_db?account=...&prepared_statement_param_style=fb_numeric....", clientId, clientSecret);
try (PreparedStatement statement = connection..prepareStatement("select $1, $2")) {
statement.setInt(1, 2);
statement.setString(2, "foo");
ResultSet rs = statement.getResultSet(); //then process result
}
```
```javascript
//Node.js allows you to make use of both parameters and namedParameters, but not both at the same time
const connection = await firebolt.connect({
...connectionInfo,
preparedStatementParamStyle: "fb_numeric", //important
});
const statement = await connection.execute("select $1, $2", {
parameters: ["foo", 1], //using the parameters executeQueryOptions field
});
const statement = await connection.execute("select $1, $2", {
namedParameters: { $1: "foo", $2: 123 }, //using the namedParameters executeQueryOptions field
});
```
## Conclusion [#conclusion]
Every driver has these features implemented and they are all ready to improve your experience with Firebolt, be it by a resource freeing aspect, a memory holdup one, or just better security. For better explanations please refer to the driver's documentation and if needed, don't hesitate to contact us!
# From Zero to 100M Users: Inside Notion’s Data Stack and AI Strategy with Sumit Gupta (/blog/from-zero-to-100m-users-inside-notions-data-stack-and-ai-strategy-with-sumit-gupta)
In this episode of The Data Engineering Show, the bros talk with Sumit Gupta, Lead BI Engineer at Notion, about his journey through prominent tech companies, modern data stacks, and how AI is revolutionizing data workflows and professional development.
Listen on [Spotify](https://bit.ly/45is6z0) or [Apple Podcasts](https://bit.ly/43Yv4q0)
**Sumit - 00:00:00:**
AI has made me a lot more productive, but at the same time, it has also made me dumber. Whereas Claude is more particularly great at programming tasks. Complexity is amazing at deep research. I'm a lead BI engineer at Notion, and I lead reporting, dashboarding for marketing and sales teams. Right before Notion, I used to work for Snowflake. I worked for them for a couple of years. If you don't jump onto the bandwagon right now, you might be left out in a year or so.
**Intro - 00:00:30:**
The Data Engineering Show is brought to you by Firebolt, the Claude data warehouse for AI apps and low-latency analytics. Get your free credits and start your trial at firebolt.io.
**Benjamin - 00:00:42:**
Hi, everyone, and welcome back to The Data Engineering Show. Today, we're super happy to have Sumit on. Sumit is a lead BI engineer at Notion right now, has been here for a bit more than a year. Great to have you on the show. Do you want to quickly introduce yourself to our listeners?
**Sumit - 00:00:58:**
Yeah, thanks a lot for having me, Benjamin and Benjamin. I've been so excited to jump on this podcast and talk to you guys about everything AI and a bit about me. So as you mentioned, I'm a lead BI engineer at Notion. And I lead basically reporting, dashboarding for marketing and sales team here at Notion. Right before Notion, you probably will see a Snowflake logo on the video.
**Eldad- 00:01:23:**
No, we haven't noticed.
**Sumit - 00:01:26:**
Yes. And then you have like Hello Data Nation t-shirt in here. So right before Notion, I used to work for Snowflake. I worked for them for like a couple of years. And then before that, I was at Dropbox leading their analytics team, marketing analytics team. So overall, like I have great experience being part of like- I like to use this term called Bay Area Darlings. So back in 2017, 18, Dropbox was Bay Area Darling, right? And then Snowflake became Bay Area Darling because of, you know, IPO and etc.. And now if you think about it, Notion is the Bay Area Darling. So I don't know. I have some affections with darlings, I guess.
**Benjamin - 00:02:03:**
Nice. That's awesome. So what got you started in data kind of like in the first place, right? Kind of like tell me, maybe a bit about that journey.
**Sumit - 00:02:13:**
I graduated from Mumbai University in 2014. And then right before my senior year, I was like exploring what my career choices could be. You know, a typical 18 year old where like he's confused, right? And this is before the age of AI. I know, I mean, a lot of folks nowadays have it easy. Where you could just go to ChatGPT and be like, you know what, help me decide my career. But back in 2014 in India, like you were, there wasn't a lot going on in terms of like tech. Like there was obviously like software engineering as a career. But I realized that software engineering isn't for me because I like, I love talking to people. And then I wanted more interaction. And I did not want to be a out and out management consultant either. Because I still wanted, I knew for a fact that tech has to be in my, has to play a very important role in my career. So as I was exploring, I was like, you know what, Information Management is something that I should probably pursue. And then I decided to pursue my master's in that from Syracuse University in the United States. And then the reason United States Information Management is because it has that mix of data management consultant and yet being very close to like tech. So that's, that's how I got introduced to tech. And since then, I've been, I've been in data slash tech for over a decade now.
**Benjamin - 00:03:29:**
Amazing. That's awesome. So tell us maybe about kind of like the, at least at a high level, kind of the work you're doing at Notion right now. Kind of like what, what are you working on? Kind of how's your data stack? Give us an overview.
**Sumit - 00:03:43:**
Yeah. So Notion, Notion is a very, if you think about it, Notion is relatively new. All the Notion was, I think, started in 2014. But the Notion got really popular due to COVID boom, right? I think if, if I, if I recall it correctly, until like 2020, we had close to like about a million or a couple of million users. Now, I don't know if you saw the news or if you saw our co-founder tweet about it. We have over a hundred million users now.
**Benjamin - 00:04:10:**
Wow, congratulations. That's so impressive.
**Sumit - 00:04:13:**
Yeah, yeah.
**Eldad- 00:04:14:**
That's amazing.
**Sumit - 00:04:15:**
It's wild that, you know, the kind of following that we have, B2C as well as both B2B, right? So if you think about it, going from a couple of million users to 100 million users, if your data stack is not nimble or if it's not modern, right? That's the term that we want to use. If it's not modern, you're going to be stuck, right? Your team will not be able to make any relevant decision in time. So that basically means our stack includes like some of the most common modern tools like Snowflake. We use Fivetran for ingesting some of our data. Obviously, we have some custom pipeline set up too. We use Airflow for orchestration. And then we use Tableau as well as Hex for our reporting in Dashboard. I'm an out-and-out Tableau guy. I mean, not to diss on other tools, but Tableau is great, right? One of the reasons I love Tableau is the fact that it has great community.
**Eldad- 00:05:09:**
By the way, you know, if you love Tableau, you know Tableau went through a long history of improving their query engine behind the scenes. And there was an acquisition done by Tableau of a startup, Hyper, and that query processing team is now driving Tableau, Salesforce, Query Stack. It's an amazing team. And the reason I mentioned it is all of them are coming from Munich. And many of- Some of them are at Firebolt. So there's a lot of geeky query processing stuff happening in Munich that drives many, if not most, of today's data warehouses and data solutions. So we have a special place in our heart for Tableau as well.
**Sumit - 00:05:47:**
Yeah, did not know that. But I know for a fact, like, the Hyper Engine that Tableau has, it has changed a lot. Like, you know, nowadays when I import or extract, like, 20, 30 million rows, it goes by really fast, right? Because of Tableau Hyper Engine. So, going back to the topic about modern-year stacks, our BI reporting layer is Hex and Tableau. I don't know if you guys have heard of Hex, but the way I'd like to think of Hex is like Jupyter Notebook, but on steroids. It does everything that you can imagine, right? Python, Spark, SQL, Matplotlib. When you think of Python, Matplotlib, your pivot tables, your charts and everything. So we have that. We also use nowadays, we also have started moving some of our data loads to like Iceberg tables. That's the new thing in the market, right? Like a lot of folks are moving to Iceberg tables because of the benefit that Iceberg offers.
**Benjamin - 00:06:41:**
So are you leveraging kind of the key benefits of Iceberg already in terms of having flexibility between query engines?
**Sumit - 00:06:48:**
Yes. So I primarily don't work with Iceberg, but I know for a fact that our data platform team does. So I'm not an expert to talk about that. But I think the whole point of Iceberg table was like there's a component of saving cost, right? Because with Iceberg, like with the whole metadata layer and how the data moves into the actual like data layer. That's why we decided. Because we use Snowflake as our analytical data warehouse, but we also have like S3 bucket in our data lake sitting in AWS, right? So we wanted to, because if you imagine, as I mentioned, like when you go so fast, your data is, your data needs are growing so fast. Every bit, right, is expensive. Like all the servers are cheap, but when you're dealing with 100 million users and trillions of rows of data a day, you have to find that one person saving, two person saving or those marginal savings. And Iceberg helps us with that.
**Benjamin - 00:07:41:**
Okay, super, super cool. And very modern stack as well. Kind of like using a lot of stuff to cool, kind of like new tech, I mean like Iceberg hacks, kind of all of that. That's all.
**Eldad- 00:07:51:**
At the front of the modern stack, even the modern stack, right? Like you have that starts to evolve as well. We see the modern stack getting a new definitions across. Across each component, it's beautiful to see it. We love Iceberg. Iceberg makes all metadata and data the same for every vendor, for every query engine, every ETL, ELT tool, every user. Yes, there are many challenges. Yes, there are many gaps, and that's why we're all here. And it's amazing to see the evolution happening, Iceberg, within information and analytical teams. Amazing, go on.
**Sumit - 00:08:29:**
Yeah, and then, sorry, I forgot to mention the... Like if you think about modern data stack, there is one tool that brought this into like in the front, right? And that's dbt. We use dbt too. We are a heavy dbt user. You cannot forget dbt because the whole term around analytics engineering and modern data stack, I think they were the one who like promoted this really heavily. Because before dbt, right? So I was at Dropbox and we used to have our own version of like Hive, right? And then there were times when it would take at least like 18 to 20 hours to run a query. Because you can imagine Dropbox is back in 2019, 2020, right? Dropbox is and was huge. We do like a couple of billion dollars of revenue a year. So and then now with dbt, et cetera, right? You could have incremental models and, you know, basically put things in Snowflake, right? Obviously, when you think of Snowflake and Hive, Snowflake obviously will be a lot faster, although there's a cost associated. But sometimes either you invest in time or you invest in dollars. In the Snowflake case, you invest in dollars because you save time.
**Benjamin - 00:09:34:**
In your data stack, like how is AI kind of like shaping it? Like what's changing? Kind of what's your take on all of it? Give us an overview.
**Sumit - 00:09:43:**
Yeah. So we at Notion use AI heavily. We have our own AI for Work tool, which is inbuilt in Notion. I don't know if you guys use Notion or not, but people or companies that use Notion with AI for work, basically everyone in the company is supposed to use AI for Work. It's like deep research. It has all the context about your data. So you can you can connect your Slack. You could connect your Google Drive. You can connect your other like third party apps into Notion now. And then just like let's say if you have a question about when was the last time a specific metric was updated, right? AI for Work will do that for you, like Notion's AI for work will do that for you. And outside of Notion, we are a heavy AI user as a company, like everyone in the EPT department or pretty much in the company has access to Cursor, right? The coding, I wouldn't say agent, but like editor nowadays, right? So we use Cursor a lot nowadays to like help U.S. Speed up our productivity. And then we have been building a couple of like GTM or go-to-market market, I wouldn't say tool, but like use cases. Like one of the use cases that we have built is whenever we are going to talk to our customers, if they are already, especially renewal customers, if they are already, if they are up for renewal, we have this AI agent or AI workflow setup where you select a customer and then it will give you all the details about total number of users, active users and users who have never logged in in XYZ days. Plus like we also ingest our call transcript, our Slack messages that the sales rep have and then create a short summary of what has been discussed in last 30 days, last 90 days. What has, was there any blockers? Was there any pointers that we can use? So that you have one page document where you can go talk to the company and then, you know, try to, because when you go for a renewal conversation, right? If you have data to back it up and be like, you know what, 80% of your company is using Notion. So it's probably prudent or it's probably wise to, you know, renew it with U.S., right? If you were to move to a new tool, there would be a lot of headache, et cetera. So the fact that we could go have that conversation and the reason we could do that now because of all the context is AI is the reason for that.
**Benjamin - 00:11:56:**
Very cool. Okay. That's exciting. How, how do you stay on top of like all of the stuff happening in AI in your personal life as well?
**Sumit - 00:12:03:**
I think my mode of AI consumption has been podcast recently. When I say mode of consumption, I was recently on another podcast called Your Everyday AI Podcast. Jordan does like 20 minutes AI news every day at like every, every morning. So whenever I wake up, I listen to that. And then a lot of keynotes nowadays, right? Like recently Google had a Google I/O, right? And I would watch that and see like what's happening latest and greatest. Claude recently launched, launched Claude 4.0, like, right? So whenever some new model comes along, I am first to jump onto it, right? And basically test it out and see what's working. And you wouldn't believe I have basically premium access to pretty much all the famous LLMs, be it Claude, be it Perplexity, be it ChatGPT, right? And my use case... Every tool is great at something. Like for me, chatGPT is when I want to, let's say, rewrite an email or need some like text written or, you know, want to review some document, etc. Whereas Claude is more particularly great at programming tasks. Right? Perplexity is amazing at deep research.
**Eldad- 00:13:09:**
Who does the cooking? Who is cooking for you?
**Sumit - 00:13:13:**
Yeah. No, I mean, Perplexity, I guess. Because with Perplexity, right? I think Perplexity was the first tool which allowed you to do deep research where you had direct access to the search engine, right? ChatGPT. And Claude, like they recently launched deep research, right? Gemini obviously do, but Perplexity is where you could go and be like, you know what, like one of the use cases, I was recently searching for like universities for my niece that she was applying for, applying to universities in U.S.. And I was like, you know what? This is the score that she has received. This is like, you know, her SAT scores, her other scores. Go help me find 20 universities that has good acceptance rate. When I was applying, that was like one week of effort. Now it's not even seven minutes of effort.
**Benjamin - 00:14:00:**
That's very, very cool. Nice. How do you think the data space as well is going to evolve over the next couple of years? What are you particularly excited about? What problems do you think will start going away? Tell us about that.
**Sumit - 00:14:18:**
I actually like to divide that into two parts. One is like if you are someone who's starting a new in Data Field, right, as an entry-level data scientist, analyst, or engineer, I would say the value of your technical skills that used to be very valuable until 2021 when there was a boom in hiring. Is not as much as, so the value has decreased, right? Your tech skills are valuable, right? But as an entry level, you don't know if something breaks, how to fix it. That's where your transferable skills or your soft skills comes into picture, right? So if you are to grow in data career, make sure that your transferable skills are outshining your technical skills nowadays, right? That's where the entry will part. But if you're a senior, let's say for someone like us, who has like seven, eight, 10 years of experience, right? Your tech skill is important. You know when things break, where to go and how to look and what to fix. But your transferable skills are going to be paramount in future too, right? Especially at Notion and a lot of other companies that I've heard, right? There's a lot of like, a lot of CEOs nowadays are coming out and be like, use AI. If you don't use AI, you're going to be out of job soon, right? So in that case, right? AI, like nowadays I use it whenever I'm building Tableau dashboard, etc. There were times when it took a couple of days to write a calculated field. It was, if it was complicated because Tableau has its own way of calculating, right? Now when I'm stuck, I just go to Claude and like, you know, write me a calculated field and I get the calculated field done in 20 minutes. So the value that I used to bring as a technical guy has diminished. But if that gets diminished, I have to increase the value that I bring as a professional with like great communication skills, right? Great stakeholder management. So you kind of have to build on that. So I don't know if that answer your question because, data is important, but the role of data as a tech skill.
**Eldad- 00:16:11:**
Be technical goes down, be nice goes up. You know, like that's it. It's really fascinating to see kind of here from someone from the inside experiencing the AI like full-blown in your face not from the like not theorizing about it really practicing it turning into your personal career advantage figuring out how to build your strength around it thank you for sharing it with us.
**Sumit - 00:16:39:**
Yeah, no Notion, like, I use AI heavily, uh, in my work too, but in personal life too, as I mentioned like I re- I was recently on Everyday AI Podcast, and the topic that we discussed there was like, AI has made a, made me a lot more productive, but at the same time, it has also made me dumber. Because I have-
**Eldad- 00:17:00:**
Who would have thought? Who would have thought getting dumber gets you more productive? Who would have thought? That defies physics, defies everything we've learned.
**Sumit - 00:17:08:**
Exactly.
**Eldad- 00:17:09:**
30 years of industry experience. I'm too old. I'm like, I've learned, you know, like, Previous generation physics, everything is changing.
**Sumit - 00:17:16:**
Yeah. And then I gave a couple of examples there. And honestly, like after that call, and I've been like trying my best to not like allow AI to drive my life. It's good. AI allows me to research. But I have like a back of hand rule nowadays where like I use AI for repetitive tasks, right? Like I, to give you an example, like I have an Instagram channel by the name Data by Sumit, right? It has around 21,000 followers, et cetera, right? So previously I used to have a social media content strategist and content researcher. We used to like sit down and, you know, research content ideas for me. Now I have a Make.com like workflow where I have identified a list of, let's say, 100 Instagram users, right? Who are, I wouldn't say competitor, but who are in my niche, right? And then I use Appify's API. Also, I've also tested RapidAPI. Appify's API, which basically I feed the users into Appify. Appify goes into, Appify's API basically helps me extracts, let's say, last 30 days of data for each profile. And then I have a GPT-3 model as part of the workflow where it looks at, I have a custom formula where based on number of views, number of likes, number of comments, I create a custom column, like a calculation of high, medium, low, like, should I be writing about this or should I be scripting about this? And once that cutoff is met, the second part of the GPT-3 model goes and transkypes the reel and then learns from it and then comes up with the new script for me.
**Eldad- 00:18:44:**
Wow. This is when information work becomes operations. Like, you're turning, like, your usage of data with those tools, you- This is operations. You're driving a business. You're driving a unit. You're moving crazy. Amazing.
**Sumit - 00:19:02:**
And honestly, that's how AI is supposed to work or supposed to be used. Someone recently asked, do you think AI will take over the world kind of conversation? I'm like, maybe, yes. But for now, I mean, if it does, we don't know. If it does, everyone would be infected, not only me. But right now, AI is helping me improve my workflow. AI is helping improve my, improvise my workflow and get better at it. Like something which took me a week to get like four or five new content ideas. I just have to click a button and get like 50 different new ideas based on what's working right now. The last seven days, last 30 days. And then I get script too. I just have to. Like one of the, so like an inside scoop. One of the things that I'm doing now is like creating another workflow where based on the script, right? I'm using HeyGen and ElevenLabs to basically replicate myself now. My voice and myself, right? So that's the next step in the process where if I'm not there.
**Eldad- 00:20:01:**
Are you real? Are you real?
**Sumit - 00:20:03:**
I am real. I can pinch myself.
**Benjamin - 00:20:06:**
Six fingers. Six fingers.
**Sumit - 00:20:10:**
I mean, I wasn't doing a lot of big motion. And, you know, when you don't do big motion, HeyGen really works. So you guys probably don't know.
**Benjamin - 00:20:22:**
In the beginning, you were like, oh, kind of fiddling with the laptop, fiddling with the phone. But really, you were just preparing your AI self.
**Sumit - 00:20:31:**
Exactly. Exactly. That's it. That's it.
**Eldad- 00:20:34:**
The free trial was over. Enter the credit card, subscribe, activate the agent. Nice. Amazing. Thank you.
**Benjamin - 00:20:43:**
Yeah.
**Sumit - 00:20:44:**
No, I mean, as I said, I think anything which can be automated, especially a repetitive task, AI is the way to go. And honestly, there are a lot of AI content creators nowadays that you can learn from. And a lot of them are actually offering free workflows, Make.com workflows and entertain workflows too. So if you are into AI, if you are jumping into AI, this is the right time. And the scariest part about the whole AI boom is this is the worst AI will ever be. We get impressed when ChatGPT was launched and initially I was a skeptic. New hype. And then I started using it. I'm like, what the hell? Like, how is GPT so good? Like, it doesn't make sense. And now when I think of it, I'm like, this is the worst it will ever be.
**Benjamin - 00:21:31:**
Nice. I also love how like as an AI content creator, you're at like the front of content creation. And we're getting close to AI content creation being done by AI. Like, we're very close to actually like coming full, full circle and just having AI create content about itself.
**Sumit - 00:21:53:**
But there's also morality, moral aspect of this where I'm like, I want to do the HeyGen and ElevenLabs like workflow. But I'm like, if I was following a content creator that I love and if I knew that all he is is an AI nowadays, I would be pissed. So I'm like, I can do it. If I sit for a weekend, I can set the workflow up. But I've been intentionally delaying it because I wrote a book about Tableau, the Tableau workshop, right? It sells really well. But one of the reasons I wrote it because it was like, if I was to read this, I should be happy about it. Like I should learn something, something. And that is how I view my life. If I will only do something that if I was in their shoes, I should be okay with that or happy about it. So if I build my own AI avatar, right, am I going to be happy following that content creator? Probably not.
**Benjamin - 00:22:43:**
Nice. That's, yeah, that's a super interesting take. I think we've had a lot of content creators kind of on the show already, but like this is definitely the most AI forward kind of take I've heard so far.
**Sumit - 00:22:55:**
Yeah, I mean, as I said, like I think AI is great, both in professional as well as personal life. And there's a con for AI too. So my wife's been searching for a software engineering job. She's an entry-level software engineer. It's been hard. Even in Bay Area, getting a software engineering job, it's been hard. I have so many friends who had offers and then their offers got rescinded because company decided, you know what, we don't want entry-level engineers. So as much as I am utilizing AI for the good part, I have the bad part use case also. Because if my wife was searching for a job back in 2021, she would already have the job. So yeah, there's the good part, there's the bad part. It's all about balancing it. And hopefully there's that 51, 49% chance of, you know, you using AI for the good part more than the bad part.
**Benjamin - 00:23:43:**
Cool. Wow. This, yeah, this conversation was really like kind of like twisted my brain, kind of so crazy. Kind of before we wrap up, right? Like anything else you want to share with the audience, with the listeners, kind of anything, yeah, kind of you wanted to chat about?
**Sumit - 00:23:59:**
Like I think... Even at Notion and Postman also, there has been a lot of focus on AI-heavy workflow, right? So if you are someone new, especially in data, scared of AI or still skeptic of AI, I would say jump in. There's a lot of tools, a lot of resources nowadays. And even if you don't want to use it... For your personal or professional work, use it as a research tool or a learning tool where if you notice AI, you should be able to recognize AI. Like my eyes are trained now to see when I look at something, I'm like, okay, this is AI, right? If you are not trained, right? If you're not aware of how AI actually, what are AI's pros and cons? And you know, like when you look at an AI video, it's easy to make out. If you look closely, the eye movement, the hand movement, right? If you're not trained, you might get flummoxed, right? You might get confused, right? So that would be my tip. Like if you are in the outside or if you're outside trying to get in and still like are not happy about AI or skeptic about AI, jump in. You don't have to use AI in your personal or professional life, but... Trust me when I say this, if you don't jump onto the bandwagon right now, you might be left out in a year or so.
**Benjamin - 00:25:11:**
I think that's a great kind of closing remark. Thank you so much for being on the kind of show, Sumit. It was great having you and we look forward to catching up in the future and kind of seeing what fraction of your content starts getting AI generated.
**Sumit - 00:25:27:**
Absolutely. Thanks a lot, Benjamin and Benjamin. If you guys are in the same time, drop me a message and we can grab a coffee or a beer.
**Benjamin - 00:25:34:**
Would be amazing. Thank you. Bye.
**Sumit - 00:25:37:**
Bye.
**Outro - 00:25:39:**
The Data Engineering Show is brought to you by Firebolt, the Claude data warehouse for low-latency analytics. Get $200 credits and start your free trial at firebolt.io.
# Fuzzing Firebolt: Catching 0-days as fast as our query processor (/blog/fuzzing-firebolt-catching-0-days-as-fast-as-our-query-processor)
At [Firebolt](https://www.firebolt.io/), we are committed to delivering great software that's devoid of security bugs.
As a cloud data warehouse company, we want to provide our customers with a secure product that they can use without worrying about the safety of their data workloads. However, in the complex world of software development, maintaining robust security is a relentless challenge. More features mean more code, and therefore potential for more bugs. And to consistently weed them out from a rapidly growing codebase such as ours, relying solely on traditional testing methods like static analysis is insufficient.
**This is where fuzz testing comes in.**
In this part of our security blog series, we'll highlight how we fuzz Firebolt's blazing fast query processor written in modern C++- a fast, yet memory unsafe language.
### What is Fuzzing, and why is it important? [#what-is-fuzzing-and-why-is-it-important]
* Dynamic software testing
* Send semi-random inputs to the target binary (most often compiled with a [sanitizer](https://github.com/google/sanitizers/wiki/AddressSanitizer)), hope for a crash, and then analyze the code where sanitizer reported the crash
* Catches bugs that if exploited, can lead to serious security issues. In our case, this could potentially translate to reading cross-tenant query data from adjacent memory, to infiltration and takeover of Firebolt's customer engines via an RCE
There are tons of great blogs on various fuzzers and techniques, so we will limit the details here to showcase how we employ some of them to hunt for 0-days in our product, and secure our customer workloads.
### Our approach (to Fire-Fuzzing) [#our-approach-to-fire-fuzzing]
Catching bugs is like finding a needle in the haystack, so effective fuzzing starts with picking the right targets. Fuzz targets are typically interfaces that directly receive and parse external user inputs. As a distributed data warehouse, a lot of our code is also heavy on parsing. To establish an effective fuzzing process at Firebolt, we identified those critical parts in our software, and tailored respective strategies around them. The following sections dive into some of those details.
**User input enters Firebolt in two main ways**:
* SQL statements (DQL/DML/DDL)
* Customer data [loaded](https://docs.firebolt.io/godocs/Guides/loading-data/loading-data.html) from S3 (using our COPY FROM / EXTERNAL TABLE features)
The first type of user inputs are processed by the **SQL compiler**, while the second type by various **File formatters** (depending on the file type being ingested, eg. CSV/JSON/Parquet).
### SQL compiler [#sql-compiler]
To fuzz the SQL compiler, we run two different frameworks in parallel:
* [AFL++](https://aflplus.plus/) to primarily fuzz the **frontend SQL parser**
* A custom [SQLsmith](https://github.com/anse1/sqlsmith/tree/master) extension (privately maintained) to **stress the later stages**
For AFL++, we maintain a harness code linked against a LocalExecutor library which encapsulates our in-process SQL query processor, seed the fuzzer with a subset of DQL, DML and DDL statements from our continuous SQL test suites, and use its [persistent mode](https://github.com/AFLplusplus/AFLplusplus/blob/stable/instrumentation/README.persistent_mode.md) to call the target function "**executeQuery()**" with the fuzzer generated strings.
Most of the time the fuzzer generates inputs that the SQL parser rejects due to incorrect grammar- and that's okay. Lexer rules are also written to handle invalid SQL strings, so we still manage to crash the target and find memory bugs from time to time such as **double-free** and **heap OOB reads**, especially in classes that deal with invalid SQL error handling.
With **SQLsmith**, we add another dimension to SQL fuzzing, as it can generate grammatically correct statements that the parser accepts, yet still find bugs deeper inside the query processing stack (**Planner, Runtime**).
The upstream version supports PostgreSQL and other PG compliant databases (e.g. sqlite and MonetDB). Since Firebolt strives to maintain SQL dialect compliant with Postgres, we were able to extend SQLsmith with minimal changes by:
* importing upstream under our private GitHub org
* implementing Firebolt classes to connect and fetch schema (**col names**, **data types**)
* overriding the [dut\_base::test()](https://github.com/anse1/sqlsmith/blob/master/dut.hh#L46) function to send SQLsmith generated queries to Firebolt
* catching the HTTP responses (when not a **200 OK**) with SQLsmith's internal error handler, which it uses internally to track error types to mutate AST's and generate fuzzing stats.
Being PG compliant, we were also quickly able to piggyback on the existing implementations for registering most of our supported [SQL functions](https://docs.firebolt.io/sql_reference/functions-reference/functions-reference.html) with the corresponding args and return types.
SQLsmith works by connecting to the target database, fetching schemas, and recursively creating random ASTs with schema info and registered Firebolt SQL functions/operators while keeping the grammar intact. The AST's are then converted back into their equivalent SQL string representation and sent to the target database for execution. If the response returned is logically inconsistent, or the server crashes, it [records a bug](https://github.com/anse1/sqlsmith/wiki#score-list).
Additionally, in order to test boundary conditions and discover what happens when SQL constructs are pushed to their limits (e.g. longest identifier, maximum number of tables in JOIN), we use the [tensile](https://github.com/themosha/tensilelib) library written by our CTO [Mosha](https://www.linkedin.com/in/mosha/). We have extended the language support in tensile for Firebolt specific SQL constructs such as lambda functions.
### File formatters [#file-formatters]
To fuzz file input formatters, we leverage LLVM's in-process fuzzer (**libFuzzer**) and our existing unit test suite (gtest), as reaching that part of the code by simply hammering the frontend parser won't be possible due to the invalidness of the fuzzer generated strings. For e.g., we took our gtest code for the CSV file formatter class, and re-wrote it into a libFuzzer target. To boost fuzzing efficiency, we employ techniques such as code coverage analysis (gcov), and feed off that information to guide our fuzzing campaigns.
If it's not covering enough code, we study the target further to identify all possible execution paths, and the conditions under which execution flows into each of those edges. Thereafter, we take two important measures:
* **Re-write the harness:** We instantiate different instances of the target class with different configurations (e.g. enabling [ALLOW\_COLUMN\_MISMATCH](https://docs.firebolt.io/sql_reference/commands/data-definition/create-external-table.html#csv-types) in one object and disabling it in the other) to cumulatively cover all the blocks and edges.
* **Refine seed corpus:** We seed libFuzzer with valid CSV data that adheres to each of these configurations, allowing the fuzzer to mutate these inputs more effectively and reach deeper code paths.
We also write the harness in a way that covers all data-types currently supported for that file format and seed the corpus accordingly, so that de-serializers for those data types are also fuzzed in the same campaign.
For complex data-types such as arrays, we use structured fuzzing to make sure that our target only processes inputs that are enclosed within **\[ ]**.
Using the above techniques, we managed to discover a critical **heap OOB write** in our CSV input formatter. It is also worth mentioning that for fuzzing with libFuzzer, we use Debug builds that enable assertion checks ([\_LIBCPP\_DEBUG](https://releases.llvm.org/12.0.0/projects/libcxx/docs/DesignDocs/DebugMode.html)) in the standard library, in addition to instrumenting with address sanitizer. This has helped us catch buffer overflows in code while iterating over STL containers such as vectors and maps that even ASan once failed to detect (we tested this against non-Debug ASan build with the same fuzzer input).
### Reproduction, Triaging and Fix [#reproduction-triaging-and-fix]
When we discover bugs, we reproduce them both locally, and in a secure isolated environment without affecting customer workloads in Production. So far, all our findings have crashed the test engines due to illegal memory operations, thereby re-invigorating our faith in fuzzing as a sure path to shipping secure software.
Afterwards, the security research team conducts a detailed crash analysis to determine the root cause of the bug. Once an RCA is done, we follow a structured process to address the issue with engineering teams through Jira tickets.
Additionally, we publish internal research blogs after each finding to help engineering understand why the bug occurred, how it was debugged, and include suggestions to avoid similar logic pitfalls in future iterations.
In most cases, bugs are fixed within a week of discovery, with patches rolled out in subsequent releases cut-off for the Staging environment, where we re-verify the fixes before finally releasing it to Production.
### Fuzzing at scale, LLMs and beyond [#fuzzing-at-scale-llms-and-beyond]
We continuously improve our processes to discover bugs at a pace that keeps up with our engineering velocity. Some of these efforts include fuzzing the latest master branch in CI **three times a day**, with automated crash detection and uploads to a private S3 bucket at the end of each CI run.
Additionally, we've developed internal tools to automate reproduction with the crash data locally before conducting the RCA, thereby reducing significant time and manual efforts in eliminating false positives.
We are further researching the potential application of LLM in our fuzz tests, such as in the area of corpus generation for fuzzing file formatters.
Every new bug that we discover and eliminate brings us one step closer to our commitment to shipping a secure product. And at Firebolt, we enjoy that challenge every single day.
# Getting Rid of Raw Data with Jens Larsson (/blog/getting-rid-of-raw-data-with-jens-larsson)
Why would you create ugly data? According to Jens Larsson, don't even go near raw data. Jens started off at Google, continued to manage data science at Spotify, caught the startup bug at Tink, and recently joined an exciting new company called Ark Kapital, together with Spotify's former VP Analytics. Jens explains how he and his team killed the notion of raw data at Tink and walks us through the Google, Spotify and Ark Kapital data stacks.
Listen on [Spotify](https://open.spotify.com/episode/1li4BomGpzfXrDytERxGbF?si=0iXPB-uRQ96AOV3Df0hxug) or [Apple Podcasts](https://podcasts.apple.com/us/podcast/getting-rid-of-raw-data-with-jens-larsson/id1561927688?i=1000554823929)
Guest: [Jens Larsson](https://www.linkedin.com/search/results/all/?keywords=jens%20larsson\&origin=RICH_QUERY_SUGGESTION\&position=1\&searchId=72d96b66-e8a4-41bd-a5df-91797dae829c\&sid=r7r), Head of Analytics at [Ark Kapital](https://www.arkkapital.com/)
Hosts: Eldad and Boaz Farkash, AKA The Data Bros, CEO and CPO at Firebolt
Boaz: Hello Everybody! Welcome! Eldad, How are you?
Eldad: I am good.
Boaz: Welcome to another episode of the Data Engineering Show. Before we start, I do not know how much time will pass since this will be published. So, if you are listening to this, probably like a few weeks have passed since we recorded it, but I did want to send out our support to our Ukrainian friends, at Firebolt and to the whole of the Ukrainian people with such a situation that is out there and we just wanted to send out our support. Now, we can dive in.
Today with us is Jens Larsson, Head of Analytics at Ark Kapital. Jens has quite a background in analytics in general. He spent his initial years doing analytics at Google. Google was too big for him. He moved to Spotify, which is a little bit smaller, and spent a few years there doing analytics and then moved to FinTech in Sweden in Stockholm, which is where he stayed and recently moved to a startup, which is called Ark Kapital, doing exciting stuff with analytics. How are you, Jens?
Jens: Hey guys, I am doing really, really good.
Boaz: Did I miss anything about your story?
Jens: If by the story, you mean where I have worked, then, I do not think you missed anything. I have not moved around that much. But, yeah, that is above my story. I am from an industrial town in Western Sweden and studied engineering and business. I got my first job at Google in 2011, moved to Dublin, which was exciting, moved back to Sweden, did Spotify just like you said. Then Tink, who recently was acquired by Visa and now Ark Kapital, and I think I am more or less 10x downwards in size, every time I have transitioned a company. So next time I join another company, it will be like 0.2 employees or something like that.
Boaz: And it is also with a pandemic, you are working from home. It seems like you are getting more secluded from society over time and so started to be worried, what's wrong?
Jens: No, that is very much true, though I have to say, since I joined Ark in November, I have actually spent most of my days in the office. It is just so nice to be around good fun people and I have truly missed that over the last two years.
Boaz: How did you get into analytics, to begin with?
Jens: I think it started out at Google. We were working with customers doing kind of sales, support, blogging, and stuff, to essentially help the ad-words business. And, I guess I just had a knack for analytics and somehow was transferred into a local analytics team doing, analyzing the performance of our sales teams, analyzing the performance of our ads customers, and so on, trying to optimize the way we sold ads at Google. I was, of course, super exciting and not so much maybe the problems we were solving, but the people I got to work with and that data stack, I do not think it occurred to me at the time I was fairly junior, and it was my first job, but what Google had already back then in 2011, 2012, must have taken me six or seven years before I got to experience anything like that again somewhere else.
Boaz: Let us talk about that. How was Google as a school for becoming a data person? Tell us about that data stack a little bit more.
Jens: Yeah, absolutely. So, we are talking about 2012, roughly, right? So Google has MapReduce and all of that is already 10 years behind Google or something. But, basically, I started out with a bunch of these patterns that people are talking about just now in terms of, let us do ELT, let us load all our data into a big data warehouse and do transformations in SQL, and I never realized how revolutionary that was, because that is the way I have always worked with data. Data stack at Google back then was all loaded into some distributor file systems and queried through a tool. I think it was called Tenzing. I think it is equivalent to Hive, which was eventually open source to backup it. While I was there, they were starting to roll off Dremel internally, which eventually would become a big query, I guess, and I just remember being blown away by the speed. Everyone has this story about how you start to query across gigs or terabytes of data, and then you would go and get a cup of coffee and you come back to see if it is done. We did that transition there back in 2012 or something like that. It was fun, transitioning, rewriting queries from the Tenzing dialect to big query, took a lot of time, was a lot of headache. I remember Dremel when it came, it did not support joins. Then, eventually, it started supporting joins, but you can only have one join in each query. It was quite painful, but the reward was the speed you got in return.
Boaz: Interesting to think back then, did data engineering exist? What kind of positions were around analytics that helped the end-to-end data flow.
Jens: I had never heard about anyone calling themselves a data engineer back then. We were part of the Google sales and marketing organization. There were not many engineers on the payroll in that part of the company. To me, raw data magically appeared in a bucket somewhere in a file system. And, then I would write all my queries in SQL. I would schedule it through the SQL UI, and I would power some internal dashboarding tool off of the back of that as well, all end to end essentially.
Boaz: On the other end, there is somebody sitting and saying, how ungrateful are these people? They get the data into a bucket.
Eldad: He had a mustache.
Jens: It just magically appears.
Eldad: Did he have a mustache?
Boaz: Probably.
Boaz: How many years did you spend at Google?
Jens: I think it turned out to be like three and a half or something, four maybe.
Boaz: And then you move to Spotify. Tell us about that. What did you do there?
Jens: Yeah. So moved to Spotify and side note, the VP of analytics at Spotify, when I started, is actually my current manager. So, if we skip ahead a little bit on the story, that is why I am at Ark Kapital today, but going back to 2014, I joined Spotify, into their still relatively small and unproven analytics team. We were responsible for everything from calculating the royalty payouts that went out to artists and labels, all the way to understanding how people were doing product analytics, really understanding how people were interacting with the application, to trying and what I spent most of my time, understanding how people were using the free product and eventually upgrading to the premium tier and see what can we do to get more people to upgrade and what different product offerings do we need to offer in order to have that? Because this is back to 2014, Spotify has just gone live in the US, I believe, a reason to go live in the US and there was only one product really and that was the Classic 999 Spotify Premium, but we fairly soon launched students tier, a family plan and so on. So a lot of workaround, how those different products would cannibalize our own users or will they complement our own subscriber base, essentially will we eat their own lunch or will someone else do it for us?
Eldad: So, you switched the metadata to playlists, songs.
Jens: Yes, absolutely.
Eldad: Stopped listening after x seconds, time becomes like 2 minutes, 3 minutes when the queen is being played, maybe 10 minutes.
Jens: Yeah, we did a lot of those fun little experiments because we had all this metadata about when songs are being streamed. We could basically detect when a local football team had won a game because the "we are the champion" song would spike in that region when that happened. There were a lot of these kinds of fun things you can do. On the other hand, we were quite limited by our capabilities. I think we used to at least tell ourselves that we had the world's largest Hadoop cluster at that time to process all this data. That does not mean it was very fast for the kinds of analysis we tried to do. That is why I said, it was like going back in time, I had just helped our team at Google migrate off of Tenzing and now, we come back to a Hadoop cluster where people are still writing MapReduce jobs in Python.
Eldad: Progress.
Boaz: The years you spent at Spotify were years of crazy growth, right? How did the increase in scale feel like from your end?
Jens: Yeah, it was pretty intense. We must have gone from 600, 700 to 5000 to 6000 employees over those four years or five years, maybe. Scaling was pretty intense and in particular around the analytics team, and I think analytics and data engineering were some of the areas that really had to scale and scale pretty fast. And in this time span, we also moved from this massive Hadoop cluster into the Google cloud, the world of data services, basically every service we are running on GCP eventually.
Boaz: Do you guys move completely to GCP or was it sort of that you have things in AWS side-by-side?
Jens: I know that we were a few things still running on AWS. The Hadoop, it took a while to properly kill off the Hadoop cluster. But yeah, I think the end goal is everything was moved into GCS.
Boaz: So from 1 to 10, how much do you miss the Hadoop days?
Jens: I have this kind of romantic view of these MapReduce jobs we used to write. I love just going really deep and optimizing the combiners, and figuring out how to avoid an unnecessary shuffling step between jobs and stuff like that. I kind of miss that. I also do not miss that because it was taking way too much of my time. But, it gives you this opportunity to feel really smart about the work you are doing.
Boaz: Yeah and now you look at the younglings of today who do not have to worry about these things.
Jens: Exactly. It is all just drag and drop this and trying to click boxes.
Boaz: Okay, you spent all the time Google, Spotify, learning how to work with some of the world's most complex systems and data sets and then you went on to the startup world, right? Move to FinTech.
Jens: Moved to FinTech to Tink. This was back in 2019 or early 2020. They had just done a pivot from being this business to consumer personal finance management application. They had kind of been a driving force in creating the whole world of open banking, forcing banks to open up APIs. Long story short, Tink started fetching data from banks, closed APIs before there was a mandate, whether banks had to open up or not and the banks deemed it was probably legal. Then the courts realized, no, it is not illegal. It is perfectly illegal and we are actually going to create these mandates that force banks to open up APIs. Tink was really early in that journey, but we are building most of their own applications. When I joined, the company completely pivoted into becoming this platform as a service or API as a service. That kind of unbundled the app fee functionality and features and sold them off to other FinTechs and sold them off to other banks. So, we did things like connecting to bank accounts and fetching all the data so that you can do various analyses, risk analyses, or actually building personal finance management. We were categorizing transactions using various AI machine learning models to figure out all these line items on your balance sheet, on your transaction sheet, what are they actually, which is a surprisingly hard problem because the banks do not include much data in those transaction lines. It is usually like just mumbo-jumbo when trying to read it, sometimes you can read like an MCD and figure out it is probably a McDonald's transaction, but that is basically all you get.
Boaz: And so the analytic stack there was part of the product, the services that the clients were consuming, went through, your stack.
Jens: Yes. The entire backend of Tink is a data platform in a way and it is completely homegrown. It is using things like Kafka and S3 and Cassandra and various databases, but it is all focused on productionizing data access to customers. What I was in charge of building was really this data warehouse, where we could learn ourselves, how our products are actually being used? Instrumenting everything from event collection so that we know what our systems are doing, to how our customers are using our systems, and also some batch processing to get the data back out of Cassandra because Cassandra is not a very nice place to run massive queries over.
Boaz: Can you walk us through the different teams in charge of the different data deliverables. You run the analytics department. There is data engineering, engineering around it. How do all of you interact in terms of responsibilities around the data platform and stack?
Jens: Yeah, so when I was at Tink, I was heading up both the data engineering side and the product analytics side. Most of the analytics we did were product-focused. And most of the data engineering we did was focused on getting this metric data, getting data out of the platform that allowed us to do analysis, create KPIs and metrics, and so on. We were, of course, leveraging a lot of the infrastructure that other teams were building at the company. So, the data engineers were heavily dependent on our infrastructure team that managed not one, but I think 11 Kubernetes clusters in various AWS and on-prem and so on, instances to create these environments that we provided out to our customers. It was an extremely complex setup in a complex environment but the data engineering team had to figure out ways to kind of standardize how we source data from all these different systems and how we put them in a unified data warehouse model.
Boaz: And what did the data stack look like? What data warehouse did you guys use and what was around it?
Jens: Yes, for much of the data, we were kind of bound to you saying, like AWS tooling. So, we were using S3 and Athena. We tried a little bit of Redshift and so on, on that. But for the data where we had anonymized it, taken out any sensitive bits of information, we moved a lot of that over to Google cloud, to use that to power things like interactive dashboards, metric collection. We even used that power, the developer console will be feeding some of these metrics back to our developers that we're building stuff on the platform.
Boaz: In retrospect, you spent a fair amount of time there building that. What would you have done differently if you were to restart that entire journey? What lessons were learned? What could have been avoided?
Jens: What could have been avoided? So much pain could have been avoided. I think what took the most time was to figure out which data we are allowed to do work with? Because you have to realize, Tink is a data sub-process around the GDPR. Does not really own the data that flows through the platform. It is processed there on behalf of someone else and at the end of the day, it is on behalf of the end-user, the person is actual financial data wizes. So, we need to make sure that the data we look at and analyze is only metadata about whether or not someone has to aggregate the data, not the actual aggregate data. And, I think if I were to start this over again, we should have created much, much clearer vocabulary around this really early on in the process, because we could have just avoided so much back and forth discussing security and anonymization and legal. If we would have just said, this data is metadata, about how Tink services are being used, we do not even need to bring that into the discussion and we could have limited the scope drastically. And, we could have probably been more proactive in how we set up contracts with customers to allow that. But yeah, at the end of the day, I think we would have made better progress if we would have been more upfront with figuring out what are the different classes of data and the different use cases we have of that data and realized that we do not have to enforce the strictest restrictions on all of it, because what would you do when there is uncertainty and unclarity in this case, what you do is you apply the strictest rules across all the data, and that kind of also limits what you can do with the data. And it creates a lot of headaches for people trying to do stuff that we have designed, that you cannot do for good reason. And, then you try to approximate or you try to not necessarily sidestep, and not necessarily work yourself through limits and barriers, but you try to create the proxies, try to make an estimate of a metric that you actually could have probably just queried if we had clearer, the clearer delineation between what data is sensitive, which data is not sensitive.
Boaz: Yeah. I guess, as you noted, it is not the kind of thing that you thought would be critical to your work when you first started off at Google. You thought one day I will be regretting, not having planned enough, not having thought about what data I am allowed to keep and what not. But it is true at the end of the day, these things bring the entire difference in the project with a lot of headache or less.
Jens: Yeah, absolutely. Then, there were a lot of good things we did there. One thing we did is we more or less killed off the notion of raw data, because with the kind of idea that there is no such thing as raw data. We are not like pumping raw and crude oil out of the ground and then we have to refine it. Data is something that we control and that we create. We could have probably created crude oil and then, we would have created a process to refine it. But, instead, we decided to create nice tabular data with strictly enforced schemers and contracts from the get-go. And a big chunk of our data engineering work we just never had to do because the data that is streamed in from all these other services, was already pretty clean when it came to it.
Boaz: That is now an interesting philosophy that I never heard anybody articulate that well. But saying, data essentially, it is not born out of a vacuum.
Jens: It is not. It is our system. It is our source code that creates the data. Why would we create ugly data? And, I think one of the reasons why you would create the ugly data is because you created it for a different purpose like you created this data because it is a log record that you want to feed into Logstash and do debugging. We had this alert when you set up this system, you are like, either we tried to build something brand new or we go with what we already have and we tried to retrofit it like a data platform on top of these Logstash data that we had available to us. And, thanks to our lead data engineer who had spent many years also at Spotify and other companies in the past. He was heavily advising against trying to retrofit and repurpose that log data and, actually now let us build a service that we speak through with proto buff, schemas, and contractually sound data. Then, we can stay in control of that data throughout the whole chain.
Boaz: So, the message may be - Let us not be victims of our raw data. Let us take control. You can actually change it up.
Jens: Yeah. Do not even go near the raw data. If you have the opportunity to say no, let us build a parallel system that creates nice data. Do that instead, I think.
Boaz: It is time to question your raw data. So, tell us about Ark Kapital. So, then you decided to join, like you said, your former boss at Spotify. What do you guys do at Ark Kapital? What is the mission there?
Jens: Yeah. So, it is really two-fold and I am obviously most excited about the data part of this. So, I might not give the most flamboyant or elaborate description of the other. But I will start with a "boring part" which is we are lending money to companies to finance tech companies and other modern companies. We do this as an alternative to bringing in say, venture capital and the idea is your current options as a company looking for capital for growth is either you have assets or you have some other security that you can use to get the loan or you go to venture capital and you give away part of your company in exchange for that funding that you need to grow and both of those have a place. If you own property or buildings or whatever, go to the bank and get a loan on those. If your business is not proven, you may not have found product-market fit and so on, go to venture capital. They are perfect for handling that risk. That is exactly what they are experts at. But, then we see all these other companies, they might be too new or they might not be established enough to really go to the bank and get money, but they might also have very predictable growth. They know that for every $10 they spend on marketing, they get $25 back in terms of sales. If you are in that position, you do not necessarily want to give away your company to a venture capitalist either. And, that is kind of where we come in. The reason they do not go to the bank is that the bank does not have access to the information really. They can look at your annual reports from a couple of years ago. If you are a fast-growing company, that is not going to work out. It is really hard to model this and excel in a way that allows you to really understand what is going on. That is where our data platform comes in. So, most of these modern companies are building their organizations on SAS platforms. Modern e-commerce companies built on Shopify or Instacart or any other of those services. They do their advertising on Facebook with Facebook ads or Google ad words. They do their bookkeeping and CRO and so on. And, we connect to all of these systems. We are using Fivetran Airbyte and other services to connect straight to the system and get the source data. Then, we apply models who we have years and years of experience in business analysis and it comes from a VC firm before joining Ark. The rest of us are quite seasoned analysts. We could analyze the business performance using all these different data from all these different sources, and we apply machine learning and we do forecasting to try to really get an idea of where this company is heading. And, based on that, we can really tailor the financing option to that company. So, we call it precision financing, where we learned so much about the company from these sources, and hence, we can tailor financing solutions to them. But, in order for us to kind of digest this, all this data and all this information, we are also building really cool dashboards for ourselves that we are also providing back to our customers. So, we are really building a turnkey solution where you connect your data, and we give you a best-in-class business dashboard with KPIs and forecasts and LTV models and all of that nice stuff that all the big players already have. So, I think of it as a bit of an analogy when you buy data tools these days. You buy a data warehouse, you spin it up and it is empty or you buy Looker and you get this dashboarding solution and you open it for the first time and you are greeted with a blank empty page, which I think is a quite boring way to buy data products. Ideally, more and more products in the future, I hope, will be like turnkey. You buy the product, you authenticate to get your data and then you open it and there is an actual dashboard there already.
Boaz: Yeah. Beautiful. This is an essence you are entering Ark Kapital with a blank slate. Please share, how do you go about implementing the analytic stack, with your experience, but now have the complete freedom of choice?
Jens: Yeah, with the complete freedom of choice. There are many of us in this company that have a lot of experience with GCP. We have decided we are going with GCP. Then, we have all this low-hanging fruit. There are all these kinds of companies that help make our lives easier. One of them is DBT. I have built tooling that works pretty much like DBT several times before, but it is quite nice to just install it, get it off the shelf. We get Fivetran, Airbyte which allows us to connect to all these different APIs. So, we are basically collecting this different software that helps us kind of do our job. But we are also taking quite a lot of time to figure out exactly what that platform is going to look like. There are so many things that we still have not decided on. For instance, orchestration, one of our big headaches. I have heard it on this podcast too, people talking about how much time they are spending, just managing the Airflow instance and trying to upgrade it, and so on. Trying to figure out what is the modern way of orchestration of all these different data pipelines, because our complexity does not really lie in the volumes of data. It is the diversity of data. We were fetching it from hundreds of platforms for hundreds, or potentially thousands of customers and we cannot really, as a colleague told me yesterday, who had been working at Spotify with similar problems in the past, we cannot really take a representative of Google analytics and put them in the room with one from Mixpanel and tell them to start aligning their data models, "Hey guys, can you please start defining amu the same way."
Eldad: Let us join everything together.
Jens: Let us join everything together. Can't you guys just agree on what the daily active user is? So, I do not have to kind of show diverging definitions of the same data. There is a lot of complexity that stems from just the fact that all the data is different that comes from various sources and yeah, the complexities are hard to grasp actually.
Boaz: How do you imagine things looking a year from now? What do you want to look back on with pride having built?
Jens: Year from now, I really, really want as much as possible to be automated, and we have this idea that a customer that comes to us and opens their dashboard for the first time and sees their metric, they are kind of blown away by how rich it is and the insights they are getting. We are currently achieving that, but with quite a bit of manual labor in between, and I mean, creating good visualizations of data or standardizing it is a craft and it is quite hard to automate and do that at scale. So, I am really hoping that a year from now, someone just goes and connects their data to our platform, and they are more or less immediately blown away by the insight that they are able to get. And, hopefully recognizing the numbers they see in our platform from what they have in their own spreadsheets and so.
Boaz: Awesome! This is great Jens! I really appreciate it.
Eldad: Yes.
Jens: Thank you, guys!
Boaz: It was great talking to you, a super interesting, amazing journey, amazing challenges in the past, and amazing challenges you are working on right now. Thanks again!
Jens: Yeah, thank you too.
Eldad: It was great to connect. Thanks.
Boaz: All the best Jens.
# GROUPING SETS as a pure planner rewrite ? Yep - it's possible (/blog/grouping-sets-as-a-pure-planner-rewrite-yep---its-possible)
In this blog post you will learn how GROUPING SETS work and how Firebolt's implementation uses smart query planning to execute them efficiently.
## Introduction - What GROUPING SETS do for you [#introduction---what-grouping-sets-do-for-you]
When working with SQL queries, reporting and analytics often require summarizing the same data in multiple different ways. Let's say we have the following table tracking orders of our products:
We might be interested in multiple aggregations here:
1. the total sales amount per region and product
2. the total sales amount per region
3. the total sales amount over all orders.
Usually we would cover any of these points with a GROUP BY statement. For example, to get the total sales amount per region and product we would query the data with
```sql
SELECT
region, product, SUM(amount) AS total_sales
FROM
orders
GROUP BY
region, product;
```
But how do we get all three aggregations in our query? Instead of writing multiple GROUP BY statements, SQL provides a powerful feature called GROUPING SETS, which allows for multiple aggregations in one go:
```sql
SELECT
region, product, SUM(sales_amount) AS total_sales
FROM
orders
GROUP BY GROUPING SETS (
(region, product), -- Sales per region and product
(region), -- Sales per region
() -- Grand total sales
);
```
With this query, the result table will look something like this:
We just did three aggregations in one swoop: The first four rows are the result of the aggregation with both region and product, the next two rows aggregate only by the region, and the last row corresponds to the empty grouping set (also called *grand total*), that evaluates the aggregate function without a GROUP BY key. Columns that are not part of the grouping set of the current aggregation are set to NULL.
## Rewriting GROUPING SETS [#rewriting-grouping-sets]
So we have seen that GROUPING SETS can be used to request the same aggregates for different sets of grouping keys in a concise way. Imagine you were confronted with a SQL database that does not support GROUPING SETS syntax. The first idea you might have is to write a query like this:
```sql
SELECT region, product, SUM(amount) AS total_sales
FROM orders
GROUP BY region, product
UNION ALL
SELECT region, NULL, SUM(amount) AS total_sales
FROM orders
GROUP BY region
UNION ALL
SELECT NULL, NULL, SUM(amount) AS total_sales
FROM orders
```
The output would be the same, but the query is much more verbose. But maybe one could implement GROUPING SETS this way? Just rewrite any statement containing GROUPING SETS into one that the runtime can already execute. The GROUPING SETS would only exist at the query planner level and would internally be transformed.
But using UNION ALL for this is probably not a good idea. We want to have as few scans over the base table as possible so we only have to read it in once. With this approach however we would have one scan per grouping set. That corresponds to the FROM orders part of the query. The amount of scans over our base table therefore increases linearly with the amount of grouping sets we have. This gets even worse when you calculate your grouping set over more than just a base table. Imagine you compute grouping sets on the result of joining terabytes of data. The UNION ALL strategy would mean that you need to evaluate that very expensive join once for each grouping set.
But there is a smarter way of rewriting a query with GROUPING SETS. Let's take a look:
```sql
SELECT
case when get_bit(gs.id, 0)
then region
else NULL end as region,
case when get_bit(gs.id, 1)
then product
else NULL end as product,
SUM(amount)
FROM
orders,
UNNEST(ARRAY[3, 1]) as grouping_sets(g_id)
GROUP BY
grouping_id, 1, 2
UNION ALL
SELECT
NULL as region,
NULL as product,
SUM(amount)
FROM
orders
```
What happens here? Let's focus on the first SELECT for now. In the FROM clause, we not only scan our base table, but also construct a cross join with a static table grouping\_sets that contains a single g\_id column that encodes the columns participating in the requested grouping set. Each input tuple from table orders is paired with each g\_id value. Our example table after this cross join would look like this:
The values of the g\_id column represent a binary encoding of the different combinations of the expressions passed as arguments to the GROUPING SETS clause. In our case, the two columns used in the GROUP BY clause are "region" and "product". Hence, we have two binary positions that can either be set to 1, if the column is used in the grouping set, or to 0 otherwise. This way, we can distinguish the different grouping sets and do a GROUP BY that covers all of them. For instance, in our example a g\_id value of 3 (binary 0b11) denotes the grouping set that includes both columns, while a g\_id value of 1 (binary 0b01) denotes the grouping set containing only the "region" column. Consequently, in the SELECT clause, we can retain the original column value if the corresponding g\_id bit is 1, and map it to NULL if that bit is 0 (we can check this with the get\_bit function). The aggregate input of the SELECT statement from our example query looks like that.
The GROUP BY clause uses the g\_id and the possibly NULL-mapped region and product columns as the key.
The special case of an empty input requires grand totals to be handled separately. This is because a GROUP BY clause with an empty grouping key should return the default NULL value for each aggregate function, rather than an empty result.
What's so great about this rewrite is that we can compute any amount of grouping sets with at most two scans and two aggregations. And by having one larger aggregation instead of multiple smaller ones we can take full advantage of Firebolt's distributed aggregation computation.
## How GROUPING SETS are transformed from start to finish [#how-grouping-sets-are-transformed-from-start-to-finish]
After the parsing stage, a GROUP BY clause that utilizes GROUPING SETS is internally represented as a special aggregation node within the query plan. This node stores all the information pertaining to the grouping sets by using bitsets, which function similarly to the g\_id values in the rewritten SQL query.
This special node only exists at the beginning of the logical query plan. This logical plan is an abstract representation of our query that we can use to perform optimizations using rules. The rules run over the tree and try to find patterns that they can optimize. In our case we use a rule that matches on this special aggregation node and transforms it subtree implementing the rewrite strategy outlined as a SQL-level rewrite above. The resulting plan consists of nodes that can already be executed by our runtime, so no runtime extensions are needed for adding support for GROUPING SETS.
Let's examine what the rule does in detail.
1. Create the grouping\_sets(g\_id) static table and the cross join. We can just interpret the bitsets in our special aggregation node as integers and build a table this way.
2. Build a projection that optionally maps grouping set expressions to NULL depending on the current g\_id value. In addition, the projection retains the g\_id value and all aggregate input columns.
3. Add an aggregation node
4. Do the same thing, but for the grand total (if requested in the SQL query) and combine both paths
5. Remove the g\_id with another projection
And just like this, we transformed our GROUPING SETS aggregate node into an equivalent logical plan that can be further optimized and later executed.
There is one more edge case that we need to cover in the query planner: duplicate GROUPING SETS. A query like
```sql
SELECT
region,
prodcut,
SUM(amount) as total_sales
FROM
orders
GROUP BY
GROUPING SETS ((region,product),(region,product))
```
has the grouping set (region, product) repeated twice. This leads to the aggregation result being duplicated:
We want to compute the aggregation without the duplicates first, and then duplicate the rows later, when we already have the correct aggregation result. That means our cross join would look like this:
Even though we have two grouping sets that correspond to g\_id = 3 we only have one 3 in the static table. We only consider the duplicates later, by adding another join on top with a static table that keeps the multiplicity:
With this setup, we make sure to compute the GROUP BY both correctly and efficiently—no matter how often a grouping set appears in the GROUPING SETS clause, all aggregate functions are only computed once per distinct grouping set.
### The grouping function [#the-grouping-function]
You might have noticed a problem with GROUPING SETS so far: The data might contain NULL values, so how do we differentiate between them and the NULL values added by the aggregation? SQL also offers a function for that, called *grouping*. Similar to our g\_id, the function takes expressions found in any grouping set as arguments and produces an integer that represents each expression as a single bit. In this case, a 0 bit in the integer indicates that the expression is part of the aggregation.
If our initial query had utilized the grouping function:
```sql
SELECT
grouping(region, product) AS grouping,
region, product,
SUM(amount) AS total_sales
FROM
orders
GROUP BY
GROUPING SETS ((region, product), region, ())
```
The result would look like this:
Grouping functions are also evaluated during planning—for each row the correct value is computed based on the current g\_id value and the arguments of the grouping function call. The values can then later be joined and added to the result.
## Conclusion [#conclusion]
**GROUPING SETS** offer a powerful way to perform multiple aggregations in a single SQL query, improving readability and conciseness. Firebolt's implementation leverages smart query planning to efficiently execute these aggregations by internally rewriting them. When transforming **GROUPING SETS** into a series of joins and aggregations, Firebolt minimizes table scans and optimizes distributed computation, ensuring high performance even with intricate grouping requirements.
# How Amplitude Engineers Process 5 Trillion Real-time Events (/blog/how-amplitude-engineers-process-5-trillion-real-time-events)
Weichen Wang, Senior Engineering Manager at Amplitude, came to meet the bros to talk about Amplitude's cutting-edge data stack and how it processes 5 Trillion real-time events while dealing with mutable data and massive scale.
Listen on [Spotify](https://open.spotify.com/episode/3o0GCNoxaq1g5Dlv65PFW3) or [Apple Podcasts](https://podcasts.apple.com/us/podcast/the-data-engineering-show/id1561927688?uo=4)
**Benjamin:** Welcome back everyone! Today, both Eldad and I are in Tel Aviv. We have the pleasure of having, Weichen Wang, here, who's a Senior Engineering Manager at Amplitude.
A few words about Amplitude and Weichen. Amplitude is actually at this point a public company. Before that they raised more than 300 million in private rounds and Weichen is based in Vancouver, British Columbia. He leads the data connections team there and built on scale, their Vancouver site. So, we're super happy to have you. Do you want to give us a high-level view of what Amplitude does and what service you guys deliver to your customers?
**Weichen:** Sure. First of all, I just want to thank you for having me today, super thrilled! My name is Weichen. Thanks for the introduction!
Maybe I can talk about Amplitude, the team that I'm working with.
For Amplitude, we are a digital analytical platform. We started serving the customers for product mix. We're giving this self-service visibility into the entire customer journey. We have all those tools for funnel analysis, behavior graphs, event segmentation, and all of that. The key takeaway here is we try to help the entire spectrum of data collection, instrumentation, governance, and all to get insight and sharing out. So, it's kind of the one-in-all solution, if you may.
For the team, I'm working on - We are the data connection team and as the name suggests, our mission is to send the data in and out to the right place, at the right time. If you think about the entire space out there, all different sorts of services, including data warehouse, ads, marketing, CRM, and CDP. Sometimes, the data has to be sent in real-time, when time is of the essence. Either time, the data will be sent in batches of different natures. So, that's where our team is building this pipeline and also the tools, and the user interface for our customers to handle those data.
**Benjamin:** All right, sounds awesome. We'll have a lot to talk about. How big is your team and data connectors, can you give some examples of...?
**Eldad:** What this means?
**Benjamin:** Exactly. Like, where's the data coming from? Where's it going?
**Weichen:** If you talk about the Amplitude as a whole, we're about 700 people altogether. And for engineering, I would say it's less than 200 at the moment. This is still a fair small team at the moment. Data collection is roughly less than 30 people, I say altogether. So again, it's a fairly small team. You want an example of the connections. I already named just a few of the categories. In the different places people want to bring data in and these years, the data warehouse has been a really prevalent choice where they served as the first step in all the data, in relation to the governance and everything, and this server, the source of choice. Where in other cases, some clients might prefer trying to get data indirectly from instrumentation, using our SD case and sending directly from the devices or websites. So, whether data hit to us directly in the streaming fashion. So, that's already a different type of ingestion.
When it comes to exports, they are different, again, for example, sometimes people use us at a CDP capacity where they do real-time event propagation. As soon as we receive the data, we probably send them to their data warehouse or anything else. And in the other case maybe they generate insights, for example. They might do queries on user segmentations with audiences and this is a new group of users. They want to start a new campaign, ads. Then they actually want to sync that audience to Google Ads, for example. That's another problem. So, we're trying to support as much as we can at the moment, about a hundred sourcing destinations.
**Benjamin:** Okay. Awesome. Can you give us a feeling of the data sizes or the data volume you guys are dealing with?
**Weichen:** Yeah. We measure the data that is ingested in a number of events. Mostly, we process somewhere around 5 trillion events altogether and most of them at moment is real-time streaming. But we also are ingesting lots of batch data as well.
**Eldad:** 5 trillion is nice. Very nice.
**Benjamin:** Yeah, definitely.
**Eldad:** Is it run on a daily basis or a monthly one?
**Weichen:** No, that's a monthly basis.
**Benjamin:** Awesome. You manage to connect your team, and this is where you have the most experience, but do you want to give us a high-level overview of the internal data stack at Amplitude and where the data goes after it made it through your connector?
**Weichen:** Sure. So, I can give you a high-level architectural overview, although I'm trying to not disclose too much information, that we have been talking about.
**Benjamin:** Of course.
**Weichen:** The most important part was when Amplitude started, there were a few principles that when came, and again, this came from one of the early engineers and the founders and I joined at a much later time. So, one of the key points here is we want to make the data query super fast.
Even with the real-time data coming in, how can you update existing queries in a real-time with as lower latency as possible? So, basically, they chose a column, the storage architecture. Cause you think about other queries you could not make, especially for product mix. So, making like say DAUs or other things, users who did this action after that action usually with just one or two properties that you are interested in among all the event property or user properties for the data. So when you index things in the column, things get you professed. We actually have it in-house column storage system. It's proprietary and at the end of the day, it just stored the raw events and then we can index them in a way. Then, we have a query system we call Nova, that actually builds on top of it. So, it's actually working in a kind of a divide and conquer fashion where it tries to utilize a cluster of computing resources based on those indexes so that you can get a query out super fast. So that's kind of the essence of the system.
For the streaming, of course, Kafka and various stuff. We also use Postgres, lots of things. We started with MySQL, but we're trying to move away from them. And then, everything was on AWS. So, we have to use an S3 for storing those raw indexes and events. So that's probably the most important thing on a high level. But of course, the bunch of things down the roof, like we were Redis with a bunch of other stuff.
**Benjamin:** Got you. Okay.
**Eldad:** Who should take credit for inventing columnar storage and databases?
**Benjamin:** Boom. Okay. Asking you database research questions to me.
**Eldad:** Is it the Stanford guys or is it the CWI?
**Benjamin:** That's a good question.
**Eldad:** Is it Store or is it MonetDB?
**Benjamin:** So as someone from Central Europe, let's give it to Amsterdam.
**Eldad:** Boom! Amsterdam wins.
**Benjamin:** Amsterdam wins, easy, but I mean, that's super cool. So, basically, at least at the core of your stack, you then have this in-house query engine in a sense. Like it's not talking SQL, but this domain-specific language that actually powers all of the stacks. That's actually super interesting. Cool.
Now, what's the hardest thing about your internal data stack? Is it lower latency? Is it high concurrency? What are the biggest data challenges in a sense you guys are dealing with and trying to give your users a great experience?
**Weichen:** If I put it in words, it's just a scale. Everything was easy when the scale is small. But as a lot of demand comes in, even the original architecture would not scale, even the actual feasibility when the query span you cannot scale vertically, just a number of open clusters and all that. And now the other part is cost.
For real-time, it all sounds really good, but of course every time there is trade-off. It's an intentional design that we have, but then that also actually has a recurring burden on us to make everything super fast. Sometimes, there are use cases. I don't know if you actually order food, like with DoorDash. So, maybe you put something in your shopping bucket, but you didn't check it out and you move away. Maybe the DoorDash, hey, you want to send the coupon or push right way in 30 seconds before you gave up and try to finish that order that is super time sensitive. So that's why we need that to be really fast responding. In other cases, that itself, maybe in the monthly data of your inventory and stuff, there's no sense of time, urgency, and that cost is kind of actual.
The other example is data mutability. If you think about the event ingested from an instrumentation SD case, most of you think, oh, the data is mutable and which is the assumption that the system was built before. But there are lots of cases where that assumption is not true. The biggest example is just GDPR. The data deletion request where they actually remove every trace of certain users and that is a big pain point where we had. We actually have a dedicated team to handle that.
**Eldad:** Thank you for that.
**Weichen:** Yeah. Then we are of course rethinking our architecture.
**Eldad:** Also European.
**Benjamin:** Exactly. Like a European, you take credit for the second time.
**Eldad:** Sure, the delete feature in the database and data warehouse.
**Weichen:** If you ever doubt what was that, it's, a definite challenge if it was not designed, to begin with.
**Eldad:** I remember that period when delete was dead and everyone was happy with the pend only, immutable. And there were these two years where everyone was happy, and life was presumably becoming simple. And then, GDPR came and everyone woke up.
Tell me, ecosystem-wise, input, output, you said Kafka and Postgres. We get that a lot. What do you see? Do you see more Kafka or more Postgres, one overtaking the other? We see a lot of comeback of Postgres. What's your take on that?
**Weichen:** Kafka is mostly for the streaming process. Postgres, which we use primarily for storing metadata. Because we have already our separate column indexing system, so Postgres is not used for that purpose. I actually was listening to some of your shows, and lots of people are coming from data engineering backgrounds that they're familiar with just the SQL or just relational database kind of thing. For us, we started with the streaming perspective where the data is coming in and we see every single stream, whereas the batch process or is actually coming in a different stage where we have a separate architectural fit where the system was not designed, to begin with to handle that. But, nevertheless, I think there are benefits that we're designing at a later stage. That means we learn the use cases. So that we know how to optimize that. So back to your question, and Postgres is a popular database. Yes! and we use it all the time. There's a reason that we move it away from MySQL, I suppose. At the same time, they are just serving a different purpose.
**Eldad:** What about output? How do you export your data? Do you see a lot of customers using your platform as a way to cleanse and prepare and then share data with others? How does it play with the existing data? Is it the Snowflakes of the world?
**Weichen:** That's an interesting one. There are a few parts to it. The first is just data governance. There are again, multiple, parts of it. But, you know, I think one phrase I really like is "garbage in garbage out." You have to really control your data ingestion. If you think about all the instrumentation or tools out there, there are certain solutions that actually offer basically with no instrumentation, that's basically auto-tracking and that tracks every button click, every event you get from the app. Of course, that's easy to set up, but then there's a time we had to dig really deep. It's like, imagine you're in this pile of garbage and trying to figure out the gold in it. So it's time-consuming. It may work.
Where's the other part, which is basically what Amplitude, at least at the current stance is we try to do tracking plans and then we try to do versioning of that. So, there are lots of tools we build so that you have a really protected data schema that actually has infrastructure shared across the entire organization. All the teams will actually benefit from it and though it might sound a lot of overhead to start with, and we'll try to make it easier for the small but low-maturity customers. But if you are actually the bigger customers out there, it's actually beneficial to start with, especially to basically make peace of mind for the data leaders. That's one part.
The other part, how we enhance the existing raw events with all the user properties. And then there's the part, we actually try to send it out, in order to touch a little bit, there's a real-time propagation we call stream. So whereas the other, batch export and other time, we send out insights you generated.
So how does that reconcile to the source of truth? I think this is a great question. I guess nowadays, because, Snowflake or the other data route is becoming prevalent, our mission is so that people don't think about that source of choice problem.
**Eldad:** People should always think, I'm telling you, I've learned that. You have been trying to build tools, so people don't need to think and those projects always ended up badly. But, first, your passion is amazing about your product.
**Weichen:** Thank you.
**Eldad:** We love that and listening to you over a few minutes, you first learn how complex it gets when you move from theory to real life, and try to practice data to drive your business, and this is why all of these little things are so, so important.
Tell me, how do you deal with data politics internally? Your team, would that be considered an engineering, a data engineering, a data team? How do they work with the rest of the engineering organizations? What can you tell us about remote and the evolution over the last few years? And then some lessons learned, maybe.
**Benjamin:** Tell us everything.
**Weichen:** You've asked multiple questions, I'm hearing too. Wonders just the internal data policy, it's one. The two is going to be working remotely.
I will start with the first one. Amplitude is coming with a unique edge because we are a data analytics company. Drinking our own champagne is actually a motto for us. We hold ourselves accountable whenever we release something, even if a new feature comes to Amplitude, we actually be the first ones trying it out. If you are shipping a custom-facing feature, you better be using, AB testing experimentation. You better use feature flags and all the other instrumentation in place so people know from, let's say you launch a new, export, the destination of data. So, how many actually click on it? What's the funnel from the user clicking and actually finishing setting up the configuration with all credentials and actually getting data through? So, we have all this tracking and we held ourselves accountable. It's never we shipping features out there. So, in the sense, everyone is a data engineer or a gross engineer so to speak.
I think that definitely provides us with a unique edge, that's one.
Two, just on a remote. I am remote in a sense because Amplitude headquarter is in San Fransico. It's a blessing to me because I was really looking for it. I didn't even know what Amplitude is before someone reached out to me and they opened their office here. So I was hired as one of the earliest managers here without an engineer. And the lot of work I was doing last year is just building the team here.
**Eldad:** Drinking champagne and drinking a lot of ...
**Weichen:** Hopefully, yes.
**Benjamin:** Much, nicer than dogfooding drinking. I would much rather drink...
**Eldad:** He talks and drinks water here. But, some companies are more involved.
**Weichen:** It's just a different way of putting it, but you got it what it is. So working remotely, I guess for us, it's lots of internal debate as well. We at Amplitude want to position ourselves because when Covid started, the situation was staying there. So there is a new reality and a new paradigm that would never go back. Certain, people would just prefer going remote 100%, and they are not even considered working, being the hybrid. We actually had an internal vote and we had an internal discussion.
At this moment, Amplitude is operating in the hybrid remote where we require engineers to be in the office two days a week. We'll still cherish that in-person time, especially with collaboration that's a lot of quality time when brainstorming or face-to-face, which cannot be replaced in my opinion. But yes, there's the time when people can definitely get focused on what they don't want to get disrupted with meetings and stuff.
So, that's where we are at right now. Of course, there's also another debate about whether, for example, the teams should be clustered, geographically. For example, maybe all the teams in Vancouver focus on data connections, and, maybe, teams in London work on something else. So, I don't think that the debate is decisive.
My personal opinion is I do not think geographic location should be a parameter in that equation at all. But you know those people who think otherwise because I do believe that having that flexibility to tap into the talent pool globally definitely provides us a unique edge and the challenge coming with managing a team, it's something that can be managed in my opinion.
**Eldad:** You see Benjamin, it can be done.
**Benjamin:** It can be done. In terms of your team, where are people actually located?
**Weichen:** At the moment, maybe I have to also dive into the kind of a bit of structure, so we run the EPD trail. So, Engineering, Product, and Design. We have pillars and parts. So in data connection, we have three parts and I run two of them. So the streaming and the integrations. For integrations, the majority of the engineers is in Vancouver. For streaming, is probably half-half.
**Eldad:** Tell us about the most embarrassing failure you've had and something that was a big lesson for you. And answering, I'm perfect, I've never made mistake in my life is also valuable.
**Benjamin:** I told you that in interviews, Eldad.
**Eldad:** That's what Benjamin answered in his interview, but we proved him wrong many times. That's why we went on this adventure.
But data teams, we talked to many teams, engineering teams, and when it comes to data, people make a lot of mistakes. So I was just wondering any project that went the wrong direction, any technology that had huge promise and turned out to be a huge flop, something smart for our listeners to learn from in terms of past experiences, the good or the bad. Usually, we love the bad ones.
**Weichen:** Okay. However, I like to share lots of things, I think there is an example I think I can share. If you want to have a third-party authentication system, if you work within a SaaS company and is big enough, they would have authentication done without actually the right keys, so, you authorize on their behalf.
Before going there, Amplitude is in a really interesting position where lots of our customers are mutual that we are their customers and that they are our customers. Also, we share common customers that we have other in a hundred customers, which are both customers of us. In the early days, when Amplitude started, we were still a startup back then, there was a lot of thinking, when we set up those data per account, we just have one and there were also legal requirements where you can only have one account per entity. But, later on, when the system expands, we're building more and more integrations, the problem occurs where the same account was used for both purposes, both for internal teams managing their own data and needs and/or for managing the OR SAP, which is used for our common customers. And this becomes a burden. Sometimes, it is a really heavy burden for us. Then, any migration would come at a risk and also both teams are interfering with the internal teams and internal facing teams are interfering with each other where migration might provide a risk as it might break the data connections for hundreds of customers. It's the burden that's have shown the repeated pattern, and I wish, I know that years ago, so we have, all separation responsibilities for those, internal, and external apps and yes, this actually came with some of the cost. That's kind of the lesson I have. So, my advice for you, try to separate your internal and external systems as much as you can.
**Eldad:** And a cold shower.
**Benjamin:** And a cold shower. So Eldad, you are a bit of a bad cop, good cop. You ask about kind of a failure. I'll do the kind of opposite thing. 2022 is coming to an end. Tell us about something you're super proud of, that your team accomplished this year that really is huge, and that you thought before it was, I don't know, impossible.
**Weichen:** Interesting! Two things in mind. First, we launched a developer portal in early the year. So basically, it's an effort to scale the number of connections we support by supporting third-party developers to build integrations towards us. Because whenever a large client comes to us, Hey, can you have these 20 integrations? And there's only so much we can do, that never scaled. So, we have to rely on our partners to build that bridge and then we provide this generic framework to allow them to submit integrations just in a configuration-based manner. We have a system that parses those configurations and basically, sets it up. So, that was quite successful this time. We probably have almost 50 partners submitting their integrations this year. So, I think that's a huge success for us. That's one thing.
The other for streaming. In the streaming, actually a new product we launched in Q3. Real-time data propagation is increasing demand because at the beginning we think about that we're analytical tools. We won't get data in, but we know we don't want that get data out. But that's apparently not true, especially for the real-time cases. It's basically the path through. They're using our governance features exactly back to that the early point, but, it's again a different system, comes with its own challenges, in relation to reliability, the latency cost and the broadcast overall. So, I think there is a huge shout-out for all the engineers if you're listening on this. So thank you for this great achievement.
**Benjamin:** Wow!
**Eldad:** Wow!
**Benjamin:** Awesome!
**Eldad:** First example by the way, was amazing. As a startup, we meet a lot of partners and platforms and we divide them into two those that have built connectivity. So third party can build their own stuff into the system and those that are not, they're black boxes and a long tail of startups cannot connect to those platforms. So it's really important and you never know how and when someone is using your system and you learn a lot by just opening it up. So congrats on that, especially to your engineers. It's an amazing effort.
**Weichen:** At Amplitude, one of the benefits as a public company, because it actually earns the trust of lots of the partners out there. So, they were linked to us, but of course there are times, those bigger players which never think about us. So we have to still in doing our job, but it definitely still makes our life much, much easier.
**Benjamin:** For us as a database vendor, this is in terms of the dialect and wanting to be compliant with Postgres, to make it easy for the ecosystem to exactly adopt. It's interesting that you guys face similar challenges or similar problems in that regard.
In terms of the second thing, you guys provide these very specific data experiences, to analyze certain user interactions and those types of things, how does this work in terms of like schema management? Do you guys basically say, okay, data has to come to our connectors in a certain schema and then we can give these use cases or like these types of analytics products? Or do you actually kind of manage the schema in your connectors in your system and can accept data in whatever form or shape?
**Weichen:** If you see the pattern for all my answers is like "it really depends." Because as Amplitude evolves, we get more and more customers from different spectrums were in those, the really mature customers where they have their own stuff or in the low maturity customers, we don't have anything which actually easier or somewhere in the middle where they have something. So, we try to be flexible. If they want to use our governing feature, they can, and if we can block all the events. Again, we think things are events. If the event doesn't fall into the tracking plan, we just block them, that's the easier way. And we can use transformations, and filters, certain data only goes to certain destinations and with different solutions, all of that. But, again, we will try to accommodate as much as we can. So that's why it involved a lot over time with all the different use cases coming in. Hopefully, that answered your question.
**Eldad:** It's all about surface area.
**Benjamin:** Yeah.
**Eldad:** Perfect. Okay.
**Benjamin:** Awesome.
**Eldad:** I think we've nailed it.
**Benjamin:** Definitely.
**Weichen:** All right.
**Benjamin:** Alll right Weichen. Thank you so much for your time. We super appreciate it. This was super interesting, learning more about what you and your team are doing, at Amplitude.
**Eldad:** We are going to check out hopefully next year, the connector framework.
**Benjamin:** Definitely.
**Weichen:** All right.
**Eldad:** Come work for us.
**Weichen:** Thank you for having me.
**Eldad:** Thank you for coming.
**Benjamin:** Awesome. Take care and have a great day!
**Weichen:** Take care.
**Benjamin:** Bye.
**Eldad:** Bye.
# How AppsFlyer manages scale without sacrificing performance (/blog/how-appsflyer-manages-scale-without-sacrificing-performance)
Listen on [Spotify](https://open.spotify.com/episode/1dmzUBJXP0WJdPo3ng18Vr), [Apple Podcasts](https://podcasts.apple.com/us/podcast/how-appsflyer-delivers-sub-second-bi-to-1000-looker/id1561927688?i=1000521449360) or watch on [Youtube](https://www.youtube.com/watch?v=MTC9TEMqXcU).
Recently on The Data Engineering Show, we had the pleasure of speaking with [Alexandra Sudilovski](https://www.linkedin.com/in/alexandra-sudilovski-b436a83/?originalSubdomain=il), Senior BI Expert & Looker Guild Master at AppsFlyer. For those who don't know, [Appsflyer](https://www.appsflyer.com/) is a leading SaaS mobile marketing analytics and attribution platform.
Whenever you download and interact with an app, data is sent through their servers. As you can imagine, this amounts to an enormous amount of data.
Appsflyer processes 120 billion events daily and saves 90 terabytes of data in AWS and 40 petabytes in BigQuery daily — and they're still growing.
Appsflyer has exploded in size, growing from a small company of 200 people to 1000 people in just three years. They went from nothing to having thousands of users and hundreds of thousands of dashboards in a matter of a couple weeks.
Dealing not only with a huge amount of data on a daily basis but doing so while growing quickly as a company can come with many challenges. Not to mention, the definition of "huge data" seems to only grow year after year.
We asked Alexandra how she and her team handle these challenges:
*We've learned that you need to adapt quickly and be ready to change your mindset and approach. After we went into production, we had all these people who were hungry for more data, and it became hard to keep up with everything from delivery of new projects, system maintenance, and even bringing on the new employees that we needed. But at Appsflyer, every single department and internal process is measured — and not just the R\&D side of things but also functions like HR, marketing, legal, finance, and more. We want to see everyday how we work as a company and if there are any processes we can improve. We constantly measure ourselves and are improving, and we have seen great results from this. We also use every code that is known on the market to be able to adapt and be equipped to always use the best solution in any situation.*
With so much data to manage, it would be easy for mistakes to fall between the cracks. But Appsflyer takes data accuracy to a whole new level — which is one of the reasons we love this company so much. Alexandra is a leading force behind this drive and mindset:
*If I saw data that was not accurate and not showing the truth, this would make me not believe in my own product. If I don't believe in it, why should others? So my priority is always data accuracy and making sure people can trust us and our product because people rely on that data to make decisions. At Appsflyer, we constantly measure ourselves and double-check our work. There are always multiple pairs of eyes on everything, and we have people dedicated to testing and QA in R\&D. We want to catch any errors immediately instead of waiting to get this feedback from stakeholders and users.*
Alexandra is truly an amazing leader in the BI space, and of course, Appslyer is a company that we believe in and love to follow. We talked to Alexandra about so much more, including how she has blended the roles of BI and data engineering in her work and how she uses some of the hottest tools in the industry today, so be sure to check out the full episode with her.
# How are those data intensive customer facing apps engineered at Gong? (/blog/how-are-those-data-intensive-customer-facing-apps-engineered-at-gong)
[Gong](http://gong.io) manages hundreds of thousands of videoconferences and millions of emails PER DAY, which add up to hundreds of TBs. The Data Bros met Yarin Benado, Gong's engineering manager to understand what is required to move to a modern data stack to support all this, what this stack looks like, and why it all comes down to data quality at the end of the day.
Listen on [Apple Podcasts](https://podcasts.apple.com/us/podcast/how-are-those-data-intensive-customer-facing-apps-engineered/id1561927688?i=1000548477518) or [Spotify](https://open.spotify.com/episode/62BDiWR7CTFZC9aQM3uycL?si=aa14aMlpTIekEtBQQ7D_HQ)
DATA ENGINEERING SHOW INTERVIEWERS:
1. Boaz Farkash.
2. Eldad Farkash.
GONG GUEST: Yarin Benado - Engineering Manager.
**TRANSCRIPT**
Boaz: Hello Everybody! Welcome to another episode of the Data Engineering Show. How are you Eldad?
Eldad: I am good. Thanks.
Boaz: Not tested positive for any variant yet?
Eldad: Dodged it. Had it at home.
Boaz: Talk to the mic.
Eldad: Managed to dodge it.
Boaz: Do not dodge the mic. I dodged it too. Maybe you think it runs in the family.
Eldad: On this version, at least.
Boaz: Yeah. Thanks everyone who joined us. We are here with Yarin Benado. Hi Yarin! How are you?
Yarin: Hey guys! How are you? I am great.
Boaz: Did you dodge the virus too?
Yarin: So far so good.
Boaz: So far. Yarin Benado - Engineering Manager from Gong. Gong, for those of you, who do not know is a really, really interesting and great product, a revenue intelligence product. It started with a product that helps you record and analyze all your conversations, typically for sales teams, but not only, and you can go back and analyze who talked, how long, what words were mentioned, and get insights into how to better, sort of, close deals if you manage sales teams, etc. Yarin joined around a year and a half ago after an acquisition. So, Yarin tell us what the Vayo was, which sort of was your home before entering Gong.
Yarin: I founded Vayo around 2017, with a good friend and a partner, and our goal at Vayo was to basically take the entire customer data that a company is creating constantly, and answer a few simple questions about these customers. Is the customer a happy customer? Are they going to leave the churn? Are there any upscale opportunities? and the idea was basically to automate all the data aggregation, collection aggregation, and reporting around these specific customer-related questions. What used to be done as internal BI and data teams, so we offered a simple solution, where Vayo is doing all the hard stuff.
Boaz: How big was the team at Vayo at that time?
Yarin: So we were a relatively small, very small startup. We ran for two and a half years. We were like 10 employees, mostly engineering, here in Tele Aviv and a bit offshore.
Boaz: Tell us a little bit about your background. What did you do up until that and how did you enter the data and engineering world?
Yarin: Before Vayo, I was leading the engineering team for, a bit bigger startup, here in Tel Aviv. Before that, I held the role of principal engineer for several startups. I think Gong is the biggest company I worked for.
Boaz: How many employees are at Gong nowadays.
Yarin: Almost 900.
Boaz: Wow! You are an engineering manager. Tell us a little bit about your current role and how much of it is dedicated to data.
Yarin: I am an engineering manager in what is called the insights group within Gong. The group is in charge of all the data that is customer-facing basically. So it is either in-app analytics or BI for customer use. Basically, everything we do is around data. At Gong, there are many, many teams and groups within Gong that create a lot of data and we have to build insights on top of this data.
Boaz: Yeah, Eldad and I know, we talked about it in past episodes, I think as well. We always loved the intersection of software engineering and deep data projects. So, let us try to untangle that. Tell us a bit about the current data stack and sort of the current active projects, and let us see what is going on under the hood at Gong with data.
Yarin: Cool. So in terms of the stack, basically, we like to keep things very simple, but at the large scale at Gong. Our data stack is, we have the operational databases, which are serving the entire product and pipelines within the product. In terms of data, we started simple. We just PostgreSQL, then moved to Redshift. Now, we have a bigger project basically to take data from multiple sources for this in-app analytics use case and move everything to a single data warehouse, with a lot of pipelines to aggregate and pre-aggregate, data for basically sub-second query times for dashboards and graphs, etc. For operational databases, we still use a lot of Elasticsearch, PostgreSQL. We have some MongoDB, and as mentioned, it is also for the in-app analytics and we are now moving also a lot of parts from PostgreSQL to Redshift and to Snowflake.
Boaz: These are a lot of things. Walk us through the different teams that are in? How do the teams that deal with data look like at Gong? What do you have between data engineering, software engineering, which teams are dedicated to in-app insights in particular? How does that look like? How do all of these interact with each other?
Yarin: It is a very good question because it is pretty complex at the pace that Gong is growing. We have within the insights group, right now, we have 2 tracks, 2 pods, with products and engineering and data engineering. One of them is to create the in-app analytics track. Another team is focused on the infrastructure of basically building the lakehouse if we can call it like that, both for internal use.
We also have the classic data analysis team and internal BI team. They all are utilizing the same data generated from Gong and others, we have data as mentioned product analysts that are doing a lot of internal work with the data. Overall, I would say there are about 30 different people working around these areas of data that is being generated, on top of Gong's platform.
Boaz: What data volumes are you guys dealing with?
Yarin: A lot. So, most of the data that is not being analyzed or being analyzed but not for data analysis is basically the videos and calls - what we call the media. The last time I checked recently, it was almost 5 petabytes, in terms of data ingestion. So we are talking about hundreds of thousands of videoconference calls per day. I would say many millions of emails per day. Overall, all the data that is not the media, I think takes around a few hundreds of terabytes.
Boaz: Wow! As you mentioned a variety of technologies. Walk us through, maybe, you know, if I am a user, from a user experience perspective, what kinds of analytics am I exposed to as a user? And then what parts of the stack are delivering it to me?
Yarin: The largest part of data, which we call team stats is analyzing how the team is performing in terms of sales calls. Gong has identified some very interesting metrics that help salesperson do better, things like the talk ratio, patience. So we have a very specific part of the product that serves this data, and then, we have some more advanced things that we can basically track what is being said in calls and track it over time. It is relatively new area that we are focusing on at Gong. What we call tracking the strategy or the sales organization strategy. Basically, all the data right now is being served either from PostgreSQL, in which all the data is pre-aggregated or from Snowflake where all the data is basically being modeled using DBT and almost no aggregation is done there. So it is query time.
Boaz: How did the data stack evolve in the year and a half that you are there? I mean you mentioned you are looking into a Lakehouse architecture right now. Is that a new initiative or was that always the case? And when did Snowflake and DBT come in and what was the driver for that?
Yarin: So the entire lakehouse solution is being in progress these days. We are trying to make it more of an evolution rather than a revolution, but still, a lot of the data is being served from the classic simple one big table on top of PostgreSQL, and now we form the new architecture basically, looking for, in terms of scale and building this new lakehouse solution.
Boaz: You also mentioned Redshift, right? Where does the Redshift come in?
Yarin: Mostly around product analytics. A lot of data that is being collected from either third-party tools ends up in Redshift anonymized. We do not have any customer data at all there. So, it is only just IDs and raw metrics.
Boaz: What are the typical frustration that you guys run into in the data stack? What about maybe the internal users like to complain about or what the new evolution may resolve?
Yarin: Users always complain obviously about performance, why this query takes so long, etc. A lot of internal users find it a bit difficult to get the real picture of data because we are using so many disparate databases and also when we try to bring everything under the same hood, data modeling still takes some effort because Gong is a company that moves really fast in the engineering, and sometimes we do not always have a cohesive data model, in terms of data engineering and analytics. So, these are two pretty interesting challenges that we are facing.
Boaz: Who is driving or who is involved in the evolution into the new architecture? How from a process perspective or a human perspective are you managing that project?
Yarin: Everyone, not kidding. The interesting part here is that it is not a classic internal data project, because we are also exposing some of this data, obviously model differently, but to customers. So, they will end up having their own access to a warehouse with all their data. So we have two areas pulling the same string, both from the product and from the internal analytics use case. It is almost everywhere in the organization, starting from customer success to product, to engineering, to pretty complex projects.
Boaz: I think you mentioned sharing data with customers. Can you elaborate more a bit about that? So what's the business need or requirement there and how are we going about it?
Yarin: I think I look at it as the next evolution of APIs. Gong has a pretty robust API, which companies can utilize and build their own products, but the biggest benefit of Gong is the data that it holds, and obviously, our customers want to build data products around Gong's data. So it only makes sense instead of consuming APIs and building the pipelines and data transformation is just hand them over all the data cleanly modeled in a way that they can either just plug in a Tableau or Looker and just start using the data, or just build products on top of data and not API.
Boaz: They are essentially exporting the data, copying it into their own environment through the APIs.
Yarin: Yes.
Boaz: Got it. What are your impressions so far you have been using both Snowflake and Redshift, how is that sort of looking for you guys? What's the conclusion>
Yarin: We are relatively new with Snowflake, but overall I think both have their strengths and weaknesses. Obviously Snowflake, now, in terms of a bit cost and Redshift is not as performant and ingestion is a bit more challenging. Overall, I think they are both okay for what they do. Some use cases are better here. Some are there.
Boaz: Is DBT used with both or only with a Snowflake.
Yarin: Only with Snowflake.
Boaz: And how long has it been since you have adopted DBT? How has that been going?
Yarin: Same pretty, pretty new for us as well.
Boaz: Any tips so far for other newbies with DBT - Things to avoid, best practices from the rather short journey for you guys there or not yet.
Yarin: Yeah, not yet. We are still in the part of, like these things, we are more comfortable with, but we are still learning as we go.
Boaz: Awesome! I wonder you being a veteran in software engineering and essentially now delivering data experiences into products, the intersection of software engineering and data, what kind of practices are important for you that are different from sort of the traditional internal analytics world
Yarin: Mostly it is around the quality of data, which we end up serving our customers. So we treat data projects almost as a software development lifecycle. What is the SLA for supports, monitoring? How can we make sure that the data that customers see, that went through so many pipelines and transformation, is accurate? Pipelines cannot break, so we cannot have data that is, "yeah, it's broken for the past 24 hours." So customers see data that is from the last couple of days. We treat it just like any other large-scale customer-facing software project.
Boaz: Let us maybe use this as an opportunity to pick on you and ask, was there any sort of a failure that you remember that we can look back and then maybe help our listeners avoid? What do you remember as a horrible day? whether it is data pipelines or production issues that we can all learn from.
Yarin: So far, nothing major customer-facing. We did have some mega failures trying to be very naive in terms of how we are going to approach this lakehouse project. We were very naive and said, yeah, let's like take something like a CDC solution that consumes all the possible data, throw it in S3 and then just have Athena or Presto or something like that and it is just a one month project and we goal in, but yeah, this one month was a battle, I think, for about 6 months ago and then, we decided to move on to something more robust.
Boaz: What is that more robust thing. Let us just untangle that a little bit. Let us spend some time there.
Yarin: It is let us not cut angles and just build a streaming service from within the products where we can make sure also in terms of data lineage. So we know when the data was modified and what kind of data was modified. So for example, if we are talking about recorded calls, even scheduled calls. So before the call is being processed and analyzed by Gong, we can know exactly when this call was changed and modified by the host, was it rescheduled, etc., and then, we write everything. We almost created our own CDC mechanism. We dump everything to S3, where we have a like staging area, I would say, being rapidly ingested into something like Snowflake, which is the journal for each entity. And from then on, we can create history tables or daily snapshots. We can go back in time and say, yeah, let us see how the actual data was looking a week ago. So, no shortcuts.
Boaz: How did you go about with the lineage challenge? How did you implement that
Yarin: We still have some challenges there, but we keep track of any transaction and data pipeline, which is internal to the product, and have a record that says that we know when was this entity modified and by whom. From then on, we try to keep a sequence or a batch ID and each transformation has its own records. When was it transformed, by which pipeline, etc? Eventually, we should be able to unroll aggregated data and find out the actual raw data that was taken into consideration for disaggregation.
Boaz: Tell us internal analytics, how does that look like? What tools are being used and which of the different databases and warehouses are they hitting?
Yarin: For internal use?
Boaz: Yes.
Yarin: We have Sisense. We now have Tableau. We have our homegrown analytics, which I am not sure at which front it is being used. Most of the data is being ingested to Redshift, either from third parties or being streamed from within the product. Pretty straightforward. We are using Amplitude for analytics. FullStory for UI analytics, like product user journey. Pretty straightforward.
Boaz: You know, we have picked on you for data tragedy. Now, let us talk about something happy. What are you proud of or projects that went very well or something that we can learn from if you can share?
Yarin: I think one example that I have on top of my mind is like we were looking to replace a PostgreSQL for one part of the product, which we served some stats, and before moving off from PostgreSQL, we map where the problem is? Why is it not working? Is moving on from 2 different datastore is the solution? and at the end of the day, we identify that with just a very, very minor optimization to this PostgreSQL, using RDS or Aurora, so we can more easily scale PostgreSQL, not that it is an easy task in itself, but we had a pretty big table in terms of what PostgreSQL can deal with, it was just research for about 2 weeks and then the solution took a couple of days, and we were back from like 10 seconds loading time to sub-second queries, just with using the right column type, the right indexes, and it was pretty interesting how far you can go with the modest tools.
Boaz: Yeah. Sometimes the simplest things work like a charm, but we oftentimes do not spend enough time figuring that out and go from a robust solution. Awesome! Great story! Maybe an interesting angle to talk about would be hiring. How do you go about hiring engineers from a data angle? How do you make sure they will be able to deliver on those big data challenges? What are you looking for?
Yarin: First, I would say it is a very difficult task nowadays, but yeah, we all can agree on that. In terms of data, we are looking for people that dealt with data, not necessarily at a data engineering level and know all the tools and how to build infrastructure, but have some experience around data, what it means to like query 2 terabytes of data? How databases and data stores are modeled and understand technically the strength and weaknesses of each. It is not necessarily the best SQL experience, but more of the understanding of performance, good data modeling. This is also something that we put some emphasis on. Have the good foundations, 3NF, Kimball, things like that.
Boaz: Awesome! Thanks! Are there any parts left in your stack that you consider very legacy and are the ones that are in line to be replaced?
Yarin: Yes. Although Gong is a relatively young company, the things that are implemented 2 years ago are considered legacy. I would say that we still store data, which is not for operational use in PostgreSQL is something definitely want to move on from, either to the lakehouse approach or maybe some other data store which allow fast analytics at a smaller scope, but at the faster scale and performance.
Boaz: Thank you. I think Yarin I am running out of questions. I mean, that has been super, super interesting.
Eldad: Yes.
Boaz: We love Gong ourselves at Firebolt, and it is definitely interesting to hear such a sort of insight-driven and data-centric product as data flowing behind the scenes. Thank you for joining our episode and any final words you want to spread out to the data engineering world?
Yarin: Yeah, I would say what we always do, one of our operating principles at Gong is first enjoy the ride and the second one is just, do not be so naive and say, yeah, put everything on S3, and then it would work. It takes a lot of effort and a lot of time and a lot of people along the way to create a robust data solution.
Boaz: But the first few days of that naive, false optimism, they are so fun. You think only worries are over before it explodes in your face
Eldad: Yes
Yarin: Feel so super-powered. Yeah! it will take us just 2 weeks
Boaz: Within your head, it is all done already. Yarin, Thank you very, very much and see you around.
Eldad: Thank you so much.
Yarin: Thank you. Bye.
Eldad: Bye-bye.
Boaz: Bye. Thank you for joining us.
# How Bolt Engineers Are Designing Its Next-Gen Data Platform (/blog/how-bolt-engineers-are-designing-its-next-gen-data-platform)
Bolt's ride-hailing app serves over 75M users in Europe and Africa and handles 500K queries every day. Erik Heintare along with Bolt's engineering team is in the midst of designing a new next-gen data platform and is sharing how it's going to solve their biggest data challenges.
Guest: Erik Heintare - Senior Analytics Engineer at Bolt
Hosts: Eldad and Boaz Farkash, AKA The Data Bros.
Listen on [Spotify](https://open.spotify.com/episode/73lYRwqhnEsPYzXn7UDTMG?si=w-HyTZFBT4-7Q8U2v7PaKw) or [Apple Podcasts](https://podcasts.apple.com/us/podcast/how-bolt-engineers-are-designing-its-next-gen-data-platform/id1561927688?i=1000544916326)
## Transcript [#transcript]
**Boaz:** Hello!
**Eldad:** Hello!
**Boaz:** Welcome to another episode of the Data Engineering Show with myself, Boaz and Eldad.
**Eldad:** Hi, everyone!
**Boaz:** How you have been Eldad?
**Eldad:** I am good.
**Boaz:** Are you ready for a Hanukkahs around the corner.
**Eldad:** Yeah, absolutely!
**Boaz:** Yeah. Do you want to come over to light some candles?
**Eldad:** Light some candles!
**Boaz:** Yeah, do it!
**Boaz:** Okay. With us today, getting ready for Christmas time, I guess, is Erik Heintare. How are you?
**Erik:** I am good. How are you? Hi everyone!
**Boaz:** Thank you so much for joining us, Erik. Erik is joining us from Bolt. He is a Senior Analytics Engineer/Lead Data Analyst at Bolt. He has been there for over 4 years. Before we let Erik introduce us a little bit further, a few fun facts about, Estonia in general, where Erik is from. Estonia, a small country, only around 1.3 million, but is actually ranked number one in the world in unicorns per capita. That's very interesting.
**Eldad:** Woohoo!
**Boaz:** Also regarding Bolt. If you have not heard about the Bolt, means that you probably have not been spending enough time in Europe or outside the US. Bolt is actually a big competitor to Uber. Next time you hop off a plane in a European country, try not defaulting to Uber and see if you can get a ride a scooter or bike, even food delivery from Bolt. A Bolt is active in over 300 cities across more than 40 countries, by now. Mostly in Europe, but also Asia Africa, Latin America, and that is exciting stuff. Bolt was founded in 2014 in San Francisco. But in recent years, especially I think saw tremendous growth.
**Eldad:** Amazing intro.
**Boaz:** Am I wrong? Am I wrong?
**Erik:** You were wrong about the San Francisco part. This was actually founded in Tallinn in Estonia.
**Boaz:** Got it.
**Eldad:** Other than that, amazing intro.
**Boaz:** In the PR, once they opened the San Francisco office, it takes over everything in the PR. But thanks for the correction. It makes a lot of sense. So far, the company is still privately held and has raised over $600 million to date serving more than 75 million end-users. So, Erik, thanks. What did I miss anything about Bolt did I leave out?
**Erik:** No. Actually, maybe a couple of things about Bolt to add more is that we are actually not focused purely on Europe and I think we are the leading ride-hailing and delivery app, not only in Europe but also in Africa. So, we should be the biggest player in Africa as well. So, Europe and Africa together are like 2 billion people. Quite a big market actually.
**Boaz:** Tell us, what do you do at Bolt? Let us here it from you.
**Erik:** Just one more thing coming back to the intro part, then, when you mentioned Estonia and the facts you gave, then actually I prepared one of the fun facts about Estonia and it included that also that we have most unicorn per capita, but when I usually do this intro to foreigners, then I also say we have most supermodels per capita also.
**Boaz:** Interesting, we should investigate a correlation between startups and supermodels, Interesting!
**Erik:** But yeah. Great intro! I have been at Bolt for more than 4 years. Initially, I started off as the first data team member ever. Before me, there was no one working purely on data. I started off as a data analyst, basically trying to get started with our data warehouse together with engineers, try to automate and move away some queries from a production or a pre-live environment to the data warehouse. Also wanted to get rid of Google sheets, all of those things, and try to optimize everything we did in a business. Within those 4 years, I gradually, since we hired more analysts, moved to like analytics managers/lead analyst role, but then after some time, still wanted to be more hands-on. So, now, I am mostly focusing on our analytics platform, and still helping everyone in the analytics also, around all of those topics. So, yeah, this is what I do.
**Boaz:** Awesome! Let us talk about your title for a second because it is interesting your course of a senior analytics engineer, we all know data engineers, analytics engineering is something you heard less about. What do you think what that means?
**Erik:** I think it is the around tooling and the things that we actually do in the analyst world. So this analytics engineer, in general, does not mean there is a big difference between like analyst work or data engineer work. It is just I am more focusing on helping to build the infrastructure for our analysts. Meaning, like I think, the biggest user of analytics engineering is a company DBT labs and they are promoting it heavily. For us, we are not using DBT, but still, the work that we need to do is basically build out a platform for analysts, for business users, for product users to do their work efficiently, smoothly with high quality, high speed. So, I am somewhere in between data engineers and data analysts. So this is how we combine those two together.
**Boaz:** How big is Bolt in terms of headcount worldwide?
**Erik:** Worldwide, I think, we are more than 3000 employees and the data team just reached over 100 members.
**Boaz:** Wow!
**Boaz:** Tell us a little bit about that. What kind of data roles exists in Bolt? And how are they spread out in terms of departments, groups, teams, etc? And where do you fit in?
**Erik:** Yeah, we have 3 different roles. I think it is pretty common. We have data scientists, data engineers, and data analysts. Data scientists and data engineers belong to an engineering organization and are like a centralized group. They report to some higher management in engineering and data analysts, they belong to a product organization, mostly, and they report to product managers. So, they are way closer to the business and to have the effect and the knowledge about everything that is going on in the business. So, this is why we went with this hybrid approach. Initially, with, I think, 3 data analysts in the company, we were thought of going with centralized also for analysts, but it did not make any sense and we are quite happy with the setup at the moment.
**Boaz:** You report to the product as well and not to engineering.
**Erik:** No. Actually, I am part of data engineering.
**Boaz:** Okay.
**Erik:** So we have 4 soft teams in data engineering. Firstly it is a data lake, then it is data transformation, thirdly model life cycle and experimentation platform, and fourth is my team, which is analytics engineering.
**Boaz:** Got it. Let us talk about the data stack and definitely, a bit deeper. Before that, in terms of data volumes, what kind of data volumes does Bolt deal with?
**Erik:** If you want to be fancy and fancy starts, I think from petabytes, then we can say that our Redshift Cluster can handle more than 4 petabytes of data, but actually we are not there yet. I think our data lake in total is somewhere between half a petabyte and one petabyte, somewhere around that.
**Boaz:** In terms of sort of a daily number of events or daily data volume?
**Erik:** Yeah. So I am talking about from an analyst perspective, then, we do, I think, a bit less than half a million queries a day in our BI tools. Then, talking about, as we serve all of the models also as a platform then, I think we do 100 million model life cycles daily.
**Boaz:** Wow!
**Erik:** So, quite a lot, and, yeah, I think there are from those 3000 employees more than half of them use our BI tool also on a weekly basis.
**Boaz:** Okay. So, let's break that down. Tell us about the data stack a little bit, from bottom to top, how does it look like?
**Erik:** Yeah. So, we stream our data from live databases and services with Kafka to S3. Then, we have, of course, as I mentioned, Redshift as a data warehouse. In the S3 and Redshift, we do some transformations in Apache Spark or with Apache Airflow. Then together with Redshift, of course, we use a Spectrum, which is basically a layer to get data from S3 directly without storing it in Redshift. Then, we use Looker as our BI tool. We use SageMaker and all of the data team members use Jupiter notebooks. So, this is, just a brief overview and the really, really high level of what are the tools we use. I think, there is nothing, really, really epically different than the other companies are doing. So, I think it is a pretty common stack.
**Boaz:** How long has this stack been active? I mean, if you remind 4 years ago how did it look back then? And how is the journey?
**Erik:** So 4 years ago
**Boaz:** Google Sheet.
**Erik:** There was not almost anything. When I joined the guys, so cool thing, since we are over on AWS already back then, they were like, "Hey, what is this Redshift. Let's spin it up and see, maybe you can use it." Of course, we still use S3. So S3 was there. We did not use Kafka. So it was a bit different then. And then I think in Redshift when I joined, we initially had like top 10 tables, maybe from the live database, just like getting orders and do understand like where our drivers or how active they are just to get the first initial. I really remember when I joined, I needed to get some data and the engineers were talking to me is like, "we do it from the live database." "We acquire it from the live database. It is like, we do not have time to like a data warehouse. Why should I bother?" And now it is like thousands of tables. I don't know even how many together with Spectrum we have in Redshift. It is constantly improving. So yeah, what's going on there?
**Boaz:** And eventually, because there is Looker that the only BI tool or do you have other methods of visualizing data apparently?
**Erik:** For front-end analyses, we also use Mixpanel, but everything that we as a data team wants to have a better control and better structure and the use of backend data also then, Looker is our, by far the biggest BI tool that we use, Yeah!
**Boaz:** I think I saw on your profile, somewhere you mentioned the use of Amundsen as a data catalog, is that is in use?
**Erik:** Yes
**Boaz:** Can you, maybe, tell us how you guys are using it?
**Erik:** Yeah, I would say it is in the alpha stage for us; but basically, the company is growing so fast. There are more people coming in. There are more tables popping up all the time and we wanted to get a better way to scalably share the information that we have around our data and data discovery has been one of the bottlenecks of scaling. So, we started off a trying out actually two different tools. So firstly, Knowledge Repo, which is basically where you can host; it is a repo filled with Jupiter notebooks where you can add some metadata on top of it and also, Amundsen. We currently use it in a quite limited scope, so we have all of the Redshift tables there. We have key tables, key columns, key schema all, like commented and we have attributed owners to them. So at least if you search for something, you want to know what is going on with food delivery couriers, and you are the first time doing it from the marketing perspective, then at least you will be easily able, with one search, to understand where are the tables? How they are structured? Whether they are like key tables, and who is the owner of those tables? from a data team perspective.
**Boaz:** Can you maybe share how does Bolt approach the data engineer versus analyst relationship? How do you guys make sure that, you know, things are delivered quickly, and the analyst and the other business departments can sort of become, stay self-sufficient and move fast. Has that typically been a friction point or any insights you can share on how you guys do that?
**Erik:** Yeah. I think it is also pretty common; but of course, we struggle all the time either to build the platform to be better and scalable for the future or support the current needs for analysts or data scientists, so they could move on with their projects. So it is a constant battle between the prioritization. What we have done is that periodically we have changed our focus. So, for some time, we focus more on the product requests, making sure that everyone gets their stuff quickly and efficiently. And, then we communicated to our users also, for the next two weeks, we are heavily focused on building our data platform. So all the requests that you have, we will only maybe solve the most critical ones. So please mark them accordingly. It is never easy, like products always try to push their own needs first because why would you bother optimizing something in data engineering. For them, it does not give any return on investment, but their feature, which they want to launch, of course, it is super easy to put a number to it. So it is a constant battle. Of course, we communicate a lot together with a team, so this helps to understand the priorities, and this is the only way to go.
**Boaz:** Let us talk about some of the use cases. So, you know, there is the big Redshift at the center, according to what you say. So what are the sort of uses cases that are run on top of it?
**Erik:** Yeah, I think, Redshift's biggest abuser is Looker. As I said we do quite a lot of queries from our business users, from our analysts. They are the main users. Of course, we have our experimentation platform using it. Then, we have our model life cycle which is basically they are also using to re-train their models, like profiling services fetches some data from it. So basically everyone fights for the spot in a Redshift query queue.
**Boaz:** Can that get ugly sometimes?
**Erik:** Oh, yeah! I think I will leave it to the latter question, maybe potential about the glorious failure.
**Boaz:** Okay. So, let's do a quick stop and move to something funnily quick - the Blitz Question Round. Are you ready?
**Erik:** Yes.
**Boaz:** Do not overthink. Let us see.
**Boaz:** Write your own sequel or use a drag and drop visualization tool?
**Erik:** Write your own sequel.
**Boaz:** Looker or Tableau?
**Erik:** Looker.
**Boaz:** Commercial or open-source?
**Erik:** Heart says, open-source, head says commercial.
**Boaz:** Batch or streaming?
**Erik:** Streaming
**Boaz:** Work from home or from the office?
**Erik:** Hybrids
**Boaz:** AWS, GCP or Azure?
**Erik:** AWS.
**Boaz:** To let people self-serve for analytics or not bother?
**Erik:** Let self-serve.
**Boaz:** Bolt or Uber.
**Erik:** Bolt.
**Boaz:** Boom! I like that.
**Erik:** I did not even leave you time to finish your question. I already said, Bolt.
**Boaz:** Very Good!
**Boaz:** What are the biggest challenges though, with the current stacks? What are your top priorities for next year given what you guys have today?
**Erik:** Since the current stack has been around for more than 4 years and actually, we are currently moving in the phase of finalizing POCs for that next-generation data warehouse, so we are potentially either getting rid of Redshift, they are replacing with some other or adding something on top of the current stack or just playing around with all the different things that are available right now to make sure that we are enabling our users for the next 100x growth also. So, this has been a big focus for us in the last couple of months and, we will try to finalize it during this year and next year, will be where we basically work hard on making sure that we have that next-generation data warehouse ready.
**Boaz:** So you are going through a traditional evaluation process?
**Erik:** No, we are using some of the methods that I have been out there and a lot of companies have been doing their POCs. So, of course, would take ideas from there, but what we do is we also apply a lot of our internal knowledge and let's say like we gathered a lot of internal queries, which are heavily used at the moment and we wanted to see how they would be formed. Because there are some tools that are really, really good when you just need to query data from one table and there are some other tools that are really good at joining together the 20 tables, so what is the best for us? We try to cover all of those things, and, yeah, it has been a heavy job to do those POC and kudos to all of the team members who are doing this. It is not an easy decision to either switch out or keep the current state. So you are only responsible for the most used and valuable asset, but then subsequently will become familiar.
**Boaz:** Any particular technology or feature that is out there that you really are upset, you don't have access to today or you would like to have in your next-gen platform?
**Erik:** I think one of the things that pushed us to move maybe a bit faster with this evaluation process was that we are currently hitting some of the limits when we talk about concurrent queries and that was basically peak hours. So this is where we struggle the most at the moment. Then, this is what we want to solve as quickly as possible, because if business users cannot see the data or need to wait for, I do not know, 5 minutes for it, then it is not worth it.
**Eldad:** So you have products blessing.
**Erik:** Yeah! Definitely!
**Boaz:** So I am assuming the coupling storage and compute will be a big deal.
**Erik:** Yes, yes! Most likely.
**Boaz:** You had mentioned the glorious failure prior. I really do not want to pry, but you know, since you brought it up, tell us about some glorious failure you remember.
**Erik:** Maybe, I hyped it up too much.
**Eldad:** Too late.
**Boaz:** Too late, you have to.
**Erik:** But yeah, one of the things we had, so basically it is a combination of multiple things. Looker, quite recently introduced native integration with Google Sheets and Google drive, which means it is easy to set up a spatial from Looker to those bases. We did it. We enabled it and we thought it is like good thing to have. Of course, business users, you will never ever solve all of the cases. Like people always would try to copy your data tools and Google Sheet and add some manual stuff on top of it.
**Eldad:** Sort the data.
**Erik:** So we thought doing this is a great idea. It reduces manual workload and all of those things, but what we did not realize was that due to some limitations from the Google sheets and Google drive APIs, the scheduling takes a lot longer than, like scheduling slack message or email. So what happened was, people were too happy about this and they set up a lot of things, of course, to the Monday morning together with all the other reports that are running on Monday morning. So, basically what happened was, that we had like, I don't know, hundreds of new spatial randomly popping up on Monday morning all of a sudden, none of our overnight schedules or works did not finish. We did not know what is going on. We saw that there is a load. It was really hard to estimate, like, what is causing it? Is Redshift working slower? Or is it because of Looker doing something wrong? So it was like, we could not find it out on the first Monday. It took us 3 Mondays to actually solve it. And it was a combination of many things, as I said, like from concurrent scaling from data warehouse side wasn't performing as we expect them and Looker was not working as expected. So basically for 3 Mondays in a row, our company users could not query the data in the first 4 hours.
**Eldad:** How did it affect your weekends?
**Boaz:** If it highlights on Monday and a little bit on the Tuesday and Wednesday. Thursday, got it right, had a weekend. It was okay and then Monday, all over again.
**Erik:** Yeah! This is exactly like, by the lunch of Monday, we were seeing like, okay, we killed the top queries. We did a couple of adjustments. It looks like it is working. Let's see how it works on next Monday. And, yeah, it didn't work then as well, of course, and then it was easy to get the priorities, from the product also to speed up all the POCs for the next-generation data warehouse.
**Boaz:** This goes back to the blitz question – Do not let people self-serve because they will over-schedule stuff on their own and ruin your Mondays.
**Erik:** Well, I still say self-serve. Yeah.
**Boaz:** But to care for these, care.
**Eldad:** Gradually.
**Boaz:** Another takeaway, you know, do the crazy stuff on Mondays and not on Fridays.
**Eldad:** Exactly, respect a weekend.
**Boaz:** Okay. But, you know, let us not be so negative. What about, positive stories? Tell us about the great win.
**Erik:** So, since I have heard a couple of episodes before, then I immediately started to think about it then. Well, there are a lot of wins that should be mentioned. But one of the things that I thought about was actually when the COVID hit first and then basically, we lost 80% of our company's revenue and, basically, we got a request from top management, that is like, Hey guys, can you scale down a bit, maybe 60% or so with all of your data infrastructures and actually what we did was we managed to reduce all of our infra costs, more than 50%, within like a couple of weeks, and it did not affect the end-users that much, everything remained to work as is, so basically it gave us a huge boost moving forward. We did quite interesting things there and yeah! from that, we learned a lot, and thanks to that we are now a way lower level than we would have been before the COVID hit.
**Boaz:** Can you share some of the details? How did you go about reducing costs?
**Erik:** I do not think there is only one thing to point out, just on a high level, I think that one of the quick switches was that we moved away much of the tables from Redshift to Spectrum that were not used that heavily. So, this gave us, like, we could easily have a smaller Redshift cluster, everything that I needed to be, like, get out from S3 would be used easily through Spectrum. But, I think there were more than 20 items that helped us to get this low.
**Boaz:** Wow! Impressive!
**Boaz:** Goodwin, goodwin!
**Eldad:** This is how you scale. This is so well spent the time
**Erik:** Yeah!
**Boaz:** Okay. So I think, we are reaching the end almost, maybe before we end, we would love to get a few recommendations from you from maybe recent technologies that you have had a chance to play around and got you sort of interested or excited, even if you do not have adopted them fully. So any tips or things that you ran into recently that you want to give advice to our listeners?
**Erik:** I do not want to point out any specific technology or tool specifically but during those POCs what we found was that be ready to change your perspective on some of the tools? So, maybe you checked out something 2 years ago. Maybe you checked out something 6 months ago. Nowadays, everything moves so fast and, it could be that six months makes a huge difference to the product. So, what I would highlight is like, if you, reject a something, like one or two years ago, and you still have this issue in your hand, take a look at those candidates also, which you rejected back then because so many things are happening in this space and huge, huge improvements all across the board. But, yeah, we saw some, like we needed to change our perspective on some tools.
**Boaz:** I do want to also close the loop on the topic we touched on prior, going back to that big Redshift cluster and how ugly it can get when so many people want to get access to the resource. How do you manage that? How are decisions being made in terms of prioritizing, who gets what?
**Erik:** You mean who gets what? like query prioritization or access to that?
**Boaz:** Query prioritization because so many workloads running.
**Erik:** For us, since Looker is the main user and the majority of the business users use it, so we have prioritized quite a bit to the business user side, so it is fine that some of your, I don't know, experimentation platform can run instead of 5 minutes, 7 minutes; this is fine. But for Looker, if it runs like a couple of seconds or 2 minutes, then it is like a huge, huge difference. So, we prioritize our business users first and also like what we have seen is also coming back to the previous point about changing perspective, then we tried to solve like a lot of this prioritization manually for quite a long time. But what we see is that a lot of tools that have, like internal auto prioritization, auto-scaling all of those that are basically built into the product, then, in the long run, I see that all the auto settings might win. Same goes with, I don't know, Google or Facebook ads from different industries. It is like, you can fine-tune it yourself however you want. In the long run, they will have more data on how to optimize all of this. So you can trust them in the long run. This is also what helped us. And also it is like, you do not need to have the manpower to it. You do not need to, I don't know, monitor it so actively that if they are screwing up or not.
**Boaz:** Also regarding the Kafka. Sorry for suddenly reminding myself of so many questions when we are moving about so close to finishing, but Kafka, are any of the reports actually, you know, closer to real-time or is everything more batch-oriented since using Kafka? What is looked at in a more sort of real-timeish fashion?
**Erik:** Together with the next-generation data warehouse that we are planning, and actually there are two side projects to it also, what we call, like next-generation reporting system, for batch and live. So, in here, we are building out infra to serve not only internal use cases but also external use cases. All of the engineers, I do not know, show some numbers in the client apps or restaurant apps or courier apps and also a batch reporting. If you are a restaurant and you want to get their weekly results, so we are also working on it to get it live already, first half next year. We are not doing that much live reporting at the moment, but we have a dedicated team working on it, so we could have it and also, maybe not necessarily a Kafka live, but still one of the next-generation data warehouse goal was to reduce the ingestion lag also to get it more closer to the real-time.
**Boaz:** Got it. Awesome! Thank you! Erik, this has been super, super interesting. Thank you so much and yeah, it has been great having you.
**Erik:** Thank you. It was great.
**Eldad:** Yes, it is.
**Boaz:** To see around the data world and when we visit Estonia, we will make sure to stop by for a coffee.
**Erik:** Sure! You're welcome!
**Boaz:** Take care!
**Eldad:** Take Care!
# How did Agoda scale its data platform to support 1.5T events per day? (/blog/how-did-agoda-scale-its-data-platform-to-support-1-5t-events-per-day)
Scaling a data platform to support 1.5T events per day requires complicated technical migrations and alignment between hundreds of engineers. What to see how Agoda did it.
Listen on [Apple Podcasts](https://podcasts.apple.com/us/podcast/how-did-agoda-scale-its-data-platform-to-support-1/id1561927688?i=1000542833635) or [Spotify](https://open.spotify.com/episode/3XwPJcJElAlU0O8KzIn7tT).
**Boaz:** Okay, so ready to get started. Eldad! Are you ready?
**Eldad:** Yes!
**Boaz:** Okay. So hello, everybody.
**Amir:** Hello, everyone!
**Boaz:** Welcome to another episode of the Data Engineering Show presented by the Eldad Farkash, right here!
**Eldad:** Hi there.
**Boaz:** And Boaz Farkash, that is me! We are related. We have the same parents, which makes us the data bros, woohoo! So, with us today, from Agoda, we are lucky enough to have been joined by Amir Arad - Director of Machine Learning and Shaun. Shaun your last name, please?
**Shaun:** Shaun Sit.
**Boaz:** Shaun Sit.
**Shaun:** Yeah.
**Boaz:** I had a hard time finding it because your name overlaps with a lot of options there, but I did find it eventually. So Shaun Sit, a Senior Dev Manager at Agoda and currently managing the data platform. So, we are having you guys on the show and, you know, I wonder before we start just, at Agoda, because you are in traveling and all that, is it frustrating working around travel data all the time and you guys are at home? I mean, don't you want to travel all the time? Are there any travel-related perks? How can we get some inside information? How it is to, you know, be frustrated, but enjoy travel life at Agoda?
**Amir:** So yeah! You must travel in order to be working in Agoda. It is not allowed to be stationary. You get fired if you stay at the same place. And lots of perks and everybody at Agoda travels all the time, kind of hard during Covid.
**Boaz:** So, yeah, exactly. So bomber for those who joined Agoda during Covid and did not enjoy perks. Any recent, exciting trips that you guys were on?
**Shaun:** I recently went to Phuket. It was lovely. Absolutely lovely. You guys should come whenever travels open up.
**Boaz:** I'm jealous. I am jealous.
**Eldad:** A few moments of silence!
**Boaz:** I recently came to the office after working from home for a long time. That was refreshing and I went back home and then went back to the office, not fair!
**Amir:** Similar, right?
**Boaz:** These guys go to Phuket and all these exciting places. Okay! So, let us start with a short intro about you guys. Tell us what you do at Agoda beyond the fancy titles? Who goes first?
**Shaun:** Sure! I can go first. As you mentioned, I'm Senior Dev Manager at Agoda. I manage the data platform teams here. I am currently managing four teams. They together manage the entire, deal with anything that is data-related in Agoda itself. So, the four teams managed the pipelines, our data lakes, the self-service data applications that we built for anyone in Agoda to use; and then lastly, we have a team as well that manages the UI, like creating UI that make the experience cohesive for all these tools that forms. So that is what I do.
**Boaz:** How many people are all of these teams combined?
**Shaun:** In my area, we have about close to 30 people.
**Boaz:** Awesome! And Amir, how about yourself?
**Amir:** So I am complimenting Shaun's effort by doing the machine learning part. We have the machine learning platform that we do in Agoda and a lot of tools and good stuff that we do to have the data scientists get their pipelines and get their models in production. And other than that, I have a few teams that do actual business applications, like personalization, marketing efforts that use all of the amazing data platform tools that Shaun's builds to improve the experience of the Agoda customers.
**Boaz:** And how many people over there?
**Amir:** It is changing all the time. But, the entire data platform, at Agoda, we have 4000. I think at the data platform, we have 300 people.
**Eldad:** Wow!
**Boaz:** Wow. Wow! Okay! That's great! So, let's talk data. I mean, at Agoda, I would imagine data volumes are through to the roof. What kind of data volumes are we even talking about?
**Shaun:** If you're asking about like, messages on Kafka, which is our main data pipeline, like how data moves around in Agoda itself, we do about a trillion or so messages a day.
**Boaz:** Okay!
**Shaun:** And then if you're talking about data lake, that comprises about like tens of petabytes worth of data.
**Boaz:** So Amir, on the ML front, a typical slice of data challenges you guys look at, how far back do you go? How many events are you looking at? How much data volume?
**Amir:** In terms of, let's say predictions done daily, we just passed the 60 billion predictions we do per day from our models. And all of that is based on historical events and future predictions that we make. So huge pipelines processing, billions of rows every time.
**Boaz:** So let's break down, you know, the data stack for a second. I wonder Shaun, you know, maybe if you could elaborate when Agoda has been around for quite some time, what does the data stack look like? Or how many data stacks do you guys have? And how do you evolve through the years? How does it look like today and how distant is it from how it was in the past?
**Shaun:** I think today, like I mentioned, the data pipeline we are using Kafka for that. We are using Elasticsearch for logging. We have Grafana with a custom time-series database called White Falcon. It is built in-house. Our data lake solution is HDFS. Then, we have Yarn, Livy, Uzi, Spark, a lot of custom ETL tools. Then, we have also some data governance and discoverability tools. We have a Schema Registry that works nicely with Kafka. Data Market is our discoverability tool. And then, we have custom-like data validation, data quality tools as well. In terms of queering, we are using Impala and Vertica. So that is our ad hoc query story. And then, for UI, we are using Hugo and a custom unified data portal. The team that I mentioned earlier makes everything more cohesive. And then visualization tools, we have Metabase and Tableau, some custom dashboarding stuff for funnels that we built in-house as well. And then that's the data stack from my side. Amir, what is the machine learning side look like?
**Amir:** Yeah. First-of-all, we are using a lot of this; and on top of that, we have some cool in-house build stuff, like a notebook platform that will be Python Opstrat tooling. We are using MLflow, which is a very cool model life cycle management tool, and a lot of Spark jobs also that are used for machine learning as well.
**Boaz:** Are you guys on the public cloud or you guys are self-hosted, self-managed?
**Shaun:** We are On-Premise.
**Boaz:** Everything On-Premise.
**Amir:** We live inside that the data center. We connect the cable ourselves...
**Eldad:** So, that's the real background for a second. Yeah!
**Boaz:** That's why you guys travel all the time. The travel perk includes you must stop a data center and do wire.
**Eldad:** Minus 15 in New Jersey data center.
**Boaz:** As I mentioned, a lot of homegrown tools, maybe a time series data. How do you call it? Falcon? Did you say, Shaun?
**Shaun:** White Falcon.
**Boaz:** White Falcon, first beautiful name. The other names are not as impressive
**Eldad:** Almost as good as Falcon.
**Boaz:** That was a good reason to start some products yourself, you can name it yourself. You get straight to white Falcon, so why not go for something off the shelf?
**Shaun:** I think it is for several reasons. We obviously do try out, like other times you use databases we have always take a look at what is out there, benchmark ourselves around those, come do the feature set comparison. It always comes down to a cost, performance ratio. We think that building it in-house, and we have had this for a long while, and it has been very much key enabler for us to store huge amounts of application matrix on it and across multiple data centers that we have around the world, and also, so it's pretty good. It's pretty great. Shout out to the White Falcon team!
**Boaz:** Awesome! The data stack, I would imagine, serves a lot of use cases. What are the most interesting ones or the bigger ones running on top of the platform?
**Shaun:** The interesting ones, obviously I think, would be, I want to say, Data Market. It is kind of similar to, I guess, DataHub or Amundsen from other companies. I think the DataHub is from LinkedIn and Amundsen is from Lyft, I believe. Did I get it right? I don't know. But basically, yeah, Data Market, it is our discoverability tool that has been instrumental in our data democratization story. We send a lot of data. It does not make any sense if nobody uses it. So, we got to make sure that there are tools out there that they make it so that it is easy to find the data that you are looking for, it makes sense, it is of high quality, and it is usable. It has really been one of the main drivers for us for data usage in a company.
**Boaz:** When was the project launched, sort of, how long has it been in the base?
**Shaun:** I want to say, let's see, 2 or 3 years ago, that is when we built it. Yeah!
**Boaz:** In your current stack, how much would you consider sort of moderate? I am happy with versus legacy. We are always in the process of, sort of, trying to modernize.
**Shaun:** I think that is like a tricky question, I guess, for any engineer. You are never fully happy with a solution that you have. You always want to improve.
**Boaz:** I never met a happy engineer. They are never truly happy. I was always almost there, almost there.
**Eldad:** Always forward-looking.
**Boaz:** Yeah!
**Amir:** It is job security, right? We are never done.
**Shaun:** I think for us, areas which we definitely can improve on, I would say is, around the area of decoupled storage and compute. I will be honest we are a little bit behind in that regard. We have not made that shift, and that is something that we are actively working on right now and it is going to give us that next-generation data platform. So that is something that, yeah!
**Boaz:** How do you go about such a project with those data volumes? How long does it take even to evaluate given the massive lift and shift it would involve?
**Shaun:** So that is one of the pain points, I guess, when you are work data. I think, the biggest, pain point is always migration. All the technologies are pretty cool, like everything; but, in order to use it, you have to migrate from what you are using currently to something new, Right? And that is where things become fairly complicated. Migrations always take extremely long times. I guess the key here is to plan, like plan, plan, plan, plan, plan, plan, plan, plan everything, and then, try and move forward as quickly as you can, as fast as you can, figure things out along the way and then, just adapt to the situation. I would say migration is always the biggest pain point, the biggest challenge.
**Amir:** Yeah! And also in Agoda since we have so many different types of data users, like the data scientists that can write their own code and wizards and they do everything on their own. And it could be like BI analysts that only know how to do SQL and then during these migrations or when you do these changes, you need it to be seamless for them.
So that makes every change even much harder because nobody should even know that something changed behind the scene. Usually, it is impossible.
**Eldad:** The users are basically preventing you from making progress and moving forward.
**Amir:** Exactly. We always ask to just the eliminate the user.
**Boaz:** Which teams will be your early adopters for new tech? Do you have a sort of some internal teams that typically champion for going next-gen, even at the expense of painful migration?
**Amir:** Machine learning, Always!
**Boaz:** Machine learning, always.
**Amir:** Like they get the GPU, the SSA to it and they already started like downloading stuff from the internet, run the latest GPU stuff or TensorFlow.
**Boaz:** If you look at your tech stack evolution, and today you are On-Premise, does that involve moving, becoming hybrid? Does that involve moving to something like S3? Because I guess storage for you is a big thing and a big part of the challenge of migrating somewhere, or how do you think about it moving forward?
**Shaun:** It is an interesting question. I think that the thing is we have always explored the cloud, right? We do constantly explore the cloud. We do have some stuff running on the cloud; but for data specifically, we have always done our research and POC, and we have never found the right motivations or the right reasons or the right…, how would I call it? Like the right thing…
**Eldad:** It is the one big thing that helps everyone make that from transition.
**Shaun:** Yeah, Yeah! Exactly, we would never find that big push, right? that pushes us in that direction. So far, we are pretty happy with On-Premise. I think also the hardware game has changed quite a bit. CPU is now the bottleneck, right? Storage is becoming way, way cheaper, and way faster. Similarly, with networks, right? And so being On-Premise, it does give us some advantage, right? to kind of leverage this system as they come along and to explore them, so I think I would not say one is better than the other. Always, I think in anything, it is just do what makes sense for you. It is just that in Agoda, we have the right people, the right expertise, and the right history as well. We came from On-Prem. So, we have a lot of knowledge in that area and so far, it makes sense for us. That is to say in the future if the cloud makes more sense, we would be 100% on-board. At this point of time, we are still On-Premise, but we are constantly exploring though.
**Eldad:** Makes perfect sense.
**Boaz:** Amir, What about you? Which use cases are the ones that are of the highest-profile?
**Amir:** Yeah. So I think for us, one thing that we got maybe a bit late in the train is like, we were mainly a Scala shop, so we are doing a lot of Spark job and huge parallel. We have jobs using like 13,000 cores for five hours; amazing huge jobs. But people had to be like Scala's expert in order to tune them, in order to get used to them, in order to drive them. So I think, we were kind of late, to see that Python is now really the go-to language for machine learning. And, so we kind of regret not building tools for that sooner; but now, we are already on par. I think very soon Python will win over Scala in at least in the use cases of data application. A lot of, let's say, the less advanced user are already write their scripts and their notebooks on Python and it made like the time to market data projects a lot faster and lot sooner. So that's cool and this is one thing that I think we tuned on maybe sooner.
**Boaz:** From a user perspective - I am a user or an Agoda customer or an Agoda visitor what kind of things happen in the background that sort of start with the melting the time, unaware of even? Can you share some cool things there?
**Amir:** Yeah, so we do a lot. If you and I both opened the Agoda website, we will get completely different experiences based on the past. Like, if I like breakfast and I like breakfast, I will see more photos of eggs and other breakfast stuff.
**Eldad:** Last time we got bacon all the time on your website.
**Amir:** Exactly, exactly.
**Amir:** So, we do try to fit the content to what we think you would like and what you care about. If you are more sensitive to price, then you will get the best offers. We anyway have the best offers, right? But then if we know that that is what you care about in your current trip, then the whole experience will be optimized for that. We do, for example, try to cut snippets from reviews that make sense for you. So if you care about cleanliness, then we will take the review that talked about it, Hey! this hotel is very clean and then, we show that one too. So, a lot of personalization effort goes there, but even before you came to Agoda, right? All of the marketing that goes behind the scene, the email, the popup notification, everything is kind of optimized to make sure that you get what you want and the information you need on the Agoda Website.
**Boaz:** What typically does the initiative for a new ML-based project come from? From your team? Is it sometimes product-driven? How does the thinking around new projects for ML look like?
**Amir:** Yeah! So, that's something that I think we do very coolly. The scrums that we have at our machine learning; they are always a mix of ML engineers, data scientists, and the PO together. And then it's like a 300 beast that kind of try to set the direction. Sometimes it comes from the data scientists, they say, Hey! this is something that we can easily optimize. Sometimes it is the PO or PM that can say, Hey! the business should go that way. And sometimes it is the engineering manager that can say, Hey! other teams did this or we see any other products doing that. So kind of a mix of ideas coming from three different directions, and then get swirled together and the best one wins.
**Boaz:** Interesting. Thanks.
**Boaz:** Shaun, I saw a piece you published a few months back on a medium called "How Agoda manages 1.5 Trillion Events per day on Kafka,"
**Shaun:** Yeah!
**Boaz:** Can you share the backstory there a little bit? Was this published to follow some sort of architectural change or just something that was there for so long and you decided to pass the knowledge out there?
**Shaun:** Yeah. Yeah! I think I wanted to write a blog piece and contribute to the Agoda Tech blog. So I thought this would be a good topic. I think it is an interesting thing because if you read the blog post, it is not so much only about the technical stuff, right? A lot of it is about the human process because you have to remember that at Agoda there are thousands of employees, right? So, certain things you need to think about in terms of scale, not only the technologies that they scale well but the human processes scale well as well. And I think that is something that sometimes we do forget as data engineers that you have to ensure that the human process is scale, so that is a lot of the things that drive towards how Agoda manages at 1.5 trillion. There is all stuff around there like cost management, attribution, even simple stuff like an auditing and monitoring and giving developers the confidence that what they send is exactly what they will receive and having them the self-service ability to just check on those kinds of stuff on their own and then the cost attribution as well, I think, that is a major portion that really allows developers to manage on their own their costs, right? You always have to think about, after all these are company resources. You got to make sure that what you are sending has some business use case, right? You do not want to just send stuff and make the data lake into a data swamp, right? So that is not very useful. So Yeah!
**Boaz:** It's super interesting. So how do you foster a culture or a way where developers are minded to that? Is that something that from day one their mentor to think about? or which roadblocks do you put in place to make sure that it does not get avoided?
**Shaun:** I think there are several ways. One way that we figured out that kind of works pretty well, like we just took a page out of the cloud providers who charge you for every single thing, right? So like, if you mess up a query, you end up paying for it. How much data you store, you pay for it. So I think just building that visibility to allow the developers to see, Hey! this is how much you supposedly will cost a company for sending this much of data or, you know, processing like maybe some, unoptimized query to the actual query engines that we have. So that itself has, you know, driven to make sure like, oh! you know, there is a cost allocated to all of these actions and I have to take that into consideration as well.
**Boaz:** How distributed is the engineering team in Agoda? Which locations are you guys spread out through?
**Amir:** So currently it's a lot more, right? Because we are working from home, so a lot of people are spread around the world; but usually, we have three main hubs. Bangkok is the biggest one where most Agoda seats there. Singapore is also big and we have a small office in Israel that is right next to you. You can jump, maybe eat a sandwich there. It is where we have a lot of smart data scientists there. I think these are the main three and now, we also opened another office in India to increase the diversity and the strength of all developers.
**Boaz:** Got it. So, what are your main challenges today? What takes the most sleep from your day-to-day? What do you worry about?
**Amir:** A lot of things.
**Boaz:** We are not talking about your personal life. We will take that offline.
**Amir:** Ah! Okay. Okay! That's right. I think one thing we find hard, I mean the success that Shaun talked about, kind of making data liability and people kind of owning up to understand that they cannot just send as much data as they want. This is one thing but still, when you try to change the way that people work with data, for example, in machine learning, each team is working with raw data sometimes with just sending SQL or building data frames. And we try to shift everyone to move to more like a feature-solid approach where you first model your data as a feature and then you start thinking this abstraction, okay! this is my feature, how it behaved, how it looked like a month ago and making this change one thing by the other, this is usually that. Like, I wish I could just press a button and everybody in Agoda would just move to work in that way, right? But usually, it is a process that takes a lot of time.
**Eldad:** By the way, this is huge and super interesting and actually one of the biggest things that are happening to engineering, to cloud-native companies or data-driven companies actually moving from engineering, building software to having engineers directly connected to the business feature or building and its cost, its value, its new terms, it is kind of broader thinking on how engineering is done and it is fascinating. So yes! It is everywhere. I can say in Firebolt as well. This is a big thing and being transparent and opening the data and giving visibility, as you said, a payload, an engineer generates payload and that payload ends up at the hand of the user in some form and that changes a lot of things. So, thanks for sharing that is super, super interesting! We should actually drill down on that on future shows as well.
**Boaz:** Yeah! Shaun, what about yourself? What keeps you up awake at night?
**Shaun:** It is interesting, for me, I think what keeps me up awake is trying to figure out what the next generation data platform will look like and trying to see if we make the right decisions, if we have made the right bets along the way. Because in the data space it is not like, yes! we are agile, but it is still like things take time to migrate. There is some time element involved. So that's what keeps me up, right? Is object storage the way to go? Is it not. Is distributed file systems coming back? You know, these kinds of stuff, right? Where is Hadoop going? Where is Yarn going? Right? All that kind of stuff.
**Boaz:** By the way I'm going to reference your LinkedIn account for the second time, looking at your LinkedIn account, your top-line message there says, "I'm hiring, we're building our next-gen data platform. Come join that." And I was wondering if that is there for 5 years or 1 year or a few months.
**Eldad:** It's a good tagline.
**Boaz:** It is always true. As you are thinking about your next-gen platform regardless of what that really means because that could mean a lot of things. Well! What is the objective? I mean, what would you like to plan for to be capable of in 4 years than is it now? Is it more about being future-ready or do you have concrete challenges you want to solve in the near term?
**Shaun:** I think one of the things that we want to solve is the agility of the systems, the agility of the architecture. If you think about the Hadoop space, a lot of things in the Hadoop space are very coupled together, right? The Yarn is coupled with HTFS. It is like Uzi only works on Hadoop, right? A lot of things are very coupled together. So, I think for us, like for me, I am less worried about which systems we ended up picking as long, like in the future, we are in a much better position to have that kind of agility to change something out whenever we need to without incurring that huge long timelines of a migration, right? So that's where I want to be.
**Boaz:** Got it. Okay! Now, we are going to do a blitz question round. We are going to ask you a few questions real quick. Don't overthink, just answer and feel free to cut into each other's answers because there are two of you.
**Amir:** I was waiting for that from the beginning.
**Boaz:** We will count counter whoever answers first, you know, but we love more than the other. We are like the parents loving one kid more than the other. Okay! Let's start - commercial or open-source?
**Amir:** Open-source.
**Shaun:** Both. I say both, do what makes sense for your requirement?
**Boaz:** That's like cheating.
**Eldad:** That's because of Vertica that is why both.
**Boaz:** Batch or streaming?
**Amir:** Batch for now.
**Shaun:** I'm going to say both again. No, the company will use one or the other.
**Eldad:** If you had to choose one, if you had to choose one.
**Boaz:** No, it is something like what makes you feel better?
**Eldad:** Exactly. There is no good answer.
**Shaun:** Okay.
**Boaz:** Are you a Batch person or streaming?
**Shaun:** Streaming.
**Amir:** Exactly.
**Shaun:** From Kafka streaming.
**Eldad:** Everyone wants to be a streaming person.
**Boaz:** We need to highlight the instructions a bit more for Shaun. Shaun! Don't overthink the answer!
**Shaun:** Okay.
**Boaz:** Whatever you are feeling is right for you. Write your own SQL or use a drag and drop visualization tool?
**Shaun:** Drag and drop.
**Amir:** Write your own.
**Boaz:** ML team versus the data development team - interesting insights! So far, no answer that was the same.
**Eldad:** No.
**Boaz:** We are not getting on anything.
**Amir:** Everyone wants to make it interesting.
**Boaz:** Work from home or work from the office?
**Amir:** Work from home.
**Shaun:** Home.
**Boaz:** That is the first agreement.
**Amir:** Yeah! But with a little bit of office here and there.
**Boaz:** To Uzi or not to Uzi? You mention the Uzi so often.
**Eldad:** The original question was - AWS, Google Cloud, or Azure? So Boaz changed that.
**Boaz:** That's true.
**Amir:** Yes! Uzi.
**Shaun:** No Uzi.
**Boaz:** No Uzi. Why not?
**Eldad:** To couple.
**Shaun:** I guess to couple. I cannot run Uzi jobs outside of Hadoop.
**Eldad:** Fair enough.
**Eldad:** To DBT or not to DBT?
**Boaz:** Do you guys use DBT?
**Shaun:** DBT.
**Boaz:** When did you guys start using DBT?
**Shaun:** We only started exploring it. We have not done it yet, but there is a lot of concept and approaches, and ideas that we like a lot, and then we are going to use it for the development of our internal tools to match, you know, what DBT can do.
**Boaz:** What is it you to use for ETL, is it mostly Spark or other stuff too?
**Shaun:** It is driven by Spark, but, it's mainly in-house right? Like we built an ETL tool that is based around spark, but it is extremely easy to do as a whole UI and so on and so forth. It is like anyone in the company just goes in, writes a few SQL, and you are done.
**Boaz:** And no commercials, sort of traditional Informatica style, kind of more these kinds of On-Prem, ETL integration tools.
**Shaun:** No.
**Amir:** Not needed.
**Boaz:** Yeah. Nice. Okay! So now after you guys, trying to be so cool for listeners, it is time to get real and tell us about one project that was horrible for you guys. That didn't go well at all. So we can all learn from your mistakes. Who goes first?
**Amir:** Yeah, that is a horrible one.
**Boaz:** What mistake are you not going to repeat again? Tell us about a project like that.
**Shaun:** I think for me is, I think the failure is that we did not get into decouple storage earlier. Like a lot of our systems are still coupled together and that has hurt us. Because obviously with decouple storage, you could always scale, compute, and storage independently whereas like, in the old ways everything has to be uniform. So, being slower on the bandwagon definitely has some impact on us.
**Boaz:** Yeah. Amir, any glorious failures on your end.
**Amir:** It's too many, every day there is a few but again, not about my personal life, right? It is just always feeling that you are moving too slow. I do not think anything specific, but yeah, getting, for example, by Spark, I think, we should have done it a lot sooner, and in terms of GPU, as we invested a lot in building an amazing project, around Uzi that will allow you to try to kind of do use Uzi to get outside of Hadoop and run stuff on some Kubernetes cluster with SSH commands and stuff. We tried to break this coupling that Shaun mentioned and we failed miserably like it ended up being unusable and we threw it away and our way of trying to rebuilt a new solution but we tried to make Uzi do stuff but he didn't like to do so if you pushed back on.
**Boaz:** But do you still, said, voted for Uzi before.
**Amir:** Yeah. But this is why we insisted, right? Because Uzi is a great tool, even though it's XML based and I think it built in the eighties and the UI is as old as like very old.
**Boaz:** As old as the eighties.
**Amir:** Yeah, something like that, but it is very robust and battle-tested, and it allows you for a lot of features and there is a lot of cool stuff you can do with it. So I wish there was an Uzi replica that is not coupled with Hadoop, but we couldn't find one so far.
**Boaz:** Yeah. We seem to have another issue with Shaun, so maybe you're going to have to answer all questions on your own until we get it back.
**Amir:** I will take it from here. Okay!
**Boaz:** Now on a better note, tell us about the project that did go extremely well or something you are proud of, that is exciting that you want to share.
**Amir:** Yeah, recently we built a very cool too for a model monitoring. Usually, I just talk about machine learning, but this one is also very close to data engineering, I guess, hardcore data engineering because what we saw is that, okay, people build a model, they send it to production. They do maybe an AB test to see that there is business value and that it is actually better than the previous approach. But then after the AB test is over, like nobody's watching over it. Okay. So people see that top-line numbers. People see that traffic is flowing. Bookings are made. People are happy with their product in general, but you don't know what is going on within your model. How good are the predictions that it is doing? What we built based on the stuff that Shaun along the great Kafka pipelines that we have with Shaun, we are sending the data from the model monitoring all the way back to Hadoop. So from the model, the inputs and outputs are sent back and we have a spun of the crunches the statistics of all this data. So kind of trying to find a needle in a haystack, we go column by column, calculate all kinds of stuff, similar to what DQ is doing maybe if you know, TensorFlow data validation, calculating the shape of the data that is flowing and trying to come to insights about how we changed compared to a week ago, compared to what we thought we had when we train the model and these turned out to be a very cool tool. So everything is now connected to Grafana with automatic alerts, and we try to find these anomalies and get them back, getting Shaun back, and, so that was a very cool win and the cool part was the human part. The model owners, they just click a few buttons, register here and there and that's it, their model is monitored. Everything goes automatically for them, and they get these amazing alerts for free and that was very cool.
**Boaz:** How did you build a justification tool to go after a project like that? And how many people were involved?
**Amir:** Yeah, that's cool. So we work closely with the data science department and we kind of search with them, what are the pain points in the beginning? For them, none of them actually complained about that because it was kind of falling between the chairs, between the ML engineers and the data scientist, so data scientists care about, Hey, I want tools to work fast. I want to be able to have a lot of resources to crunch a lot of data. And usually, once the model is in production, they cared a little less about that. Well, the ML engineer kind of felt that, okay, this is a machine learning model, like the data science responsibility. So we kind of know that nobody kind of owns that area and maybe it should be owned by the platform and that was a reason for us to go in and do that. And also there was a streak of failures and actual incidents that happened that caused, like a platform degradation. And we said that, okay, it justified to build a tool for that and a platform from that.
**Boaz:** Awesome. Shaun, what you missed is the question because we lost you for a minute.
**Shaun:** Yeah! Sorry, my place got a blackout like it happens in Bangkok.
**Eldad:** It happens.
**Shaun:** Yeah.
**Boaz:** So now it's your turn to share a project that you were proud of or a great win that you're happy with.
**Shaun:** I think that the Data Market, like the stuff that I mentioned previously, really has beyond that discoverability that it gives to everyone in Agoda. It also serves as a central place to get the information about a data piece, so you can get data quality information from there as well. You get to figure out who is sending this, from where, all that kind of cool information, all in one single piece. And that really has been instrumental.
**Boaz:** Awesome! Okay guys, I think, we are reaching the end of the show. You've been great. Absolutely, exciting to see what is happening behind the scenes with data at Agoda. I'm going to think of you guys next time I book a trip or something.
# How do Canva's engineers and analysts scale data platforms to keep up with growth? — with Krishna Naidu (/blog/how-do-canvas-analysts-and-engineers-scale-data-platforms-to-keep-up-with-growth-with-krishna-naidu)
Canva is one of the hottest, if not the hottest, graphic design platforms out there. With 55 million active users and around 500 million dollars in annual revenue, Canva is an unstoppable powerhouse. They were recently valued at 16 billion dollars!
So how do Canva analysts and engineers scale their data platforms to meet the company's insane growth?
To help us find out, we recently invited Krishna Naidu—Data Engineer at Canva and expert in building large data platforms—to speak with us on The Data Engineering Show.
Listen on [Spotify](https://open.spotify.com/episode/1zMGZ39RmRwMel41EyFPNA), [Apple Podcasts](https://podcasts.apple.com/il/podcast/how-canva-fosters-collaboration-between-analysts-engineers/id1561927688?i=1000521449359) or watch on [Youtube](https://www.youtube.com/watch?v=btF3Twk2_q4).
## How much data does Canva deal with? [#how-much-data-does-canva-deal-with]
Canva processes a huge volume of data. At the time of speaking with Krishna, the Canva data warehouse consists of about 400 TB of data, seeing daily volumes of about 2 TB of raw data. Their biggest data set is their event tracking table, which tracks all of their analytics events.
## How many people work on the data team? [#how-many-people-work-on-the-data-team]
At Canva, the software is always changing so the company requires a large team to keep up. There are currently about 20 data engineers, 20 data scientists, and 40 data analysts—and the team is still hiring globally.
## What does the data stack look like at Canva? [#what-does-the-data-stack-look-like-at-canva]
Canva has historically used both a data lake and a data warehouse, and will likely continue to use both in some form.
Streaming is the main source of incoming data. Incoming data goes to the data lake and is stored nicely in Delta format. Delta format is the foundation of the data.
From there, the data is loaded into the data warehouse, which uses Snowflake.
They also keep a raw data lake. Currently, both the warehouse and Delta Lake consume from the raw lake but Krishna and the team are working on transitioning with Snowflake to consume more from the Delta Lake to cut down on repetition.
Krishna was hired at Canva to revamp their data warehouse. As Canva began its explosive growth, they struggled to scale the existing data warehouse. They needed a better way to gain the performance and storage that they needed.
## What is the data team at Canva focusing on now? [#what-is-the-data-team-at-canva-focusing-on-now]
Now that the warehouse has been revamped and contributions are up, Krishna and his team are working on enhancing analyst and engineering productivity.
With a team of 40 people who might be working at any one time in the development environment, things can get tricky. Using Snowflake, Krishna's team is working on a new workflow that will allow for better testing and rebuilding.
They are also prioritizing giving more control and ownership of the massive raw datasets to backend and frontend engineers, to enable contributions from the broader organization.
[Listen to the full episode](https://www.youtube.com/watch?v=btF3Twk2_q4\&t=1s) for more insights from Krishna and subscribe to our YouTube channel to never miss a podcast episode.
# How Eventbrite is Modernizing its Data Stack (/blog/how-eventbrite-is-modernizing-its-data-stack)
Archana Ganapathi, Head of Data & Analytics Engineering at Eventbrite, shares Eventbrite's data stack modernization process, and how you get engineers to adopt new technologies like dbt which may be outside their comfort zone.
Listen on [Apple Podcasts](https://podcasts.apple.com/us/podcast/how-eventbrite-is-modernizing-its-data-stack/id1561927688?i=1000563268516) or [Spotify](https://open.spotify.com/episode/7sVUVmqQcA28H7kjsyomZf?si=V2johcevQ3mLdLfWtUrSEw)
Guest: Archana Ganapathi, Head of Data & Analytics Engineering at Eventbrite
Boaz: A little bit about Archana. Archana is Head of Data Analytics and Data Engineering, Data Science at Eventbrite. She has been there for almost a year, spent a lot of time at Splunk before, knows data in and out, and she is a little bit upset with us for not using Eventbrite for this event. We cannot please everyone.
Thanks for keeping your smile.
Archana: Will you promise me that next time around, you are going to use Eventbrite.
Boaz: I do not know, talk to marketing about that.
Eldad: Disqualified us for being a startup.
Boaz: Archana, tell us about what you came to do at the Eventbrite?
Archana: I joined the Briteland as we call it about 10 months ago to lead all things, data, essentially what this means is traditionally we have data engineering, the data science sitting somewhere else, and analyst sprinkled all across the board in functional business units. Eventbrite came to the realization, that we need to bring all of these different roles and functions under one big umbrella. So, we can think through this end-to-end and share context as much as possible and solve data problems and look for data opportunities more holistically. We can leverage the scale of all the rich treasure troves that we are sitting on.
Boaz: This is why Archana has one of the best of the things, titles, which is Head of Data Analytics/Engineering/Science at Eventbrite. We need to come up with a new name that can umbrella.
Archana: It is all data.
Boaz: It is all data. So, maybe before we go back to Eventbrite, you spend part of the time at Splunk. What did you learn in that career at Splunk, doing a variety of data positions, that how you feel, lead you to this phase of being ready to take on this new challenge.
Archana: That is a very good question. I think if I reflect back, the journey started much before I even joined Splunk. During undergrad and grad school at Berkeley, basically, a lot of the research I was engaged in was constantly centered around data. This was before big data was a thing or data science was even kind of a formally recognized profession, so to speak. Really, it came down to how we take advantage of traditional computer science systems, research, and technology out there to allow us to scale up insights and get people value from everything that we are accumulating, all shapes and sizes and forms of data. And then, scaleup, compute scale-up storage and make it just easier to drive consumption. That was kind of the backstory, straight out of grad school, I thought if I do not join a startup now, I will probably never go back to it. So, I joined this late-stage start that was then around 200 people at Splunk. I was part of the core engineering team, building some of the platform capabilities to leverage the data that was being ingested, stored, and queried in Splunk. A few years then, I realized, that maybe we are not solving the biggest pain points for our customers. So, I moved to the field to understand where the true pain points are for customers that were trying to use Splunk at that time. And a lot of it came down to people and process gaps, but also, some nice to have again, to drive self-service insights from the platform. Long story, short, soon after we realized we should be drinking our own champagne, just like Slack uses Slack a lot, back in the day. Why should not we use Splunk for internal insights, and I built up the data and insights team, advocated for heavy investments to instrument our product, collect even more data, and then triangulate it with instrumenting our processes to enrich the context there to drive business value. After a good chunk of time, 11 years later, it was time for a change. Then pandemic was a forcing function to want and create that next chapter.
Eldad: Oh! it was a great excuse.
Archana: Yes, it was a great excuse. We are also just partly reflecting on where are the gaps in my own journey here? I thought really thinking about the true scale of impact from data, I need exposure to something that is a bit more consumer rather than enterprisey, and hence, Eventbrite and the mission here is to connect the world together through live experiences, and what better north star for really leaning in on data to bring that to reality.
Boaz: I know even at Eventbrite there is a modernization process now having. Tell us a little about what is in place, what you got in, and what is data stack or environments look like.
Archana: Yeah, absolutely. A bit of history here. When Eventbrite started primarily thinking about ticketing. The data stack as well was designed primarily for ticketing and transaction management and reconciling our books and, those kinds of use cases. Over time, as the demand for insights and analytics grew, there was a lot of duct tape that was fit on top. Early stages, everything threw it all into my SQL database. And then slowly you see, okay, maybe we need some nicer pipelines here and there. So, Spark on EMR was primarily used for compute and the query engine is Presto, storage all our data, moving around S3, HDFS, a lot of the metadata over time is sat on Hive. Then, Tableau was used for dashboarding, and Luigi for orchestration. A lot of technology was right at the time, decisions that were right at the time but if we really think about future-proofing, the infrastructure, this is not the stack that will get us to the future state. So, that is where we are right now, thinking about modernizing.
Boaz: What do you think was your tipping point on that moment of realization, Hey, it is time to reconsider the change.
Eldad: Join. That is as simple as that.
Archana: That is a part of it, but frankly, I think a lot of that, demand also came from bringing data science and analytics under the same umbrella as data engineering, and saying, "Hey, here is what we are trying to do. Here is where the current infrastructure is not meeting our needs." If you think about it, there is reporting, there is ad hoc analytics and then, there is really driving some of these insights from the data back into the product, data power product functionality, whether it is even simple heuristics or fancy machine learning models, requirements change for what needs to happen end-to-end to make that a reality and make it a good experience for our customers as well. I think that was the forcing function and as part of that, we just took a clean slate to say, okay, if we were to design this from the ground up, what do we need to solve for it? Some of the pain points are really like stuck in this chicken and egg loop, our platform is kind of killing over so we cannot support some of these use cases and scenarios, so hold off. But then, in the process of ripping off some duct tape and adding more stuff, you are still stuck in legacy and more things that, snowball and cascade, and any incidents or bug on the product side that impacts data quality, for instance, now becomes the data team's problem, also to stop, undo, redo replay, and it is days of cycle time to get back to clean slate state. I think all of these... it is a domino effect that happened together. And to your point, me joining was a good checkpoint, to say, okay, you know what, let us figure out where we need to go.
Boaz: How do you go about that? The organization needs a big stack of support and then there is a huge journey ahead of us. How do you manage that? How many people are assigned to the new project or are the same people working on the data stack, is there migration plans? Is there a deprecation of all things plans?
Archana: Yeah, that is a very good question. I am under no false misconception that I have all the answers, but I can share what I have done. I guess. The first step, really when I joined was to listen. I get a better understanding of what people want to do, where they are getting stuck or where the challenges are, where their blockers are today, and then figure out how much of this is a fundamental infrastructure technology problem. How much of it is a process issue? How much of it is just like knowledge and awareness? Do people even know this is the place to start? And there is a rich set of data and dashboards that they can leverage already and is there an access problem? Really just going through and figuring out where all the current challenges are and then, that just turns into the requirements for what we need to build. Based on that, the first thing we realized was now we need to modernize our data warehouse. That is one decision we need to make. We also need to make it much easier to instrument and collect richer data at the right granularity and this is upstream of data, back to the dev teams to say, "Hey, this is the information that is missing that people need for what they are trying to do with the data." So, almost kind of teasing apart, data producers' requirements and constraints and data consumer's requirements and constraints, and then on my team's plate is how do we build out a platform that simultaneously solves both?
Eldad: Switching from XML to Jason.
Archana: No comments!
Eldad: Long term, long term that is long term.
Boaz: You managed the part of your origin of stories, putting this under one's roof, the various teams. Tell us a bit about that. What happened? I mean, is there still science to data engineering, so how was the change?
Archana: I think the key is really more communication, more shared context, and shared aligned goals, that matters a lot. If you think about it, if incentives do not align, there is really no benefit to solving for something truly end to end. That is the first step, to say, our biggest goal for North Star here internally is to enable all bright links. So everyone internal to Eventbrite, to have access to the data they need to do their day jobs, arm them with the insights they need to run their own part of the business and make sure we are that bridge between the data producers and data consumers. So, getting folks to talk to each other, put themselves in each other's shoes, and empathize with the challenges or kind of what are each function trying to optimize for and building that awareness went a long way because there was a lot of realization that was not happening around. Here is where I am getting stuck. I didn't realize that it is not a platform limitation. I just did not know that this was the best way to do it, encouraging that dialogue.
Boaz: Now that there is more progress that has been made from the time did you start with your stack, what do you know, for sure, once in a while, how are you getting...?
Archana: I think the top of my list is probably Luigi.
Boaz: Does everybody like the name?
Archana: It is a cool name. I must say that it is associated with the Mario brothers and pizza. Who doesn't like pizza?
Boaz: So Luigi is gone, what else?
Archana: I think next would really just be figuring out how we move folks from having to build pipelines in Spark to just up-leveling that. So dbt as much as possible. Some of the bad behaviors, were kind of, workarounds to the old stack where folks are leveraging Presto as a way to kind of shortcut into the pipeline building which is not the right thing to do in the longer term. That is another thing we will be doubling down on dbt is kind of on the radar.
Eldad: How do you do that? How do you like it because you have a team who writes Scala and build some engineers stuff, and then you counter to switch to dbt and it is such a different universe, different skill set, different people? How do you manage that transition between the team so that nobody gets freaked out too early?
Archana: I think I would almost turn that around to say like, I think historically we were trying to force people that were not comfortable doing some of those things, to use those tools that were outside their comfort zone. Now we are just saying we have the foundations to make it easier for you to do the things that you have to do. Just double-clicking on that a bit. We have a data platform, an infrastructure engineering team, and an analytics engineering team. Analytics engineering should just double down, focusing on driving consumption and building those gold layer data large stack, making everything easier downstream. But that was not necessarily where all their time was spent historically just by nature of the tool stack and toolset that we were using. So, I think it is a big welcome to the change and they are eager to modernize and learn the things that will make their lives easier.
Boaz: Also, Archana is under the BI side. The reports and the dashboards will often have to be remade, are people worried about that?
Archana: That is a good question. Well, it is okay for right now, but I know that there are better tools out there that we need to lean into and I know, Apun mentioned, that Looker was one that they use. There are others as well, that are much easier. I almost think that there is not going to be one size fits all. Also, maybe I need to do the same exercise around, like, what is the preferred interface that each of our data consumers has? In some cases they want dashboards, sometimes they just want the handholding, like, this is the thing you should focus on. I will give you that full service, analytics experience to handhold you through insights from the data and how that should drive your decision? The third flavor, I would say, there are tools that people are already using, and they just want data fed into the tools that they are comfortable with. For instance, sales prefer CRM tools that they are already using. So, the more we can push things automatically into the interfaces they are comfortable with, that is going to matter.
Boaz: That is going to be painful.
Archana: Well everything is painful, but everything is also kind of like if we take it apart into a piecemeal thing that we are solving for each pain point, then we will see some things that we can generalize into patterns and solve for and maybe that is bringing in a third-party tool. In other cases, they already have what they need in terms of the interface. Now, it is just how do you route things to the right people in the right way?
Boaz: What is one thing that sort of so far you are super happy with the specific pain that is already solved for, that you are just happy about?
Archana: I think they just love the partnership with the data team and really being able to lean in on, like, tell me more proactively how I should be thinking more data and really working through with end-to-end from. What should I collect, if these are the metrics that I am solving for and optimizing for and bringing in some of that end-to-end thinking? It is almost like being a consultant in many ways. And kind of having more of the analysts, like, channel the inner consultant in the way they approach solving for the internal stakeholders.
Eldad: I miss the days when you just guesstimate the stuff and use intuition, and gut feeling, and now you were freaked out about following through with the wrong KPI. Makes sense. Life is getting more complex when data is...
Boaz: Migration stories, everybody, some people probably think, ah, you know, I would not bother data stack, those shouldn't care about. One thing is for sure if you are on a data stack from the bottom for a while. So migration stores all is more important than we think
Questions from the crowd.
Audience Speaker 1: Consuming your data at Eventbrite for a long and actually also using it but that is pretty interesting. You have descriptive tags. Do you use a tool like Yokozuna and pipeline into their warehouse? And because you can let people rank the term that maps back to the event in lights more or you let people say I follow this and they get notifications. So you have a feedback loop, not just for your internal consumers, but for your external users and event holders where you can weigh what terms really map to the events people want to sign it for, register for, pay for, and to convert him later. That is the area where you just literally let them follow the tags better. They will forget about the really good tool like this, it is super fast, textbase, and then out of that, you can let get more discovery of earlier events. You can have a higher conversion. They will probably pay for the marketing part.
Boaz: Hire him!!
Archana: Yeah, I am eager to hear more, but I also want to plan, I don't know if you've looked into our new marketing tools suite, but highly encouraged that.
Audience Speaker: I have downloaded a new Eventbrite app, but I was like this Oh it was painful enough to use the old app. So, I didn't look at it.
Archana: Okay, we will chat offline, but certainly something that we are solving and moving on.
Audience Speaker: Actually more useful for me. We will definitely talk about it now.
Boaz: Anybody else?
Audience Speaker 2: Okay. What was the final nail in the coffin for Luigi and why do you think that there is any escape?
Archana: I would not say it is the final nail in the coffin. You still have not done that migration successfully.
Audience Speaker 2: Okay.
Eldad: There are multiple coffins here.
Archana: Yes, multiple coffins. Yes, multiple opportunities to be born again. How about that? It really comes down to just the Daisy chaining of pipelines and the capabilities around automating that better and that is where I think airflow frankly has done leaps and bounds better in that user interface.
Boaz: I have another question based on that, how much do you feel, data engineers in general, prefer working on the data, most interesting. Do you feel the teams that they are excited to do that or it is modernized, or do I know what I am more concerned about that keep doing as I do?
Archana: That's a very good question. I have been doing a lot of thinking about that. I would say it is a 50-50 split. There are folks that are just in their comfort zone with the tools they have been using and I know folks who have gone from one role to another, who just want to follow the stack that they feel happiest about or feel like they are most knowledgeable about. Then, there are other folks who at least right now, the data engineering team at Eventbrite are super pumped to just rip off all things, legacy, and just modernize. It actually makes their lives easier day today. And some of the more modern capabilities around like scaling up and scaling down to zero and the cost benefits of that, the performance benefits of that, and checkpoint rollback recovery, those kinds of capabilities that people have lost sleep over not having. So, that is where a lot of that excitement comes from and push for, yeah, let us do this as quickly as possible.
Boaz: Awesome.
# How Klarna Designed a New Data Platform in the Cloud (/blog/how-klarna-designed-a-new-data-platform-in-the-cloud)
Klarna is one of the leading fintech companies in the world, valued at $45B. While many corporations are "stuck" on-prem, Klarna made the move and today is a cloud-only company. Gunnar Tangring, Klarna's Lead Data Engineer tells Boaz what this new modernized stack looks like.
Listen on [Spotify](https://open.spotify.com/episode/1xZvW4lvciSW8qv7pmGLLj) or [Apple Podcasts](https://podcasts.apple.com/us/podcast/how-klarna-designed-a-new-data-platform-in-the-cloud/id1561927688?i=1000565772037)
**Boaz:** Welcome everybody to another episode of the Data Engineering Show. Welcome, Welcome! With me today is Gunnar Tangring from Klarna in Sweden. Hi, Gunnar! How are you?
**Gunnar:** Hi, I am good. How are you?
**Boaz:** I am very good. Thanks for joining us. For all you out there who do not know Klarna, Klarna is a super, super interesting company. Klarna is in FinTech from Sweden. It is one of the most valuable privately held FinTech companies in the world. Last year, the last round of investment came from SoftBank, which valued the company at $45 billion, unbelievable. Klarna has been run around since 2005 and has shown an amazing journey and started around just making payments online smoother and a team that kept evolving and pushing throughout the years. They are rather famous in recent years for this kind of, I do not know if you have seen this, but shop-now-pay-later kind of experience. Throughout the years, Klarna has also evolved into being licensed as an actual bank, having its own credit card, which is relatively recent with already more than a million consumers using it, I think. So, a really exciting and interesting FinTech company. Gunnar, is there anything I missed? You have been there for six years.
**Gunnar:** Yes.
**Boaz:** Gunnar is a lead data engineer, architect and more. Tell us a little bit about somebody who has been there in the last six years, and then, we will dive into what you do there and beyond.
**Gunnar:** It is an incredibly good description. So thank you for that. But just my personal perspective more from the data angle, is actually that I remember that I walked into a building and I thought it was kind of a growing startup and it turned out to be a slightly larger company than I thought. But today it is something very different that is incredibly noticeable on the data side. I remember when I joined, there was this BI team, which consisted of a handful of people who handled all the different data requests across the company. The scale we are at now means that we have an entire, what we call the domain of around 60 people doing similar tasks where we have been just five people doing. So it is a completely different ball game, of course, with the type of volumes and there has been a lot of growth on the pivoting into new areas all the time. So, it was always good fun.
**Boaz:** Amazing! Tell us about your roles throughout the years?
**Gunnar:** I started with reporting BI work, traditional working in Cognos at the time and the majority was merchant reporting, but anything people would want to have covered like I want to know this, I want to know that, and that was not exactly what I wanted to do. So, I was working a couple of years on our Hadoop infrastructure, basically building fairly large scale data flows at the time in various forms, a lot of Hive SQL and building frameworks around it. We built a tool called the HiveRunner, as an example, to do unit testing of the data, which is kind of cool. And then, we moved to the cloud. Then you know what you are doing for a couple of days.
**Boaz:** Cognus is not around anymore.
**Gunnar:** No, it is not. But the journey of "Hey, let us move everything to the cloud," I think is when I understood what Klarna was all about. Because it was not some kind of let us do this. It will take time. It is more like, "Hey, let us do this now, fast and get it all done." We did that. We pulled it off within reasonable timelines. So, that was nice. But we shifted our entire infrastructure to a cloud-based only, which obviously data is one of the parts where that becomes tricky because you have to migrate the data itself, but you also need to migrate your tooling. We did not go for a Hadoop on cloud type of setup, so we shifted quite a lot.
**Boaz:** Today, Klarna is cloud only when it comes to the data stack.
**Gunnar:** Yes.
**Boaz:** When was the final switch off for an on-prem? We know we have got companies throughout the years that the length of time it takes to complete is between years to indefinitely. So, just reaching the state where it is completely off is an achievement on its own sometimes. So, how long ago was that?
**Gunnar:** I do not remember 100%, but I think it was probably like three years ago or something like that. But the nice thing was that we were quite happy with our work because we were not the last ones out. So, that meant that someone else was running something on some service somewhere. But when we managed to pull the plug there was obviously an announcement video where someone was going to the computer all that, like unplugging the final computer and then, we ran into a new era.
**Boaz:** That is fine. So many people would not be able to ever experience that. People nowadays are born in the cloud. So, you deserve a medal for being there when on-prem was unplugged for Klarna.
**Gunnar:** We are a fairly modern company, but I still think kids these days do not understand why you have things under your desk, "oh, it is to keep your computer." The computer was the server, which was giving people what they needed. It is a different area, but it has been a kind of a fast transition in the industry. The debate of cloud or not, is fairly done to me.
**Boaz:** This debate is definitely over, I think, even though there are a lot of on-prem activities still around. More than we typically think? Every time there is a survey comes out and it turns out there is so much workload still happening on-prem. Some portions of the industry are much slower to move than we, sort of, on the more advanced side of data and tech tend to realize. Because we live in this modern data stack world and sometimes we forget that we were just a segment and that there is so much workload out there on-prem, but yeah, it is all coming to the cloud for sure. Klarna is over 6,000 people worldwide according to what I see on LinkedIn, more or less, but how many people are spread across the different data teams? How many people deal with data and how is the variety of teams structured at Klarna, if you could walk us through that?
**Gunnar:** I think there are two different stories to tell. One would be that we worked in the data platform in Klarna's domain where I work as a domain architect, but our structure internally is basically that roughly half of our domain of around 60 people is infrastructure, people building the platforms. So, providing tooling for other people to do data processing, in a sense building frameworks and making sure the databases are running and making sure that you have the correct setups of the access to data on block storage, and building the data catalog as well and the other half is doing what we call, core models, basically being end-to-end BI teams. So, they work with more central data that would be not possible to own within a specific domain of the company, but that is the next step. Like a lot of our data teams are either something that is working like in the finance department with data. And, be like a team that is explicitly working on data. And, we have teams doing big data processing in various phases, everything from risk positioning to defraud detection to anomaly detection with problems that can occur with our merchants. So, I see more and more of this, like, "Hey, we need to spin up a data team for it," for something somewhere in the organization, less and less, "Hey, can you please do this centrally at the company because we need this to be done from you" and that is very much in line with how we are trying to build the company, that we want to have. We want to have the domain knowledge close to what is going on and, I think for data that is in particular extremely important because I see a lot of cases where someone is being asked to do something but I do not really know why and they are deemed to be the data people who know exactly how to handle the data. But how to handle data will be very much a product of what the data, like, why is this data generated this way? What does this field mean? All of these things are impossible to know if you are working in some kind of decoupled function, central at the company, you might be extremely good at sparse indexing or sort of really good for performance tuning, but it does not really help if you do not know what you are doing.
**Boaz:** You guys are looking into sort of a data mesh implementation?
**Gunnar:** I would say so. We have looked a bit at data mesh but the thing for us with data mesh is actually, it was not something that came from the sky, like, "Hey, let us do data mesh." We looked at what we were doing. And, then we looked at the data mesh and we realized that this is very close to what we are doing and we can learn some things. We can try some things into terminology. But for me, the key takeaway with data mesh is the ownership aspect. I want to have strong ownership.
**Boaz:** You are saying, it is not about us picking up a data mesh guide and implementing it by the book if the book even exists. As you know, the data mesh in itself at the end of the day talks to something that the industry has always gone back and forth. So centralization versus decentralization in essence, and you are saying you are leaning now towards decentralization and domain ownership much more at this stage where Klarna is, makes more sense for you guys?
**Gunnar:** Yes, but I think we have always been a bit towards that angle, but I do not think we have had the vocabulary for it. But, what you are saying, there is a data mesh book now and I have not read all of it but I will read it at some point and I think it is good inspiration. But I also think it is kind of unclear exactly how you would implement it. Then I see a lot of flame wars of, is this actual data mesh. And, then I just realized that that is not what I want to discuss. I want to discuss, what is the thing that drives our company forward in a good way. I think ownership and how to set the boundaries, those types of discussions are needed. But, I think the data mesh in itself does not really give the answers. It just gives you a framework of how to talk about it in a sense.
**Boaz:** Yes, I completely agree. It is a framework, and it looks different in every company because it depends on people as well as practices and the details of the organization. It is like to some extent they do not feel like being agile. How how do you become agile? There was this decade where everybody was talking about becoming agile in software. There is no one way of becoming agile and there are many ways to implement changes from company to company. But that absolutely makes sense. But, what about data engineering though? There is BI, there is data engineering, how are they split? And is not data engineering more centralized compared to the BI teams or is that also spread across the different domains?
**Gunnar:** I think there are two answers. One thing is the terminology matters. I think we have been discussing if we should release the title, analytics engineer is similar. Because the people we have and who are working with, what a lot of people would call BI would potentially be labeled as data engineers. And that could be the wide-scale, you have everything from someone working with building automation of pipelines and someone implementing the use cases. So, I think there is a space there were titles could make a difference. But, I also think the type of data warehousing work or whatever you would label it, that we are doing might also be different in a sense that you fast fall into the scale of things. So, you do not really have maybe a writing SQL all day, but you are still like, if you do not know how to perform this on SQL then you will have problems like this. I think it is a title question in a sense, but I agree the things we are doing that I would label as data engineering and not analytical engineering are more centralized. If I take the clear example of building and implementing and adopting frameworks for making sure that we can build analytical pipelines, that is something that we considered to be something we should centralize and offer as a standardized component. If you talk about data mesh again, I think this is building the sidecars, but you need to run your analytics products in a company. For us, it makes sense to centralize. It makes sense to standardized because you get so many things out of the box and the way we work is if you want to build your own thing, go ahead; but if you want to integrate with 10 different available tools, if you want to be compliant, then it is probably a whole lot easier for you using the framework that you already have, but it is a give and takes, like if you have a super-specific use case, then you can always build, what you need for that basically?
**Boaz:** Saying though, everything above data pipelines to some extent you prefer an analytics engineer, maybe somebody that can do full stack data essentially, a mix of data engineering and analytics. And, I think in general, that is a trend we see in the markets. The term analytics engineering is half picked up, but people are starting to like it, but actually, I would like to call it full stack data developer or something like that. That is a kind of combination of meshing BI and data generated together because you know, you cannot do much today if you are not able to go further down the stack, roll up your sleeves and do some kind of data engineering to some extent. And, I think more and more data professionals are finding that out and that mixture of bridging the tools is picking up. Interesting to hear that you are going about that at the Klarna. But do you guys already use the title analytics engineer or not yet?
**Gunnar:** No, we are discussing it. But I mean to come with them, like the full stack, I think of it very much just like data being a field, it is not unique to data, but I think in general, I think it is more and more important nowadays to be a bit T-shaped than actually having one technology or something that you are good at and then having something else to combine it with, and that might be that you have, maybe a great building analytics pipelines and you are good at domain knowledge of finance or something or it might be that you have like DevOps capabilities to help with other things and I agree that a wider stack is a strength that you need to showcase. I think it is incredibly hard for everyone to have enough to fill up the entire scale. So, from that perspective, I am sometimes thinking, "Hey, it does not matter what type you have, because I will still have to probe and understand what you are doing." I think that is particularly, I realized as being an architect for a while, because when you talk to other architects, it is just such a scale of everything from, I do not know any details to, I know all the details, to hydro houses, like literally of course, but I have that confusion with some neighbors that they thought I was growing abscess, but I just asked, do you have a database? So, then maybe I can help. But, I do think that what are my core things? What would be my two or three things to pick up on, some kind of strengths? That is how I view profiles in general and I think I am more inclined to want to hire someone who is really good at two different things, as opposed to someone who was more like a Jack of all trades type of person. That is where I might post full stack, but I do hear what you are saying.
**Boaz:** Got it. Walk us through the current data stack. Now everything is in the cloud, how does the data stack at Klarna look like?
**Gunnar:** We have what we call a data lakehouse and we thought we branded the term and the print hats and things, but then it turned out that someone else used the same term before us, but we were happily unaware and just thought we had invented something new, but in essence, we are running on AWS analytic stack. So, we were quite heavy use of.
**Boaz:** By the way, you guys should make some noise about it. Go online and shout from the roofs. Hey, we coined the term data lakehouse.
**Gunnar:** I have told Databricks but then I Googled it and I actually found some reference where I think they were using the term before us, but...
**Boaz:** Never mind the facts. Let us rewrite history.
**Gunnar:** But I mean, we did demand, like, in that sense, we did coin the term, but it is not something super complicated in the sense, but if you think of it from more of an API perspective, it is a platform where you can publish data and as a producer, then, you can consume it as a consumer and you do not really have to necessarily worry about the exact location of the data or exactly how things are working within this box. But it is a combination of basically S3 and Redshift and EMR clusters on the spark jump running and we managed that centrally with the configuration possibilities for the users and iterating on it every day, of course.
**Boaz:** In your transition to the cloud, Redshift was selected as sort of the enterprise data warehouse, right?
**Gunnar:** Exactly. And that was like if you looked at where we were coming from, we were coming from a fairly interesting scenario where we had a mix of PostgreSQL and Hadoop and the actual data warehouse was implemented in Microsoft SQL. And, when I say PostgreSQL, that was not just one machine somewhere, it was not like 10 machines having the same structure either, it was just a wild mix of a lot of different databases all over the place. So, we went for Redshift as being, like the main data warehouse engine with capabilities of also processing less refined data. So, that is where the Lakehouse term really refers to being able to access through one logical environment where you do not have to go to a different place because you need data from a different domain. You can go to the lakehouse. You have the lake and the house, both order and disorder in the same place.
**Boaz:** There is also, like, Finna used around it.
**Gunnar:** We are not using Finna heavily. It is one of the things we are looking at how to leverage more potentially, but it is expanding in usage, I would say. But, we used it extensively, when implementing it; it was like the go-to tool because it is extremely powerful to use the data for interactive results on fairly large data volume and I was so surprised when using the Finna, but I would expect things to be slower than they were being used to, but a lot of the things were quite snappy. But, then when you are running into some limitations, of course, some tools. It is kind of built to have the pet tool and the Hive is very much the opposite where you are focused on resilience and jobs just running until they finish, the Finna is more or just did not work, I am not going to tell you, but you do not get a response. So, it is a different experience, but it worked well, like when we were implementing the phase and needed to quickly look at the data and draw some conclusions. It has just been very helpful. Also very good for doing sanity checks, if you have a data gap for whatever reason, you can refer to it easier to just stop that.
**Boaz:** What else is in using the stack sort of Redshift?
**Gunnar:** We use Airflow for orchestration, and we have built our framework surrounding it. We have a team that is focusing on building frameworks. So, instead of exposing Airflow to end-users, we are leveraging our own CI/CD set-up for it. So, you are kind of forced to come into our setup where we guarantee that you have virtual control for transformations and you have some support outside of what you can get from Airflow and we do not get very skilled people, messing up too much because you have to go through our Jenkins to get things running basically. One other thing that I would mention but I did not is, our data catalog as well, which is the thing we built recently, based on a data hub from LinkedIn. So, this is one of the things we realized was a big gap when we rolled out their first citation, but we did not have a good way to just give the user a way to discover all the data in a sensible manner. So, we decided to adopt a solution that would be flexible enough for our needs. So, we rely on being able to ingest metadata from other services by pushing the data to the data hub. So, that was a conscious decision to go for a flexible open-source project that seems to have some traction.
**Boaz:** Got it. What data volumes are you guys dealing with?
**Gunnar:** Well! It is petabytes at least. So, the timing varies depending on the domain. Our growth of data is quite big and a lot of the things that have gone live later. When you talk about how much data we have and how much it is growing, we do have hockey stick curves for a lot of it and that is a challenge, of course, but we always have to. My experience has been about when you go live with something new, you tend to be at the state where you are producing a bit more data than you need because you are going for a fairly naive setup. So ironically, even though the volumes are growing, you tend to be able to manage it more over time and it becomes more predictable because you can determine if this is useful or not. But the overall influx, I do not have the number.
**Boaz:** How much does all the data end up in Redshift? All of it or do you do it year to year?
**Gunnar:** No. Not all of it. We obviously want to keep it sensible. But this is always a friction point because it is tricky to work with a situation where you would need to send people to different places, depending on what data we need. I would say we have more than we would wish for.
**Boaz:** What do you guys do for an ETL or ELT batch processing and Spark or other things?
**Gunnar:** We run Spark and some Hive as well, but we run a combination of Glue and EMR. So, that gives us the main use case for using best Glue is really that we get access to a run time of serverless Spark and it is quite convenient because that means we have an API that is well known. If we would want to run things on a different computer that would be next to impossible. SQL is a big thing in terms of standard languages, but Spark is gaining some traction and it is interesting to see that. I would predict that if Spark is being replaced with something, I would expect them to try to keep the compatibility with the API to be able to help people to migrate if possible because it is becoming standard for running replacement for heavy lifting.
**Boaz:** Tell us about your day-to-day. So, what are you working on now? What does your job look like?
**Gunnar:** My main focus is iterating on our architecture. So basically, more technical, reworking the typology of how we have our sizing of different components in the data stack is my main thing. Obviously, I am doing various other things, but that is the big thing.
**Boaz:** Where do you want to see two years from now? Where do you want to see Klarna in terms of data capabilities?
**Gunnar:** My dream would be to have CRO concurrency concerns basically. I would want to have a situation where everyone who just wants to do some data processing would be able to do it, and just pay for what they need and not have to worry at all about where the data is. I think that is the main point, even though we have a fairly consistent environment to stabilize the situation where you might be sent to a specific place because of having specific data needs. I would want to break that down entirely and just let people choose the capacity they want and ideally the tools they want. But, I think that is a utopia and I do not think you would not be able to offer everything, but at least like, do you want SQL or Spark, that type of decisions and have frameworks that just support you to the work you need.
**Boaz:** With the data organization that is so big, how do you guys even go about making these decisions? Making decisions that would affect the long run? Can you tell us a little bit about the culture of decision-making around data at Klarna?
**Gunnar:** Historically, it has been a bit unclear, but what we are doing now is going full-blown with the RFCs and the ADR processes for everything. So, to make sure that people have an opportunity to raise their voices about the different decisions, but typically these things also take some time, so it is a process of like, "Hey, we are doing this." Maybe I would propose an ADR for it and then I would get some pushback and then we would move forward with it. But, it is becoming more and more formalized and I think some people love that, some people hate that, but it becomes a necessity when you are growing. You just realize that other people have solved this problem of having growth and it is like some kind of administration, but you can no longer tell people like the coffee machine. We are going to make it change and then it is like, "Oh, but I'm in the Toronto office." "I did not hear what you said at the coffee machine."
**Boaz:** Absolutely, there are the challenges of how you need to adapt the way you work for scale. We feel it also at Firebolt but we have grown tremendously, we spread worldwide and we need to invest more time in writing things properly, sharing them properly, and encouraging people to access them.
**Gunnar:** Exactly. But, how big are you at the moment?
**Boaz:** It is like a drop in the sea compared to you guys, 200, doubled within a year. So it feels huge for us.
**Gunnar:** Yeah, but you need to be prepared for the growth.
**Boaz:** Looking at that journey now, imagine you would do everything from scratch. If our listeners go through the same journey, modernizing their Hadoop and leftovers from on-prem and now sort of are all in AWS, what would you have done differently now that would have saved you some time or headache, that you know today?
**Gunnar:** I think it is a boring standard of things, but I think I would not necessarily focus more on testing documentation, but on strictness on the get-go. I think this is something that will always bite you when you end up making decisions. "Hey, we can do this a bit faster if we cut the bit down on the strictness that you want to have them." And I think going forward with more strictness, this is how we expect data to be produced. This is exactly how things should work, as opposed to letting us try to solve this local problem for now and later, we will implement some chemistry. So that is a very tough transition to make. So, I think that is what I would change. So try to be stricter from the start, and maybe the way you would handle the exceptions because you will always have someone breathing down your neck and forcing you to take some shortcuts. I would probably have some kind of exception process and just gather the poor technical decisions that were made with a clear ambition to move faster. Because I think that would be helpful not to solve those cases, but to get an overview of what we need to learn for the good future cases. So, that would be what I would change I think.
**Boaz:** Awesome, thanks. I am thinking, what else we have not covered? For Klarna, you are dealing a lot with architecture today. When was the point in time where architect for the data became a full-time job, became something people decided to do, we need architects, people to do this as a side job?
**Gunnar:** That is a good question. I think it probably must have been like four years ago or something. And, at the time, it was a product manager who stepped into the first architecture role. I thought it was a bit weird then because I again like titles and how we think about it. But now, it makes a lot of sense to me, I think. I come from a data development background entirely. For me, I realized the type of things I need to adapt and pick up. It is a lot of actual product management because you are building a product thing and you need to think of it that way and we have been looking at architecture like the way we work. I access the architect of the domain on the actual technological platform. But then looking at how we drive architecture, on data modeling, that is more of like team responsibility in a way as well and I think that is all scenarios where it is a bit tricky to like find the exact sweet spot for help to do that. Like how much you should, again, like centralization versus Federation. Like, how do you get everyone to build the data models that are consistent with each other, without having someone centrally telling them exactly what to do and I think that is one of the things that is a bit of a challenge. I do not think there are some standard solutions for doing it, but this surely I need for like, having syncs on how to do that type of architecture and we have some things that come centrally in terms of how you should name your fields or the different like counter codes, trivial funny example, but like, it is easy to standardize exactly how you do it. But then it is just a never-ending list.
**Boaz:** Interestingly, the product person moving into the data or like you said more and more thinking about data, as product, which is by the way, natural link back, sort of to the data mesh story, because in there the causes of the data as a product is also heavily included in the story, but it is true. At the end of the day, I think more and more companies are doing that without noticing just as you guys are going to notice we are doing something that feels like it is a data mesh. Many companies that did, have realized that unless we treat data as if it was a product and understand end-users who are, the internal, could be analysts or whatever, it will definitely make our life easier down the road and it just has picked up like crazy, which is true. What are the top data use cases, workloads that run today, that the company is very reliant on and is interesting?
**Gunnar:** I think that is a hard one to answer, but obviously there are processes that are more important and central and they might have a smaller scale of what they are doing, but be extremely valuable, but then you also have the actual scenario where almost all the different product development teams are doing, like AB testing of their features and when you follow up like train models for how we are doing our decisions and every company is doing bookkeeping and the financial reporting, I think as a company, we are more skewed towards both the product development and underwriting aspects. I think that those are what sets us apart a bit, and it is also partly why we have probably a different load profile from a lot of other companies. I would predict that other banks are not doing the same type of large data processing in the same sense. Because if I look at their web pages, it does not quite look like they are doing AB testing of every feature, it looks more like someone thought something very sensible through and then built it. Then it is Okay. But, for us it is more like to the core; if we release this feature in our app, what will the implication be? And, those types of things. I think that is the type of data use cases where an analyst in a team that is working with a specific module, would want to know, does this work better than if we do this little tweak and this thing, will more people realize that they need to look at this thing now.
**Boaz:** Klarna in essence is very modern when it comes to everything, the data, maybe not what we are used to seeing in the traditional finance world, but definitely representative of the new age, the modern FinTech companies, where everything has to be data-driven and data is entrenched in across the departments, in their decisions and how they work.
**Gunnar:** I think, there is never a scenario where we take positions, and it is okay to not have any form of data. We really need to back up the company and a lot of these things to make sure that the people are not having to do cut, paste calls.
**Boaz:** I wonder for our listeners try to look up Klarna has a very cool commercial that was viral with the fish, sort of moving down, how do you call it, a slide and then smoothly sort of moving across the floor and then the message is just smooth or smooth payments or something like that. So, I wonder how many fish were part of the AB test, types of fish, maybe Tuna, Salmon, and, and there was an AB test and the right fish was selected for the commercial.
**Gunnar:** Unfortunately, I was not part of that, but, AB testing is such a central thing when the core of what you are doing is just removing friction and the fish had no friction. That is probably part of the message.
**Boaz:** Very creative piece of commercial. Okay, Gunnar, this has been super, super, super interesting. Thank you so much for sharing those stories with us.
**Gunnar:** Thank you.
**Boaz:** I hope you had a good time as well.
**Gunnar:** I did. Thank you very much!
**Boaz:** Thank you, folks. See you next time. Bye-bye. Have a great day.
# The Creator of Airflow About His Recipe for Smart Data-Driven Companies (/blog/how-preset-built-a-data-driven-organization-from-the-ground-up-podcast)
According to [Maxime Beauchemin](https://www.linkedin.com/company/40719957/admin/#), CEO & Founder at [Preset](https://www.linkedin.com/company/40719957/admin/#) and Creator of Apache Superset and Apache Airflow, building a thriving company is not so straight-forward. So how did he do it?
Choosing the right system and services is key for a successful start, and can help you avoid the chaos of having too many tools spread across multiple teams.
Max walks the Bros through his recipe for a smart data-driven company, and the genesis of Airflow, Superset & Presto (with some great tidbits about Airflow's old school marketing approach and how the open source platform took on a life of its own).
**Listen on** [**Spotify**](https://open.spotify.com/episode/6FptytEqK20n6ZLDX8pIwO?si=gDBNIlVnQ9-ufTvoi0aDAQ) **and** [**Apple Podcasts**](https://podcasts.apple.com/us/podcast/how-preset-built-a-data-driven-organization-from/id1561927688?i=1000574866747)
**Guest:** Maxime Beauchemin - CEO & Founder, BI Platform - Preset
**Hosts:** The Data Bros, Eldad and Boaz Farkash, CEO and CPO at Firebolt
Boaz: Welcome to the Data Engineering Show! We are here again, this time.
Eldad: Different setup.
Boaz: Eldad and I are not next to each other at the office. We're actually at home.
Eldad: I have my own mic.
Boaz: With us is Max Beauchemin? Did I pronounce it correctly?
Max: Yeah, as good as it comes, except for people who actually speak French as their first language. But yeah, you did well. I am in a beautiful South Lake Tahoe. So, not too far from the lake and somewhere surrounded by mountains. It's beautiful out here.
Boaz: Awesome!
Eldad: Lot of people who live in those amazing places, they're polite, and then they put the background like the live background to see how amazing the place is and then I guess you are kind of enjoying the view yourself. So, thank you for that!
Max: You might be able to see glimpses of the lake in the background and a lot of pine trees, but yeah, I'm saving that view for myself here.
Boaz: I cheated by the way, by pronouncing your last name correctly. Do you know what I did? I used the amazing LinkedIn feature where you can click to listen to the pronunciation of your name. So that's how I heard you pronounce it yourself.
Max: Oh, nice. So, I recorded that at some point, I guess.
Boaz: You probably forgot about that feature, but it comes in handy sometimes.
Eldad: I never knew it existed.
Boaz: Yeah. So if you go to Max's LinkedIn page next to his name, you see this audio button, you click it and you hear Max pronounce his own name.
Eldad: Nice. We'll share the link after the podcast.
Max: That's a pretty cool feature. No one has excuses about mispronouncing my name.
Boaz: I won't record myself. I would like to see the struggle, the first attempt.
Eldad: When you put this signature, your LinkedIn signature kind of puts in action in the link that takes you straight to play off your name and the page opens up and you get Linked in telling your name, pronouncing your name properly.
Boaz: Always more innovative. Amazing, amazing!
Eldad: Tons of innovation.
Max: Yeah. Really don't mind it. Like people pronounced my name. I just go by Max. Yeah, just call me Max. And I don't mind it very much.
Boaz: Max is quite the data guru. If you haven't heard about Max. Max has actually started [Apache Airflow](https://www.theseattledataguy.com/what-is-apache-airflow-data-engineering-consulting/#page-content) back in 2014 when he was at Airbnb. Shortly after in 2015, he started Apache Superset. Later on, he moved on to found Preset, a commercial version of Superset. And going backward in time before Airbnb, he spent time at Lyft and Facebook. So, he is an amazing data professional. And you know, the rule for the data engineering show is if you listen so far, we bring in data practitioners, not vendors, nobody to sell anything they're building. So Max, even though he is the founder and CEO of Preset, will actually talk about Max as the data practitioner and...
Eldad: Burnt scars.
Boaz: From the data world and less about what he is selling in the world out.
Max: Yeah, it's interesting. Over time, I started the company a little bit more than 3 years ago and I was coding a whole bunch at the beginning. I was wearing all the different hats. Then I stopped coding very much in the past year or two, but I still do a lot of data engineering. So I'm still in the data pipeline, still building a dashboard, and still analyzing our data. So, I'm holding onto that data analyst and analyst-engineer-type role for maybe like 10% of my time, but I don't code. I don't develop as much anymore. I don't contribute as much to Airflow and Superset as I used to, just because it requires a lot of contexts, a lot of time.
Boaz: Yeah, it seems the listeners quite surely understand that Max has a problem with delegating. He is a CEO who cannot let go of his passion for data engineering, still hands-on coding until this very day.
Okay, cool! Let's get started. Before we go into actually, Preset and it's super-interesting to hear how you've built a modern data stack there when you started a company, let's go back actually to your time maybe at Airbnb. Tell us a little bit about how you got into data throughout your career and the role you landed at Facebook and then in Airbnb and we touch on all the projects that you did there.
Max: So, I started my career as what you would call today [a data engineer](https://seattledataguy.substack.com/p/tips-for-hiring-junior-data-engineers), but I was a data warehouse architect. I did a little bit of web development. Then I became a data warehouse architect/business intelligence engineer. So, I had a good run, almost a decade worth of using the previous generation tools. So things like business objects, Informatica, and a lot of ELT back then too. I would just write a lot of store procedures either in SQL server or Oracle. So, writing a lot of ETL, building a lot of dashboards, and organizing the data for the whole organization. At that time, it was at Ubisoft. So I did that at Ubisoft video game company. It was super fun. Got my foundation in data. So tons of data pipelines, data modeling, dashboard building, that sort of thing.
And then I joined Facebook in I believe 2011 or 2012. Well, so I skipped Yahoo. So I went to Yahoo. It was the birth of a Hadoop. I didn't stay there very long, maybe 2 years or so, but I remember meetings with the people who went on to start Cloudera. So, my manager's manager Amr Awadallah was there and then some of the...
Eldad: He was always there. Every, you know, impactful, bigger than this world event in the data evolution, he was there, somewhere in the background, in the foreground. It's like always seeing the picture, always seeing Max there. Two peers in Yahoo, exactly the right time, and then, yeah, sorry! go on.
Max: Yeah, I was at the right time and then I joined Facebook, which was a few years later. So I was at Yahoo in 2008, then joined Facebook in 2011 or 2012. And then, there's like a big renaissance of data tools there. So people had to rebuild a lot of things from scratch on top of Hadoop and other things. Because the scale of Facebook was just too big for the Teradatas of the world. There's just no commercial database that could scale to petabytes at the time.
And, I think like during my time at Facebook...
Eldad: Hitting Big tech companies, no database company in the world that can build something that is big enough for us and we just have smart people, so let them do it. And I always wanted to ask and since you did it so many times, so successfully, kind of incubating something in big tech, in a big company, how was it back then? How is it today? Is it still happening today? Or do people just live and open a startup? What's your take?
Max: Yeah, there's got to be phases of a new renaissance kind of era. But, I know at that time we had to rebuild everything on top of Hadoop. We built everything on top of MapReduce and there were no better databases. Oracle's very advanced database or Teradata is actually, really good. Database, it just wouldn't be like parallelism, and scaling horizontally, was just not as much of a premise as it needed to be for a company like Facebook. So, Facebook had to rebuild everything on top of MapReduce or rebuild everything with the premise of things, adding to scale to thousands of machines horizontally. So, they created this culture of, "Hey, we're building everything from scratch." A lot of experiments too. So, we had a bunch of different data pipeline tools. One of which is called Dataswarm became the inspiration for Airflow along with other ones. But there was probably like for every project that stayed on Facebook and got used by people, there's probably like a dozen other projects that didn't go anywhere. So, there's probably been like 20-30 different schedulers built at Facebook and then one or two...
Eldad: So, many dead startups. So many startups that could have happened and didn't happen because just Facebook is like let's have 50 projects on data pipelines and one of them will win. And that's you again, by accident again, you were there in that single project, again, go ahead, sorry.
Max: Yes, I was there at that time when these things were happening. Interestingly, I don't know if it's the engine or if the big tech companies like stopping innovation by putting the brains kind of inside their wall gardens, or are they actually stimulating progress with all the wealth that they generate? So, I think it's a mix of the two, but in the case of Facebook, there's a lot of really good open-source help to break the wall gardens of tech. Because people like me, I'm going to join Airbnb if I can work on open source and then my stage for impact is not Airbnb, it's the world. So, there's been a lot of people before me too, that I've had a lot of success with open source. So, I'm just kind of following those footsteps and say, maybe I'm delusional enough to think that I can do the same as people like Jay Kreps or the people behind Hadoop and some of the open-source technologies like LINSTOR I think have been an inspiration to us all too. So, I was like, oh, maybe I'm crazy enough to attempt this thing and other people have done it. How hard could it be?
Boaz: When you joined Airbnb, what did the variety of data teams look like?
Max: Yeah, it was kind of interesting. So, I really grew up while I was there. So it's hard for me to close my eyes and think about what exactly it looked like on my first day. But, I remember, well, there were like a handful of people, about 3 people, Johnson Parks, Aaron Keys, Sid, think all 3 were working on something called core data. So, I was like, people at Airbnb had suffered enough from handling raw data and they're like, we need to do some data engineering. We need to create some core data sets that we can trust and rely on. Later on, these data sets, there's too much pressure and too much pull from the different teams, so I tried to evolve these things centrally. But for a while, people working on core data, we started using early, early Airflow in production, within a few months, to build the core data stuff and some of the core data sets at Airbnb and soon after, we migrated a lot of the pipelines that existed in some previous scheduler called Chronos that was built on top of Meso and then we just migrated that.
At the time, there were like 3 data engineers. They didn't call themself data engineers. It was not a popular term at the time. I think they called themselves ETLeans or Eliens, so that was the name of the team and then there was probably a data platform.
Eldad: Data mart team, back then, the formal names.
Max: Yeah, little fun name. And then, there were maybe 10 data scientists, 10 people on the data platform and then the team went on to become, I think, there were like a hundred data scientists by the time I left.
Eldad: Phew.
Boaz: Wow!
Max: And there's just an army of data scientists. The data engineering was probably like 15-20 people.
Boaz: That was sort of between 2014-2017, right?
Max: Yeah. So the data platform included all the data functions, and was definitely north of 150 people or so
Boaz: Tell us about the context and the role and how it went about with Airflow during those years?
Max: I started a project in between gigs. So, I left Facebook with the premise that I was going to work on something like Airflow, at least as a side project. I didn't know the name at that time. I talked with people at Airbnb and they're like, we need something to manage our dags. We have tons that we're crumbling under the weight of our own pipelines. So, we need something better than what we have today. So, I was like, "oh, that sounds fun. I want to come and work on this." And then in between jobs, I started working on what now became Airflows, I had I think a two weeks sprint. So, instead of taking a vacation, I just decided to start coding this thing and I put it under my personal GitHub. I would join and say, "Hey, it's already open source." Like, what are you going to do about it? So, it was under my personal GitHub and then the moment I got there, I think we got something in production very, very quickly. So it was like within a month or two with me being there. We had some data marts in production very quickly and within 3-4 months, we had all the core data pipelines and we migrated a lot of the Legacy pipeline to Airflow. Also, moved the experimentation framework as a gigantic dag of thousands of tasks to compute all the metrics and all the experiments data. So, there was a big hunger internally. There are a lot of data professionals and people who need to schedule arbitrary workloads. It was also at a time when things like DBT did not exist. So Airflow was preferred for SQL, for scheduling mountains of SQL too. And then there's a bunch of, you know, people doing all sorts of crazy stuff with RPython and IPython and notebooks and Java or whatever it might be. So, there was really a need to schedule thousands and thousands of jobs, big hunger for that. So, that's how it took off.
Boaz: When did you notice that this is being picked up and is extending its reach beyond this project of mine and getting popularity.
Max: Internally, I cannot understate how much I visited or I did some evangelism around Airflow. I was super early. I would visit any company in the valley that would show interest in me coming and talking to them. I would just go and visit them and answer their questions, talk about their data engineering challenges and kind of convince them that Airflow was probably a good solution to that stuff and the Landscape at the time was oozy.
Eldad: That's how you do it, baby.
Max: Yeah. That's it.
Eldad: You go out there and knock on doors and you just do it old school.
Max: Yeah. And I never wanted to start a company or that was not my intention at all. I just wanted to build something relevant and impactful. So, when the VC started approaching me to say, "Hey, why don't you start a company?" It's like, "are you crazy?" Like, that's like, "why would I do that?" And I'm not an MBA. I'm happy. I used to call it doing open source for the right reasons. I'm just here to evangelize and build something useful, for the longest time that was really what was driving me. And then, it was really progressive. The popularity comes like one issue, one PR at a time on GitHub. You're like, "Oh, here's someone that seems to be associated with this company name." Then the mailing list. So, it is very progressive. So it goes from a handful of people showing a little bit of interest to a small crowd and eventually, it's like a mob. And now I heard recently there's I think astronomers did some analysis and there's probably north of like a hundred thousand companies using Airflow today
Boaz: Wow!
Max: It's just insane.
Eldad: Insane.
Boaz: Insane. From Max refusing to start a company.
Eldad: Shut it down, now.
Max: It has a life of its own. So people ask me, when do you know that your open source product is very successful? You know when, if you would try to stop it, you couldn't, right? If I would try with all my might and all the resources in the world that I have to stop Airflow, I could not, at this point.
Eldad: Amazing.
Max: That's when you know, it's successful.
Boaz: It's like the terminator, once machines take over, you can't stop.
Eldad: AI all over again.
Boaz: And then you also started Superset?
Max: Yeah. The genesis kind of story for Superset. Well, first there's a delusion of like I can build a BI tool. I think that came from Facebook as Facebook people had built all sorts of little visualization tools that were very simple to use, very fast time to chart and time to dashboard. You have a data set ready, you can build a chart and dashboard of that data set in no time. It doesn't have...
Eldad: It's so funny, you mention it. One of our engineers joined us from Facebook, maybe 4-6 months ago. And one of the first things he mentioned was how nice and how well data visualization and data consensus is at Facebook. If you send a link, it opens up a page with all the right charts. There's a discussion on the data. So, just now you're mentioning took me back to that conversation, and Boaz has a kind of query history and all the discussions we're having on how to embed data within conversations. So, yeah, it's super interesting.
Max: Yeah, no, totally. I think it's great. Like to talk and you know, I hate to overpraise any big tech giant or whatever, but like we got to give credit where credit is due. I think like a lot of what we call today, the modern data stack, there was like a microcosm of all innovation and modern data stack happened 10 years before at Facebook. They had something like a, Idata which is a data portal, data catalog with a full lineage of everything at Facebook, you can navigate the metadata graph of all the data objects pretty well and do like lineage analysis and impact analysis. There are all sorts of little visualization tools, little schedulers, and then the databases like a database called Scuba. It's a little Druid and Firebolt maybe too in some ways. Like really fast, real-time in memory.
Eldad: They have a really strong theme by the way. Writing a vectorized query engine is really strong.
Max: The Presto team too.
Eldad: Respectful.
Max: Yeah. Very impressed by the quality of the gray matter at Facebook. Like people are empowered to build new things. Things like Scuba, things like HiPal was like a little bit of a SQL, notebook type thing. There's something called Unidash, that's a dashboard building tool that came a little bit after my time and just this culture of like, if there doesn't exist, I'm going to build it. So I think there's like...
Eldad: Now, they migrate everything to the Metacloud.
Max: That I don't know about. Metacloud I guess is their new data center, I don't know.
Eldad: So all of your friends at Facebook, everyone was sitting there in the room and someone heard, it became this huge, huge open source success beyond control. So you had to feel special. So you went on and said...
Max: I'll do it from Airbnb. I looked to open source on the Facebook stuff internally. It was just difficult because everything was tangled up as you know in data, all the different systems, it takes a fair amount of duct tape and chicken wire to kind of hook a data platform together. And it is really hard after the fact to take a piece of that data platform, that's all duct tape and chicken wire, the rest of everything and cut that out to serve it as an open source project. So, I think it's been done in the past, refactoring an internal technology as an open source project. Some companies have done it with some projects, but I'm guessing it's always with the Premise that it might be open source one day. So let's keep this as a microservice that works well by itself, but yeah, I realize..
Eldad: If all the engineers leave us, we can solve it by open sourcing it and kind of getting engineering love back.
Boaz: We need Max to evangelize it.
Max: That happened in the past with Premise. If we open source this, like the community's going to build it, I think in my experience it is not, as if you guys are like, Hey, we're just going to open source Firebolt so that we get hundreds of contributors for free, that typically does not work. It needs to be open source from the get-go and I think it's really often like a handful of core contributors that are bred, that are very close to the core of the project that built the bulk of it.
Boaz: Now, you started Preset in 2019.
Max: Yeah, early 2019.
Boaz: Early 2019. Okay. So, here's the...
Eldad: So what happened?
Boaz: Yeah. What happened?
Max: What changed? Well, I didn't like my life's goal pretty much to push Superset forward since 2015, so it has been like 4 years that I've been working on this thing. I was like, I want open source to come and compete in business intelligence, and data visualization. That was my life goal. I want to build something relevant that's in every other company. So that every tech company, every company who does data, everyone has heard of Superset. So, that was my goal for the 4 years before.
I'd been sponsoring or incubating my project inside Airbnb and inside Lyft for a little while and these companies are super nice and I had this small team of people working with me on these things, pushing this thing forward. But when I started talking with investors, they were telling me that or it became really clear to me that it would be a really great way to take on capital to really be able to push the open-source project forward. So really it's in the vein of like, if I raise my A round with $12.5 million, that allows me to hire a bunch of people that are very dedicated to working on Superset and making Superset great. So, there's always this duality too, like, hey, you need to build an open core. You need to build some crust. You need to build a successful company that makes money too. But I've seen other companies do it successfully, like companies like Databricks, Confluent, and companies of different shapes and sizes. So, I thought I could navigate as well as anyone else on how to give back to the community to grow an open core, but also like to build something that we can sell in and around it
Boaz: Let's get back to Max, the practitioner, for a second. So you're starting a company. Obviously, you're super experienced with data. Walk us through how you thought about building an organization that is data-driven from the ground up and how you went about that.
Max: At first it's like, when you don't have a product out too, I mean, building a company, there's just like six months or a year just like trash. You need to like set up a bunch of stuff, make a few hires, just kind of get going and there's not a whole lot of data at that point in time, but you know, one thing I've been talking about lately is this idea of like data native companies, the same way that there's like cloud-native companies or digital native companies, companies that were born in a certain era act in different ways. And I think like this generation of companies, like Preset, we add just like really easy access to things like BigQuery and Snowflake to things like DBT and Airflow. Like, we didn't have to build our own scheduler. We just like to pick one up. So you assemble your data platform. Now Fivetran. Now, there are open source counterparts, like Airbyte and Meltano. So for us, we just like to sign up for Fivetran. We get our HubSpot data and our segment data all centralized in a data warehouse pretty easily. Like you can kind of assemble these, pick up these pieces off the shelves and they're all like pay as you go. So, they're really cheap or free if you don't have a lot of data. That's how they get you though but...
At Preset, we offer a premium of up to five seats. At Preset, that's a really good offer. So if you're a small startup, if it is like 2, we make sure that works. And then it's like pay as you go.
Eldad: Is it free, full-featured, like any big limited edition or really kind of about 5 people?
Max: It's like 90% of the features are in. I think we block some things like alerts and reports, where alerts are delivered.
Eldad: Held up, held up and stuff like that up, 5 people and you need that to go away. That's how exactly
Max: But if you're 5 people, 2 or so, then the premium tier, so there's premium and then the premium is like $20 per user per month. If you have 5 users that's $100 a month, and then you can have the SSO and LDAP on that tier too. And, we have an enterprise tier too, but what's nice with these tools is you can sign up for them in 10 minutes and so you can start sending data into BigQuery and the bill is going to be like nothing. So, you can assemble a data platform for us. We picked up Fivetran segment, DBT, BigQuery and just started pushing a bunch of data into BigQuery and started reorganizing the data with DBT. We have Airflow now, too. It took a little bit longer for us to really need an orchestrator, but as you assemble more things, so we have things like Hightouch now, too, which is a reverse ETL tool to send maybe product usage data back to your CRM systems. So, now that we have a complex enough data platform, we need something like Airflow, but it took a little bit longer for us to really have a need for that. But super easy to pick up Astronomer just like it's installed in a half a day and you can start jamming on that stuff very, very quickly.
Boaz: How many you...
Eldad: It's scary and crazy, how fast you can kind of play with those building blocks today and how hard and painful and long it was just a few years ago, which is crazy because it was not long enough in our space, you don't appreciate, you said something, you said kind of as new companies get born, they get born differently, and we are getting old, right? We found companies, but your engineers, your people, they are fresh and they are coming with a new mindset. Just, it's crazy. So, it's a lot of ecosystem things.
Max: It's late if you compare, like, say cloud-native companies, like companies like Airbnb that were born, say like 2000. Airbnb is a little early for that, but companies born after 2010 or so are all built on AWS. And that is also hard to appreciate just how hard it is. We used to order servers and machines and we had to wait for the boxes to make it to the data center. That we were renting and we had to wait for this, this and then, usually balding with a ponytail, to like install that server somewhere...
Eldad: I missed that. I missed that. One of my best friends from my previous startup Sisense, Shlomi. Hi Shlomi, if you see us and listen to us, we love him so much and he used to do it old school. He used to fly to New Jersey and he used to install stuff there and sit there in the basement and everything. Now he's running AWS. It's not the same anymore but in a good way
Max: People would make a nice setup for the network wires, like to kind of weave them together in a specific way and use zip ties of different colors that meant different things. But yeah, so the super transformational companies who were born after, EC2 was born, EC2 and S3 as core components of AWS completely changed the game. You can just be like, hey, I need a hundred machines and you get them. But then I think the data space, it's a prolonger for us. It should be like, I need a data warehouse and I need a scheduler and I need a data visualization tool and you don't need to talk to salespeople. It was not expensive either. All you want is a database tool. That's like 50K, now it's like $20 per user per month. Like why? Like just set it up. You can use it today, you know?
Eldad: Now!.
Max: And I think that's now transformative. Like today, yeah, you don't even need to talk to someone.
Eldad: You can though. If you want, you can and that's when you raise money.
Max: Yeah.
Eldad: Because again, 5 people, it's an amazing team, but it grows and from my experience, everyone needs that at one point or another everyone in the company, almost everyone in the company needs that. And as companies get younger, more people need that. So, it's amazing to see, I've been in this space, and Boaz as well, for so many years and it's just amazing to see that the need for data is never kind of...
Boaz: Never stopped.
Eldad: Nobody gets tired of that, so creative.
Max: There is a question though, it's like, why is data 5-10 years behind some of the software engineering practices or like data engineering is like notably behind I think.
Eldad: Expensive.
Max: Because it's expensive. Yeah. It's not a priority too and we've had tools for a long time that was kind of okay. So maybe for a long time you had, I don't know, business objects or Cognos, and you had like Informatica and Teradata and that was your stack and it was okay. And, maybe that prevented the kind of innovation that we see today. And the data team used to be just like a handful of specialized people, like 2000 to 2010 for me it was like if we had like 4 or 5 data specialists at Ubisoft and we were taking care of all the data needs of the company and that was okay back then.
Eldad: You needed IT back then. That's why and now you don't need IT anymore. That's the only difference.
Boaz: But every data that goes by data engineering and software engineering are getting closer together in sort of bi-directional ways. I always say like a decade ago, a software engineer tasked to do a data project or do something with a database would say that's not for me, that's a job for a student and look down on those tasks. And, today you see that more and more software engineers want to do data-related projects and tasks. So much of the more interesting stuff is happening there. And you see also the way data engineers work is so deeply affected by what's happening in the software engineering world and that's the direction everything's headed.
Max: ML and data science have brought a lot of excitement to data too. So, being able to train models and to predict at scale, I think it is something that brought a lot of attention to data and data engineering. If you want to have the algorithms, you want to have some good ranking and feed ranking, and you want to run a lot of AB tests and know what to ship in your product, you need to have good data. So, I guess that pushed more requirements on being data-driven
Boaz: Yeah. So, back at Preset, how many people are in the company these days?
Max: We're about 70 people or so
Boaz: 70. How many people are in-charge or are on top of that data stack?
Max: We have 2 data specialists now, but those are new folks that started less than 6 months ago. There are an analyst engineer/data analyst and a data engineer now, but it took quite a while for us.
Eldad: And Max.
Max: I mean, but I do spend a fair amount of time doing so...
Eldad: Nobody wants to join that team.
Max: There's definitely pros and cons of joining a team with a super experienced data engineer and I'm kind of anal and picky about certain things. But at the same time, I can be a good peer, a good mentor on some things. Yeah, so we're like two and a half people or so on the data team itself. I think it is right, what you said before, like building a company, you don't realize just all the functions that are required. As a tech founder at first, you're like, Hey, I can do a little bit of everything and it's mostly about building a product that sells itself. And then you realize, "oh, no, I need a sales team. I need SDRs. I need like customer success folks that can help my customers be successful in my product." We need education and the whole marketing side. I'm not even going to talk about it because there are a lot of specialties there. There are a lot of data-driven processes there too. So, it's hard to understand, I think, as a tech founder, what really you're getting into and the vastness of the skills that are required to build a successful company.
Boaz: So for other companies that maybe now in the process of building data stacks from scratch or modernizing. We enjoyed talking about how fun it is nowadays with all these building blocks, but still sort of what lessons learned in implementing a modern data stack can you share? I mean, if you had to do the same thing all over now, what would you have done a little bit differently?
Eldad: Faster. Just faster, for three years everything.
Max: I think the recipe that we chose works very well for us. Fivetran works very well to do data sync. We're a small startup, we have more SAS systems than we have employees. We have more than a hundred SAS services that we do, which is kind of insane. There's a whole portfolio of things that you want to use and there are some really core ones. Your CRM is really important and we use HubSpot, would probably pick HubSpot again, and then Fivetran to bring the data from the different systems into and land it into a warehouse. So that's a really easy data acquisition thing. Then, we have to build our own like scraping and analytics events, inside your product. I'm sure inside your database, you have a lot of sensors and you have a lot of logging, every time someone runs a query, every time someone does an action for us. It's like anytime someone creates a chart, saves a dashboard, alters a dashboard, invites someone, all these analytics events we need to bring to the warehouse. So, we use Segment as a little bit of an ingestion layer for that.
Eldad: You are obsessed with observability?
Max: Yeah. To me, there's operational reporting on one side and that's just Datadog for us. So that's more technical like the hardest systems and machines doing and then there's like their analytics stack which is kind of interesting because they're two different worlds and maybe they don't need to be and I think they're less different worlds than they used to be in the past. I'm sure, at Firebolt, you probably have use cases across the chasm of things like operational analytics and more traditional business analytics; but for me, I'm much more on the product analytics side of things.
Eldad: We're very flexible. We let employees use whatever tool they want, as long as it's Firebolt and if they're not happy, if they don't like using Firebolt, it's okay. It's also okay of course. Then, they have Kafka Streams and they should just manage on their own. I've heard Honeycomb, by many people, and I think there's a great project going on. I think, on our end, we use observability now mostly to define success, enable engineers, and have them figure out how to kind of define the success of a feature they're releasing with observability. So it kind of focuses them and removes a lot of noise, it's just going crazy on matrices and events and just choke the system. So kind of, we went through that journey. How is it with your startup? Where are you now on the observability evolution?
Max: On the observability side of the house? So there's a pretty big chasm between what we do for, I'll call it business analytics or product analytics for us. So this is all for us, it's BigQuery, DBT, Fivetran at Preset to analyze this data. On the observability front, we take advantage of everything that Datadog offers like from their traditional logging to metrics logging. They have some sort of time series database behind the scene. And this is a world that I know a little bit less about, but yeah, Datadog got also, APM, like Application Performance Monitoring.
Eldad: That's when you get into that, don't go there.
Max: Yeah
Eldad: If you go into APM, it gets expensive.
Max: Yeah. So, we used that stack for observability and then we used a different stack for business analytics, and then that's much more, the place that I come from. The lines are getting blurred. Databases like Firebolt, I think, now can take either workload, like you don't need a time series database for your observability and a different database for your data warehouse as much anymore. I don't know how you guys position yourself in relation.
Eldad: Consistency is key to getting that done properly, and we can spend the whole podcast on why consistency is so hard on low latency databases, but why it's so needed, but yeah, we use also multiple systems, not just Firebolt obviously, and we've gone through multiple different products and we just realized that sometimes different tools solve the same problem for different people in a better way. And that's also okay. But we do kind of try to manage cost, kind of adding those tools adds up and get this kind of one point overlapping feature sets and you get confused like 5 logging systems. So, we are also kind of trying to go on a diet, and once in a while stop for a second and say, okay, we've been trying things out. Let's stop for a second, and just pick one or two winners, like the fact that you can connect stuff so fast today only gets simpler, makes that...
Max: But that creates the reverse problem, right? If I can spin up both like Firebolt, Imply, PnO, and like three other databases like today, then maybe I will start using all of them. And then you have this accumulation of systems. I talked about that in my Airflow Summit talk, setting up our data platform at our small startup. And yeah, it's so easy to set up systems that maybe that creates a need for more orchestration and more metadata catalog type things. Because all of a sudden, maybe you have BigQuery and you have Snowflake and you have Firebolt for your super hot data sets and you have a bunch of BI tools because I don't know, they're easy to set up and different teams prefer different tools. So, you end up with chaos in some ways. So, you need to keep things somewhat constrained.
Eldad: Spend on marketing. So you need to spend more on marketing to focus and then help people understand.
Boaz: It's easy to set things up, but there are also things that have a lot of information in the community. There's all of the knowledge sharing going on today. It is much easier to talk to colleagues who have tried things hands-on and you know them in person, you trust them. It's not like far out anymore. It's very easy to get information and find somebody who is really talkative, who's experienced.
Max: It's easy to try software too. You can just go on and try it. But yeah, the reverse problem is like maybe you end up with a startup with 70 employees and 500 SAS tools. But here's, I think that's interesting, like talking to the parallel between data engineering and software engineering. On the software engineering side, we accept that you need multiple databases, multiple languages, multiple frameworks, and multiple libraries like there's such a diversity of systems, services, frameworks, libraries, and languages and it's well accepted, right? There's no one that says like, "oh, no, we should have like one language to rule them out" or there should only be JVM. So, there are probably still people that think that. But I think we accept that it's going to be a highly diverse, different solution for different people, a lot of microservices that we accepted in software engineering. In data engineering, it's like picking one data warehouse and sticking to it, picking one BI tool and sticking to it.
Boaz: It's like in software engineering, people accept that there are multiple ways to reach a great outcome, even though you chose a different path to get there and in data, it's like, No, this is the perfect stack he should've chosen and no, any other whatsoever.
Eldad: No, it's just licensing guys. It's just licensing cycles. That's all, that's the whole difference, and people go and build and decide on a stack and they need to close the license and they negotiate and everything takes time and they close the year subscription or buy credits a year in advance, whatever, and then they take the project to production, and then they move on, it works, they move on. They add another project with a new tool because they like or think that is better and they move on again.
Boaz: New license.
Eldad: But I think like and it's a good and bad thing. On one hand, people have much more flexibility and options and on the other hand, they can just pick another tool. They can pick another tech. So you constantly need to justify yourself. You constantly need to get better, and at Firebolt, for example, consumption, it's even more. You constantly need to earn your customer's consumption. If they consume, it means they're finding you valuable. They don't consume, you're not valuable. They're not using you. And it's very out there. There is no way to hide it. There's no way to hide or postpone the contract. So, advantages and disadvantages.
Max: The trend seems to go in a direction of things becoming more or at least at larger or faster work organization, there's more pressure to decentralize things and to democratize things. So every team can decide what they're going to do, and what tool they're going to pick and use. And then of course that creates a different set of problems. But if you want to move fast, centralized structures don't work as well. Because you need consensus and consensus is expensive.
Eldad: Return of business objects, like retro-release, SuperBGA, business objects.
Max: Yes. SuperBGA, business objects, high resolution.
Eldad: Exactly.
Max: Not working on Macintosh, going back to the desktop products.
Eldad: Old pricing. So you get the retro $1 million license pricing as well. It's amazing. A lot has changed since then.
Boaz: Yeah, good. Max, this has been awesome being in the company of two data geeks and founders. I'm sure we can keep talking for too long, but we are running out of time. So thank you, Max so much. It's been super interesting, and good luck with everything at Preset, and we'll keep definitely watching out for what you do next.
Eldad: We'll see you soon.
Max: Yeah, it has been super fun. There's a bunch of things we haven't talked about, so we got to solve a bunch for next time. But I learned from a pretty good conversation with Eldad that he worked on early MDX compilers, which is MDX.
Boaz: Wow!
Max: MDX like multidimensional version.
Boaz: Don't get him started.
Max: Yeah, so we should do it. I wanted to talk about data apps. I wanted to talk about what it takes in a database, and what properties we need from a database engine for the next wave of highly data-centric applications. Like those data apps. So, it'll be material for another show, maybe in a few months, we connect back.
Eldad: Absolutely.
Boaz: Absolutely.
Max: And talk more.
Boaz: Okay, Max! Thank you so much!
Max: Thank you for hosting.
# How Rising Wave Is Redefining Real-Time Data with Postgres Power (/blog/how-rising-wave-is-redefining-real-time-data-with-postgres-power)
In this episode of The Data Engineering Show, the bros sit with Yingjun Wu, founder and CEO of Rising Wave, to explore the innovative world of stream processing systems. Yingjun shares his journey from academic research to creating a Postgres-compatible streaming system that drastically reduces resource usage. They discuss how Rising Wave's S3-based architecture and Postgres compatibility provide advantages over traditional systems like Flink, and explore the increasing role of Apache Iceberg in data pipelines.
**Benjamin - 00:00:00:**
This is interesting, right? Because this seems to be a general trend in data management at the moment, in a sense that when Snowflake came up, they were the ones who invented in many ways, okay, decoupling storage and compute, using object storage as the lowest tier. And now you're seeing these kind of, I would say, new age systems using the same underlying principle to kind of disrupt that space. So RisingWave for streaming, you had WarpStream, which was acquired by Confluent, kind of doing it around Kafka and so on, right? So I think it's super interesting how this dynamic right now is actually playing out throughout the entire data infrastructure stack.
**YingJun Wu - 00:00:46:**
Yeah, we are definitely the first one to build a kind of RR3-based or systems who are in the stream processing domain, even earlier than WarpStream.
**Intro - 00:00:56:**
The Data Engineering Show is brought to you by Firebolt, the cloud data warehouse for AI apps and low-latency analytics. Get your free credits and start your trial at Firebolt.io.
**Benjamin - 00:01:09:**
All right. Hi, everyone, and welcome back to The Data Engineering Show. Today, we're super happy to have on YingJun Wu, who is the founder and CEO of RisingWave. Welcome to the show. For the listeners who haven't heard of RisingWave and maybe Eldad, do you want to tell us what the system is all about, kind of what you're working on? I think it's super cool, so I'm sure everyone else would love to hear more, too.
**YingJun Wu - 00:01:29:**
Sure. First of all, thanks for having me here. And I'm YingJun Wu, and I'm a founder of RisingWave. So, yeah, people always ask me about, okay, what are you currently working on? Like, I would say, okay, definitely, what is RisingWave? RisingWave is a stream processing system. But this day show a lot of buzzwords about the Iceberg. We could probably discuss that later. But essentially, we found a company four plus years ago. Before that, I would say the Amazon Redshift as well as IBM Research Almaden. So, actually, people sometimes will ask me, okay, why Redshift is like a Bash base? So, the data warehouse, right? Well, why you started working on something like stream processing? A little bit of history of myself. I have my PhD in stream processing and database systems.
**Eldad - 00:02:16:**
You had no choice. You had no choice, basically.
**Benjamin - 00:02:19:**
And you spent some time in Andy Pavel's group as well, right? Like who's a previous guest on the podcast. So I love how we're like connecting. Exactly. Former guest of honor. And now we have you as guest of honor.
**YingJun Wu - 00:02:30:**
Yeah, I'm not sure whether we have enough time to dive deep into details. But essentially, when I was with Andy Pavlo, we were doing something like transaction processing. And essentially, at the time, there was a concept called deterministic transaction processing. Basically, it's like, okay, reorder the transactions before execution. And essentially, the concept is the same as stream processing. And that's why I have that. During my PhD, I studied both stream processing and transaction processing. Because at the time, there was just some concept that would link these two things together.
**Benjamin - 00:03:03:**
Nice. Cool. And then fast forward after your PhD, you said time to build a new database system.
**YingJun Wu - 00:03:10:**
Oh, yeah, definitely. I mean, Redshift was, yeah, IBM, I mean, dbt, right? Well, yeah, Redshift actually at that time was probably 10 years old. If you look at Redshift or batch-based system, but well, how about, I mean, stream processing? Because well, essentially, in Redshift, a large percent of the data is actually came, was ingested from Kinesis or Kafka, right? And these are streaming data, but there was no stream processing systems. Well, I mean, there were some stream processing systems like at the time, Spark streaming, Flink, these are still popular these days. At the time, there was also KSQLDB and Apache Storm, but these kinds of systems were not that easy to use. And there were some fundamental issues with the key concept was called state management. We can probably dive deeper into details, but yeah, there were some issues. So I feel that's right. Yeah, it's the right time to build something new. And that's why we find that company.
**Benjamin - 00:04:03:**
Nice. That's awesome. So tell us about those early days. I mean, like, this is one of those, like, building a system from scratch is like... Crazy daunting challenge, right? Like you need to do everything, like parser, planner, runtime, kind of all of it for streaming system, integrating with storage. Like how long did it take you to get to a first version of the product? When did you launch it? How did your first users like it? Take us through the journey of RisingWave, basically.
**YingJun Wu - 00:04:30:**
It's definitely what's appearing. I mean, during the first two or three years, whenever you started building something, let's say a database system, so I have very hard called low-level data systems, beta-infra system, then essentially it's not about your concept. It's not about your, let's say, the design principles, right? It's more about, okay, how old the system is. It's more about the trust. If you say, tell people, okay, my system was just two years old. Okay, then people will say, okay, that's a new system. That's pretty cool. I will probably take a look at that. And I probably will follow you, but nobody will take it seriously, to be honest. And in our case, we did not really have any customers for the first two years. Actually, the first two and a half years. And the first POC user was like tiny startups. And we found startups because, I mean, they actually just having startups, trusted startups. Also, we know each other. And yeah, so they adopted RisingWave. And I definitely also pitched the RisingWave to some big companies. So, but the world would ever say that, okay, like what, RisingWave is like our new system. And you do not need to put it into production. Just to use it for in some of your testing environments. Because it's so easy to use. We have actually have the single binary. We are Postgres compatible. And I mean, RisingWave itself is not single binary, but we actually made a single binary version just for some customers to self-deploy. No Docker, just a single binary, just like DuckDB. But well, anyways, for the first two years was just like tough. And I think we were definitely blessed that we got first a few customers in the year 2.5. We have at least probably three of them. One of them was like a crypto company. At the time, it was also a pretty tiny company. But actually these days, well, they are still all customers and paying us six figures every year. They've become big. And they grew from a startup to a big company. And the other two are both pretty big companies. One of them actually are Fortune 500 companies. So I feel that's why it's like just a blast and we got lucky. But this is why it's like because we have already had a pretty good customer base and we never need to answer questions like, okay, whether your database system is trustworthy or not, or we just have a page showing all customer logos and that's it.
**Benjamin - 00:06:53:**
Nice. Cool. Zooming out a bit, like take us through the current stream processing space, right? Because there's, nowadays, actually a fair amount of systems. I think like the OG is kind of Flink. That's been around for quite a while. There are systems like Materialize that are also Postgres compliant. There are systems like RisingWave now, which are like built in Rust. Okay, Materialize as well. Kind of like, how are these systems different? What makes RisingWave special? And why would I choose RisingWave, I guess?
**YingJun Wu - 00:07:21:**
Okay, so I believe that's for RisingWave is definitely a number one stream processing system in the world today. I mean, if you want to add something, for sure, I mean, go ahead. I probably removed my sentence. But, well, I mean, personally, I feel that's for we are the number one stream processing system at the moment. I mean, definitely Flink. Well, for Flink, well, people use it for, I mean, if people need to have, let's say, a Java API, for sure, yeah, go for Flink. But, well, if you just focus on the SQL, then it's really hard for us to lose to any other systems. If I talk about design principles, well, then we have several things I want to highlight. The first one, actually, I think we have already mentioned we are Rust-based. I mean, Rust-based, well, it's not like a design principle, but, well, Rust always gives the people the feeling that it's cool. So, yeah, we attract a lot of early adopters, probably just because of Rust. And second thing here, that's well, S3 is the primary storage, basically the corporate-companion storage architecture. And some people would say, okay, it's like Snowflake-like architecture, right? So we started from day one, it's like S3-based architecture. And if you check out some other systems, well, I mean, like Flink, well, they probably just recently announced that, well, they were supposed to put that. But we actually have already adopted this architecture from day one and basically baked it for in production for over four, lots of four years. So there are a lot of like tricks I probably could discuss later.
**Benjamin - 00:08:50:**
This is interesting, right? Because this seems to be a general trend in data management at the moment, in a sense that when Snowflake came up, they were the ones who invented in many ways, okay, decoupling storage and compute, using object storage as the lowest tier. And now you're seeing these kind of, I would say, new age systems using the same underlying principle to kind of disrupt that space. So RisingWave for streaming, you had WarpStream, which was acquired by Confluent, kind of doing it around Kafka and so on, right? So I think it's super interesting how this dynamic right now is actually playing out throughout the entire data infrastructure stack. You have like Estuary, I think, who we had on the show, like Danny Palmer before, who were like building ELT Tooling on top of S3. So I think it's like interesting how that's just happening everywhere, basically.
**YingJun Wu - 00:09:35:**
Yeah, we are definitely the first one to build a kind of Raya 3-based system in the stream processing domain, even earlier than WarpStream. We will be funding in 2021 and all the other systems will probably later. But essentially there are a lot of, there are a lot of like tricks and I have to tell you that it's pretty transparent. Essentially over the last two or three years, we were also thinking about, okay, whether we were doing the wrong design decision. And I can hear you more about that because for a lot of, we are stream processing systems and S3 has two characteristics. Well, one is S3 slow. Second one is S3 expensive. So, I mean S3 storage is not expensive. It's just the 23 bucks per terabyte per month. But if you look at where the S3 is access rate, they actually are charged for get input in all these operations. So I mean, that's super expensive and essentially it costs us a lot of money. And that's why we actually revisited whether we were asking us, okay, if we want to rebuild the system again in 2025, should we adopt the exact same architecture? But essentially we did a lot of optimizations. And now our answer is that's for years. If we build a system again in 2025 or even 2006, we'll stick with this architecture and there's no change. But definitely we prioritize some of our roadmaps. But essentially it's super good for several reasons. The first one is that you never need to worry about the state management because everything is in S3. So yeah, we always trust S3. And secondly, it's about the elastic scaling. In Regional, if we can achieve a second level elastic scaling, but for the other systems like Flink, probably, yeah. Whenever you want to do some elastic scaling, then it probably takes hours.
**Benjamin - 00:11:29:**
Which isn't very elastic if it takes hours.
**YingJun Wu - 00:11:33:**
Yeah. So they actually have dynamic scaling to be honest. Well, I mean, I don't really want to hide anything, but they do have dynamic scaling. But the issue here is that people do not really use that swing adoption. Because for the people who think that it's not good for that, which is true. But in our case, well, people do dynamic scaling all the time. Yeah.
**Benjamin - 00:11:54:**
Gotcha. So RisingWave itself is open source, right? Kind of an Apache license. At the same time, you also have a commercial offering. Take us through that because it's such a common model nowadays, right? Like you have kind of ClickHouse doing something similar. A lot of other players in the space doing something similar. What is free? What is paid? When would I become a cloud customer, basically?
**YingJun Wu - 00:12:15:**
There are several things. I mean, we have the cloud offering, we have BLC bringing on cloud, and we also have on-prem. So on-prem, people say auto-powering business. Well, I mean, but we have to say that so like, if you want to enter banks, all these kind of highly regulated industries, you actually have to be on-prem. We actually enter banks. We also enter the aerospace. Nice. Yeah, I mean, this industry is what I mean.
**Benjamin - 00:12:41:**
Is RisingWave running in space? One of our questions, we had Hannes Mühleisen on who said DuckDB is running in space. So is RisingWave running in space?
**YingJun Wu - 00:12:50:**
I mean, it's definitely not in a satellite, but I actually don't know where they deployed.
**Eldad - 00:12:54:**
It might, it might. For all we know, it might. Tell me something. Who would be the typical user? Where do they come from? They decide to build a new streaming system? Are they coming from a Snowflake? Are they coming from a Kafka, Redpanda? Do they come from Flink? Like, how would you categorize that person?
**YingJun Wu - 00:13:14:**
So we definitely do not really compete with Kafka. We do not really compete with Redpanda. We have super good friends with Redpanda. They're awesome. We're definitely a good friend with WarpStream. But we do compete with Flink, directly compete with Flink. And so basically, there are two cases. The first one is that, okay, people probably are new to stream processing and they want to adopt a new system to process their streaming data. And then they do an evaluation with us. Well, they probably always start with the easiest solution. And we are essentially the easiest solution. Why? Because we are Postgres compatible. That's it. We are Postgres compatible. Challenge me. I mean, you can go with Flink, but I mean, you learn the SQL. But in all case, Postgres.
**Benjamin - 00:13:56:**
Don't have a hard tough crowd here. It's like since Firebolt is Postgres compliant, we're all in on your vision of the world. Like everything should just be Postgres compliant.
**YingJun Wu - 00:14:06:**
Definitely. I think Postgres definitely helped us a lot. Well, I mean, people always love Postgres and there's no such kind of headache learning something new. So basically people are new to stream processing system and they want to adopt a new system. And typically, RisingWave is probably the first choice. And second option is as well, okay, they have already adopted, well, the Spark streaming on Flink. And then they want to, they feel some pain and they want to migrate. So a very reasonable case, which I'm super proud of is as well, I cannot shout the company name because we signed on NDA, but it's a super big company. And they actually run a certain system, either Spark streaming or Flink. I could not tell you which one because there are some issues. And they actually run a system with 20k CPUs, 20,000 CPUs. Wow. And then they use RisingWave and they only use 600 CPUs.
**Benjamin - 00:15:02:**
Wow, that's massive.
**YingJun Wu - 00:15:05:**
I have my PhD and I always challenge such kind of number, right? Well, how you can reduce the, I mean, the number of C2 from 20,000 to 600, how that's possible. So I also dot that for it, but I actually talked to them. So the reason here, that's for their like state management, where they do joins, multi-way joins. So for all the other systems that are not scalable, but for in writing wave is super scalable. And we did a ton of optimization. So that's basically migration case. People feel the pain. They feel that's okay. They spend too many resources in the other systems. And sometimes what people say, okay, I used to use Java, but well, it's so too slow. I mean, in terms of development, I want to shift to SQL-based systems. Or sometimes people say, okay, I don't really want to spend four hours waiting for the Flink to get recovered from failures. Well, that was from a bank. And yeah, they got some issues with failure recovery and they decided to pivot and switch to some other systems from Flink. And that's all these kind of opportunities we got. Yeah.
**Benjamin - 00:16:13:**
Nice. That's very cool. So switching gears a bit, like my link and feed recently has been full of actually posts by you talking about Apache Iceberg. So you're very good at kind of amplifying content there. When I think of Iceberg, I mainly think about it in terms of batch processing, right? Because this is kind of, this is what we're building it for. Take us through how Iceberg is changing the game for streaming.
**YingJun Wu - 00:16:34:**
So there are two things I want to mention. The first one, Iceberg is Bash. I mean, they actually compete with Snowflake, Cresthaft, BigQuery, whatever. And in the past, we basically avoid a vendor lock-in because it's open table format. As long as you adopt Iceberg again, then probably you could use DuckDB to use Firebolt to use any other system to query the data from there, right?
**Eldad - 00:16:58:**
Didn't they say that on CSV5 as well?
**YingJun Wu - 00:17:00:**
Yeah. I was in data console yesterday. Yeah. I know that's where you guys were also there.
**Benjamin - 00:17:05:**
Yeah. Some of our folks from the Bay Area.
**YingJun Wu - 00:17:06:**
Yeah. Yeah. They actually didn't meet. And some other guys who I talked to. So basically, yeah, if you adopt Iceberg and there's no vendor lock-in, right? So you can use any query engine. So for stream processing, you asked me about, okay, well, what's the relationship? Well, so basically in the past, well, I mean, our customers were essentially, after processing, they essentially send the data into Snowflake, BigQuery or some other systems, Redshift. But nowadays they think that's okay. I don't really want to be vendor lock-in, right? Well, let's send the data into Iceberg. So essentially that's why Iceberg became, I think, our top three destinations, right? Because of here.
**Benjamin - 00:17:43:**
This is super interesting to me because when we think of Iceberg, we mainly think about it like as a source, right? It's okay. Like use a Snowflake managed table on Polaris Catalog and Snowflake or something. Can we be more efficient in terms of query processing on top of that? What you're saying is that like for a streaming system like RisingWave, the sync story is actually more interesting than the source story. So you're like the fabric that populates the Iceberg cable. Is that fair?
**YingJun Wu - 00:18:09:**
That's fair enough. Yeah. We do ingest the data from, also from Iceberg. We have the capability. By the way, I mean, most people, I mean, sync the data into Iceberg because they are free of and they're locking and they just want to adopt the Iceberg. We see this pattern pretty obviously in all these kinds of like enterprises. There are VP who will say, okay, like, I don't really want to pay Snowflake too much money. I just want to shift. What's my strategy? Okay, let's do Iceberg. That's the thing. That's always a conversation we've heard about. And yeah, basically the vendor lock-in is a big thing that's a big thing. From the stream processing system perspective, definitely we need to think about the destinations.
**Benjamin - 00:18:47:**
So, Staying with this destination thing, because I want to understand how a data pipeline like that looks, right? Like, would it be, I have this event stream, which is like, has like just bunch of data. I don't want to ingest like the raw data, for example, into my Iceberg Table, but use a streaming engine like RisingWave to compute me one second averages over some metric. And then I ingest that into Iceberg. Like, what is the actual data processing part that RisingWave usually does in these data pipelines, basically?
**YingJun Wu - 00:19:17:**
So like, well, there's always debate about ETL or ELT, right? So basically, ELT means that's okay, let's just ingest all the raw data into my data warehouse. And the ETL means that, okay, let's do some preprocessing before sending my data into data warehouse. I think in Iceberg world, there are typically people do ETL instead of ELT. Why? Okay, GDPR. The reason here, that's what, like, I mean, you don't really want to send out PII data, right, but private data into your lake. Because if you send it into lake, then who to delete it, right? Well, I mean, that's always a challenge. And Iceberg is super good at handling a lot of updates. So essentially, people actually need to delete the data or remove all this sensitive data before sending the data into Iceberg, right? In some other case, it will be like, okay, so people probably have data across the different regions. And they actually want to do some drawings and aggregations, unions. Well, before sending your data into Iceberg. And also definitely people have the requirement of doing enrichment. So let's say that's where I have the Clickstream and I want to also have the user info, right? Well, I want to enrich the data. Why people not just do it into Iceberg? I mean, you can definitely do that for using whatever system you want, right? Like Spark, whatever. But I think here that's where it's already like give people the impression that's where it's historical data. And I don't really want to do a lot of like ETL inside of lake. So people do the ETL before ingesting into lake. But I think what things will be changing because as long as people ingest more and more data into Iceberg, I believe that's where they will think, okay, let's just do ETL inside of lake. Yeah, I'm actually following the icebergs back quite closely. And in v3, they actually have a feature called Iceberg materialized view. And that's essentially for the ETL in Iceberg. And I think Verizon will probably become one of the first vendors that's a part of our Iceberg materialized view because we are essentially ready. And we are just waiting for the Iceberg of v3 got merged. Yeah.
**Benjamin - 00:21:25:**
Nice. Okay. That's really cool. In v3 spec, the MV definition is that like, I mean, one problem is like SQL dialect differences between systems. How do they handle that within like the Iceberg Specification? Like if I define an MV in like an Iceberg Table, what SQL dialect is that specified? And like, that seems like a nasty problem.
**YingJun Wu - 00:21:44:**
Look, in our case, it's just like Postgres, right? For we don't really care about all the other dialects, right? For we just care about Postgres. They might
**Eldad - 00:21:51:**
Actually focus on the schema definition, consistency, and kind of how materialist views commit versus on running on the SQL. I don't see any value in Iceberg to own the SQL, the business logic behind the MV. I can see how Iceberg can expand their metadata layer to support materialist views just from a materialize view object kind of entity aspect.
**YingJun Wu - 00:22:15:**
That's right.
**Eldad - 00:22:16:**
Let's not get confused. Iceberg is not a processing engineer. It's a metadata catalog that sits on top of Arcade. One of the things that people get confused with Iceberg is it opens up a lot of possibilities, but at the end of the day, they're still doing the same thing they've done before. But what does change with Iceberg is actually the last mile of many data pipelines. So if you think about it, if you were using Snowflake before, to get data out of Snowflake into a different system, you needed to talk to the Snowflake team. You had to extend the Snowflake data pipeline to now export into Parquet. Let's not get confused again. Nobody does low latency exports from Snowflake. People do batch every hour, every day, a few times a day. They export their data to Iceberg. They export their data to an S3 bucket for the same matter. The difference is now they just write the end of the data pipe to Snowflake. So they kind of avoid having an extra export, right? They remove that extra export. But that extra export is not just about cost. It's about enabling the data to move to new locations. If one team owns, if the team that owns Snowflake is kind of the barrier within the company on getting right data, correct quality data to more systems, this is a problem. But if the Snowflake team, just like any other team, when they're done with an EAP or a data pipe, it doesn't matter, batch, streaming, it doesn't matter. As long as you write it to an Iceberg Table, you save a hop on the next stage of having to export it. So instead of exporting it, you import it. So people pull data from Iceberg Table and that changes the whole dynamics. But the same old problem still exists. Parquet is not designed for necessarily for streaming format, to do on the meta low latency out of the box, just like it's not designed to do low latency reads. It's not designed for many workload types, right? Scale out is a challenge. So there is a compromise here. You can put Iceberg and get consistent metadata across systems. You can say, actually, I need multiple search formats within Iceberg. I actually expect Iceberg to grow in terms of format. Now that we have consistent metadata across systems, having a global format is less important now. If every system can read any format and everything is protected via Iceberg, then why not introduce extensions and improve formats so that actually workloads can run better through Iceberg? So Iceberg doesn't make workloads run faster. It makes different systems see the same data and it makes different systems lose their kind of ownership over exporting the data forward to the next system. And I think that's kind of where we'll see a lot of innovation. I've heard someone telling a joke, which is kind of becoming something that kind of floats in the industry that says, well, Iceberg was that intention of opening up and unifying metadata across all vendors, but it ended up just helping Databricks deal with all their own formats. So now Databricks will work for two or three years on Iceberg just to get their own data. And like the 20 catalogs that they already support and they already sell some service to, now they can get all of them under one Iceberg. But still, Databricks, Data Warehouse Engine, which doesn't have much to do with Iceberg, is the highest selling product for Databricks. I think for startups, it's a huge opportunity. I think Iceberg for Snowflake and Databricks is kind of basically a done game, right? The architecture is obvious. I do a dbt on Snowflake. I make sure it's getting written to an Iceberg so I can read it from other systems and vice versa. But the real innovation around Iceberg will come from startups and will come from the new data ecosystem, from new companies introducing productivity tools on top of Iceberg. There's a lot of discussion on how to merge data behind Iceberg, right? A streaming system will need different compaction than a system that maybe does post-applegation. Who owns that? Who owns the tablet behind Iceberg? So a lot of open questions, a lot of opportunities for kind of refiguring the ecosystem. And it's super exciting to see, I think, in this show that streaming databases are here to stay. In fact, they're getting so much stronger and through Iceberg are actually getting much more attention. A normal person would not think about going through the streaming journey without a PhD. That's obvious. Like, you shouldn't even mention your PhD. Like, everyone would be surprised if you wouldn't have one. But opening up to Postgres using S3, just figuring out consistency on streaming with that. Like, you need a PhD for that as well. As you said, everything is connected. So it's amazing to see that complex systems like streaming, PhD-grade systems, are getting the right treatment from the right startups. And I'm looking forward to see how RisingWave and Firebolt can coordinate with more startups around that ecosystem, around that Iceberg. Everyone wants to move out of Snowflake.
**YingJun Wu - 00:27:21:**
Yeah, definitely. Iceberg is essentially, people say it's called the Single Source of Truth. So, I mean, like we ingest all the data into one Single Source of Truth. So that's where you do not need to think about, okay, where I have my data copied. Do I have my data, copy my data in, let's say, Postgres or have another copy of my data in Snowflake or have a third copy of my data in some other systems, right? They just sort of think, okay, likewise, the Single Source of Truth. Iceberg is a Single Source of Truth. And once you've got a Single Source of Truth, you can have all these kind of like engines, right? Essentially, my hot take, probably not hot take, but for an obvious take, that is for all the databases will become a query engine on top of Iceberg.
**Benjamin - 00:28:07:**
I don't think it's such a hot take anymore. I think you're two years late.
**Eldad - 00:28:10:**
A Single Source of Truth justifies the source. The problem with data is not the fact that you move it around. It's not that you clone it and write it into a CPU register or RAM or SSD or a replica on S3. It's the fact that when it changes, there is no bookkeeping on that. So you need to backtrack the whole data pipeline, figure out in which step of the data change that source of truth changed. Now with Iceberg, you get a single, unified, consistent source of truth, and that can now tickle down anywhere. And any system that runs on Iceberg that supports transaction can support it. So you can time travel. You can do a lot of stuff you couldn't do without copying everything into a proprietary environment and then using it end to end. And having streaming on one end, just not one end, actually through the ELT as well sometimes, because sometimes ELT is streaming ELT. But then having that switch to a different engine, doing something else like a scale-out group buy, that becomes very interesting. And that monopoly over, you use everything, use a single stack by single vendor. The compute, the metadata, the workload, everything needs to be done by single vendor. That claim gets shaken and we'll see how that plays out.
**YingJun Wu - 00:29:25:**
Yeah. Like, well, I mean, Snowflake can just the data probably can read the data. I mean, yeah, it's great for, but sometimes people just have an impression like, well, I don't really want to get under locking. Right. And all problems with people are saying that, well, like, so I have some, let's say specialized data type and that, which can be only be read or probably write by certain systems. And in the past world, you probably need to build an integration between that kind of system with probably Snowflake. But no, we don't need to do that. We're just to build an integration with Iceberg. So it essentially simplifies the development for a lot of vendors, right? So, I mean, in the past, we probably need to think about that. I mean, we do have an integration with Snowflake. Well, yeah, essentially, I mean, if Iceberg came early, we don't need to do that.
**Eldad - 00:30:12:**
You don't need to talk to Snowflake. That's the beauty. If Snowflake rides to an Iceberg Table, there is no locking. The data is not locked behind the Snowflake wall. It is actually completely open to anyone that can scan or write or read from and to Iceberg.
**YingJun Wu - 00:30:28:**
Absolutely. Yeah. Yeah. In the past, it's like, okay, I need to think about the work. Okay. I'm in front of a vendor perspective. Okay. I need to think about the work. Okay. How to maintain the pipeline to Snowflake, how to maintain the pipeline to integration with Redshift, how to maintain the pipeline integration with Cray. Now that it's no, I don't think you're worried about that. That's right. I mean, even though we have already had that for, I mean, but these days for me, mainly focus on Iceberg.
**Benjamin - 00:30:53:**
Perfect. I think that's a beautiful summary to actually end our show today. Thank you so much for being on. I think you guys are running a bunch of meetups in the next couple of months and we'll be kind of participating in some of the European ones. So listeners, kind of take a look, kind of go to some RisingWave meetups. I think it's going to be a blast. It was great having you on the show.
**YingJun Wu - 00:31:12:**
Sure. Yeah. I mean, we do have our being Europe in May and June and hope to see you there. And I know that's where we have a lot of collaborations. And if people are interested in either RisingWave or Firebolt, just join us.
**Benjamin - 00:31:26:**
Lovely. It's going to be fun. Thank you for being on.
**YingJun Wu - 00:31:28:**
Thank you.
**Eldad - 00:31:30:**
The Data Engineering Show is brought to you by Firebolt, the cloud data warehouse for AI apps and low-latency analytics. Get your free credits and start your trial at Firebolt.io.
# How Similarweb Delivers Customer Facing Analytics Over 100s of TBs (/blog/how-similarweb-delivers-customer-facing-analytics-over-100s-of-tbs)
According to Yoav Shmaria, VP R\&D Platform at Similarweb, the best way to manage data warehouse costs is to tag every table, database or ETL running to have good granularity over every feature. Besides handy cost management tips, Yoav walks the bros through the tech stack he implemented to analyze 100s of TBs of web data to serve fast customer-facing analytics. Full disclosure, Similarweb is a Firebolt customer, but the bros kept it objective, and there's no Firebolt talk in this episode. Let's go!
Listen on [Spotify](https://open.spotify.com/episode/5wCvyjR5DgCGUHbR1wM2u1?si=spxiYNkcTcuXrmP1juT2Bg) or [Apple Podcasts](https://podcasts.apple.com/us/podcast/how-similarweb-delivers-customer-facing-analytics-over/id1561927688?i=1000569868238)
**Guest**: Yoav Shmaria - VP R\&D, SaaS Platform - SimilarWeb
**Hosts**: The Data Bros, Eldad and Boaz Farkash, CEO and CPO at Firebolt
Boaz: Welcome to another episode of the Data Engineering Show.
Eldad: Welcome everyone.
Boaz: Eldad, How are you?
Eldad: I am great. Yeah, it is always good to visit and feel again.
Boaz: Yeah, you did not come over for a shopper's dinner last night.
Eldad: It is the only place we meet, on podcasts.
Boaz: With us today is Yoav Shmaria - VP R\&D, of the SaaS Platform at SimilarWeb. Yoav, how are you?
Yoav: Never better. How are you guys?
Boaz: Thanks for joining. You are from SimilarWeb. SimilarWeb, if you have not heard, it is an amazing data product. SimilarWeb, our market intelligence, and research platform were publicly traded as of last year. Yoav, tell us a little bit about your journey, to where you are at SimilarWeb or how did your career evolve?
Yoav: My personal journey in SimilarWeb?
Boaz: Yes. Even before. Because you have a nice mix of software engineering, data technologies, and now management of R\&D groups.
Yoav: Yes. I started my journey as an entrepreneur doing my studies. I had some touch in doing everything from everything, from full stack development to marketing and sales, and we had some specialty around payments and registration platforms, and I found my career journey starting in SimilarWeb as a Frontend Developer, nothing data. I have a very strong connection with product management, with business management, and I think, these capabilities took me to start managing, within the organization and eventually, two and a half years ago, I am already six years in SimilarWeb, I joined when my big daughter was six months old and now, she is six years old, so that is quite a journey. For the last two and a half years, I am leading the R\&D group of our main B2B product. We call it, pro internally. When I started my current role, we were a bit less than 30 employees. Today, we are over 80, 15 of them are located in Ukraine, the rest in Tel Aviv. I think that my biggest leap was when I started leading the entire group. So, I had to deep dive into the entire data ecosystem in our organization, which was amazing and still an amazing journey. In my group, I have two main positions, separated into backend engineers and frontend engineers. And what is special about our group in SimilarWeb is that the backend developers are doing some kind of full stack backend engineering. We call it a data server. It is a mix of data engineering, focusing on ingesting data into different databases and the backend layer for the API.
Boaz: Before we dive into that super interesting stack which we want you to dive deep into, tell us in your own words about what SimilarWeb does and the role data plays at SimilarWeb?
Yoav: Getting how to understand about SimilarWeb is that we are not giving you some insights above your data. We are giving insights about everyone else's data and the simplest way to describe it is Google analytics for the entire internet, which it is. We are giving analysis across almost any domain and also a mobile app. We divide our use cases into five separate solutions. So, we are starting with general research. Today, most of the marketing budget goes to digital, not to paperwork, and not to radio. The journey usually starts with research when we get some strategy decisions around how my market looks. I want to get into the website, build their market, who are the main players, in which countries they are, and how the audience looks like.
Boaz: If I am, for example, a digital marketer at Zava, a clothing company, what do I do with the SimilarWeb application?
Yoav: First of all, you would like to explore the fashion industry worldwide, maybe to get some trends around rising countries or some rising audiences to find new competitors, understand maybe the market is getting a lot of traffic from, let us say display ads. Let us say that on average, most of the fashion websites get 10% from display ads and I am getting only 2% for my traffic. So, maybe I am doing something wrong. Maybe, the audience for fashion loves banners. This is very, very high-level market insight. And, then, we are going into competitive research. We call it digital marketing to optimize for each channel for the SEO, PPC, maybe by an affiliate manager. These personas are doing tactical actions, like tracking search engine position, tracking the campaigns, understanding competitors, performance, and doing this benchmark with Zava.com against any other brand and understanding what are the differences in terms of traffic performance?
Boaz: Given this is such a data-centric product, let us go back again to your team. How do you call them the data backend team?
Yoav: Data server.
Boaz: Data server.
Yoav: In my group, we are doing everything. I mean, it is data engineering, backend, frontend, and QA, but I do not have a single backend developer, doing only API, coding, and data engineers. It is like a full stack data server engineer and we changed it four years ago. We had a separate group for data engineering and a web group for backend and frontend and we realized that especially for a product like us, it is super important that the person who is modeling the data and the person who is serving the data should be the same person and this is how we started to get data engineering stack for our backend developers.
Boaz: What titles do the data engineers in that group have? Do they call themselves software engineers or do they call themselves data engineers?
Yoav: In our HR system, it is a data server engineer, but most of them are on LinkedIn, I guess there are somewhere between backend engineers, software engineers.
Boaz: So it is a software engineer, building data-centric products.
Yoav: Yeah, we do separate the frontend from the backend, but the backend is not a trivial backend at all.
Boaz: That is super interesting. We love that. I think one of the courses came up, let us say 10-15 years ago. If you asked the software engineer to build a database or data warehouse or query engine kind of project, people used to look down on that as something beneath their payroll. Today, the most interesting software projects, the hottest thing in software is to build data, data-rich applications and it is a huge shift in the market.
Yoav: Yeah, I love to call it that in SimilarWeb we are doing data engineering upside down. In most of the organizations, I and probably you also, find data engineering is some offline processing. It is saved for BI and analytics. It is not part of the real production. Maybe we are hosting some database, a lot of data to serve something, but we do not have an entire data engineering operation for production. In SimilarWeb, it is the opposite. We are analyzing the entire internet, and we have this funnel to give real-time insights to our customers.
Boaz: Let us talk about the data stack. What runs there in the data server?
Yoav: Everything. I want to deep dive into the entire stack of our data collection methods and machine learning and everything, as I said, it is data engineering all over the place here. But in my group, we have shared the Data Lake but we are still on AWS, which is a very nice name for tons of files, we are using the Glue Catalog of AWS and very important detail for SimilarWeb. We are managing a branch system, a data versioning methodology for our data that is because we have multiple algorithms running to calculate our traffic estimations, and we are changing them from time to time. We are always running with some master brand for our data, and then, we are having in production a mechanism to show our customers a better branch. So look, we upgraded our estimations, take a look, we are validating it for some period of time and when we are deciding it is good enough, we need to make a decision whether we are starting a new breaking point with a new out algorithm or we are running back three months or three years of data. Very soon in the next quarter, we are going to run the entire three years of our mobile web data.
Eldad: What you are saying is the customers are moving away from software versions to data versions. To them, that is much more interesting to be able to play with versions and to be able to kind of understand the data that is driving the results, and that is also a very big shift from how engineering looks at the world. I wanted to ask someone with your background, what was the experience in trying to really reorganize an engineering organization that is more traditional, looks at data as something that is decoupled? How do you reorg from 20 engineers to 80, but also not just grow in size, but really rewire? How does that organization operate on data?
Boaz: How was the experience?
Yoav: I think something that helped them is that we are running, in my group, a totally very strong horizontal teams methodology. That means that we have a metrics organization, and there are a lot of methodologies in HR development. What we are doing is that I have a data server team and each team member is or a couple of them taking part in a different squad delivery and this helped me to keep knowledge in this team. So, we are not losing control but in every team, anyone and everybody doing whatever they understand. So we have a very solid layer of data engineering stack. Now, what we did change when I got into the role, I think we had, two ETLs running on a very premature Airflow and what we did is that we started to take, and that is a great lesson, two or three engineers every quarter from all around the squads and dealing with the infrastructure. I think our two main tasks that we took when I entered the world were first, creating solid infrastructure for an ETL abstraction so that every new engineer can pretty easily run a trivial ETL. We are not talking about a very complex stack, but let us say we are using DynamoDB a lot. If I want to just take a very simple collection from S3 buckets and write it to DynamoDB, it would be like two hours of work. And you have a running ETL supporting branching, everything, it was a big deal. We invested a lot in it. And the second was to find more in databases that we can sell as production because we Edgebase base a lot, which is amazing for concurrency and really, really, fast, querying like the data, but the structure is very key value-oriented. It is like you are keeping a very dummy data structure for a specific key. You can scale, but you cannot really run complex analyses over the data. So, these were the main two initiatives that we took with some kind of virtual team of infrastructure.
Boaz: What data volumes do you guys deal with?
Yoav: It depends. It is basically per feature. I think our largest database in production is around 150 terabytes, compressed, of course. This is for a part of our big data research product and in the digital marketing for the pure dataset. We also have a data set that is some dozens of terabytes, but it is crossing the trillion rows of data. Other than that, we have more than 100 ETLs of data ingestions running every day or month. We have, I think, over 30 or 40 tables in DynamoDB, and in general, I think, we are running over a petabyte in production and that is, of course, only what we sell, which is three years of data and we have more in our Data Lake.
Boaz: When you set out on the journey as you mentioned before on the versioning of the data, I think that is something that is becoming more interesting. How would you recommend it for people who want to go in that direction to take it on? Because it feels like there is no clear sort of market standard on how to go about it. People are trying to figure out if there are even startups and companies being built around that area like lakeFS, what is your take? What would you recommend?
Yoav: Wow! Take for data versioning. First of all, let us understand the challenge. Because theoretically, you can say, I have a new version, let us override. The problem is that you would like, first of all, sometimes you would like to show the customers in production like us two different versions. And, of course, before you release the production, you would like to have some staging product that you can test. So, you want to be able to walk seamlessly with those two versions, maybe more. We are doing a lot of infrastructure work right now on it, but I think the main area to plan is for the actual serving layer for the databases, because in Data Lake, as you say, like lakeFS, where you have a lot of methods to manage like data versions. But, eventually, let us take DynamoDB to the table. I have one table and now, I have another version. So, whether I want to manage a table version, and then if I want to take 12 months from one and then from another that is one kind of complexity. We totally use a lot, prefix or suffixes like the branch name. But you will need to design it in advance because otherwise, you will get into a situation that you want to write about, and again, I am talking about the serving itself. You want to write only part of the data, like for the new branch and you are willing to think about how they both merge together and in big data, if you want to do a union data or join, it is not always a trivial task and you can lose performance for that. So, I would say modeling the serving data, I think, that is the most complex part, because as you say, there are many, I think also Data Lake now has some solutions for the versions. But when we are talking about serving, that is not trivial at all. We are still struggling by the way, what is more relevant and when you are running some analytics engine on the data, it is even more complex.
Boaz: What do you mean by struggling? You told me everything was perfect.
Yoav: Nothing is perfect.
Eldad: The versions are related to the models, right? Different model versions require different data versions, which are translated into a feature version, and the challenges across every step. And you have mentioned up to the serving part but then it starts much earlier. This is why versioning in lakes is becoming such a hot topic. It is connected to ML, it is connected to models.
Yoav: And, of course, you need to hold some metadata store for your versions. Now, I have multiple versions all over the place, somebody needs to hold this information.
Boaz: In which case is what metadata store.
Yoav: So, we are using it, it is an in-house solution. It is a simple database that we are holding in relation to the database with some services that we hold. But again, think about it. It is something that is in your critical path, right? Because, unless you have some cash in place, you need to go through this service, always to ask, wait, where do I get the data from? This is your router. So that is also a critical part.
Eldad: Wired into the product versus just kind of looking at different versions to figure out which report.
Yoav: I always want to rewrite my entire data. I would rather only write parts of it. So, I want to hold the changes somewhere that I can serve seamlessly, let us say a perfect graph, but it is combined with five different versions of data.
Boaz: Let us go down that path, testing. So, how do you go about testing your data?
Yoav: Again, testing is all over the place as part of the modeling of cost, data rating, and everything. In our part, we have a very loud stack of automation testing. That is running on the product and seeing the final results. What is important again in our life is that we have a window release and snapshot list. We have a daily release of daily data and then we have a snapshot where all the monthly calculations are taking part. We have nightly tests, making sure that we have first of all a full sync between our data lake and our production databases, and that we are showing new data. It is supposed to happen that if you have, I know hundreds of data sets that somewhere will get a zero. So, we are always testing that we are not missing anything. This is like the critical test I am always talking about with my teams. Please, when you are releasing a feature with new data, make sure that next month we will see numbers. It is like the dumpiest test, but you always lose something. I think this is the main area. We have multiple tests in place for Linux, but I do not think it is the most relevant now.
Eldad: Testing is moving from units to data units and results.
Boaz: At the end of the day, you are delivering an experience to users, they slice and dice, they look at data from different angles, and users' experience is great and fast, but what are you guiding, I would say product principles. What do you think about the user experience? How slow is the query not good enough to be in the UI? How much does engineering need to be pushed to come up with ideas, to make things smooth from an experience perspective?
Yoav: There is general user experience science, and there is also the part of how easy to use the product. We are talking about basically time to insight in SimilarWeb. How many actions and how long does it take to get an insight from those huge data sets? I think in terms of performance, I believe that and Google has a lot of articles about it, that in some point, let us say around five to ten seconds, this is well, you are starting to lose it, especially now that we are so used for instant messaging and the user will wait for this report, but he would not play with the data and in one slice it over and over again and drill down if every action takes now 10 to 15 seconds. It will not happen and we see it in the data. We did several evolutions in the last year or so, taking the dataset from operational databases or starting a PLC with a low engine and then taking it with a more robust cluster or whatever the solution is, and we see the difference. We see more queries per user as we increase the performance. And, this is from the performance side and the other part is how do we take the most relevant piece of data and expose it to the customer as soon as possible. We have not cracked it perfectly yet, but we are having more and more steps into it and the onboarding experience will be that you define who you are, what is your market or competitors and we will give you the most interesting part of the data.
Boaz: Another question I had in mind. How do you guys hire engineers for the data server team, but the skills you require are not a trivial one. So what is the strategy there?
Yoav: So, in general, hiring is not easy these days. Hiring backend developers or data engineers is really tough, especially experienced ones. We are focusing on hiring smart people, good engineers with one of the aspects of our stack and we have an understanding that we will have to have this ramp up and learning curve for the other part. So, if we have a strong backend engineer, we want to believe that he will be able to pretty quickly take over our backend stack. And, then we will give him all he needs to know about data engineering and if we are getting a strong data engineer, we believe that he will be able to track with our backend stack. So, I do not think we ever hired someone that is fully stacked with both data engineering and backend development.
Boaz: So for that team, essentially, you are looking for both, you would get both data engineering-oriented people and backend engineers.
Yoav: I think I will tell my backend team that part of my problem is that once you are in SimilarWeb, we are like a superstar. You can do everything. So, it is a very, very desired material in the market. But yeah, our stack is not trivial and again, I think also, BI developments got more and more complex over the years, but they usually have a different mission because most of the processing is happening offline and we are taking care of a really big stack of the production data stack, which makes the challenge bigger.
Boaz: For all these years running a data stack in production, which were the most sensitive or error-prone, or risky areas that ended up causing on average more errors in production that you had helped for? Like, if you would go back in time, knowing what you know now, which errors would you have tackled to make for less incidents in production?
Yoav: You were asking like, what should we improve?
Boaz: I mean, most at the end of the day, the incidents that you guys happened that affected the experience, were they more the data versioning, the ETS were serving mismatch of Schema issues. I do not know. What were the most sensitive areas in retrospect?
Yoav: I think, above all-important to say we are running full. We have two regions that we are maintaining. We have a hot backup in two regions. Because that happens. We are managing a totally HPS cluster and things happen and we have a very, very structural methodology of shifting from one region to another and that is something that we know how to do and usually happens with machine errors, this happens other times. We realized that data validation, again between the Data Lake and the actual database is a very, very critical thing. Again, our customers are buying data from us. Saying that it is a very bad experience if you are querying data and getting one response, and the next day, we will get a different number. This is also something that we experienced. By the way, in various areas, it can be that we had some historical bug when loading data because we were writing in batches. We have a daily batch or monthly batch. So, writing to DynamoDB, we had some bug in the Spark client, that we realized that we are just losing rows on the way to Dynamo and it was not logged very well. So in that case, you have keys in the Data Lake that do not exist in production. It happened when we just loaded data that was not synced well. So the numbers, we did not have the same rows count or same aggregation count between the query engine in our Data Lake. So, we had many tests around these areas. And, eventually, you would be surprised, even UI can create some issues, because sometimes doing teamwork, so we say, "okay, it's just a simple average. I will do it in JavaScript." But then, the customer gets maybe a different experience because one developer wrote the code for the client, for the graph and one for the Excel in the backend and one for the API and, then you have some tiny number typos that usually will say who cares, but our customer cares. It is important to mention some of our customers, they have many data analysts that get our data in batches, from API and they are doing the math. They do not need the UI. So, in their world, if we have tiny biases all over the place, it is a big deal.
Boaz: Okay. What parts are of the stacks today do you sort of consider legacy and our plan to be phased out in the next year or so?
Yoav: I do not know we phase out. We are doing a lot of exploration around and changing the way we are maintaining the Edgebase cluster today. It is a technical death during the years. We used to have some very nice hacks in the old versions for how we prepare the data files to Edgebase and how we manage the clusters. So, this is something that we are working on strongly. Other than that, we need the ability to be able to sell as much as we can to sell from the same data source for features that are using the same data. Pre-calculation again, it is great in terms of performance, but the developer experience can be very bad. Imagine that now it happens a lot that you need to take one aggregation over an existing dataset. So, you have this dilemma. We calculated it again and, then you have hundreds of transitions all over the place, and it is really how to track or you want to execute the query on the fly and then the performance penalty is always the question.
Eldad: AAP, aggregation analysis paralysis.
Boaz: Awesome. What did we cover, by the way?
Eldad: Did we cover huge failures while I was away.
Boaz: Maybe you could say because we talked about a variety of things that have gone wrong but we can do that again. If there is one single sort of meltdown that you remember, that you would want to warn others to learn from your mistake and to not get into it again, what would that be throughout the years?
Yoav: Meltdown.
Eldad: I was having a quick question given that you are like, aggregating everything that is out there in terms of data. Can you give us or share with us how fast the universe of data is expanding? Meaning that kind of it is constantly growing. How much of a challenge is that? How fast is data growing, given that it is not per user, it is just massive, massive granular data points across so many industries? How fast is it growing?
Yoav: So you are saying if you are storing more and more data. I can tell that we have data sets where we are loading every month, let us say, two terabytes for seeing the database. So, we are seeing an increase also in coverage. So, we want to cover more and more from the internet. I believe we are not seeing everything in all the darknet and areas that do not exist, but we have more and more websites and, also in the internet area, I think that you have new websites every day. I think this is where the main challenge is because you do not have an absolute number of entities. The entities are growing. So, maybe the data is not always like exponential growth, but the number of entities is endless, keywords are something that all three of us want to buy genes, but we will search for different terms.
Eldad: I search for skinny.
Yoav: Yeah. I had Eldad for the skinny, but yeah. So, these data sets are going massively because the entities are being changed. Of course, you have the baseline, but by the way, we are exploring it a lot and we are seeing that keywords especially are changing on a daily basis. Think about our lives. So, today we have the Russia-Ukraine war. Ten years ago, you had something else, and today you have some fashion trend and tomorrow you will have some fashion trend and you have NFT today and then SpaceX and Tesla, and every day it is something else. So, the information is endless. And for that reason, you cannot assume that most of your data is the same. It is the opposite, most of your data will be changed.
Boaz: Cardinality is a nightmare.
Yoav: But, that reminds me, eventually we serve the entire internet, but our customers are not querying the entire internet. So in that matter, something that they want us to be able to do in the next couple of years is to find solutions, where we can optimize what our customers are more interested in and then to have a better, more cost effective solution to give a very robust solution for the data customers, are querying and having some cold storage for whatever just need to be there and can be queried some time, somewhere.
Boaz: Product cost we did not cover and it is interesting for a company like SimilarWeb.
Yoav: Yeah.
Boaz: How predictable is the cost these days. How tough is it to manage?
Yoav: Also from colleagues in SimilarWeb, I know it is tough. This was one of the first actions I took when I got into this role. I am really glad that I did it. Because we did some gross action again for the data server team. And now, we are tagging every Spark cluster, every database, every table, each ETL running. So, I have a very, very good granularity to know for a specific feature, the entire cost from the ETL and ingest and the data serving, the databases, the staging, the development. So, I have very good visibility. But, otherwise, it is impossible. I mean, I think the number of roles and tags that I have today in our stack is something that you cannot manage if you do not have a standard.
Eldad: You need a data warehouse to manage your tags.
Yoav: That is a very good tip. I think, as soon as you do that, you can control it. Again, shit happens, but at least you can know in real-time what happens. It has happened to me every week. I think there is no way that I am not slacking on one of our engineers and what the hell is that? Why is that on? I see some anomaly in a specific table for a specific team. So that is very, very important. Because otherwise, you do not know what happened. You just know that you have a bias of $10,000, now go figure it out. So, imagine your tags.
Eldad: Keep your tags close.
Yoav: Yeah.
Boaz: Yeah close, as close as can be. This is awesome, Yoav. Super inspiring. It is rare to see a product that is so deeply driven by data. And, I have personally played with SimilarWeb products and it is absolutely amazing. The levels of experience you get there, the breadth of analysis, coupled with such a great user experience instead of speed and insight is something definitely to learn from. So thank you for joining.
Yoav: Thank you, Boaz.
Boaz: And see you around.
Eldad: See you around. Thank you, Yoav.
Yoav: Thank you, guys. Take care.
# Making Observability a Key Business Driver (/blog/how-statsig-engineers-do-observability-right)
80% of the code that you write doesn't work on the first try. And that's fine. But knowing which 80% is not working and which 20% is working is the actual challenge. After 10 years at Facebook, managing and scaling the Seattle site to over 6000 engineers(!) Vijaye Raji founded Statsig to make observability automated and real-time. How is the semantic layer managed? How was the Statsig team able to build an observability product that handles real-time ever-changing metadata? What are Vijaye's main takeaways from engineering at Facebook? Tune in.
Listen on [Spotify](https://open.spotify.com/episode/5hcGDxIPtZ7VmEZposgb1z) or [Apple Podcasts](https://podcasts.apple.com/us/podcast/how-statsig-engineers-are-doing-observability-right/id1561927688?i=1000587915342)
Eldad: Just before we start, I don't know if you've noticed, I've adopted a new brother, his name is Benjamin, as you can see here. I tried to get the hair close to the family's DNA, but it didn't work very well.
I'm joining Benjamin. Benjamin will run Season 2. He is also our query processing tech lead based in Munich, building and scaling the office there. I've met Benjamin 2-3 now, how long ago Benji?
Benjamin: Probably two.
Eldad: Two years ago, we got in touch. He was an intern, thinking about his PhD in Munich University and this one thing led to another. And here we are today with you, talking about data. So I'm really looking forward and with that, Benji, it's all yours.
Benjamin: Awesome! So, you're introducing me and I'll just go on, and introduce our guest, Vijaye.
Thank you so much for joining us today. Vijaye is the CEO and founder of Statsig, and he founded Statsig in February 2021, if I'm not mistaken. They're based in Bellevue, Washington, and their original office was in Kirkland, where Firebolt also has an engineering office and Statsig is building a product observability platform. And, we'll talk more about it later. Basically, it makes it super easy to manage feature flags and experimentation and yeah, we're excited to have you. So, they raised a series A in August 2021, and now in April of this year, 42 million Series B, led by Sequoia and Madrona. So, that's awesome.
Vijay didn't start out, as a startup founder, he started out in Big Tech actually. So he spent his first 10 years, at Microsoft doing a variety of different things, and then joined Facebook in 2011 where he stayed for another decade and there he led Facebook video and gaming, and was also the site lead for Facebook, Seattle. So, he scaled the site there from, just a handful to more than 5,000 engineers, which is pretty impressive.
Eldad: So, one out of those 5,000 used to work for you. He works for Firebolt now, and we got great, great feedback. So, we are happy and excited.
Vijaye: I love it. I love it.
Benjamin: Awesome. Perfect! So Vijaye, do you want to give a two minute high-level overview about yourself, about what Statsig is doing, before we dive into some of the nitty-gritty details.
Vijay: Absolutely! Hey guys! Thanks for having me. I'm honoured to be here on the first episode of Season 2. Excited to be here and chatting with you both. So, Benjamin, you covered the history a little bit. I started Statsig about 20 months ago, and this is after spending 10 years at Facebook. So, I joined Facebook in 2011 when Facebook was still considered a small startup. And then, obviously the company grew, both globally as well as in Seattle. So I spent, my time also growing the Seattle offices, from you said, a handful of folks to I think when I left about 6,500 people.
One of the things that struck me as I moved from Microsoft to Facebook, I was always an engineer, so I grew up and learned programming and wanted to be an engineer and I spent 10 years at Microsoft as an engineer. And, during these 10 years, it's formative when you see the software processes evolve.
And it went from the shrink box software that sits on Best Buy shelves to shipping every single day and then eventually continuously shipping code. Those are the things that I learned first hand at Facebook. And I was fascinated by that whole concept of how you decouple feature launches from code launches, and the tools that were associated with it, the tools that kept this chaos from getting out of control.
And also, I started understanding how you measure the impact of every single feature you ship. And then, throw away code that doesn't work because just because you thought it would work, it doesn't mean it would and then measuring every single change actually gives you a lot of insights, a lot of humility, I think, and then also a lot of cultural changes. And so, those definitely resonated very strongly to the point where I said, Okay, this is important, that we should go build a company around these sets of tools. And that's why in February 2021, I left Facebook and started Statsig. I brought with me a few folks who joined and built the tools and then now the company is going strong. We have 52 people as of yesterday. So, pretty exciting growth.
Benjamin: Awesome! Yeah, that's super cool! So, I mean...
Eldad: How was it switching from this huge, huge organizations, engineering organization to a startup that just now got to 52 people. How does it feel?
Vijay: It's crazy. Going into a startup, I thought I knew some things, but then over time I realized I had absolutely no idea what I was doing. It's really jumping off a cliff without a parachute, but just instructions to assemble a parachute and then you have to figure it all out before you land. And that's kind of like how it feels.
Managing an org of thousands of folks is a very different skill than building a startup from scratch, hiring the first person, the first engineer, the first product manager, first designer and then getting a product out. It's kind of like when you're building in the early days, you're building it in a vacuum. You have this vision, you believe in this thing, and then you go build and then you put it out. You're really vulnerable, right? So you build out your product and you put it out for somebody to come and take a look at it and you hope somebody would like it, and then, will use it, give some feedback, and then for six months you sit there and nobody is touching your product and you're wondering, Oh, what have I done, I've made the worst decision of my life. But then slowly, the traction happens, slowly people start to notice and then they are like, Oh, this is actually a pretty useful product. And, then you go from this one stage to another stage where people are now even trying out your product for free. And then eventually they want to actually use it and then eventually they want to actually pay for the software, and then eventually they love the product and then they actually want to evangelize the product. So you go through these stages, and each of these stages you're constantly questioning yourself and questioning your decisions. It is a fascinating journey that all startup founders go through. And, this is my, like you said, I spend all my life in large companies where you don't have to worry about this kind of stuff, and then when you throw it into a startup, every little thing matters.
It's been a fun journey, very much a learning journey, very much humbling, just to look around and see how many people that have gone through the same journey as me before and been successful.
Eldad: Nice! And you just started, there's a long journey ahead. Yeah, it is.
Vijaye: I know, definitely. And it's only been 20 months.
Eldad: You picked great timing as well, because I think of raising round be that perfect. Perfect, perfect, perfect! Those are crazy times for building startups. It is always crazy building startups but those times are especially crazy.
Benjamin is young, so everything is new, but we've been there through previous recessions and it's always surprising how different each time it is.
Vijaye: Yeah.
Eldad: But I wouldn't replace it for anything. And, really, we look forward to hear more about what you're building.
Vijaye: Yeah. Thanks.
Benjamin: Awesome. So let's dive right into that. So maybe for our listeners who weren't exposed to, I don't know, A/B testing, experimentation, all those things yet, do you want to give the two minute pitch of why this matters and what it actually is?
Vijaye: Yeah. One of the most important things is when you're building a product, you have a hypothesis and you think that's something that you're building is going to be beneficial for your users or your customers or for your business. And that hypothesis is baked into every single feature that you're building. The idea that we build those things out and then ship it to people without really instrumenting or measuring whether there is that particular belief is actually true or not, is no longer valid. It's no longer okay.
And, at Facebook, I remember this, where 80% of the code that you write or the products or the features that you build, don't work on first drive, which means that you have to go back to the drawing board, iterate on the product until it works or sometimes throw away your idea and it's okay because you can't be right all the time.
But knowing which 80% is not working, which 20% is working is the hard part. So, just understanding and measuring the impact of the changes that you're making to your product, to your users, to your customers is extremely important.
The stage number one is always about measuring, measuring things, measuring things that you build and ship to people.
Stage number two is taking the measurement back and then actually addressing some of these causal inferences. Why did the metrics do what they're doing? Can we attribute it to the changes that we made? And then make the decision, Okay, well what do we need to do in order to... whether double down on it because it's doing really well or wind it down because it's not doing well.
Those are the kinds of tools that people need access to in order to be able to make the right decisions on the product on a daily basis. The state of the art in the industry currently today about experimentation is still A/B testing. And A/B testing is the gold standard for causal inference. However, it's very, very time consuming and manually there's a lot of overheads because you have to first come up with a hypothesis, you'd have to build a variance and then ship out the variance and then you have to allocate the samples, isolate the experiment, run it, for however long it takes for statistical significance, maybe two weeks, maybe three weeks. And then you have your data science team go back and look at and analyze all of these results.
This whole process takes a long time. Most people don't run that many experiments. They pick some of these features that are really, really important, and then they run it through the A/B testing, experimentation pipeline, rest of the features just go out untested.
So, this is where I think companies Facebook, Uber, Airbnb, and even LinkedIn, some of these companies have sophisticated tools that basically take every single code change you're making, feature change you're making, and then have tools that'll automatically run these A/B testing. And then will give you back the results of the impact of those changes. Thereby you don't have to do the previously mentioned long process of A/B testing. You get these results on a daily basis and then you make decisions, which will be much better decisions than before. And that's what we are trying to build with Statsig, it's where we're going with this product observability.
Benjamin: Gotcha. I mean, you have a background in gaming at Facebook, for example. Do you have one or two grippy examples of things where this mattered or where these types of experiments led to interesting insights?
Vijaye: A lot of them. I say, one of the key metrics that I share is 80% of the code or 80% of the features that we think are going to be beneficial are not. And so just going back to the drawing board is a regular thing and it's okay. And, that way people don't get attached to the code they write.
So, I'll give you one anecdote. In video specifically, we have run plenty of experiments, and the general belief is that the higher the quality of video that you're able to provide, the more consumption that'll happen because obviously people like higher quality videos and so, at Facebook, we built, this automatic bitrate detection code, which basically constantly analyzes your network stability, last mile, bandwidth and all of that stuff to be able to, Okay, well we're going to give you the best possible bitrate for the connection that you have and so far, that's been experimented and proven and we continue to increase and increase and increase, and coding have gone up from a 480p to 720p to 1080p and then eventually, 4k, and the belief is, yes, that is always good.
And then, in 2020, the pandemic hit and when most people stayed home, the media consumption grew pretty heavily, almost over a course of a month, it doubled and tripled. And some of the backbones, the carriers and the ISPs, they couldn't basically handle the load, and there some of these folks reached out to Facebook and, Hey, could you help? Because you know, a lot of the traffic that is going out is going to Facebook and a lot of it is for video, could you throttle these for us? And we were, Oh my goodness. If we start throttling, that's actually going to reduce the consumption. That's actually going to leave people not feeling great about the quality and the experience, but we also understand that this is not normal terms. It's crazy, crazy times that we're dealing with. So, we went ahead and started tweaking our ABR algorithm to actually give a couple of clicks lower bit rate than what was actually possible to give. What we found...
Eldad: I always knew this 4K wasn't real 4K when they saw it. Something felt fishy there.
Benjamin: Yeah.
Vijaye: No, we actually showed that we are dropping and then if people wanted to pick 4k, they could go ahead and pick the higher bandwidth. It's just the automatic default that we pick.
When we change the default, normally, obviously, we experiment with everything, this is one of those things when we changed the defaults, we were actually looking at the metrics that were coming in. We were shocked because the usage actually went up. People were using video a lot more and consuming video a lot more with the lower bitrate. Obviously, this was surprising, this was new. It's , okay, well we would've only known because we ran these experiments for every single change we make. Then, we went back and we looked and we did some case studies and we talked to folks, and then later on, we understood that there's a segment of population that is very bandwidth sensitive, and whether they're throttled or whether they're limited bandwidth and so on, generally, they get constrained by how much per day that they can use.
Now all of a sudden, a whole batch of people were now able to use or consume more video than they did before. Then, there's a whole segment of the population that were on the phone that were old outdated phones where 4K or 1080p doesn't actually make a difference. And so, trying to push more and more bitrate was actually not necessary. So there's a set of learning that we had that was very interesting. We wouldn't have had those learnings if not for this experimentation, constant experimentation. Anyways, it is pretty eye opening for me as I was running the video org.
Benjamin: That's pretty interesting, because when I looked at this first and I was, oh, maybe this is about, I don't know, the color of you are pay now button on the shopping cart, but these are really deep algorithmic, core parts of your infrastructure to which you then apply those experiments. So that's cool to see.
Vijaye: Yeah, that's absolutely right. I mean, a lot of people, when they think about A/B testing, you're naturally gravitating towards, Oh, well let's change the text of the button or the color of the button, or change the layout of where things go in order to get more conversion. I think those are also valid A/B tests, but they're more marketing centric and more shallow because your metrics that you're trying to move are directly correlated to the product changes. Whereas when you touch, Okay, well I'm going to change the parsing library, that's in the backend, and then, I want to see if that increases any latency on the large segment population, especially at scale.
Those are the kinds of things that you actually care about when you're modifying a product and you want to verify that it's actually doing what you're expected to do. And then secondarily, you want to verify that there are no extreme side effects. Sometimes you don't know what's going to happen when you actually make these changes. So monitoring those things are important.
Benjamin: Right. So let's talk a bit about what powers actually infrastructure this? So the actual experimentation infrastructure.
The first question I would have is, say I want to run this experiment now, right? I'm an IC at a company, I have this type of experimentation software, and now I'm saying, okay, my video and coding algorithm, I want to change that and see if XYZ increases. Do I have to be the one formulating the hypothesis? Do I have to be the one saying, I changed things and I think XYZ might increase? Or is it there's some core metrics, someone else from the product team defined, and then each experiment is actually tested against these same core metrics? How does it work?
Vijaye: Generally, what you want to be keeping track of is there are some business critical metrics. You don't want to drop those business critical metrics. You want to make sure that they're healthy across all of the product changes that you are making. And then there are all these metrics that are the primary metrics that you expect to change, expect to move. Okay, well, if I'm changing my video and coding algorithm, I expect more video watches to happen or more time spent on videos to happen, and then perhaps even lesser lag or lesser interruptions in the video because of bandwidth issues. So those are the kinds of things that you would normally be watching as a direct result of the change you're making.
I think those are two separate sets of metrics that you should be watching. And one of them is probably picked by your company level product or growth leader. You're kind of like, okay, well I don't want to ever drop my DAU or engagement or retention, or obviously, revenue and things like that.
And then the other one is generally determined by the product team that is making product changes very close to those metrics. I think both are extremely important and then generally, more sophisticated tools Statsig. So you can basically specify if this particular metric, say, my DAU metric drops by below 3%, then I want an alert to fire and people to be notified. And so at any time anybody is changing any product you, if that drops the metric by 3%, you want to be aware of that and you want to be able to get right to the problem of what's happening?
Obviously you can also have these trade off conversations whether it's worth it or not. But that's up to you.
Benjamin: What actually powers these things in the background. There must be some pretty sophisticated statistics or something going on and one thing I just thought about, what about correlation and experiments, for example. I, on my team, am changing the video encoding, another team is changing how much video gets buffered and in the end, we are dropping this into production at the same time, somehow you have to figure out which change actually changes the core metrics you're interested in.
Vijaye: Yeah, this is a very good question. So obviously there's a lot of teams that are changing a lot of things. One of the things obviously you would not launch every product that you're making 100% to the user or splitting the audience into smaller portions and rolling it out.
Typically, we use this exponential rollout model where you take a new feature, you roll it out to 1% of people and then see how it affects the metrics, and then 2%, 5%, 10%, and then 20%, 50% and 100%. So that's the sequence that you normally follow through and making sure that at every stage, you're not dropping any critical metric more than necessary.
When multiple people are doing those, salt or the randomization for each of these rollouts is different. And so different sets of population will get these different sets of features. There will be some intersection and the rest of them are getting their unique experiences and that's how our system determines how to attribute metric change back to the actual feature change. And that's why we also have error bars in our confidence intervals. And then you'll be able to once the error bars are within statistical significance, and then we turn them into either green or red, depending on if it's positive or negative. And so that's how our stats engine determines the causal inference back to the features.
Now, obviously, there are lots of things that you could, also get, employ the tools to validate some of these. So, if you have multiple features that are highly interactive in nature, so, okay, I'm changing something in the product that are very closely tied to each other, then what you do is you have isolated experiments. So you create a layer, which is called the universe in other words. And then you actually say, I'm going to allocate 25% of this universe or layer to this particular experiment and another 25% to this experiment. And so you go like that and that way no single user will be getting two different experiences. They'll actually be isolated and getting one particular experiment that they work on. That way you can also make the confidence interval better, just decreasing the confidence interval and at the same time, also attribute it really clearly, Okay, well this experiment is doing these things to my metric. Those are the kinds of tools that are available.
Now, there's also something that you mentioned, which is, okay, what if I launch these two things at the same time? What happens? The interaction affects the cumulative impact of multiple sets of features. So, we have something called holdouts. So long term holdouts are able to actually measure the cumulative impact of multiple sets of features that you're launching over a course of time. We use this extensively. So basically what that means is the beginning of a quarter or the beginning of a half, you specify 1% or 2% of the population that are in the holdout group, and then you launch a whole set of features and then you go back and analyze, compare all of the metrics to these 1% or 2% of the people that have been held out from these features. That gives you the cumulative impact.
So you have all of these tools at your disposal to get all of that information.
Benjamin: All right, super interesting! Sorry, Go ahead, Eldad.
Eldad: How do you manage this semantic layer of all of that? Because it gets so complicated. I wouldn't consider myself an engineer anymore, unfortunately, but observability to me always seemed that it's easy to collect the events. But it's impossible to connect those events to a semantical model where you can actually drive the business based on that and having so many cloud native companies driving their business through their data, through their product, it seems those platforms are really not just about observability anymore, right? It's really about driving your business. Is there a standardized way to do it? In the industry, is there a shift towards a semantic, like a DBT for observability or something that will make an engineer's life easier to communicate on the inside?
Vijaye: Yeah, that's basically what we're trying to build, which is, you know, the sophistication is actually in the stat's engine and that should not make your product building process any more complicated than it already is. And so the idea behind everything that we're building is engineers should still be building features and when you decorate a feature with a feature flag, we then take care of attribution, the analysis and then also try to do it as real time as possible because you want to get to diagnostics as quickly as possible. You do not want to put out something that is broken experience for your users for longer than is necessary, and then, catch that as quickly as possible and then fix those things.
While it seems like, okay, there's lots of stuff that is happening, all of that is encapsulated by the complexity of the stats engine. But there is also a layer of like, okay, well if you make it complicated for people to understand what's really happening, then you've lost. You have not actually achieved anything.
So how do you take all of that data and simplify it? Some of the visualization innovation that we're doing is to like - How do you make engineers, product managers, even designers, be able to understand the insights easily. However, when a data scientist wants to come in and dig into the code, you want to have this progressive disclosure of complexity, want to actually have the ability to dive into the data if somebody is inclined to do so. So that's the challenge, right? The tooling should make it extremely easy for everyone, but at the same time not block you from getting into the details. That's precisely what we're building.
Product observability is like, if you think about it as an extension of, I always say, tools in the data observability space have become so sophisticated, and then if you look in the ops observability, you kind of like Data Dog or so, in real time, you'll know when one server is misbehaving among a forest of thousand servers or 10,000 servers, in real time, and that's pretty amazing. So systems have gotten so much more sophisticated, than what I remember, 10-20 years ago.
And then when you talk to product folks, you still are in the, Oh, yeah, we're going to launch V2, three weeks from now, and then when V2 launches we're going to wait for three weeks for our product adoption to happen and then we're going to have analytics or data scientists go and look at the analytics. And then have some way of correlating, well, are the metrics going up after the feature launch or the product launch or they are going down? Why are they doing those things? You see how slow and...
Eldad: And it's done manually outside of the system, so it takes your alpha feature eight months to go into beta while trying to make observability work.
Vijaye: Yeah.
Eldad: I think it's amazing. I think the next step is really getting observability to be impactful.
Vijaye: Yeah.
Eldad: It's always been a black box for engineers to do their stuff and seeing observability moving away from just helping engineers do their day job to really driving your business. I think it's a huge step forward, and it's also if you think of engineering, the more data they deal with, the more data that's being impactful, using those kinds of tools becomes not just optional anymore. It's really changing how you build products, and we've seen that firsthand at Firebolt, by the way, how it affects. This is not a hacking, it starts at, okay, let's put a feature flag, but it quickly turns into kind of a way to drive your engineering culture. As you've mentioned at the beginning. Amazing stuff!
Now tell us how complicated these are... like building observability is the hardest part, right? I mean, it's data driven, it's real time, metadata is actually changing unlike your product, which makes it even more challenging. Can you share a bit with us, how does it work? How do you do it?
Vijaye: Obviously, the credit goes to the data science and the infrastructure engineers that we have at Statsig. These guys are pretty amazing. I'll give you the overview of how we ingest the data and the infrastructure that we rely on and then the details of that is actually beyond me.
So, let's talk about… We have a set of SDKs that help you get started relatively quickly. Now, obviously building these SDKs is also a pretty interesting problem because as an engineer you're excited, okay, well there's all these technologies. It's like iOS and Android and React, and React Native. And so you got to build SDKs in every single technology because people have all of those out there. And this matrix continues to grow, especially if you have a new feature, you have to test it across. We have 25 SDKs, I think. So, 25 SDKs...
Eldad: The ecosystem team is going crazy.
Vijaye: Oh, Yeah! We have this on this whiteboard, I'm not kidding. There's this huge matrix where people draw, Okay, well have I implemented this feature in this SDK? And then, the matrix continues to grow. So, that's fascinating. That is step number one.
And then once the SDKs are integrated, they're obviously collecting event data and then sending it back to our servers. On the server, we have a queuing system called Event Hub. So basically the Event Hub is the first stage that actually acquires all of these events and cues them, so we don't lose a sequence. And then we actually have a redundant system as well. So once the Event Hub is receiving all of these things, they actually throw it into a file system, a big large file system so that if anything happens to the Event Hub, we can go back to the data that's sitting on the file system and replay it when necessary.
Beyond that it goes all of this...
Benjamin: Vijaye, if I can interrupt, can we take a step back maybe? What is an event actually in this world?
Vijaye: Yeah. No, absolutely. So, an event could be as granular as, somebody purchased a product in an e-commerce setup or someone, added to cart again, in an e-commerce setup, in a gaming setup, somebody reached a new level or achieved an accomplishment or achievement, and then in a FinTech scenario, someone used a credit card, and so those are all events and those are the ones that we be able to gather and turn them into metrics, really. Imagine if you have a purchase event and that purchase event happens to have a value, which is actually the dollar amount, then you can aggregate a sum of all that stuff, and then you can see revenue as a metric. And now you can verify, okay, if I'm making all of these product changes, is it changing my revenue up or down? And that's how we correlate it.
Benjamin: These events are then basically annotated with the specific feature flag configuration of that customer or user at that time?
Vijaye: No. The events only have user ID information and then on the other side where when we actually deliver the features you ask, for this user id, which features should we show?
Benjamin: Ah, got you.
Vijaye: So that's the correlation. The join is actually...
Benjamin: Join then in the background. Okay.
Vijaye: Yeah. That's why...
Benjamin: I love joins as someone building database systems.
Vijaye: Oh, I love it too because that actually..
Eldad: There's always a join Benjamin.
Benjamin: Always a join. Wow!
Eldad: Always.
Vijaye: Everywhere. Join is actually powerful because it simplifies the two responsibilities. So if I'm a feature builder, if I'm an engineer, I only have to care about, okay, which feature flag am I using? I don't have to worry about all the metrics and annotating the metrics and we do that for you. And so it's part of actually simplifying how we use the system.
So, once the events are in our backend, then we throw it into a data lake. And so the data lake is where we do deduping because sometimes when users are, especially in React setups, constant rendering, you have multiple events generated at the same time, and so you want to de-dupe, do some sanitation of the data and stuff that and so we do all of that on the data lake side of things.
Then what happens every hour, we have this gigantic data bricks job. This data bricks job is where all of the statistical correlation, causation, all of the analysis is all encoded, in Python. Jobs pick back up and then they basically spin. Can you guys hear me?
Benjamin: All good.
Vijaye: Yeah, they basically spin off thousands of servers to go orchestrate all the analysis and then outcome is basically a set of analytics. Those analytics go back to our data warehouse and then, where we actually break it down into lots and lots of pre-computed metrics, so that when you actually go to our console, you'll be able to get these things in real time. So you don't have to recompute anything. The UI is extremely responsive. You can click around and actually have things pop up very fast because that experience is important. Otherwise, you're sitting for even one minute for every one of these queries to run, then you lose the whole fluidity.
We take a lot of care about making sure that those things are working as quickly as possible.
Eldad: Nice.
Benjamin: Nice. It's not about querying anymore. It's just a feature. The fact that there is a query running in the background, users don't care.
Benjamin: All good. Cool. You have these hourly batch processes running, which means, say, I launched something, this experiment starts. After an hour or two hours, I'll start seeing some data on my experiment. What if something goes terribly wrong? Did the worst thing ever and it just breaks everything? Is the safety then that it's only with 1% of my customers and so these couple of hours don't hurt a lot, whereas to a fail safe or crap door if things go terribly,
Vijaye: Yeah.
Eldad: Benjamin, you want them to fix your bugs as well? I mean, come on.
Benjamin: I'm sorry.
Eldad: Don't stretch it. Don't stretch it too far. Sorry, go ahead.
Benjamin: No. I think that's important. I think you, you're hit in the nail because, it's important to roll out slowly in stages. So typically we see most customers when they create a feature flag, it's open to only employees, and so there's usually this dog footing process. If something is terribly wrong, you catch it with just employees before it even hits your customers or users. And then what you do is you roll it out to sometimes some customers have these early adopters that are a little bit more okay to get some beta experiences, and they're okay to tolerate some of these things that are broken. And so you turn it on to the next ring and which will be the early adopters. And then obviously you are monitoring all these metrics. Sometimes you get a little bit of scale, which is good. So it's stress testing your product and so if something is broken, you pause the rollout and you go back and fix it.
Then the next stage is, okay, well now I'm going to open it up to 1% of the general public or you can pick a country. Okay, I'm going to open it up only to the United States. Sometimes people do that because they don't want to invest in localization, and they want to test it in English-speaking countries first before and if everything works, then you invest in localization and so you roll it up to 1% and see how things are operating.
The good thing about this is if something is broken, you catch it within an hour. So, which is as real time as we can get. And so then you are going back and fixing things versus waiting and waiting for your customers to tell you or some support channel file a ticket, things like that. So, those are, I think, why it's extremely important to catch these early and fix them.
One interesting thing is because I'll add this here too. A lot of people think that, oh, you need lots of samples in order to even catch these things. But if something is broken, you don't need thousands of samples. You need 10 samples to tell you, Oh my gosh, this particular metric is dropping, so let's go figure out what's going on.
Benjamin: What volumes of data are we actually talking about here? How much data are you processing every day?
Vijaye: Oh my gosh, terabytes. We have about, 15 to 20 billion events every single day, give or take, based on how one of the large customers ran a sale for three days. And then what happened was their volume tripled for a week and our system is able to scale and take care of that. But give or take 15 to 20 billion and it's been growing. Just eight months ago, we had a million or so events a day, and now we're talking 20 billion events. Those things are raw events, right? And so the job that we need to do is to quickly dedupe and reduce that as much as possible before we toss it over to the data bricks jobs.
Eldad: Benjamin, this is a huge insert into the group by, by the way.
Benjamin: I see.
Eldad: Translating to relational algebra, insert into group.
Benjamin: Ah, now I understand...
Eldad: Billions, billions of unique events.
Benjamin: I always need those relational algebra trees to understand what's going on.
Eldad: That's why I'm here. You're welcome.
Vijaye: And then what happens after that is actually store all the raw events for when our customers need and then they want to be able to go download this thing and when you download, you get this a terabyte file, which is pretty crazy. It takes days to even download.
Then, obviously, we offered several ways to slice and dice, filter, so you can get your data as quickly as possible. But we store all of that stuff and so we're not quite at the petabyte yet, but we're quickly approaching that.
Benjamin: Awesome! So what are actually the scalability limits on those pipelines? What's harder basically growing to 10X more customers or a single customer of 10X the size. Is it similar? Is it very different? How is that looking?
Vijaye: Yeah, it is very different. I think 10X customers is actually easier than one customer growing 10X especially if that customer is extremely large. Because what happens is the joins and the analysis that we do are within the same customer's data. We actually have separation of data. These things take much longer and require a lot more orchestration. It's interesting, some of these things that I never thought we would run into, we are running into. For example, one of the things, where in the data bricks job runs, it spins out thousands or thousands or spot instances, spot VMs, then actually go and do the actual heavy lifting. And then to communicate with each of these spot instances you need local IP addresses and you run out of IP address allocation. Really! I never thought that you would actually run out of those kinds of things and apparently those are resources and then you're, Okay, well how are we going to solve this problem? And then you have to start thinking about, okay, well..
Benjamin: IPv6.
Vijaye: Pretty much, right. It is fascinating, when you're running some of these jobs at scale with so much data. Even the choice of how we store the data, what kind of databases we use, and how we run these orchestrations. It's been affected and it's actually evolving too. Obviously, we didn't foresee all of this stuff, so we built something that worked six months ago, and then you throw it all away and then you rebuild based on, well, we know now, and I am pretty sure six months from now, we'll probably throw all of what we have right now and then rebuild something for whatever scale that we will be dealing with. Because those are the times we're actually going to identify some new problem that we haven't even thought about.
Benjamin: Right. So, what's, for example, now on your current stack, an experience where saying, Wow, we would really like to provide this, but it's kind of something we just can't do on the architecture we have?
Vijaye: Well, I mean, wouldn't it be awesome if we're able to give you more real time than one hour? So imagine if we're able to give you in minutes, 5 minutes, 10 minutes, that would be amazing. Currently, our systems or architecture, our expense, the cost of infrastructure is prohibitive enough that we cannot provide that. So we have to work for an hour, as real time as we can get at this point, but I would love to keep pushing that limit as much as we can because I think we've gone from many, many weeks to an hour, which is a pretty huge step forward. But we're not going to stop there.
Benjamin: Awesome!
Eldad: I smell subscriptions coming along soon.
Vijaye: Yeah. I think especially as we go into this Web 3 world, I'm assuming that that's going to become a real time, an important issue. So we will have to keep investing.
Benjamin: Awesome!
To wrap up the conversation about your internal data stack, I assume you guys are dogfooding basically, so that you're also using Statsig internally to try out the things you are launching. Do you want to quickly talk about that maybe?
Vijaye: Oh yeah. Everything we do is behind the feature flag. So we don't ever throw out any code that is even a little bit sensitive or a little bit, we want to validate it. And so we throw everything behind a feature flag.
Obviously that gives us the separation of, when we can roll out a feature. Sometimes we do an announcement to everyone and then when we do the announcement is when we open up the feature flag. Obviously, we are also monitoring how people use these new features and when we build a new feature, is there adoption to this feature? And then we also constantly consider, okay, should we invest more in this feature or not. Those are product decisions that we make on a daily basis based on the data that we know how people are using our own product.
Our marketing side is full, feature flags. If you go to our marketing site, the button text then gets a demo, that text is actually different for every person, and that is driven by a multi-arm banded experiment, which is automatically deciding which text actually yields the most number of conversions. And then we'll start to show more and more of the winning variants. So we have 8 different variants that we throw into that system. So, we use stat's pretty extensively.
Benjamin: Awesome! Cool!
Maybe to close up today's podcast episode, we'll shift gears a bit to more on the advice side for our listeners. On my end, Okay, Eldad said it earlier I have just more academic background, so I'm relatively new to industry and you've seen all of it, right? Over three decades or well starting your third decade at Big Tech now at startups, do you have any advice for aspiring engineers basically?
Vijaye: Yeah, I think, as an early engineer, I always chase technology. So in the beginning I was, Okay, I want to be, for me, the compiler engineers are like Gods. And I was like, Oh, I want to be a compiler engineer. So, I went in... I was in Microsoft, I worked on compiler technologies. I worked on language services, incremental compilation and stuff like that.
Eldad: Benjamin's upcoming paper is on the query compilation, I'm sorry.
Vijaye: Oh, really.
Benjamin: I haven't told anyone yet.
Eldad: We cut it out, but we don't cut anything out. We'll list it after we release the paper. Sorry, go ahead.
Benjamin: Three months.
Vijaye: Yeah, and I went in, worked on compilers for a while and then over time I started realizing technical problems actually ladder up to user problems. Okay. What user problems are you solving? And so you have a view on, okay, this is what the problems are, and those translate into technical problems. And so you just start chasing specific user problems that you're interested in solving. So developer problems and so I worked on lots of developer frameworks in Windows and such.
And then over time, you start to realize a lot of the user problems after you peel the first couple layers are about the same, the technologies that you deal with, the problems that you solve are repeated over and over and over. You start to pattern match and stuff.
Then, it starts to, okay, there are people that I want to follow, go from technology to use problems to people, you start following people because you pattern match, like this person has done things that I want to do. I want to aspire to be these people. And so, pick those folks. And then, because you don't have to chart your own path, some of these things have already been done. So, we follow them. Pick good mentors. Pick good managers. And learn from them because there's so much to learn from people that came before you.
As you grow, then there's an element of turn around and pay back, because just as you grew on the coattails of other great folks, now there are a set of folks that are joining and learning the trade, turn around and mentor them and coach them, creating that followership. I think that's the journey of an engineer as you grow in your scope, as you grow in your craft.
Eldad: Boom! I love it! It doesn't matter where you work. If you are an engineer and if you love being an engineer and if you actually think about the problems that you are solving as an engineer, there's no limit and Benjamin, I hope I've been mentoring you enough, and if not, I will try harder because, yeah, it's future engineering is all about.
Benjamin: What am I supposed to say now, Eldad? Awesome!
Well, I think those were great closing words. Thank you so much Vijaye, for joining us. This was awesome. I think we've all learned a lot of new stuff. Cool!
Vijaye: Well, thanks guys.
Eldad: Thank you Vijaye.
Vijaye: Thanks, Eldad. Thanks Benjamin. Thanks for having me on your show. I'm looking forward to watching this.
Benjamin: Awesome!
Eldad: Absolutely!
Vijaye: All right, guys. Take care.
Benjamin: Take care.
Eldad: Bye-bye. Take care.
# How Substack's Data Platform Supports 500K Paying Subscribers (/blog/how-substacks-data-stack-supports-500k-paying-subscribers)
Substack is an amazing — if not the most amazing — content publishing platform out there. Essentially, it allows anyone to become a journalist or to start their own newsletters and charge subscriptions for them. So how did they build a data stack that can support all of their 500K paying subscribers?
Listen on [Apple Podcasts](https://podcasts.apple.com/us/podcast/how-substacks-data-stack-supports-500k-paying-subscribers/id1561927688?i=1000530877276) or [Spotify](https://open.spotify.com/episode/0sfJrGQXZ2ddTEo93m1GeK?si=rLvA6cyvROWFuYnsEjLmlw)
**Boaz:** We are very proud and lucky to have Mike Cohen with us today. Mike is a super talented data engineer at Substack. Now, for those of you who don't know Substack, you should! Substack is an amazing — if not the most amazing — content publishing platform out there. Essentially, it allows people like you and me to become journalists or to start our own newsletters and charge subscriptions for them. And Substack has been growing. In February, they reported that they have 500,000 paying subscribers.
**Eldad:** That's old news.
**Boaz:** Yeah, that's probably old news by now. Every month that number grows like crazy. They've been getting a lot of media attention. I think that we should consider stopping this podcast and moving to Substack or something. That's the place to be right now. So, Mike has been working in the data space for quite some time prior to Substack, which we'll hear all about. He also spent time at companies like Capax, Venmo, and a lot of other exciting places.
**Boaz:** At Substack, how much data, in terms of data volumes, do you guys deal with?
**Mike:** I don't know what big data means when people say "big," but I think we're in the small to medium data camp still. We're in the tens of terabytes, not in the petabytes or anything like that quite yet. But the data volumes are growing quickly as more and more people come on. We have a lot of event data that we're logging and that's where we're at today.
**Boaz:** I think the rule of thumb to "big data" is that everybody starts with apologizing that they have data, but it could be bigger. Yeah, we only have hundreds of terabytes. We're not petabytes scale yet.
**Mike:** Yeah.
**Boaz:** I think that is considered big data. And what's the headcount at Substack these days?
**Mike:** We just surpassed 40. In the last couple of weeks, we broke through the 40 mark. We're on a big hiring spree. Come check out our jobs page!
We're trying to scale up the team. Fingers crossed, we'd love to be somewhere in the seventies by the end of the year.
**Boaz:** And how many people deal with data?
**Mike:** We are a very small team in an already small company. We're a two-person team at the moment. So, it was just me for the first 15 months or so and then I recently brought on someone so we have two of us now since March.
**Boaz:** So, you've grown really fast. In terms of subscribers and visitors, you've grown dramatically also in a little bit over a year. How does that look like from the retention perspective?
**Mike:** It's been exhilarating. It's been super fun to watch. And the problems to tackle have grown with the growth of the company in general. Exhilarating is the best way to describe it. Constantly thinking about "this thing that we were doing a week ago — now how do we do it at a much bigger rate and faster pace? How do we design our systems for a couple of weeks and months from now?" Stuff like that. So, it's been super fun.
**Boaz:** So, let's talk about your data world. Tell us what it looks like and what kinds of things do you do with it.
**Mike:** We have a Postgres production database and what we call our events pipeline, which is effectively a Kinesis stream of data that gets processed in parts into S3 and subsequently then dumped into Snowflake. Separately, we also have a process that will mirror our data from production into the data warehouse. Other data sources are getting piped in there too. And so, everything ultimately lands there. Then we have our BI tooling set up on top of that. From there we do transformations internal to Snowflake, and then we pipe that back out to places. I think the phrase I've seen a lot of these days is reverse ETL. But we send our derived or transformed data back out to a separate Postgres database such that the data can be accessed in the product with indexes and be super-fast. So that's our high-level structure today.
**Eldad:** What about BI? Which BI tools are you using there?
**Mike:** We use Periscope Data acquired by a company called Sisense.
**Boaz:** So how much of the data stack at Substack do you consider legacy versus modern? How much has it changed since you joined or is it something that's already built to scale into the future?
**Mike:** That's a good question. A lot has changed since when I joined. When I first joined, we had very little in terms of BI tooling or any data warehousing. So, all of that was in the last 12 to 15 months and that's gotten us to where we are today. As a company, today we are thinking about how we start to really put the data to work now that it's much easier to work with and accessible, and we have the systems in place to put it back into the product or to do BI and analytics. And then I think after that, there'll be the next chapter of asking "What do we do next? How do we build towards more real-time? How do we build towards faster insights?" And unfortunately, as a small team, we have to take things in little chapters. That's how I kind of think of it. So, I think we're on chapter two now and then chapter three will be about figuring out how to ramp this up even faster and do even more.
**Boaz:** Who is driving the requirements for BI? With 40 people, is it specific departments or cross-departmental?
**Mike:** It's a mixture of data exploration work, which can be driven by either the product or data team. We also have other internal teams, whether it's our support team or what we call our partnerships team, and they have data questions. As a data team, we will help them answer those questions by giving them reports that they can monitor in Periscope.
**Boaz:** So how much of a bottleneck do you end up with? It sounds like you have a lot of supporting to do.
**Mike:** Yeah. That's one of the reasons for doubling and why we're seeking to double again, hopefully this year. I'd love to end the year with around four people on the team. And so it's definitely a factor. And I would say it was only somewhat recently we found our team chemistry... we used to be more technical users than non-technical users. In other words, more SQL users than non-SQL users. And only very recently has our growth shifted that equation to where now we have more non-SQL users. And so that bottleneck has started to become more apparent than it once was.
**Boaz:** What does your morning routine look like? Which tool do you open the most to check in on things every day?
**Mike:** That's a good question. I check Slack to make sure there's nothing in the data channel that someone has reported or asked about. Then I have about six Periscope dashboards that I look at in order every single day that are basically pinned in one of my Chrome browsers. They're high-level company stuff. And then I have a spam publication detection dashboard where I look for any bad actors and try to handle that, too.
**Boaz:** It's an interesting use case. How do you find the right actors in the spam publications?
**Mike:** I can't answer that. That'll give away my fraud rules and everyone will know how to beat them.
**Boaz:** Good point good point. I was just testing you, but okay. Let's do what we call a lightning round...
**Mike:** Okay.
**Boaz:** So don't overthink. Shoot straight. Let's see what you come up with. Are you ready?
**Mike:** Yes.
**Boaz:** Commercial or open source?
**Mike:** Commercial.
**Boaz:** Batch or streaming?
**Mike:** Streaming.
**Boaz:** Write your own SQL or use a drag-and-drop vis tool?
**Mike:** Write my own.
**Boaz:** Work from home or from the office?
**Mike:** That one is hard.
**Eldad:** Both. You can have both.
**Mike:** Yeah, I think three days at home, two days in the office.
**Eldad:** Yeah, exactly.
**Boaz:** There are Hybrid modules now so it's legit to say both. AWS, GCP or Azure?
**Mike:** AWS.
**Eldad:** So, you can pick one. To DBT or not to DBT?
**Mike:** Controversial, not to DBT.
**Boaz:** To delt delete or not to delt delete.
**Mike:** Not to delt delete.
**Boaz:** Not to DBT, I think is the first time we had somebody said, no.
**Eldad:** This is the first time.
**Mike:** I know.
**Boaz:** Let's talk about that.
**Eldad:** You're probably mistaken. You probably got that answer wrong.
**Mike:** All your listeners just turn this episode off.
**Eldad:** We put it in the trailer.
**Boaz:** So, elaborate on that a little bit. So, what's your take on DBT and why not?
**Mike:** No.
**Eldad:** Big no. It was a big no, that's why.
**Mike:** It's not, it's not. I don't have that strong of an opinion. I've used it, but I have another system that I kind of put together that does similar behavior and I think allows a little bit more control. Ultimately, with DBT, my understanding is you still need to have something that schedules and orchestrates the jobs and so I've just kind of created a system that does a lot of that. I don't want to say it's anywhere near as comprehensive as DB, but does a lot of that. And it's all just based in Airflow, so it's Python-based.
**Boaz:** You said you guys are on Snowflake. How much processing? Did you guys do ELT exclusively in Snowflake? Do you do a lot of processing also outside of Snowflake?
**Mike:** No. A hundred percent of the processing is happening in Snowflake.
**Eldad:** Have you ever considered using spark to do that? Or just started a clean sheet with Snowflake? No need to migrate anything.
**Mike:** That was my thought. Yeah, it was clean sheet. Start from scratch and we'll see when we need to go bigger than snowflake "can handle." I'm sure they don't want to hear that, but I'm sure there's a point where using something like Spark in a more distributed fashion — where you can have a lot more control — might make a lot of sense. But we're not there yet at least.
**Boaz:** Looking at your pie chart of time spent on which activities, how much time do you spend supporting the BI users and the BI tools versus supporting the warehouse or supporting the pipeline, and so forth?
**Mike:** Yeah, not to cop out on the question, but I'm pretty evenly split at the moment. And there's like another administrative chunk, which is hiring and building the team out so that I can be more places all the time.
**Eldad:** So basically, like most high-growth startups, 70% of your time goes to hiring and then the 30% that's left goes on real stuff, which is great.
**Mike:** Yeah, I would say I'm at 35% hiring, 35% support of different people or functions and meetings, and then the remaining 30% is split evenly between either data engineering work or just my own data analytics work.
**Boaz:** Always hiring is also always a good excuse because if you tell people, "I'm sorry, it will be fixed the moment we hire another person, so I'm actively hiring." It's not your fault essentially.
**Eldad:** Boaz loves hiring. He discovered hiring a few months ago.
**Boaz:** Eldad is always complaining, "Why didn't you do this? Why didn't you do that?" And I tell him I'm hiring for it.
Okay. So, tell us about an awesome win at Substack.
**Mike:** I mentioned before we want to get to a place where there's more real-time analytics and more real-time insights in the product. But right now, it's we're living in a batch world where our definition of "real-time" is really every 20 minutes. We're kind of updating data in place. But I'm pretty proud of the system that we have. We're running a bunch of interesting, complex queries that create meaningful tables that are de-normalized and great for analytics, but also great for serving up things in the product. And then the way we're piping that back to Postgres with indexes in a way that is efficient and scalable is pretty neat. So I'm happy with that system that we have in place.
**Eldad:** Connecting data back to the product and feeding the product experience with data is huge. And you're right, it is super satisfying to get there.
**Mike:** Yeah. And when you send a newsletter a lot, some people want to just refresh, refresh, refresh, refresh, and watch the numbers tick. We're not there yet and I want to get there. There'll be other exciting things between now and then, but that will be a really exciting day for me.
**Boaz:** Now enough with this self-gratification, and then the win stories, tell us about an epic failure.
**Mike:** Okay. That's a bigger list.
**Eldad:** Everything that happened before we managed to get data back to the product.
**Mike:** I guess I should say, thankfully, there haven't been catastrophic errors that we can attribute to the data team, but there have been things we've done poorly. For example, we were writing data too aggressively to this Postgres thing I keep talking about and we ended up filling up the write ahead log and knocking over the database. All queries started to time out and then the site went down. So, we've had a number of learning experiences about how to do things that keep the site running and how to test things a little bit better. We also use some of our metrics and monitoring tools like Honeycomb to have a sense of when things might be going wrong and then we try to prevent that from happening in the first place. So, there's been a lot of small disasters, nothing too catastrophic yet. The keyword is "yet" because I'm sure that it's coming.
**Boaz:** What's the top challenge for data engineers or the data team in general at Substack?
**Mike:** I prefer to keep the surface areas small. So, in many organizations, there might be a data warehouse or something like that and then data is sent back out to a lot of different other services. And I am trying to not send it out to too many places because my fear is that ends up leading to a situation where you have one person looking at something in Google sheets, or Excel, or Air Table and saying, "Oh, I see this number here, but over in the BI tool, the number is different." There's a lot of ways to try to control for that, but one way I think to control for that is to try to centralize and keep things in one location.
And so, the thing I'm thinking about a lot recently is how to make our data and our analytics more self-service? And more self-service for not necessarily technical users. Whether it's building canonical data sets that are easy to query and we give everyone a little SQL lesson, or we get some sort of tooling that doesn't require SQL knowledge at all. How do we democratize the data access a bit more? So, I think about that a lot, and that's maybe more on the analytics side than on the engineering side, but I think that they're two sides of the same coin, really.
**Boaz:** How much of your responsibilities are on the analytics side as well?
**Mike:** All of it.
**Boaz:** Building out company dashboards and stuff like that. All of it? Wow. What gets on your nerves the most in your daily work with data?
**Mike:** Reconciling data from different data sources. For example, yesterday, I spent a long time trying to reconcile data that we're seeing from a test that we're running in Optimizely with our own data events logging and it's hard. It's an example of what I was talking about just a moment ago where Optimizely is a bit of a black box. You want to be able to put trust in the tool but sometimes you verify it yourself and in verifying it yourself, you end up in a rabbit hole. And so that can be kind of frustrating but it's important.
**Boaz:** Yeah. I feel you on that one. It's important, it's frustrating.
**Mike:** Yeah.
**Boaz:** For sure. Okay, so we're close to reaching the end. We want to get your advice on which companies, leaders or people to follow that inspire you or that you find interesting online.
**Mike:** Data Engineering Weekly is a cool Substack newsletter. Tristan Handy has a great newsletter, which is not Substack based but is still a big newsletter.
I'm also interested in what you guys are doing at Firebolt. I love Snowflake, but one of the reasons we're sending data back out to Postgres is because you lose the ability to index or to have functional aggregation that are very snappy. And so, I think getting the data warehousing or the OLAP databases to look more like databases where the index is going to be very, very interesting and compelling in the near future. So that's very interesting to me.
**Boaz:** Thank you so much. I think this is it. Eldad, what do you think?
**Eldad:** I think his interests are really spot on.
**Boaz:** Imagine Mike is not here with us. What would you tell me about him behind his back?
**Eldad:** So, I think the stack is perfect and frustration is hard, but it's a daily frustration we all deal with. And really, I think I wish you all the best. I wish you can scale fast. I wish you can hire a team that you love and can work with and accomplish meaningful things together. Always great to see you.
**Boaz:** And congrats for being a part of the success of Substack. So, keep in touch and thank you everybody for hopping on this episode of the Data Engineering Show. See you soon.
**Eldad:** See you soon. Bye bye.
**Mike:** Thank you.
# How Vimeo Keeps Data Intact with 85B Events Per Month (/blog/how-vimeo-keeps-data-intact-with-85b-events-per-month)
How does the Vimeo data team deal with 2 PB of data and 85 billion events per month? What made them recently build a data ops team? What data tool does the team love? And why (the hell) did they call their legacy platform Fatal Attraction?
Listen on [Spotify](https://open.spotify.com/episode/4hQMUFl974ipOfrUHGSfmL) or [Apple Podcasts](https://podcasts.apple.com/us/podcast/how-vimeo-keeps-data-intact-with-85b-event-per-month/id1561927688?i=1000532388799)
**Boaz: We recently had the pleasure of speaking with [Lior Solomon](https://www.linkedin.com/in/liorsolomon/), VP of Engineering at [Vimeo](https://vimeo.com/).**
**Of course, Vimeo doesn't really need any introductions. We all have watched a video at one point or another on the platform. Lior has been with Vimeo for almost three years. He joined as the Head of Data, then moved his way up to VP of Engineering. He is an industry veteran, having been in data leadership roles and engineering leadership roles for quite some time.**
**Anything I missed?**
**Lior**: The only thing I'll elaborate on is what Vimeo is. It is known brand and video platform for a lot of people. But you know, for the past five or six years we've been focused mostly on helping small to medium businesses scale their business and bring an impact with video.
So, in the context of data, that's where it becomes interesting. If we're trying to provide these businesses insight about potential competitors or other businesses that are driving an impact, the data should be intact before we try and do that. So, I joined Vimeo with that objective.
**Boaz: "Data should be intact." That's a good tagline. We should make t-shirts. That could be a big seller.**
**Okay. Thanks. So, let's get to it. Let's talk about data at Vimeo.**
**Let's get started with the sheer numbers. What kind of data volumes are you dealing with?**
**Lior**: Last time I looked at it, we collect about 85 billion events per month. We have a couple of data warehouses, unfortunately. We have about 1.5 Peta of data on Big Query, and about half a Peta on HBase.
Most of the data is streaming in as viewership — you know, the analytics around who's watching what and how frequently, the quality of experience, if it is buffering, which regions people are watching from, and if the videos are distributing properly. That's the motive of most of the data coming in.
Beyond that, just like any SaaS platform, is just gathering the user experience, analytics, and any events and trying to make sense out of it.
**Boaz: Ok we'll spend more time diving into the stack because it's interesting how you spread all the eggs across more than one bucket with all these technologies, Snowflake and BigQuery.**
**But, people-wise, I think Vimeo is a little bit over 10,000 people strong, more or less. How many people are in data-related roles? How are those teams structured?**
**Lior**: We're about 35 data engineers. We have a data platform team that focuses on the real-time processing and the availability of the pipelines —mostly Kafka stack. We used to have a managed Kafka clusters, but we moved to a manage one of them in Confluent.
The data platform team is also working continuously on building a framework to make it easier for other engineering groups to consume data from Kafka — just like having their own data syncs dropped or whatever.
There's another team called the video analytics team, which focuses on the consumer-facing video analytics, basically serving the consumer on the website and on the mobile apps. They work on getting the data closer to real-time with all the aggregation and all the challenges around that.
There's another team for enterprise analytics, which focuses mostly on the CDN aggregation. We get a lot of data from vendors on a daily basis that we need to aggregate to understand how much bandwidth is being consumed by each one of the accounts. There, the stack is mostly Dataproc using BigQuery, and some of that data set eventually leads into Snowflake for the BI team. The enterprise analytics team is kind of the gatekeeper of the data warehouse and Snowflake. They're the ones that actually do the ETL integration bags, Airflow and so on.
The data ops team focuses on the time to analysis. They help product teams to onboard their event, payloads, work with them to do the event, and data modeling. They're the team that actually provides a framework internally for the big picture. We like to think of it as a structure stream where any PM or engineering team can go and create their own schema. And once that schema has been created, we provide the SDKs to whatever platform they use. And that framework basically validates the schema and lets those teams find where exactly they want to drop later.
**Boaz: Has data ops always been around at Vimeo or is it a newer addition to the team? Tell us about evolution of it.**
**Lior**: It's pretty new. We launched it maybe eight months ago.
**Boaz: And what drove you to launch it?**
**Lior**: That'll take us into the legacy and the history behind Vimeo.
**Boaz: So, let's roll up the sleeves. Let's talk about the history. How has the stack evolved?**
**Lior**: So originally, we had another home group kind of pipeline for collecting data from different names, called Fatal Attraction. Don't ask me why. That's the name.
It's basically just like unstructured data. You can push whatever you want, but you have like, basically three different columns where you can push. The original idea was like, send me the component where the event actually happens. Send me the page, the actual payload and unsurprisingly after many years, all the massaging of data happened on the ETLs downstream by the BI team. You see ETLs of thousands of lines of code that basically extract the data. In some cases, it's JSONs someone decides to send.
**Boaz: Were these Spark-based?**
**Lior**: No, it used to come in from basically backend logs. We'd aggregate all the backend logs, push it into Kafka, drop it into Snowflake. And then on top of the raw schema in Snowflake, we'll build ETLs that make sense out of it.
**Eldad**: One big variant field with all the adjacent in it. Thousands of ELT processes to parse and structure it.
**Lior**: Absolutely. Yeah. So, you know, there's a couple of challenges here. First, it is so easy to break this pipeline. You know, an analyst worked for quite a while to find his way to the data, and then add a couple of lines of code to those thousand lines of code. And then exactly a couple of days later, someone changes it upstream. The analyst is not even aware of it and we have a broken pipeline.
That's the world we're trying to shy away from as much as possible. Today we're focused on improving the analytics efficiency, letting them be as fast as possible and letting them focus on driving insights. We're probably not the first company to do that.
But going back to Fatal Attraction and the unstructured data. We got to the point where we said, okay, let's build a different framework. Let's make sure that the there's a contract with the client sending the data and we can validate the data. That way we can find the ownership to who's the one submitting the data and emitting the data. And basically we're using schema registry to validate on top of all those contracts, all those like schemas.
For each event, we have a valid topic and an invalid topic. So, once it goes away in that topic, we know, oh, something's wrong. We can go and ping the team that actually is responsible for that pipeline and tell them, "Hey guys, something's wrong." They can go fix the issue. And we can rerun the events because we don't lose them. We still have them.
Problem solved, right? It's awesome. Now everything's like intact. If anything breaks, we know immediately.
**Eldad**: There's always someone to blame.
**Lior**: We don't do that.
But, you know, we created a new problem that goes back to your question about data ops.
From the engineer's perspective upstream, like the one building the applications, you know, working on the user experience, they're like, "oh, that's awesome. I can go into like the big picture UI. I can set up a new schema. I can focus on the questions that I have because I don't care about the other teams. I have a question I want to answer. And I personally just want the data in amplitude." Because, you know, that's how my PM is using… That's how they want their analysis and they don't really think about needs of the analyst or the marketing guy and so on and so forth.
So really quickly you end up with like hundreds, if not thousands of different schemas. The life of an analyst is like: they come in. Someone knocks on the door like, "Hey guys, can you tell people this hypothesis? I want you to analyze this user behavior." And they spend weeks to understand which events they should cherry pick from the list of events that exist.
Yes, they are more reliable, but still, it's not quite clear what the context is, what you should use, And so on.
So, data ops came to that for two reasons. It's first of all, to stop that behavior and say "guys, before you go and create a schema, let's sit down. What are you trying to do? What are you trying to achieve?"
Say if you're trying to track if someone paused the video, I think that already exists. Don't go and create another one. Or if you're just missing some data, let's think about how we can extend the existing event or coordinate with the other teams and make sure it's event data modeling.
That's one objective. The other objective is to create the automation that we need for that specific framework because downstream for the analyst, it doesn't make sense for them to be so familiar and savvy with each one of the schemas. At the end of the day, they want to have click stream.
There's a set of properties that they're looking into, they're looking for some sort of standardization. So, the data ops team was working on creating what we call internally "global properties."
Basically, we have roughly three engineering organizations within Vimeo. It's about 500 engineers. Each one is responsible for a certain application. We signed a contract with all those engineers saying they have to provide us with that information. That's like the basic bread and butter for any analysis. That's how we create that.
So, a lot of the automation is done by the data automation team. And that team is actually a cross-functional team. In that team, you have a front-end developer just for the UI and all the "making fancy," and a data engineer that builds the ETLs.
**Boaz: How big is that team?**
**Lior**: Four right now.
**Boaz: But you mentioned that they're essentially in charge of standardization and consolidation of models.**
**Eldad**: Its basically going full self-service and then turning that into kind of a fully managed self-service. Removing tons of friction, trying to find common data sets, common metadata.
This uncontrolled service, pretty much going back to the Excel spreadsheet nightmare. We all grew on just a better scale with data engineers instead of information consumers.
**Boaz: I wonder, putting that team in place and suddenly telling the org "let's stop for a second," like you said, how do you enforce that? How did you manage the process around that? Because you know, an analyst I might guess that they can just go ahead and do what they did before. Sometimes it would be faster to not even ask if we're around the data ops. So how do you manage?**
**Lior**: Yeah, that's a great question. I won't be able to say, "Hey guys, stop for a month, we're moving." We actually moved, we worked with both of the pipelines, so we still have the legacy to Fatal Attraction, which is slowly draining because as the new products are going, everyone's speaking the framework. So, we are slowly draining that pipeline. I'm having active discussions with leadership to be more aggressive. But that's it: we need to be completely deprecated, even from the legacy parts of the website, because we're going towards using the new framework for experimentation and AB testing, which we do internally.
One of the ways that I got the engineering teams to actually bear with me and work with me on the new framework was to find the carrot at the end of the stick of and say, "Hey guys, how about you get some self-serve analytics on aptitude?" And they were like, "That's awesome."
The analytics team is more focused on business. You know, KPIs, correlating, retention with FTS and subscription. More of the advanced analytics data science work.
And when it comes to, "Hey, tell me how many people click the button. Or show me a funnel." Applicant is just amazing at that. So, I say "go create that event." It immediately appears on aptitude that they created a self-serve world for a lot of the PMs and engineering.
That also created the problem because now again, going back to the original issue, there's not multiple schemas out of team and really hard to control this. So how would you get from this mess? Because it seems like you created a great framework where anyone can create whatever they want and they're happy about it, but the collective is not. So, in the coming quarter, we're working on an enrichment layer and we're looking into KSP over confluent.
The idea is that we want to dumb down as much as possible for the client sending the data. I don't need you to tell me about what subscription the customer has because I already know it. And I have those dimension tables. I don't need you to provide you all that information. I don't need to tell you which team ID they belong to.
So, the reason we can do it so far is because a lot of the obligations. And the business logic is maintained on the data warehouse and Snowflake. And when I'm serving the data from Kafka into amplitude or any of the end points, you know, it could be a Prometheus, whatever you're using, I don't have the access to those dimension tables.
So, using KSP, we're basically planning to have those tables closer to Kafka, KSP runs under the hood with persistent data. And then once that happens, we're going to create basically no more than 20 or 30 generic events, but just asking for the facts and whatever needs to be enriched.
That's going to be done through KSP. It's a framework that you can use in order to drop it to whatever data sync you need. So, you will be able to self-serve. The fact that the event's already designed. The data model is all in place.
We're dumbing down the client. It will speed up the deployment of new products.
And we're keeping the data modeling for data engineering and analysts.
**Boaz: Is this already launched in production or is this an active project?**
**Lior**: No, we just started working on it.
**Boaz: You mentioned before transitioning from managed Kafka to Kafka confluent. What was the story for that transition?**
**Lior**: I think overall, when you need to measure where you want to put your efforts versus maintaining your infrastructure versus putting it on managed, even though you pay for it, it's not for free. Right now, we are focusing our putting our CPUs into driving data insights and helping BI, helping data scientists, machine learning teams to actually move forward with their initiatives.
So right now, as a rule of thumb, if there's a service, I can put it as a managed service and offload that work from me.
**Boaz: You mentioned a huge variety of things at the end of the day that you guys do with data, and it's pretty impressive to see Vimeo being so data driven. Can you share what sort of workloads or use cases you enjoyed seeing that brought a lot of value?**
**Lior**: Yeah. I would say almost any department is actually consuming data from the data warehouse for their own initiatives all the way from marketing to return on targeted ad spend.
A lot of the data science projects are also utilized for Snowflake. I would also say the machine learning team uses some of their payloads. For example, we use Kubeflow together with Airflow to ingest and pull data from Snowflake. An important team is also 'Search and Recommendation', which is running Elasticsearch.
They're basically running our personalization and recommendation algorithms. Also they're utilizing data. The data comes from Slope. And the personalization data is actually stored into a big table and Snowflake. As we talked at the beginning, originally the reason we had both was just because we moved to GCP a couple years ago. When I joined, we just moved from Vertica to Snowflake, Snowflake had only the AWS implementation.
So, we kind of run both and I don't see us moving to Snowflake over GCP.
**Boaz: But what's your strategy there? Is the intent to stay on both AWS and GCP? I mean, you're running both Snowflake and big query — Do you consider that sort of a legacy tech debt, or is it sort of a strategy of using the right tool for the right purpose?**
**Lior**: The way I'm thinking about it is Snowflake to me is where I want to make sure there are high-quality data that we use by this different business. BigQuery and the stack on GCP are for data stores. There's something more engineering-oriented.
The easiest way to think about is like, I won't put any report which is coming from BigQuery in front of leadership. I'm not owning it. It's not my problem. Anything that goes into Snowflake, that's what we do.
**Boaz: So what are the big plans for the next year within your data initiatives?**
**Lior**: We are doubling downloads. For me personally, I'm really, really focused on the data availability and building trust with the overall organization.
We are actually expanding our machine learning teams and really trying to go in that direction. We are trying really hard to advocate for hiring more and to take more risks as a business. We spent a lot of last year on the data availability aspect, but you know, creating SLAs for some of the data stack, making sure that teams have the expectation what they don't like, the business, what is supposed to be done, what's done in response to any data outage. The more we can create that world, you set up expectations with the stakeholders because the easiest statement is saying "the data is wrong." It is not easy to draw those lines like where does the data engineering start? In a lot of the cases, I'm more facilitating the sessions that are not necessarily related to data engineering, but for example, we're trying to help the team by being their ambassador to find where the data problem starts. There's a transaction scope and deal, or maybe payments not getting in on time. It could be third party like one of the vendors, it's not an engineering issue. We're still trying to find this engineering language to help them get there. It's the world of human engineering required to be successful in data engineering.
**Boaz: We don't talk about it often enough, the politics of data.**
**Lior**: Yeah, it doesn't matter how good of a technology we build. It's a combination of people and processes and technology. We implemented Monte Carlo like September last year, which was super successful. I'm really, really happy with it.
**Boaz: Yeah. So, Monte Carlo are trending like crazy. Tell us a bit about the value you got out of it.**
**Lior**: We literally jumped to the future because we were actually trying to build some of those things, and I'll give you an example. So, let's talk about a little bit about data validation. So upstream data comes into the Kafka. Now we have a schema registry and fancy framework that know to tell if the data is right or wrong. Well, what we don't know how to do that yet. To tell tell you if there are any unknowns or anomalies in the data. So, you go downstream a little bit in the ETL and Airflow. We put great expectations there, and maybe the expectations are great, but it takes time to implement for the whole pipeline, all the ETLs. You have to prioritize. And unfortunately, usually you're prioritized by who's shouting at you more.
So we experimented with it a lot. We found it is good for specific anomaly detection, specific business logic. What Monte Carlo did, which wasn't the main thing in my mind, was that it hooked up to Snowflake and Looker.
And basically without setting up anything, they started listening and building those metrics for all the tables, the data sets we have. And suddenly, I'm starting to be aware of problems I wasn't aware of at all.
My first reaction was that sounds like pretty risky, because now we're going to spend our time jumping on like a thousand alerts. How's that going to go?
So, what we did was to actually start focusing on specific schemas or specific data and understand who's the owner of those nodes. Because you know, sometimes you cannot leave the issue with yourself. You need to go talk to the engineering team, understand what's going on. So, we started building those relationships where if I know what dataset, I know who's the team actually driving the data set. I can set up a Slack channel where any alert goes there. I'll always be there as a data engineer. So, I'll be aware of those alerts, but also I'll make sure that the stakeholders that also on that channel and the publishers are also in that channel so we have a team informed.
Now in reality, we need to build that relationship. Some teams are more data driven and they're excited about solving those problems. That's not the biggest problem. The biggest problem is we need to just start building those reports.
I'm getting a report and it tells me, from all the challenges you have right now, from all the teams you're working with, that's the responsiveness SLA. And for me, that's kind of like a get out of jail thing, because whenever someone says, hey, data is bad. I'm like, data is bad where?
Oh, it's bad on the CRM. That's the CRM. Well, you guys are not responsive to any of the alerts for the past month, obviously it's bad, you know? So that starts the conversation in the board where you're setting up the expectations with your stakeholders.
**Eldad**: That's the amazing thing about how Monte-Carlo takes your existing modern stack, your older mess, all the confusion and kind of backwards, built up draws a picture, shows you where the problems are.
That's a new way of driving data. A few years ago, we would have to have a dedicated team just for that. It doesn't work, especially when there's so much self-service going on. It's just amazing to see and we're super happy to hear about Monte-Carlo. We hear that a lot. So, we expect great things from this company.
**Boaz: You shared a lot of great things that you do at Vimeo but we cannot let you brag forever. Now it's time for you to share an epic failure with us. Doesn't have to be from Vimeo, can be from your data career. Let's hear about some things that didn't work.**
**Lior**: When we built this big picture framework, we were so focused on the self-serve aspect. We thought that would be amazing. They can go and create their own schemas. And it's so great. If it fails, we'll find it. That was the mindset. But then we said, hold on. Who's going to be the one that sets up the mandate in regards of like, why are we even creating that? And at what point do we bring the customers data? The analysts and the data scientists should be part of the discussion about what we need.
**Eldad**: Will be interesting to talk to you in a year and see how that experiment went. I mean, in many cases, companies just jump between two extremes. So, they either go full self-service and then they kind of get burned out and then they go full managed and four people try to route every request to the company.
I think that the modern data stack is here to help. So, will be amazing to see kind of how that plays out. Always, as you said, manage expectations, manage people. And prioritize.
**Boaz: Ok, maybe a weird question, but what would happen to the business if the budget of the whole data engineering and data-related initiatives were cut in half?**
**Eldad**: No ads on videos, for example.
**Lior**: Great point, because we don't do ads on videos. Do your research.
**Eldad**: Ahh, plugged on purpose. Of course I know that.
**Lior**: So, going back to the question about managed, not managed, I think right now we're focused on insights.
We strongly believe that data is on an exponential mode because we are a company that has existed for 16 years with no space on video. We believe that if we'll be able to land the processes and teams and infrastructure properly, we can start extracting insights about videos and help businesses thrive.
And really we need that. We won't be just the video platform. We'll be providing video insights for customers. So that's like, that's the main focus and how fast can we get there.
We just had the IPO and are in that sweet spot. It's not as money doesn't count, but right now anything that we can just delegate and just not focus on so that we can just focus on building those insights, that's going to promoted right now. But when my CFO looks at the Snowflake budget, he's really worried because we did grow it. I think from the first year, we grew it by 55% and the second year, I think 70%.
**Boaz: Just by the number of times you mentioned Snowflake, I'm sure your CFO is super worried looking at that bill.**
**These are amazing times to be in data because in the past data budget talks were "how can we, with the least amount of spend, get these annoying reports out of our way." Whereas today, modern companies understand that data is an investment. It's value. It is suddenly an X factor for business models for companies.**
**And it's okay to invest more than we used to, not less, to try and drive value out of it. So, it is super interesting to see how Vimeo does that.**
**Okay, good. So, we want to lighten things up a little bit as we near the end. We'll do a quick blitz round of short, quick questions. Don't overthink. Just answer quickly.**
**Commercial or open source?**
**Lior**: Commercial.
**Boaz: Batch or streaming?**
**Lior**: Streaming.
**Boaz: Write your own SQL or use a drag and drop visualization tool?**
**Lior**: Write your own.
**Boaz: Work from home or from the office?**
**Lior**: For me personally, at home, but I think it's more feasible to work from the office.
**Boaz: AWS GCP or Azure?**
**Lior**: Well, right now, absolutely GCP because we're all hands down into it. But it depends.
**Boaz: To DBT or not to DBT?**
**Lior**: To DBT. Absolutely, all the way.
**Boaz: Delta lake or not Delta lake?**
**Lior**: No.
**Boaz: YouTube or Vimeo?**
**Eldad**: YouTube Premium or Vimeo?
**Lior**: Oh that's a good question. It depends. If you're trying to monetize for ads, definitely Youtube. If you're a business caring about your brand mission and making sure you have clean video with no ads, then that's Vimeo!
**Boaz: That's the difference when we have executives on the podcast versus the hands-on people. If we interview an engineer at Vimeo, "of course Vimeo!"**
**Eldad**: Just true or false. There's no in between.
**Boaz: Okay. Very good. I think we're about to end. Anything else you want to dive into?**
**Eldad**: One quick thing. So, you've mentioned that you want to drive insight from Voice. People communicate and then you get tons of value out of kind of analyzing what multiple voices say on the call. Are you planning to make videos more interactive and embed more? Are you planning on making it less about broadcasting and more about getting some feedback from viewers so that it can actually drive some kind of a video analytics insight in the future? Is that planned? Anything you can share?
**Lior**: Short answer, yes. But I can't share too much.
**Eldad**: Nice! So, there's a big thing planned. Okay.
**Boaz: No comment, no comment. Lior, this has been super interesting. Thank you so much for joining our podcast. We that said we need to talk to you in a year. Please do not disappoint us, we want you to have all these milestones achieved with excellence. Thank you so much.**
# How we built Firebolt (/blog/how-we-built-firebolt)
## Introduction [#introduction]
To understand how we built Firebolt, we need to understand first what we built Firebolt for. We describe Firebolt as a "Cloud Data Warehouse for Data-Intensive Applications". Huh? What does it even mean?
Well, let's start with surveying the landscape of how people do analytics today. Today, we have a generation of modern Cloud Data Warehouses (CDW) such as Snowflake, BigQuery, Redshift, Databricks, etc. They are mature, versatile, battle-tested, and support a wide range of analytic workloads. Today's CDWs can handle BI, reporting, ad-hoc query and exploration, ELT, data science, ML, and, let's not forget AI. They deal with different types of data - structured, semi-structured, even unstructured, geo-spatial, time-series, and, more recently, vector embeddings. CDWs can handle data as batch, micro-batch, and streaming; they connect to a variety of systems - from OLTP to LOB, and, of course, integrate well with Data Lakes and Lakehouses. Amazing!
But there is one thing that the current generation of CDWs struggles with. They are not very fit for supporting data-intensive workloads which require low latency (tens of milliseconds) and high (thousands of) QPS. Workloads which are common in user-facing applications - like mobile apps or Web apps. Today, developers don't dare to connect directly to CDW from an iPhone app and have SQL queries against CDW power interactive experience. This simply won't work, or in rare cases where it might work - would be prohibitively expensive to run at scale. Again, today's CDWs excel at many, many things, but being the backend of interactive user-facing data-intensive applications - is not something they are good at.
So, what do application developers do today? They have to introduce another system for data serving that the application will work against. What system - really depends on the application access pattern. It could be a simple cache; if the application needs key lookup - Redis or Valkey is a good choice; if it needs text search - then Elastic fits the bill. For aggregation queries - Druid, Pinot, or Clickhouse will work. This serving layer could even be MySQL if the data volumes to power the app are small.
There are many specialized systems, each one being very good at what it is designed to do, and solutions employing a dedicated serving system generally work well. However, there are multiple problems with this setup. They all stem from the fact that the data now lives in two systems - one is CDW, where all analytics is done, and another one - the serving system which is exposed to the application. Having two systems is not too bad by itself, as enterprises routinely run dozens (if not more) of systems. However, this approach presents its own challenges:
* Data freshness is compromised, since that data needs to be copied from CDW to the service system.
* Each specialized system has its API, programming model, security model etc. It's not too hard to learn a new API - backend developers will pick it up quickly, but it's a different skill set than doing SQL and database development. And there are many more SQL developers in the world than backend developers.
* These systems all have their preferred data models that they work best with - so data needs to be prepared and molded into that shape. Again, this is not the hardest thing, but yet another set of pipelines to maintain and babysit.
So, we at Firebolt saw an opportunity here: Wouldn't it be wonderful if we could build a Cloud Data Warehouse which can handle these demanding data-intensive applications? The data won't need to leave such CDW, and it will handle both common CDW tasks as well as serve as a backend for workloads requiring low latency and high concurrency. It will have to be a real database system, meaning that it has all the properties expected from DBMS - real SQL (and not "SQL-like"), ACID transactions (and not "eventual consistency"), etc.
So here you have it - that's the Firebolt's vision in one sentence:
*Cloud Data Warehouse for Data-Intensive Applications*
Or, if you prefer in the poem form:
*Low latency*
*Data intensity*
*SQL simplicity*
*Cloud elasticity*
Nice vision and nice rhymes, but does the world need Firebolt? Who needs to expose their data to user-facing applications that require low latency and high concurrency? Or to rephrase it - why doesn't every enterprise build such applications on top of their data warehouses? Those are good questions, and we think that if it were easy to build such applications, every business and every enterprise would do it. Today, it's just too hard and too expensive. Well, Firebolt exists to make it easy!
## The challenge [#the-challenge]
Of course, building a **Cloud Data Warehouse for Data Intensive Applications** is not easy. The rest of this blog is going to give an insight into how we did it, where every decision was guided by the two simple principles:
* Our system has to be a real analytical DBMS. No compromises.
* The most important workloads are data-intensive applications. Optimize for them; everything else is secondary.
While building analytical DBMS, which works at petabyte scale with low latency and high concurrency, is difficult, there are some things which make it easier. The most important one is that application workloads, unlike ad-hoc queries, are very predictable and contain a limited number of query patterns. We fully take advantage of it by being able to set up indexes which fit the workload, utilize caching of sub-plan results, and automatically adjust query plans based on history. This will be covered in more detail below.
And we also want to be very clear that like always in engineering - building a system means making tradeoffs. In software (like in life), it's not possible to have a system which excels at everything. Getting excellent at A often comes at the expense of B.
And it is, of course, true about Firebolt. In this blog, we will show multiple examples of tradeoffs we took in order to comply with our two guiding principles. Some of these tradeoffs will even look counterintuitive, but, I promise, it will all make sense in the end.
One more question that might be nudging you - is it really practical to build a real user-facing application with SQL database as a backend? Answering queries in tens of milliseconds. Well - why don't we take as an example one of Firebolt customers:
## Example of real production workload [#example-of-real-production-workload]
To give an example of such a production workload, let's look at the latency chart of one of Firebolt's customers. It shows median and P95 values for latency in milliseconds - e.g. half the queries finish under 120 ms.
What's more interesting is that these are not simple queries. Let's take a look at a couple of them:
```sql
with O as (
select …
from D
where x IN (...) and y < … and z between … and …
union all
select *, …
from M
where x IN (...) and y < … and z between … and …
),
C as (
select …, coalesce(sum(case when … then … end), 0)
/ sum(case when … then … end) - 1
from O
group …
)
select …, sum(...), …
from O left join C on …
where … between … and …
group by …
```
This query has a couple of Common Table Expressions (CTEs), UNION ALL, aggregations and LEFT JOIN. The input tables D and M are tens to hundreds TBs in size (although WHERE clause filters are very selective).
Another query:
```sql
SELECT …,
COUNT(*),
ARRAY_COUNT_GLOBAL(...) / COUNT(*),
ARRAY_SUM_GLOBAL(TRANSFORM(x,y -> (y - x), …)) / ARRAY_COUNT_GLOBAL(...),
ARRAY_COUNT_GLOBAL(ARRAY_FILTER(x,y -> (x = y), …)) / ARRAY_COUNT_GLOBAL(...),
SUM(...) / ARRAY_COUNT_GLOBAL(...),
FROM (
SELECT …,
ARRAY_FILTER(x,i -> (i = 1), …),
FROM (
SELECT …,
ARRAY_FILTER(t,p -> (MATCH_ANY(..., ['(?:^|[^\\p{L}\\p{N}])(?i)(word)(?:[^\\p{L}\\p{N}]|$)']), …),
FROM D
WHERE a = … AND b NOT IN (...) AND …
)
) GROUP BY …
```
Here, we have aggregations and no JOINs, but a lot of array manipulations, including array aggregations and multiple array lambda functions, as well as the complex regular expression matching function.
These queries power the user-facing application, and with 125 ms median latency, and even P95 around 200+ milliseconds - it provides instantaneous answers to **user experience**.
## SQL API [#sql-api]
We set out to build a database system, and databases speak SQL. In some parallel universe, the term "speak SQL" would've been unambiguous, but in ours - "it is complicated". The SQL language has been [accepted by both ANSI and ISO](https://blog.ansi.org/sql-standard-iso-iec-9075-2023-ansi-x3-135/), since 1986, there were nine edition since (SQL-86, SQL-89, SQL-92, SQL:1999, SQL:2003, SQL:2006, SQL:2008, SQL:2011, SQL:2016, and most recently SQL:2023). Despite all this standardization - every single DBMS system has its own unique SQL dialect. Some are more aligned with ANSI SQL, some are less. We had to decide what Firebolt's SQL dialect would look like, and it was the most natural decision for us to align with Postgres's SQL: It is widely used, people love it, it works with virtually all ecosystem tools, and it is also probably the closest one to ANSI SQL. By aligning with Postgres on our SQL dialect, we achieve two goals:
1. **Ease of use:** Postgres is very popular, and many developers used it at some point and are familiar with its SQL dialect. So they would be immediately productive with Firebolt.
2. **Simplifying ecosystem adoption:** The success of the platform depends on the ecosystem around it. And virtually any application working against a database has support for Postgres. And if the SQL that these applications generate for the Postgres connector works without modifications in Firebolt - it becomes easy to have a Firebolt connector.
Choosing to align with the Postgres dialect is a popular choice in the industry - many other systems have done it as well: [CockroachDB](https://www.cockroachlabs.com/docs/stable/postgresql-compatibility), [DuckDB](https://duckdb.org/docs/sql/dialect/overview), and [Umbra](https://umbra-db.com/#features) are good examples for this.
## Both compliant and fast [#both-compliant-and-fast]
Choosing to be Postgres compliant does come with certain costs and tradeoffs. Particularly, being compliant, means that SQL functions should have exactly the same behavior, including error handling. Let's take the simplest example of the addition operator: x + y. This operator needs to throw an error if the result of addition overflows the result data type - this is both Postgres behavior and is prescribed by ANSI SQL. In real-world data, such overflows are unlikely, but the code has to be ready for them. It turns out that there is a non-trivial cost for doing such checks in a query engine which uses vectorized execution. So it seemed like we had a hard conflict between our two principles - we want to be a SQL compliant system, but we also don't want to sacrifice even a little bit of performance, as we need to power low latency workloads. Both requirements were equally important to us, so we invested engineering resources to satisfy both of those. Details about how it was done can be found [in this blog](https://www.firebolt.io/blog/making-a-query-engine-postgres-compliant-part-i-functions).
## Beyond Postgres [#beyond-postgres]
Postgres compliance is great, but modern data applications need many features beyond what it has to offer. Here, we will discuss one of them - support for arrays. Arrays (alongside with structs) are the two most important building blocks for semi-structured data support. And semi-structured data is ubiquitous in today's data world (thanks to JSON, but also protobufs, Thrift, Parquet, Arrow, and many similar formats). Postgres, being born in the world before that, implements arrays very faithfully to the ANSI SQL specification. Particularly, multidimensional arrays are expected to have same number of elements:
```shell
psql=> select '{{1},{2,3}}'::int[][];
ERROR: malformed array literal: "{{1},{2,3}}"
LINE 1: select '{{1},{2,3}}'::int array;
^
DETAIL: Multidimensional arrays must have sub-arrays with matching dimensions.
```
This is ANSI SQL compliant behavior, but it will be breaking pretty much any modern application. So Firebolt adopted a more flexible and relaxed definition for multidimensional arrays:
```shell
fb=> select '{{1},{2,3}}'::int[][];
?column?
-------------
{{1},{2,3}}
```
Moreover, the ANSI SQL's (and therefore Postgres') way of working with arrays - is to convert them into a relation using a lateral join with UNNEST, and rely on correlated subqueries. This is a very general approach, which allows to do any operation with an array that can be done with a relation using the full power of relational algebra (through SQL, but still). However, it is both cumbersome to program, and query optimizers struggle with efficient query plans on many types of correlated subqueries. Let's take a simple example of filtering out array elements. Suppose we only want to leave even numbers. The Postgres's version of SQL would look the following:
```shell
psql=> select a, (select array_agg(e) from unnest(a) e where e % 2 = 0) f from
(select array [1,2,3,4,5,6] a) t;
a | f
---------------+---------
{1,2,3,4,5,6} | {2,4,6}
```
There is a lot going on here for such a simple task - unnest, correlated subquery with aggregation to reconstruct an array after filtering - ouch.
Firebolt recognizes that if arrays are common, then operations with arrays are common too. So Firebolt has a wide arsenal of so-called "array lambda" functions - functions which operate on an array, and one of the arguments is a lambda function telling what operation needs to be performed on each element.
The above query can be rewritten in Firebolt as:
```shell
fb=> select a, array_filter(x -> x % 2 = 0, a) f from
(select array [1,2,3,4,5,6] a) t;
a | f
---------------+---------
{1,2,3,4,5,6} | {2,4,6}
```
The `x -> x % 2 = 0` is a lambda function here. OK, I know what you are thinking - what about [Codd's theorem](https://en.wikipedia.org/wiki/Codd%27s_theorem) which establishes an equivalence between relational calculus and first order logic, and aren't we escaping the boundaries of the relational model by introducing lambda functions which look like second order logic? Don't worry - everything's still fine; we are still within the relational model. There is proof, but the margins of this blog are too small for it.
But jokes aside, it is much simpler for the optimizer to reason about scalar functions (even if they operate on arrays and lambdas) than about correlated subqueries and unnest, and it can generate better query plans. Our customers use these a lot - in fact, query 2 from "Example of real production workload" was very lambda functions heavy. As a bonus, array functions (including lambda ones) are allowed in Aggregating Index definition, allowing Firebolt to accelerate them.
## Planner: Learn from history [#planner-learn-from-history]
We already discussed that data application workloads tend to be very homogeneous and repetitive. Since the same query patterns repeat over and over again, it makes a lot of sense to learn from the past execution in order to build better query plans for similar queries in the future. This is what we refer to as History Based Optimization (aka HBO). Many HBO techniques have been proposed - ISOMER \[SHMMKT06], STHoles \[BCG01], simplified ISOMER \[BMHM07] and more recently QuickSel \[PZM20].
Of course, Firebolt's planner also uses traditional data statistics, but given our primary focus on predictable and repetitive workloads - we have invested most of our efforts in HBO. The diagram below shows the overall HBO architecture.
To make it work, we first normalize query plans. The goal of normalization is to let queries sharing the same semantics but differing syntactically use the same plan. The normalization process includes but is not limited to
* Remove projections so column pruning and ordering do not matter.
* Join graphs are normalized so join ordering does not matter.
* Filters are sorted so that the order of predicates does not matter. But note that concrete literals do matter.
* Order of window functions applied on the same window frame does not matter.
We fingerprint every subtree in the plan (not just the root), and attach observed execution metrics to it, e.g. row counts. When a new query arrives, the planner tries to match its plan subtrees with the ones learned from the history service and uses historical statistics in the estimation process.
For queries where exact matches are not found, but the plan shape matches (for example, if literal values in the new query are different), we use a variation of QuickSel. A simple example would be the filter operator. Suppose we saw queries with filter predicates `x > 10` and `x > 50`, and learned statistics for them. A new query now comes with `x > 20`. The QuickSel algorithm allows us to estimate cardinality for this unseen predicate. To do this estimate, QuickSel models relations as k-dimensional spaces, and it models query filters as hyper-rectangles in these spaces, where k is the number of relation attributes.
Learning from history is very important for Firebolt main scenarios, and investing in more sophisticated HBO is a priority for our planner team. The tradeoff here is that focusing on HBO means less investment in more traditional CBO (cost-based optimization) and fancier data statistics.
## Runtime: Smaller rather than faster [#runtime-smaller-rather-than-faster]
We already discussed that our production workloads tend to be very homogeneous, with 10s or 100s of predictable query patterns. Such repetitive workloads can benefit tremendously from reuse / caching. In analytics systems, caching as a concept is ubiquitous: from buffer pools to full result caching to materialized views. Firebolt employs a surprisingly little-used approach: caching subresults of operators. The idea itself is not new, with first publications appearing in the '80s, e.g., [\[Fin82\]](https://dl.acm.org/doi/10.1145/582353.582400), [\[Sel88\]](https://www.sciencedirect.com/science/article/abs/pii/0306437988900142). More recent results include [\[IKNG10\]](https://dl.acm.org/doi/10.1145/1559845.1559879), [\[Nag10\]](https://homepages.cwi.nl/~boncz/msc/2010-FabianNagel.pdf), [\[HBBGN12\]](https://vldb.org/pvldb/vol5/p1436_alexanderhall_vldb2012.pdf), [\[DBCK17\]](https://cs.brown.edu/~kayhan/papers/hashstash.pdf), and [\[RHPV et al. 24\]](https://www.amazon.science/publications/why-tpc-is-not-enough-an-analysis-of-the-amazon-redshift-fleet). It also would be appealing to use HBO (history based optimization, discussed above) to help detect common subplans across consecutive queries.
This whole subject is fascinating, and we have a dedicated blog [going into depth on it](https://www.firebolt.io/blog/caching-reuse-of-subresults-across-queries), but here we want to focus on something else.
Since caching is so important for Firebolt's main focus - data-intensive applications - obviously the more we are able to cache, the better. The amount of RAM in the engines is fixed, so the only variable we can control is: using less memory for the cached data structures. And this is exactly how we have designed our runtime data structures. Let us take the one used by our Hash Join implementation as an example. The hash tables built for the Hash Join are great candidates for caching, as there could be many queries with different left sides of the join, but the same right side of the join (e.g., when joining against a dimension or lookup table) - if the right side is chosen as **build** side of join, it will be the one for which hash table is built.
Usually, designing for performance means coming up with the algorithm with the lowest run time. However, since our priority was to squeeze "more data structures" into the fixed sized memory, for building JOIN hash tables, we did not optimize to have the necessarily fastest algorithm, but aimed for an algorithm which produces smaller hash tables. The details will be described in a dedicated follow-up blog, but the general idea is to make multiple passes over the right side. In particular, we compute statistics so both join keys and values can be packed into as few bytes as possible. Additionally, we even sort by hash values, in order to be able to do delta compression on offsets. Yes, all this may take more time, but if we build this hash table once and use it many many many times, and it takes less memory, there is less of a chance for it to be evicted - i.e., it is worth it!
## Distributed execution: Shuffle dilemma [#distributed-execution-shuffle-dilemma]
Firebolt supports multiple dimensions for scaling the processing power of a cluster. One is scale up, where the user simply selects a more powerful node type. However, changes in node type are somewhat coarse grained. If more precise control over sizing is needed, or if there is a need to have more processing power than what a single node can offer - there is a scale out dimension. With scale out, the user can increase the number of nodes in the cluster (and, of course, can also scale up or down the node types within that cluster). Logical query plans for single node and multi-node clusters are the same, but physical plans are different, as multi-node clusters need to add dedicated operators for data exchange between nodes. This data-exchange operator is called Shuffle.
Here is an example query, computing a join and an aggregation:
```sql
SELECT r.c, SUM(s.d)
FROM r JOIN s
ON r.a = s.b
GROUP BY r.c
```
The logical query plans are the same for both a single-node and a multi-node cluster:
```shell
[1] [Aggregate] GroupBy: ["c"] Aggregates: [sum("d")]
\_[2] [Projection] "c", "d"
\_[3] [Join] Mode: Inner [("a" = "b")]
\_[4] [StoredTable] Name: "r", used 2/4 column(s): "a", "c"
\_[5] [StoredTable] Name: "s", used 2/4 column(s): "b", "d"
```
The physical plan for a single-node cluster can execute the entire query in a single stage:
```shell
\_[1] [Projection] ref_0, ref_1
\_[2] [Aggregate] GroupBy: ["c"] Aggregates: [sum("d")]
\_[3] [Projection] "c", "d"
\_[4] [Join] Mode: Inner [("a" = "b")]
\_[5] [StoredTable] Name: "r", used 2/4 column(s): "a", "c"
\_[6] [StoredTable] Name: "s", used 2/4 column(s): "b", "d"
```
In contrast, the physical plan for a multi-node cluster splits query execution into multiple stages which are connected by shuffle operators:
```shell
[1] [AggregateMerge] GroupBy: ["c"] Aggregates: [summerge("d")]
\_[2] [Shuffle] Hash by ["c"]
\_[3] [AggregateState partial] GroupBy: ["c"] Aggregates: [sum("d")]
\_[4] [Projection] "c", "d"
\_[5] [Join] Mode: Inner [("a" = "b")]
\_[6] [Shuffle] Hash by ["a"]
\_[7] [StoredTable] Name: "r", used 2/4 column(s): "a", "c"
\_[8] [Shuffle] Hash by ["b"]
\_[9] [StoredTable] Name: "s", used 2/4 column(s): "b", "d"
```
The Shuffle operator is the fundamental building block for distributed query processing, and there are many possible designs for it - e.g. \[HBEAM18], \[SAJHJ19], \[MYC20], having different strengths and tradeoffs. Usual considerations are how to make shuffle very scalable and resilient to failures. This makes sense in the context of huge batch jobs, which process petabytes, use hundreds or thousands of machines, and run for hours. These, however, are not the main scenarios for Firebolt - we are focused on data-intensive applications, which need to process queries at low latency. So, we made different tradeoffs - instead of prioritizing scalability and resilience, we prioritized latency and network utilization.
One example of where this tradeoff affects design is - whether shuffle should be pipelined and streaming across stages, overlapping their execution, or whether it should be checkpointing state. The benefit of checkpointing state is that in case of failure, execution can restart from the latest known checkpoint. The drawback is that such checkpoints introduce additional pipeline breakers, and also require additional disk or network I/O to actually write the checkpoints. Since Firebolt optimizes for fast queries, we were less worried about the possibility of entire node failure in the middle of query execution, and therefore chose a pipelined and streaming shuffle. In the unlikely event of node failure during query execution, we can simply restart the respective query from the beginning (with mechanisms in place to preserve idempotence of such restarts, i.e. we would never have INSERT adding same data twice). Note, that our shuffle is resilient against transient failures such as network connection drops.
By being able to overlap different stages of distributed execution, we achieve much lower query latencies, since we can usually hide network I/O in one stage behind processing work in another stage. Again, this strategy works very well for user facing low latency queries, and is not optimal for long running ELT batch jobs - but user facing low latency queries are what Firebolt optimizes for.
When pipelining different operators, it is important not to make writing and reading to shuffle a bottleneck, or the entire pipeline will stall on it. Since most SQL operators are CPU or memory bound, we also needed to make sure that shuffle doesn't use too much CPU or memory while trying to utilize all available network bandwidth. We were able to achieve this by doing all network I/O fully asynchronously using the [io\_uring kernel interface](https://unixism.net/loti/index.html). But it turns out that even with io\_uring, it is not trivial to saturate network bandwidth, when AWS gives you up to 200 Gbit/second. For example, AWS imposes limits on a single TCP connection flow, throttling it at 5-10 Gbit/second, so in order to get higher throughput, we are using dozens of TCP connections between each pair of nodes. Of course, since we are no longer using single TCP connection, we needed to build flow control to maintain ordering of packets over multiple TCP connections. The shuffle code became more complex, but we were able to get very close to full network bandwidth - so for us this tradeoff was worthy.
Another consideration is that 200 Gbit/second is getting close to the memory bandwidth, so we had to be very careful not to do extra copies of data in kernel- or user-space. Finally, shuffle flow control has to include backpressure mechanism to prevent excessive buffering of data on either sender or receiver side.
## DML: Automatic incremental index maintenance [#dml-automatic-incremental-index-maintenance]
The classic definition of Data Warehouse was given by [Bill Inmon](https://en.wikipedia.org/wiki/Bill_Inmon) in his "Building Data Warehouse" book \[Inmon02]
"*A data warehouse is a subject-oriented, integrated, nonvolatile, and time-variant collection of data in support of management's decisions.*" \[Inmon]
The book goes on to explain that "nonvolatile" means that data is loaded into Data Warehouse, but not updated, keeping the history of changes. So one would think that in the perfect world, the only DML that Data Warehouse needs is INSERT. OK, and maybe some simple form of DELETE to remove outdated data. But certainly not UPDATE, right ? Well, our world is not perfect, and there are many use cases that require doing mutations in Data Warehouse. Upstream systems need to issue corrections, laws such as GDPR and "Right to be forgotten" require deleting individual records, and sometimes people get carried away and build applications which treat Data Warehouse more like Operational Data Store (ODS) or even a system of record.
Well, we are meeting our customers where they are - if they need UPDATE/DELETEs - we got them. And just like everything else in Firebolt, we always look at these SQL statements in the context of data-intensive applications. For one, it means that these statements themselves should be fast, especially if they update a small number of records. The technique for doing such small deletes is well known - soft deletes, i.e. keeping an auxiliary data structure which remembers which rows were deleted, it is sometimes called [deletion vector](https://delta.io/blog/2023-07-05-deletion-vectors/), and apply it during reads. Firebolt uses [Roaring Bitmaps](https://roaringbitmap.org/) as deletion vector implementation. Roaring Bitmaps is very compressible, yet fast to access data structure.
Below is an example of production workload in Firebolt, where application issues DELETE statements, each deleting a single value from a big table. The charts below show QPS of about 6 per second, and average latency for these deletes is 140 milliseconds (which includes uploading deletion vectors to S3 for durability).
But while performance of DML is important, what is much more important to Firebolt - is performance of reads in the presence of writes. Firebolt is there to support low latency high concurrency data intensive applications, and the latency should remain low even in the presence of updates.
For example, one technique which enables low latency query serving in Firebolt are [Aggregating Indexes](https://docs.firebolt.io/godocs/Guides/working-with-indexes/using-aggregating-indexes.html). Conceptually they are similar to Materialized Views, but with the restriction that they only support a single GROUP BY operator, but with arbitrary aggregate functions. Since Aggregating Indexes contain precomputed results of GROUP BY, query planner can automatically rewrite queries matching (exact or subset) of GROUP BY keys and aggregate functions to redirect them to the Aggregating Index instead of base table.
What should happen with precomputed results during DML - especially for UPDATE and DELETE? It is a tradeoff between write and read latency. Some systems don't update Materialized Views for every DML, and instead recompute them lazily in the background. But we cannot afford doing this in production workloads, because then once DML transaction commits, the subsequent reads will suffer performance drop. So we update aggregating indexes in the same transaction as DML, and we do it incrementally. There is extensive research literature about incremental maintenance of materialized views, particularly the ones containing aggregate functions - \[GMS93], \[GM99], \[DQ96], \[MQM97]. These papers are kind of complex to read, but the idea is relatively simple. Incremental update is straightforward to do for "good" functions like SUM. We can compute SUM (optionally per group) for all values in the transaction. For INSERT we simply add it as a new delta, and for DELETE we also add it, but as a negative delta. It is more tricky with more difficult functions, like MAX. It can still be added in INSERT, and we can quickly calculate MAX between the existing values and the delta. But what to do in DELETE - how do we know if we removed the maximum value per group (maybe there are ties), and what the new value would be ? It is possible to mark affected groups as invalid, but it means that they will have to be recalculated for every subsequent SELECT. In the spirit of doing more work at write to optimize reads, instead we selectively recompute values in the Aggregating Index, by natively deleting the old group by entries, and inserting a fresh values from the origin table after filtering the deleted values, but only for groups affected by the DELETE.
## Transaction manager: ACID at no cost [#transaction-manager-acid-at-no-cost]
Finally, to be a DBMS, Firebolt needed to support transactions. This is something we couldn't compromise on. Firebolt had to have real ACID transactions, not eventual consistency, not "almost transactional" - but true ACID. There are many ways to build Transaction Manager, but for it to fit the Firebolt mission, we put together the following list of requirements:
* 100% ACID
* Within distributed environment, across multiple compute clusters
* With storage compute separation, where any compute cluster can access any data
* With full support for multiple writers, any node of any compute cluster can modify any data
* Immediate consistency: Once a transaction commits, changes should be immediately visible to all other nodes in all other compute clusters
* Strong Serializable Isolation Level for all metadata operations (DDL, DCL)
* Snapshot Isolation Level for data operations (DML and DQL)
* At scale of analytics, where a single transaction can modify hundreds of terabytes
* But also allow low latency "trickle" DML, i.e. DML updating only a handful of rows few times a second
* Optimize for workloads dominated by reads, with writes being a small fraction of the queries
* Allow long running multi-statement transactions
* Enable transactions to span across multiple databases
And if that was not difficult enough, here is the kicker:
* All the above without performance impact on low latency high concurrency data intensive application workloads
To illustrate the last point - the fastest query that one of our customers is running in production to power their application takes 5 milliseconds. So if the transaction manager's overhead is only 1 millisecond - which would be fantastic achievement given the above requirements, that would still be 20% slowdown, so we strive to reduce transaction manager overhead to zero.
Designing such a system is very challenging, and we are not aware of precedents neither in industry nor in research. One tradeoff that we decided to do very early was to prioritize the "happy" path of the read-only DQL SELECT query. Given that analytical workloads have many more read queries than write queries, we wanted to perform best if there were no changes since the previous query, and we could use all the cached metadata - definition of the schema objects - tables, views, indexes etc; snapshot versions of the data (i.e. list of tablets per table), statistics etc. All we needed was one RPC call to the transaction manager at the start of a read-only transaction to verify that indeed nothing changed, and that cached snapshots are still valid in the new transaction.
Since we were optimizing for read-mostly workloads, but with writes that had to be consistent across all the compute clusters, we decided to persist transaction log (aka Write Ahead Log or WAL) in [FoundationDB](https://www.foundationdb.org/). FoundationDB is a proven datastore which gave us production readiness, robustness, scalability, fault tolerance and transactional semantics. On top of that, it offers extremely low latency even at the P99 tail. So in a sense, we delegated the most difficult part of transaction management to FoundationDB.
In order to be able to quickly interpret whether changes in WAL impact the current query, we built indexes over WAL which are used by the most common requests to Transaction Manager, including one which checks if anything changed. Not only that, these indexes allow to easily compute a delta of changes, and replay only the relevant portion of the log. Having WAL indexes has the drawback of slowing down transaction commits which need to build these indexes, but it was acceptable to us, since we were optimizing for reads.
Other techniques include having asynchronous notifications about WAL changes post commit, so all compute nodes have a chance to see them and update their snapshot (and possibly prefetch data as well) before a user sends the next query.
We also invested heavily in network topology of our deployment both for Transaction Manager and FoundationDB, to ensure that within a regional deployment, we cross availability zone boundaries at most once.
So coming back to the "happy" path of the query: A compute node that handles a user query needs just one RPC to Transaction Manager to check that nothing has changed, and Transaction Manager uses WAL indexes and reads few FDB keys in parallel to confirm that nothing indeed changed. We target 1 millisecond for that interaction. But even 1 ms could be too much for low latency queries. We plan to optimistically start the query execution in the hope that the cache is up to date. In the unlucky case when it's not - the query execution would need to be canceled and restarted with an updated version of the metadata. This part is still work in progress.
Since Transaction Manager stores all of its state in FoundationDB, it is a stateless service, and scales easily. (Of course, FoundationDB being a stateful service also needs to scale well - but FDB is a very mature technology, and it does scale). We have put our Transaction Manager through stress testing, loading it with tens of thousands of QPS both read and write transactions, and it holds up well.
## Conclusion [#conclusion]
In this blog we started with answering the question - what did we build Firebolt for, and why do we think the world needs Cloud Data Warehouse for Data Intensive Applications. Once we articulated our vision, it became very clear what our focus and engineering priorities should be in realizing this vision. We defined our two guiding principles - building a no-compromise analytical DBMS and making it perform the best for interactive, user-facing, low-latency, high-concurrency applications. These principles were guiding us in designing and implementing all components of our system. Like always in engineering, we had to make tradeoffs, and the above principles helped us stay focused on our vision, and make tradeoffs that fit the purpose of our system.
The blog gave a handful of examples of how we navigated these tradeoffs in different components of Firebolt - from the API frontend, to query planner, runtime, distributed execution, storage, and transaction manager. And while we didn't cover many other parts of our system - cloud infrastructure, compute, client SDKs, connectors, security manager, metadata service, control plane etc - they all followed the same principles and decision making process.
We believe that being laser focused on our vision, helped us to build a great product which excels at what it is designed to do. But ultimately - it is up to you - our customers - to judge!
### References [#references]
\[SHMMKT06] U. Srivastava, P. J. Haas, V. Markl, N. Megiddo, M. Kutsch, and T. M. Tran. ISOMER: Consistent Histogram Construction Using Query Feedback. In ICDE, 2006.
\[BCG01] N. Bruno, S. Chaudhuri, and L. Gravano. STHoles: a multidimensional workload-aware histogram. In SIGMOD, 2001.
\[BMHM07] A. Behm, V. Markl, P. Haas, and K. Murthy. Integrating Query-Feedback Based Statistics into Informix Dynamic Server. In: Datenbanksysteme in Business, Technologie und Web (BTW), 2007.
\[PZM20] Y. Park, S. Zhong, and B. Mozafari. QuickSel: Quick Selectivity Learning with Mixture Models. In SIGMOD, 2020.
\[HBEAM18] Zhang, Haoyu and Cho, Brian and Seyfe, Ergin and Ching, Avery and Freedman, Michael J. Riffle: optimized shuffle service for large-scale data analytics. In ACM 2018. [https://doi.org/10.1145/3190508.3190534](https://doi.org/10.1145/3190508.3190534)
\[SAJHJ19] Qiao, Shi and Nicoara, Adrian and Sun, Jin and Friedman, Marc and Patel, Hiren and Ekanayake, Jaliya. Hyper dimension shuffle: efficient data repartition at petabyte scale in SCOPE, In VLDB 2019, [https://dl.acm.org/doi/abs/10.14778/3339490.3339495](https://dl.acm.org/doi/abs/10.14778/3339490.3339495)
\[MYC20] Shen, Min and Zhou, Ye and Singh, Chandni. Magnet: push-based shuffle service for large-scale data processing. In VLDB 2020. [https://dl.acm.org/doi/abs/10.14778/3415478.3415558](https://dl.acm.org/doi/abs/10.14778/3415478.3415558)
\[Inmon02] Inmon W\.H. Building the Data Warehouse, 3rd edn. Wiley, New York, 2002
\[GMS93] A. Gupta, I. S. Mumick, and V. S. Subrahmanian, "Maintaining views incrementally," In Proceedings of ACM SIGMOD Conference, pp. 157-166, 1993.
\[GM99] H. Gupta and IS. Mumick, "Incremental maintenance of aggregate and outerjoin," Technical Report, Stanford University, 1999. [https://www3.cs.stonybrook.edu/\~hgupta/ps/aggr-is.pdf](https://www3.cs.stonybrook.edu/~hgupta/ps/aggr-is.pdf)
\[DQ96] Maintenance Expressions for Views with Aggregation Dallan Quass Stanford University, [http://ilpubs.stanford.edu:8090/183/1/1996-54.pdf](http://ilpubs.stanford.edu:8090/183/1/1996-54.pdf)
\[MQM97] I. S. Mumick, D. Quass, and B. S. Mumick, "Maintenance of Data Cubes and Summary Tables in a Warehouse," In Proceedings of ACM SIGMOD Conference, 1997. [https://dl.acm.org/doi/pdf/10.1145/253260.253277](https://dl.acm.org/doi/pdf/10.1145/253260.253277)
# How Zendesk engineers manage customer-facing data applications (/blog/how-zendesk-engineers-manage-customer-facing-data-applications)
This time on the data engineering show, Eldad abandoned his brother Boaz but it's ok because Boaz got the full 30 minutes to talk to one of the most interesting people in the data space.
Ananth Packkildurai is Principal Software Engineer at Zendesk and runs one of the strongest newsletters in data – Data Engineering Weekly. He talked about data applications at Zendesk and how they're built, technologies that excite him like data lineage and data catalog, and the best routes for software engineers to get their hands dirty in the data world.
INTERVIEWER: Boaz Farkash.
ZENDESK GUEST: Ananth Packkildura - Principal Software Engineer.
Listen on [Spotify](https://open.spotify.com/episode/3VdCM7JyQWDtkvRr0TagVw?si=K9c4Nd5dRAKD0Im9YRgh7A) or [Apple Podcasts](https://podcasts.apple.com/us/podcast/how-zendesk-engineers-manage-customer-facing-data-applications/id1561927688?i=1000551399658)
**Boaz:** Welcome everybody! Thank you for joining us in another episode of the Data Engineering Show. Today, sitting next to me is nobody. Eldad, my co-host and brother, disappointed me. He could not make it today, and he trusted me. Ananth is here with us. Help me welcome Ananth Packkildura, who is a Principal Software Engineer at Zendesk, used to work for 4 years prior at Slack. Has a bunch of very interesting stories for us today, from both. Ananth is also somewhat of a veteran in the industry, has been in software for a long time, and has moved to data for many years. I would like to talk about that as well. But, Ananth is also a mini-celebrity in the data space. He runs the data engineering weekly, newsletter. Tell us about that. How many subscribers do you have today?
**Ananth:** Okay. Great! First-of-all, thank you so much for having me. It is really amazing to talk about data all the time.
**Boaz:** Thank you for joining.
**Ananth:** Subscribing, I think, we just crossed over 1700 or so, I think we are around that mark.
**Boaz:** Nice. If you are not registered to Ananth's newsletter, the data engineering weekly, you have to. If you are in data, you have to. It is a must-read. I follow it. Everything you need to know is in there, so look it up. I mean you are by yourself or do you have a lot of people involved there? How does it work?
**Ananth:** Thank you so much for the kind words, first of all. No, it is all on my own. I started, the background of the story, why I started? Towards the end of my time at Slack, I started working in our observability monitoring infrastructure, essentially applying data principles into the monitoring stack. I started to kind of feel of missing out and there was a pretty good data engineering weekly newsletter before that and got stopped and that was my first go-to source of information before everything, and that is how my morning starts. I had a special alert to read that newsletter and then, I also started to feel that I am going to miss out on something, so I started like, okay! I am going to read something and I just started writing. So, it is out of my own learning purpose, and I still consider this as a good way to learn. I think I would encourage everyone to write most of what they learn and that is how you are structurally learning by yourself. So, it is really, really amazing. Anyone listening, please start your own data engineering newsletter.
**Boaz:** But I am sure it has been getting tougher and tougher because there is much more content to create from out there. I think it is time you find help with that newsletter. It is too much on your shoulder.
**Ananth:** Yeah, that is true. Last week, I was going to curate. I had at least 18 articles to curate and shortlisted. Now, I do not have time to write all those things, shortlisted 10 items or 9 items to do that, and if anyone viewing that we already have data engineering weekly, GitHub link, that, anyone read some article, they like it, you can create a pull request and then we can add that as part of the newsletter in this case. So, there is also a way for more community contribution to the data engineering newsletter.
**Boaz:** Nice. Awesome! Ananth, I would love to hear about your personal career. You started from software and over time, by me looking at your history, it seems that data crept in slowly, slowly, until eventually, you could say, took over. So walk us through that. So when did data become such an important part of your career? What was the tipping point?
**Ananth:** Yeah, I think I started my career as backend engineering, mostly writing code on Java and other stuff like that and it is funny that when I was starting a career because data warehouses were not that much kind of winded off. In most organizations, there is very less visibility on those things, and it is often viewed as you will be using some tools, maybe SSIS package or any of the tools, I will never go to the that saving my time, but that also is a thought beginning at the career. I would say I got carried away and pulled into this whole big data wave or the buzzwords kind of thing, so I started looking at developing systems in Hadoop as part of experimentation in one of my works. I started looking at Hadoop 0.2 at that time or something like that. It was kind of a very early stage and that is the first time I am actually reading a lot about not producing better than when I kind of realized that most of the problems can be solved if you started to think not to produce patterns and most of the analytics problems. So I think that is kind of a very pulling point, and at that point still, the work is more of a backend engineering because you still have to write a bunch of Java code in order to kind of build any kind of analytical ecosystem at the time. I think I just jumped onto the bandwagon of that hype and then, slowly travelled and realized, okay! This is what we are going back to, where I just kind of reinvented the wheel, and then I actually went back and learned about the data warehouse concept and I took a course from the Kimble group. I think that was the last training program he took, so it is very fascinating to learn. Then, I go back and then learn the basics, and then you just continue on those cases.
**Boaz:** I think it is interesting, you mentioned that SSIS part and if we are to be completely honest, I think in the past, software engineers used to, to some extent, even look down at data technologists, thinking it is a different position. That is not for us, you deal with your data warehouse. Whereas now, it has become such a big deal, and most of our engineers are actually looking to be more involved with data technologies. It has become one of the most interesting parts of software engineering in general. Thanks for sharing!
Let us start with Zendesk. You have been there for a year and a half. What did you come to do there? or what do you do at Zendesk?
**Ananth:** Yeah, Zendesk is a very interesting company, right now, as a leader of customer experience platforms, and most of the support platforms essentially. What I am trying to do right now, is involve our customer-facing analytical platform, which is, Zendesk as a company has grown by acquiring more companies. It is like a disparate system involved, but what our clients see or want to see is a unified view of how the customer interacts with their sales and then as a support secret system. Also, not only do they want to view what is the customer experience, but they also want to see more deeper level. My product catalog is there. I wanted to send it to them and tell me which product is doing good or not. They want more than a simple support ticket analytics in this case. So my primary goal - I started in Zendesk to focus on building a data platform and analytics solution to address that particular problem in this case, and as with any data infrastructure, when you start to solve the last mile problem, you realize the problem is actually existing ahead of the problem, like, How do we make sure that we instrument data properly? How can we enable scalable analytics systems on top of it? Right now, I am playing a bridge role to make sure that we are gathering sufficient data, more domain ownership around, and then building scalable solutions.
**Boaz:** Zendesk has been around for many years. I, myself, have been a client of Zendesk for years. For a company that has been around for years, I wonder how does the data stack looks like? How much is legacy versus modernized? How do you go about modernizing a stack and how has it evolved throughout the years?
**Ananth:** Yeah, to my surprise. I would say that Zendesk has some kind of a legacy. We have an analytical system on Mongo DB that is actually still so. There is some system there, but surprisingly most of the part, it is pretty much up-to-date technologies. We have some systems using Google BigQuery. All our enterprise analytics is running on Google BigQuery and then cloud storage in this case and we recently started to adopt Apache Hoodie, which is kind of building this use case.
**Boaz:** Are you guys running multi-cloud or is everything on GCP?
**Ananth:** Yeah! So our enterprise analytics is actually running on Google cloud. Our product analytics running on AWS, various different companies Zendesk had products at some point of a time. That we could call it a legacy. We have two different cloud services for sure.
**Boaz:** Walk us through some of the more challenging use cases, workloads that are currently in place that you are involved with? How far long is that new product you described?
**Ananth:** For unifying the system.
**Boaz:** Yes.
**Ananth:** We are just barely scratching the surface. The way I am looking at this problem, not necessarily from unify the cloud services perspective, but how do we do data sharing between these two different disparate services, I think our challenge or compliant from all stakeholders is not necessarily we are running two different cloud services because two systems are atomic in nature, and then they just doing a fine job. I think the problem is at - if I do the data sharing from AWS to Google cloud, Is there any context that we are missing? Is there any lineage that we are missing? How does the consumer on the other side trust whatever you are sending? Right? because the business logic exists in one part and you just send the derived data set to another cloud and the context is missing in the middle range. So we kick-started this whole data lineage, data catalog project trying to possibly build the full bridge, establishing the full context of it, and trying to give more and more understanding to the consumer side of it to figure it out, and I hope at some point of a time, we will be able to merge these cloud services, but that is like a large project one to take.
**Boaz:** That was my next question. Has consolidating into one cloud been on the table?
**Ananth:** No, not now, at least. It works, but I think adding additional context right now will give us much more visibility to what is going on.
**Boaz:** Tell us how the data-related teams are structured at Zendesk? What kind of teams are there? How big are they? How are the responsibilities split?
**Ananth:** Yeah, that's a good question. I think there is a common pattern I started to see even in Slack and then Zendesk, it is like, we have this product analytics team that is focusing on instrumenting data from our product usages, collecting the data, and then building those, mostly they end up using kind of a Lakehouse architecture. And there is a whole bunch of business operation analytics, sales analytics, and marketing analytics, and these analytics teams have self-contained data engineers and data platforms inside to support the business operational aspect of it. Other teams focus, most of the cases, on the SAS application where you have to deliver customer-facing analytics on top of it. So that requires more and more coding kinds of things other than the SQL kind of workload. So they have a separate team. Zendesk data teams are also organized in these 3 different business orientation aspects of it.
**Boaz:** What data volumes are you guys dealing with?
**Ananth:** I do not have a top of mind, but I think we all log everything all the time. So I do not have a very finite number because it is not like one stream of data that we have been consuming.
**Boaz:** Yeah! Let us say customer-facing stuff, for example. How is that managed? Are these dev teams at the end with customized UI running the show, applying it into APIs, to run queries? or Is it more of an embedded analytics solution? What is going on there on the customer-facing workloads?
**Ananth:** The way we look at customer-facing analytics, it is more of an application. It is kind of a product on its own. We have a product manager to see how fine at SLA, which will be applied there. We source information from our customers, and then we can build those data pipelines over that period of time. That is a good question. Largely, the customer-facing analytics encompasses more of a backend engineering rather than a pure data engineering perspective. I am just slowly introducing to them the data pipelining concept, but very backend engineering focus in this case.
**Boaz:** What is the query engine behind the scenes?
**Ananth:** It is interesting, we right now have two query engines, quick databases, like we are running Redshift and then Postgres, for some historical reason that we keep running on those cases. There is a large project that is going on right now, to kind of unify the data store both in real-time and batch infrastructure to make it more customer-facing analytics. So we are looking at that particular solution.
**Boaz:** Yeah. I think trying to combine historical and real-time is something humans cannot stop trying to question. What is the solution? Any selected approaches already?
**Ananth:** I mean, we are looking at various solutions right now, potential contenders like Druid or ClickHouse or Pinot is one of the exciting ones.
**Boaz:** Which is the last one?
**Ananth:** Pinot.
**Boaz:** Pinot, yeah
**Ananth:** It is kind of a really exciting one. It is a question of right now, what we are doing in the real-time, we do join on the streamside a lot. So I am debating back and forth, join in the stream versus join in the database. I like joining database now because that is what the database is supposed to do. We can always do optimization on the stream. Like why do you want it to wheel on your stream processing? enrichment, yes! But do you need to build up a full flown join system? So those are the interesting concepts that we are exploring right now and dignify our system. Hopefully, we will do something.
**Boaz:** As you mentioned you are looking at Pinot as well. I ran into PC road from your Slack days about Pinot, you adopted that over there as well. Was it over Druid or something? Tell us about that project back then.
**Ananth:** Yeah, Druid vs Pinot. These are some of the things that might have been not relevant because every system always improves at any point in time. I think most of the real-time systems coming out of the use case were doing an ad serving engine or having ad serving capabilities. I think we were kind of an ad engine solution, and ClickHouse to an extent the same thing. The nature of the system is mostly immutable in nature. Even though, always immutable in nature, there is no upsert operation that you wanted to do. So you just see the event and you are just continuously running some time series over a period of time. I think that kind of solution is going to work really well, but in companies like Zendesk, companies like Slack where the SAS application fundamentally tries to solve the workflow over a period of time, right? So tickets have been created. Tickets have been deleted and the ticket can go through its full lifecycle, and we are trying to capture an object and we are trying to produce insight for an object lifecycle. So, we wanted to produce the current state of the object and then we also wanted to kind of do a historical view of how the data transformation goes. So when we wanted to preserve the current state of an object, that is where the upsert operation became absolutely crucial, and Pinot does support upsert operation and work reasonably well, but it is still kind of an afterthought, because, Pinot added upsert operations to solve the use case for Uber eats, which is a similar business process engine. I think maybe that is why we are not able to bridge that real-time and batch essentially building some system in real-time from the ground up, that supports ability and immutability. That could be one reason, I do not know.
**Boaz:** Yeah, this is a tough nut to crack.
**Ananth:** Yeah.
**Boaz:** You were in Slack back in 2016 and you were there for 4 years or so. How different was the data stack at Slack when you joined versus when you left? Because I know you, you pretty much helped build it from the ground up.
**Ananth:** At least at that point when I left, it has predominantly remained the same. One good thing, I do not know if it is a good thing or a bad thing, the early people started to involve in the Slack data infrastructure came from very good previous experience building data infrastructure at scale, and from the good-to-go, we choose some tools that are kind of high on the support side. Just Kafka, Presto and storing everything in the pocket format, tried to use Airflow programmatically to author our system and so on, how do you do modelling and then using structured logging, using a script for logging, not click. So, these things are, we got it right. We spend less time on detecting whether this is an integer or a stream. That kind of a problem I see more and more companies trying to do. Anyone approaching me asking, like, how do I build a data platform? I would say firsthand, please do not use JSON as your data format in this case. So, these are the things we got really well, programmatically author or data pipeline, structured even logging and stuff like that.
**Boaz:** Because of this JSON, that was an interesting comment, because all the new data platforms essentially are encouraging people to use JSON. Everybody is releasing JSON first features, JSON capabilities, so you are saying stay away from that! be wary!
**Ananth:** Yeah. Maybe, I do not know. Maybe, they start thinking of a Schema registry to solve this problem. That could be one reason. But again, the compiler is good at type checking and where do you want it to type-check after the fact. When you are doing a compilation itself, it is better to do that, you can reduce a lot of errors in this case. I think most of the Slack challenge at the point of the time was scalability because the rate at the company had grown, every assumption that we have made in every three months we just invalidate at some point of time we made over three months. So, we have to constantly reinvent or try to scale our system to kind of cope with the amount of volume that we are getting. When I started in Slack, we were ingesting maybe 10K events per second and that is what the Kafka cluster looks like. Then rapidly, nine months down the line, we were just ingesting 3 million to 4 million events per second and we barely supported only one type of use case, on the business side of it. We want the operational side of it and all the other aspects of it. So scalability is a bigger challenge in this case.
**Boaz:** So maybe it sounds like if there ever was a scalability challenge, it is this one at Slack.
**Ananth:** Yeah.
**Boaz:** I am sure not everything went smoothly. This is a good part of our beloved epic failure corner. We grow and learn from things that did not work. Any memories to share things that are a good lesson learned?
**Ananth:** Yeah. I think many things, especially, I do not know how the lesson was learned. I think the Airflow page we had one time, has stopped us from running any code for 8 hours and a lot of small accidental problems. I think this is a very interesting one. Airflow at the time had, I do not know whether it is even now it is true, the scheduler has a problem that it is not able to schedule the rate of spinning off. What we have done is we introduced a Cron job that keeps everyone in our background, just restart the scheduler, just in case the scheduler is exhausted, we just restart in an hour but what we did not realize that that Cron job had a bug. Instead of waking up at the top of the hour and then just restarting one time, it is restarting for one minute continuously. We did not realize that. It was just working fine because it is a very minimal in difference, and one day, we started migrating Airflow from one version to another version and we did not realize this bug when we are iterating that and we are continuously monitoring whether the upgrade is successful or not, we keep seeing this weird behavior and we were so confused, what is happening? Read through all the source code, read through all our deployments. Nothing is happening. What is really happening? That took us like one day to figure it out. Oh my God! there is a Cron bug and we almost forgot that we put that Cron job.
**Boaz:** The last suspect that nobody even bothered to think about was the one. Thanks for sharing.
**Ananth:** The good news is that now I am able to read all the Airflow coding in just one hour because we had to read it and understand.
**Boaz:** If you go back in time, we put you back in 2016 at Slack, starting from scratch, knowing what you know now, what have you done differently?
**Ananth:** I think one thing I would have done differently towards the end of my time at Slack. I think people started to lean more towards cloud data warehouses, 2016 cloud data warehouses might not be much more mature or much more sufficient to handle our scale. I think at this point of a time, I feel like systems like Firebird, Snowflake or something like that could have been our first choice to do that and then more using the tools rather than trying to build everything else that would really give us much more velocity in the way we want it to handle that thing, because we had to manage all our EMR cluster and because we had to manage all of our Presto cluster and Airflow cluster, typical out of toll for us to kind of support the growth that our company kind of going through and that reduced the trust level of the velocity more and we put people adopting to.
**Boaz:** In recent years, which technologies or tools or both did you put your hands on, in recent years, or ones that excited you or that you are excited about?
**Ananth:** In recent years, I think I am excited about a lot of development going on, the data lineage and data catalog. I think this is something that we have not thought through when we started our data team at that point of time. And most of the companies, the data catalog, and data lineage will always be an afterthought. I am so excited to see so much literature, so much talk about lineage data, data discovery systems, and then the data quality aspect of it. I think that is one of the things that I am very excited about. I think adopting that we finally acknowledged we came a long way from Hadoop world to be kind of a heavy hacky backend engineering to acknowledge this is a data system. This is a data management system, properties of the data management system and I think we are coming through the full cycle.
**Boaz:** Which tools have you been looking at so far at Zendesk with lineage, logging, and quality?
**Ananth:** There is pretty good tooling available right now. I think it is a good thing a lot of options are available for the consumers. Now I think datahub from LinkedIn is one of the examples of that. Amundsen is another great. I think a Clan is another interesting tool to do that. I think what I am really looking at is how this tool essentially embeds into our workflow of our analysis that the data engineers integrated into their workflow rather than introducing a data catalog, and then just do a checklist and we have a data discovery and then checkmark is done and then nothing. I am sure you are aware of Apache Atlas, which is an open-source data lineage story developed long back in 2012 or something like that. We do have an Apache Atlas from 2017 in Zendesk, except that no one uses it or half of them are not even aware of it. The thing is that the pattern that I noticed why people are not able to use it, is it is incredibly hard
first-of-all to do that. So the analysts, I observed the behaviors, and again, one interesting thing is an analyst came and said, "Hey! What does this table look like? and I wanted to get an insight. Tell me what are the tables that I should query about that?" And someone trying to point out, well, this is a lineage tool that you should use.
And it is a separate effort. They had to go to a different UI. They had to understand that UI and then they are just trying to query, that it can have information, do not have any information, and then if they are not able to find that information, they immediately go and ask, who is the senior of the team? and they will go and ask the same question, right? You will be using an analysis of the data discovery system again and asking it and there is a high chance that a person has the knowledge about it because of an experience and they answer the question. When the next time a similar problem comes in, the analyst will not go and look at the data discovery system. They directly go because their workflow is now changed because this particular tool is not trying to solve this problem. So, I look at this as a workflow problem rather than a lineage problem. So they need to sufficiently solve that problem, which will be really helpful.
**Boaz:** Awesome! What do you recommend if I am in software, but I have not been involved with data so much and I want to start? What is a good route for a software engineer to get his hands dirty in the data world? How do you approach such an educational program?
**Ananth:** My take on that is, SQL by default is the easy language of data, right? The first thing in terms of the skill set, I would like to say, is to pick up SQL, it is a first-class tool for you to start navigating the data across. And if you are a software engineer, I think one of the good things they can do is take a look inward rather than outward. Many people started taking, wanting to get into data engineering and looking at - I am going to solve a business problem or predicting, sales analytics or marketing analytics. If you are a software engineer, there is most likely that you will be working with some kind of a system, some kind of software you will be deploying to a production system. So that the production system will emit some kind of logs. I think the very first step is to take that log and apply the SQL and try to understand your system from a different perspective.
**Boaz:** Interesting.
**Ananth:** We do have the observability tooling right now, metrics and logs and you do search and try to find the needle in the haystack. But if you try to take that logs and try to understand the long-term perspective, it is particularly helpful, A, because you already know the domain, you already know the system, so you have a very good intuitive scale that you can start to work around and improve your skill and then, once you understood that, the techniques and the domain is transferable, right? The domain is more of acquiring knowledge, the tools, and your way of thinking to figure it out, the pattern is going to remain the same. I think that would be my suggestion - Start with the tool, pick SQL and then look inward in your own domain, trying to figure it out.
**Boaz:** It is beautiful. So, a log-driven approach. If you master the logs, you know everything about your platform. Awesome!
**Boaz:** Okay, great, Ananth! It has been amazing, with so many great insights. Thank you so much for joining and again, keep doing what you do with the newsletter. We love it!
**Ananth:** Awesome!
**Boaz:** Stay safe with the virus and all.
**Ananth:** Yeah. Thank you so much.
**Boaz:** My pleasure.
**Ananth:** Bye.
**Boaz:** Bye.
# How ZipRecruiter and Yotpo power self-service data platforms that work (/blog/how-ziprecruiter-and-yotpo-power-self-service-data-platforms-that-work)
Data engineers are not paid to do support. Liran Yogev, Director of Engineering at ZipRecruiter, and Doron Porat, Director of Infrastructure at Yotpo talk about building resilient self-service products that keep customers happy and engineers calm.
They walked the bros through their data stacks and explained how ZipRecruiter is completely rebuilding its data layer from scratch.
Listen on [Spotify](https://open.spotify.com/episode/076y8o8AYFgVl13r4agedU) or [Apple Podcasts](https://podcasts.apple.com/us/podcast/how-ziprecruiter-and-yotpo-power-self-service-data/id1561927688?i=1000605558364)
Benjamin: Hi, and welcome back, everyone to The Data Engineering Show. It's super cool. Today, we have another set of experienced podcasters joining us, Doron and Liran.
Liran: I love it.
Benjamin: Thank you.
Liran: You tried really hard. It's just...
Benjamin: I speak German, that's my native language, so we have some hours in there as well. So I practiced really hard before the episode.
Doron: Sounds good.
Benjamin: Awesome.
Doron is a director of infrastructure at Yotpo. I hope I pronounced that correctly as well.
Doron: Perfect. You're the only person in the world that can pronounce this.
Benjamin: Awesome. Liran is a director of engineering at ZipRecruiter and also was at Yotpo before. Do you guys just kind of want to give a brief intro tell us what you're up to, tell us about your podcast, and then we'll dive right in?
Doron: Yeah. Go ahead. I'll start. Yeah. I'll start with me, and then we'll go to the podcast.
Liran: Born in the Yavne.
Doron: Yeah. I was born in Yavne.
Liran: 47 years ago.
Doron: 1923. So, I worked at the Yotpo for 25 years, just a lot longer time.
Eldad: Was born at floor four.
Doron: Yeah. We started at floor number one and we reached up to floor number 26, right above a Firebolt. But, I worked at the Yotpo for a long time. I started as a team leader for the data engineering team. Later on, we became an infrastructure group and the team grew. I became the group leader for the data infra. We built an amazing data platform here together, Liran and I.
He was actually my predecessor. He was the infrastructure group manager, and I replaced him recently.
Liran: Doing a much better job.
Doron: Yes. So I do everything better. Now, everything is better here.
Liran: Less people, right?
Doron: With less people. Yeah. We're running very lean.
Eldad: Benjamin, don't get any ideas in your head.
Doron: No. It works only if you're a woman. We have a joined child. We don't have a child.
Eldad: Oh my God.
Doron: We have a podcast that we've been podcasting for about a year and a half. Also about, data engineering and all the surrounding world.
Liran: But it's in Hebrew.
Doron: So it's in Hebrew. So we're not really competing. It's not the same.
Benjamin: I tried preparing for the show and Tamar said your podcast, and then I was shit.
Liran: So, there is one episode in English, which...
Eldad: So he learned Hebrew.
Liran: There you go.
Eldad: Hebrew. Then you didn't hear the second half. Yeah, he's still...
Liran: If you even started, it's so fast,
Eldad: It's too modest.
Liran: How fast we talk in the episode, so I don't think it's good for you as, as the beginner, wave to learn Hebrew.
Benjamin: Okay.
Liran: Sorry. With friends or something in Hebrew. I do not if there is a version.
Now me. I'm kind of always around the one, so all the stories intersect. I was at Yotpo before that, I did something else. I've created the data platform there, but also I did a lot of other different roles, in my end position I was running, all of the platform engineerings at Yotpo, so backend data, and front end which Doron is doing right now.
Then about eight months ago, I moved to ZipRecruiter. I'm doing the same there, but a bit different teams. So I also run the platform engineering, or you call it enablement and around the area of ML experience, we call it, or ML tools, and all around ML data. And also experimentation, which is a team that is, building an experimentation platform for internally for our organization. So doing heavy testing and measuring everything. So, it's really been fun. And, the podcast.
Eldad: Data sets. Data sets, data sets. More output. More output.
Benjamin: Awesome. For listeners who are kind of have never heard about Yotpo or ZipRecruiter before, do you want to give us a quick, high-level overview of what product the companies actually building or different companies?
Doron: Yeah, sure. So, I think it's funny because I've been in the Yotpo for a while, so we kind of started off as a branding ourselves, as a marketing e-commerce platform and we really became a real platform only in the past few years where we offer a set of different products under the same platform to help e-commerce businesses just, just do better, bit bigger and stuff that.
But recently, I think given the latest changes in the ecosystem and the financial macro environment, Yotpo is trending towards being this retention platform for e-commerce businesses and we do this with the same set of tools, but we're really focused on how we can help e-commerce businesses, online businesses preserve their customers and enlarge their customer base through our products, which are review solutions, loyalty programs, referral programs, communication channels and customer data platform, and more and more products this.
Liran: ZipRecruiter is a Hiring marketplace, I think that's what you call it, basically helps both job seekers and employers find matches and we do that for customers from really small mom-and-pop shops up until customer Amazon. We do have different approaches for each of those customers, from enterprises to small businesses and we heavily rely on AI to do the matching and other things in science systems. So, that's our forte. Helping them really find good matches for both sides. And also balancing the market itself just completely. So, that's the gist of it. And we're based both in Israel and in the US.
Benjamin: Awesome. Cool. Nice. This is the data engineering show, and you guys obviously have a bunch of experience in data at a variety of companies. Take us through the types of data challenges you guys have in your day-to-day and maybe tell us a bit about your stack.
Doron: Okay, cool. So, maybe I'll start with the stack and then I'll go on to the challenges. I think that might make more sense.
Stack, I'll start from the bottom up. That's comfortable for me. So we're ingesting data into the data platform from all sorts of different data sources, whether it's operational databases and third parties event data, whatever and it's basically all streaming into the data lake. So, the whole solution or platform is built around the data lake and it's very data lake centric. Based on AWS. That's all the workloads running there.
Then in the dat lake storing data, different formats, and using different techniques for transforming the data and different engines, mostly Spark these days. I think we're running a long time with Spark, what Liran and I did together in Yotpo for many years is making Spark...
Eldad: Buy a perpetual license, gives you the ability to use it for free on unlimited resources forever.
Liran: Yes.
Eldad: Sorry, go ahead.
Doron: Our challenge years ago was how to make big data tooling available for the generalist developer and later on also for BI developers and stuff that, and that was a big challenge and how to democratize data sets as well as data tooling and we used to do this... I think we had a different approach for this a few years ago, which changed and evolved over time as the platform grew older, and the company also evolved and had different needs and requirements.
Orchestration, we're using Airflow, most of the past pipelines are running using this framework that we built internally in Yotpo. It's also open source. It's called Metorikku where we write YAML files with SQL statements to describe the data pipelines.
Also, with streaming, mostly Spark structure streaming, also Flink pipelines recently. Then, the whole analytics area things.
Plus we have DBT that we started using at Yotpo for the past year or so, well over a year. But we built this whole framework around DBT to visualize the new way of thinking about how we should manage data in the organization. It's also an internal tool that we built and it really connects data producers and data consumers on the other end where we have Looker which we use for internal analytics or external B2B analytics, either embedded dashboards or API, which is also, something we are leaning towards more and more as time goes by. Maybe add kind of analytics is also a big part of the thing, making data available for everyone. So everyone are using Databricks clusters to work on top of the data lake. And it means engineers, BI, analysts, support engineers, solution engineers, and everyone working on top of the data lake.
Yeah, I think that that's the big picture.
And if you ask about challenges, I think that in the podcast, we talk a lot with people and I think a lot of the times we go and talk to people that have big data challenges, it's still a thing. I mean, you think that it's solved already, but people have big, big data challenges.
I think at Yotpo, it's more what keeps me busy. It's more data manageability and how to architect this thing to work well at scale, serving a lot of people, here we have an R and D of 250-260 people and more and more outer circles using the data, and how we optimize this huge machine of money into something that's much more coherent, robust, scalable, and resilient over time.
Data manageability, I think, a wider term, because I can talk for hours about this. I would say that's where I am focused at the moment.
Eldad: Quick question, seems data and everything you do with data is part of the feedback loop within engineering, within product building, , everything you do as engineering and product is using data, how is that translated into user experience, the data? and a lot of what you've mentioned is internal, right? So, it's for building products, how does that get translated into Yotpo's business, for example?
Doron: I think Yotpo is not per se a data product. I think that ZipRecruiter maybe is, is more an example of how is data, centralized and within the product. Data is a big thing, and when I talk about experience, I mostly talk about, and that's what my group is focused on, we talked about front and backend and data, but we are very, very focused on developer experience. I think that where our customers, B2B customers, make the data is, the places where we make it .
Well, I'm not talking about all the machine learning, data science part of things, it's not really under my responsibility. But, analytics is becoming bigger and bigger and I think it's also a part of what's going on, in the world that we are really focused on observability and demonstrating ROI and helping them make the right decisions on. Yopto is a complicated machine and part of it is, I think it's almost actional BI, but external. It's the way that we organize the data in a way that helps them understand how to navigate through the different products to bring more value. So, I think that's where they touch on data the most.
Liran: So I want to add that. I think that what is happening is what we went through in last couple of years is more and more and more use cases were added to the world of the data lake. First, what type of consumers we have for data? So we are mostly being focused on internal, but also, external as well. The data meet the customers in both companies. You don't have to be a data company to have the data reached somehow into production systems.
So, there are more and more use cases just being added. More and more types of consumers. They require different things and their experience or even their capabilities are different. How can they access data? How can they produce data that is high quality? So I think that's what I'm focused on and what is always changing in our ecosystem.
I can but I do not know if we care about that, but our ZipRecruiter stack or not or we just moved on. It's okay. Move on.
Eldad: So, what you're saying is if your data platform shuts down, then internally engineering product business won't be able to operate. It's so much embedded. It's so much interwined. It's as you're saying, self-serve. What is self-serve? Is it opening the data for as many as possible? From your experience, is that real? Is that kind of something...?
Liran: Yeah. So we try to build a decoupled system, right? I don't think it's great if the big data platform, which again, both of our companies is very data lake eccentric. I don't think that if this drops, as soon as it's down, then everything goes down.
I don't want to be at that position. So we need to have some kind of differentiation between all of these backend processes, batch processing, even stream processing that happens, managing both for ML or Funnelytics in the production systems, which actually needs to serve something to the customer, they actually see.
So again, the business will suffer, maybe some late data will arrive to the system. It may be visible to the users, but in the end, we don't want to be, where something actually is not working anymore in production. I think in my opinion.
Eldad: I've seen many peoples saving data sets eventually after they use all the stack, all the tools, they save it in Excel. So, it's a backup, but it's also a failover mechanism and eventually, it all ends up in some report. So, we hear it all the time and I think companies went all in on data, you'll be surprised how dependent they are on internal data.
I'm not talking about external. External is easy to justify, but justifying internal, asking how do I optimize internally? What do I optimize for? Those are new questions we're hearing more about.
Liran: It's even more than that. We need to ask a question, do we need all this data? And it's a question I don't think a lot of companies are asking.
Doron: No, no, we don't need it.
Liran: We don't need all this. Yeah.
Liran: Because Doron and I have been to a lot of discussions about retention periods, for example, for our big data. At some point, just delete it. What will happen? Well, we'll be fine, and I think in a lot of... I'm not actually saying that.
The discussion needs to be made about because at some point it reaches such a high complexity in cost and so many moving parts that you need to ask yourself, do I really need all this? And I think that's something that each company needs to always reiterate on, and ask these questions. I think that's a good culture.
Doron: I would to add something you were talking about questions that we ask ourselves, and I can say that after many years working in data, it sounds dorky.
But I find myself asking myself different questions as time goes by, my concerns shift and I ask myself, recently we started talking about how we should better structure the infrastructure group. And then I started asking myself, where do the lines cross? where does data start and ends when there's backend infra starts and ends, and we have all those interfaces between them. And I think it's a really fascinating question to ask. And it's also in the way that how data stack connects to the APIs or event-based architecture and where do the lines cross. I think it's very, very interesting.
Liran: Yeah. I think we're actually seeing the world move a lot. It's becoming more an engineering world than it was before. It's less and less about just being data. So data is coming from somewhere. It has been produced by someone. It's even been managed by someone that's probably in the product or engineering world.
And I think before we used to have these silos where we had analytics teams just hand-managed data and I think that we're more mature companies. Question is, can they really manage it? Do they really have enough information? Are really taking the responsibility off of the engineering teams?
I think it's because we just talked about boundaries, I think those are also changing all the time.
Eldad: Crazy times, huh?
Liran: Oh yeah. We love it. It's great.
Benjamin: Nice. Maybe take us at how these boundaries look at, the specific companies you are working for? In a sense, you're providing core data infrastructure for your company. Say I'm a neighboring engineering team, I'm trying to build this new, I don't know, data application or internal data experience, whatever, where do I interface with your teams and what are the types of services in a sense you provide them?
Doron: Do you want to start?
Liran: Yeah. We are trying to build the methodologies, processes, and tools around producing and consuming data by all those types of customers you just talked about. That's where we are. And that means that in most cases, when you build the data application or some data experience, then you'll be interacting one of the tools that we have. So it's either going to be something that we bought and we basically implemented and integrated or something that we built that's very specific for just an organization. And what we like to do is optimize that all the time. So that's what we do
So we figure out, okay, we have this customer and we want them to have, to create the best data set or the best data application, so it's going to be the highest quality. It's going to be really fast to create, it's going to be really easy to consume for consumers. So how do we get there? And that's where we add all the different layers of the tools that we either build or buy. So, I think that's kind of where I see the interaction, specifically around the data. Do you want to add some?
Doron: Well for us, first of all, because we talked about teams being really lean, but we're not kidding the teams are very, very small and we support a lot of people, given services. So, we are really focused on building a self-service data experience. I'm going to borrow your words because I like it, but it's all about the experience and how we make this experience better, and help the developers and engineers be more self-sufficient and free to operate within their domains.
It's always maintaining this balance between allowing them to run freely. I'm trying to find the nice word, destroying our vision for how the...
Liran: And also, taking care of our people. We don't want to be just giving out support every day. We are product teams, in a way that we are actually creating internal products, just the Firebolt, for example, does for its customers. But in the end, we both are lean. So, we cannot do support. I mean, no one pays us to do support all the time, and I think it will be really a waste of our time. Our engineers want to build things and not have to support them. So self-service is a really big thing and providing with our customers, with the tool to debug the systems, to run it, to own it, to have...
Doron: And to enforce best practices as well in a way that will not ruin their lives and experience, because it can be a real drag to being blocked in CI with every step that you try. So, it's a matter of how to enforce these in a smart, elegant way, and through this, create the things that you believe in and you think are instrumental for the data platform.
Benjamin: Got you. So going back to this previous example, of retention periods. The day I'm an engineer, I want to use the awesome data utilities you and your teams provide and I said, wow, I really need a 10-year retention period here. At what point does someone question this choice? So it's this part of the core education you guys are doing inside of the companies or is this ultimately up to the consumers?
Liran: I think it's both. We always have a choice between creating validation in CI/CD or creating whatever it is you do. Adding some rules on top of it. We're getting some alerts and monitoring, so we can do everything and we can also do education. We can also go make sure that we have really good guides, and we do sessions with everyone and talk to them about what is actually happening.
Eldad: And it depends on the weather. It depends on how they feel on that certain day. So for example, if they're angry on that, there will be more about consolidation, and measuring best practices applied. When they're happy, then it's about self-serve, pushing more data tools, it's a never-ending cycle.
But the truth is even myself, you constantly ask, does it need to be centralized? Does it need to be decoupled? all of those complicated terms. The truth is you need a team that knows better and that team knows better most of the time, not all of the time. And that team learns faster because they're domain experts. And if they're actually capable of translating that into "best practices", which are amazing, if they work, then they make everyone better. Because the truth is most teams don't have a DNA for data. And the truth is that if you look at data, how it's being applied today versus a year ago, then most teams are not even close to having a DNA for data.
So, I think, The IT Crowd, I do not know, maybe some of you know the show the British, that is how it all started, right? It was all about conflict and they think teams that win with data, they get addicted and they start depending on experts and domain experts like you.
First, we salute you because I think the reason we asked it, it's because it's hard for those teams and in those times, it's even harder because they're now inbound, outbound, they do a lot of stuff.
So, first, we salute you, second, you are important, and third, if you don't deliver, then yeah, the business is shut down in so many ways that you can even imagine.
Liran: I think it close down and there are issues going on. We just help protect and make sure that people do good things and not cause...
Doron: No, I like Eldad's version. I feel very important.
Liran: Okay, we save the world.
Doron: I feel very important.
Liran: Yeah. We're on the critical path. Get that.
Doron: Yeah. I want the critical path.
I wanted to add something. I wanted to add two things, but I forgot one.
Liran: But I'll give you to add your things first.
Doron: No, but I think that another conflict that we have is following what you said, we do data all the time. We practice data. We breathe this. We eat this, and, and we, that's all we do. We do more stuff now, but basically.
Liran: We also have hobbies.
Doron: That's our DNA. We have that DNA, but the problem is working with generalist developers. They have an epic on some data pipeline or something that they have to do with data infrastructure. But the specific developer might not encounter anything related to data for a long time. And by the time they get back to that feature, to fix something, to add something, two years can go by and a lot of the time this stack has completely changed within those two years. And they're like, what the hell just happened?
And I think it's different between the culture and the DNA between the ZipRecruiter and Yotpo, but at Yotpo, the full stack developers, they do pipeline building if they need. All the teams, all the product lines, all the domains, they all have features and stuff that are related to data and data infrastructure. But it's not a day-to-day thing as it is. I don't know, talking about infrastructure related to Java, for example. Because it is not something they do and practices every day. And it's also a battle that we keep trying to solve and get better at how to engage our users to understand what are the pain points.
I remember what, the other thing I wanted to say is that...
Liran: I want to react to what you just said,
Doron: But forget. I'll just say it and then you.
Liran: Okay. Alright. Very fast, I'll forget mine, so.
Doron: One of the things that we use, more and more now, is creating the right observability around data and it started mostly around cost because it's the big thing, right? In the past six months at least. but it's not only cost. It's cost and it's performance and creating this observability in a way that is actionable for the teams. First of all, it's engaging and also really it helps to bring them closer to the material in a way that they can comprehend and understand and relate to their actions. So I also think it's something very stronger we invest in.
Liran: So, I want to add to...
Doron: On the first thing or is the second...
Liran: I don't know. Not about a cost difference, but you now confuse me.
Anyways, I just want to add, that Doron mentioned that the organizations are different and I think when I got to zip, I was under, like, we are the same person, basically, just different.
Doron: I'm a woman.
Liran: Yeah, she's a woman, I'm a man. But the same person.
But in general, we were trying to have our generalist developers use big data, which really, they didn't really know how to do it. So we helped them and created a lot of structures around and interfaces. So, they can do it with really easily.
Doron: In ZipRecruiter, you mean?
Liran: No, in Yotpo. But when I got to ZipRecruiter, I saw something else. They have a new persona that we didn't have before called the data engineer. And I think that's related to what you said before Eldad is actually in both my teams, actually maybe in three of our teams that I have, we're not the experts actually. We are experts in something. We know to how to build infrastructure. We're good at that. We understand the product. We understand our users. We can collect feedback. We understand technologies.
Doron: The platform, architecture.
Liran: The platform, we're good at that. But how to actually, and what kind of data pipelines to write or ML even. We're not data scientists. We're building an ML platform. So that's really different. We actually try not to be the expert in that, in their day-to-day, but on our day-to-day and figure out, again, like Firebolt does for its customer. You don't have to be the best in the data or understand all the different types of data pipelines people use, but more understand what your customers need.
Eldad: Oh, no, our work is harder. Okay, so I apologize.
Liran: I get it
Doron: You need to do both. Yeah.
Eldad: No, it's nothing to say, right? Benjamin? building a database. It's the hardest thing in the universe.
Benjamin: By far.
Liran: Thank you. So yeah, I think for me it's very challenging. I think in data it's a bit different because we do understand data pretty well, but in ML, we have to understand our data scientists where they're not just one person. They are many different people with many different cultures and needs that they need and create a platform that will help all of them at the same time. And that's very difficult, which we are not and really understand what they do. So I think that's also in the data world, for us as well. So our experts are ever in the company and we know how to gather feedback and how to build a really good product. So, but in Yotpo again was different. So, I'm just giving two perspectives.
Benjamin: Right. So, that sounds super interesting, right? Then, coming to ZipRecruiter and kind of adjusting to kind of this different company and different team structure. How's that going basically?
Liran: I need to go on basically, but that's what happens.
Doron: But we do podcasts together.
Liran: But we do it together. But it's neither on a day-to-day basis and it's really missing.
Eldad: So basically, you went from data engineering to data science and it's hard. And, there's a lot of science there, but it's the same data, so at least that, right?
Liran: Yeah. I have one word left from everything I did before. No, I think, whatever I did at Yotpo, it still helps me with the challenges, but it's a different organization, it's a different size.
Doron: The culture is a big part of what we do. It's really important to understand what culture you're stepping into and what's the culture you want to push.
Liran: And I think what I did... I'm going to be really blunt here. What I did when I entered ZipRecruiter, I was like, oh, I know how to do data. We're experts we know what to do. And I tried to push my agenda and I figured out really quickly that this is not what this organization needs. This needs something really different.
Maybe even if I am right, I cannot just push it. Culture is something and culture change takes a lot of time and maybe in some cases you can't even completely, you have to live with something else. And I think that's what I experienced when I joined the ZipRecruiter.
Eldad: Do you have kids?
Liran: Oh, I have too many kids.
Doron: Too many kids. Too many.
Eldad: Okay.
Liran: Three for me, two for her.
Eldad: Your same experience. So, we trust you'll manage and you'll figure the kids out and because it's a new family, but you have lessons learned from the previous family. But it's amazing. It's actually, we kind of never got, at least on our show, to hear that version that those challenges from that perspective. Thank you for that.
Okay. Benji, go to the... We need something formal. So, pick the next formal bullet that Tamar gave you.
Doron: Can we do something formal?
Eldad: We haven't even really opened the formal questions and an agenda yet.
Doron: We're not formal people.
Eldad: That was all intro up until now.
Benjamin: Then that to get more formal, tell us about your goals for the next year because you've already hinted it at six past months, things change, right? Things are focused much more maybe on cost efficiency now and those types of things. So, what are the things you're aiming for with your data teams over the next year?
Liran: Can I start? So I think, Zip is in a strange position. We basically are rebuilding our entire data layer from scratch. It's quite a big company to do that in this time, so it's very challenging. So we're moving away from previous architectures and even technologies and just removing everything to just behave differently. So, right now our challenge is the quality of all of this migration and in general, just quality
Up until now, a lot of the different use cases created, data is not at the top quality. There have been testing before and ways to measure the quality of data and monitoring, but in the end, there were many, many unstructured things along the way. So, we are now trying to create, this culture by technology. So, we are building the different tools to help make that into a structured process that's repeatable. It's easy to use. and it's just no-brainers just give you high-quality data. So, we started with Schema.
So we built a way to document Schema, write Schema in Protobuf for each of your data sets, be able to kind of document everything about that dataset in a way that their consumers can actually, you know, understand what it is they're consuming.
We're creating new ways to create data pipelines based on events or in different ways, either by SQL or in Scala, and we're creating infrastructure around that. So to help the producers create high-quality data as easily as possible without having to actually deal with a lot of the complexities of doing that by yourself. So, adding think a lot for automation on top of that. We are going to build a new semantic layer.
We have a data set that's great but how do we actually use it? How do we aggregate on top of it? How do we join it to different other data sets? So creating that extra layer that explains the consumption patterns of the data. That's something big that we're going to build. We need to switch a query engine. We are right now using Athena, so we need to think of something else.
Doron: Firebolt.
Liran: Firebolt, right? That's an option.
Eldad: It's always the first and last option, but freeze for a second with that thought, and question. You talked about how you wire, you're using Protobuf, are those things that you embed within your legacy or existing, let's not call it legacy, with your production system, assuming that over the next year, step by step, you will kind of be able to unplug and plug and play new stuff or are those practices and principles that you apply on your new projects? Because you're saying you're moving the business to a new platform?
Liran: Moving everything, the end goal is basically deprecating everything that is, old and we have deadlines for it. It's quite aggressive. It's good in a way. We're actually moving. We're not just, doing everything new is going to be like greenfield and everything old is going to remain and crappy. We have to move everything. So that's going to be something that in the end, we'll just have something new and everything else that's not new will just be destroyed.
I think that's why I talked before about the question of what we actually need. So these are questions that I'm not being asked because I'm more on the technological side, but our BI developers and analysts are asking those questions all the time. Do we really need this report? Who actually uses it? Why is it so complicated? Do we really need all the data set behind? So building that data layer is not just about the infrastructure behind it, it's also about the data itself.
And I think all of those layers are being rebuilt, so it's interesting times.
Eldad: Nice.
Doron: I guess it's an entirely different story at Yotpo, but I think it can sound very, really similar if I say it, but I can say looking for the year from now, I think the challenge divides into two parts. One is in the world of analytics, it's actually combined, you know, and again, we're also restructuring the data lake, and also I think the whole escapade with dealing with cost, very, very seriously, has also got us to think of are we doing stuff, efficiently enough?
Are we doing a lot of things not in the right way or suboptimal and I think that the rearchitecting of the data platform, I talked about before about dbt and the infrastructure that we build around it, and it really embodies all the agenda that we have towards how data should look in our organization? We also working on migrating everything. This is very difficult and it starts from the actual S3 buckets and where data lies, it goes on to data contracts and data owners and how we preserve this.
It goes on to data quality and all the way to exactly like everyone talking about semantic layers and data catalogs and how we push this thing forward and I think that, we could have worked on this stuff a year ago, maybe not semantically because something is really happening now, these times it's really, really hot.
But I think even talking about the data catalog, we could have started working on this two years ago or even three years ago. But I think that we needed to get to this, level of understanding and have the organization understand what they need from, from data and for us to reach the same point where we see things eye to eye and the importance of things, and that's returning to what I said before about data manageability and the other part of it that goes together is that, currently a huge part of the data platform or all the raw data?
Most of it, not the big data by the way, but a lot of the data in terms of a number of data sets comes from CDC where we stream data from the operational databases with the BDM into the data lake and we understand that this method of replicating normalized databases and tables into a data lake and having everyone or not everyone, it depends how you layer the data transformations, but having people need to reconstruct the logic and need to understand what the team that built this architecture of how that data is modeled for an application and build this into something that makes sense for analytics, for example, or even for the product when you want to push data into a moderation view for B2B customers. It doesn't really make sense and it doesn't make sense in so many ways.
Liran: It's also about the coupling. You have someone build an application that is used in production to serve some data or to serve something for the customers that is built in a way for MySQL, which is you have a couple of tables, you have some JOIN, that doesn't really make sense for analytics. It wasn't built for that and when you couple that means that a production team or the team in charge of those tables are stuck with those like the Schema forever because someone in analytics actually utilizes it, but it's their tables. Why is anyone using those tables? So I think it's also about that.
Doron: Yeah. It's really understandable that we need to publish these data contracts or these data facades and have the operational teams in charge and be the owners of these data facades and for the analytics to rely on that, moving onwards. And, this goes on to, for us, it's probably going to go to the direction of using the output box pattern, and it's a big thing because it means that we need to restructure the whole bronze layer, and it's just something quite big, more technical than what I talked about before, but I think it's a big part of it when you're talking about re-architecting the platform.
Eldad: So, Benjamin to someone who is just an engineer, not a data engineer, even though you're building a database.
Benjamin: Which is the hardest thing in the world?
Eldad: Think about a lot of that stuff that's being discussed is when we used to write and read design pattern books in software engineering, right? So you'd go, you open a book and you had 25 patterns, and you know that there's so much experience being embedded in each pattern that you even learn them by heart because you just assume that's how it should be done, and data doesn't have it. So, a lot of what that team is doing is trying to figure it out. A lot of the staff is common and you could say, okay, we could start treating that as a data design pattern. A lot of that is unknown, right? I've heard about semantic layers since I was 16 when the first time started dealing with data. I'm 44. We invented the semantic layers. It was 20 years ago, but now\...
Doron: But it belonged to a BI tool, and now it's also decoupled, everything getting decoupled.
Benjamin: Eldad goes all game, for this podcast was just to show how visionary he is.
Eldad: Exactly.
Benjamin: Set out, in the beginning, to just drop this.
Eldad: But the truth is we called it semantic and it was just because it was so nice...
Doron: Rebranding to make it cool.
Eldad:... to have a diagram, the JOINs are just, you drag and drop the boxes and the JOIN line follows you, etc, etc. But you're right. I mean back then data was so easy in terms of metadata, in terms of who's using it. It was nothing compared to now. So just reinforcing everything you were saying, we need it, most of it is not solved yet.
Liran: Yeah.
Eldad: But it's nice to see that there are pattern emerging.
Liran: Just to add, one of the things that we do actually is actually write playbooks for different data scenarios. With the Data Guild, which is another, entity that we have at ZipRecruiter. But that's part of the job to actually create those, and I think each company probably has its own, and that's probably why there's no single book for how to write. Data Mesh tried to, I think create some kind of standards around it. But I mean, it does not dive really deep or into each technology and each kind of a pattern.
Eldad: It's because it's culture driven and it's kind of product-driven. What's the product right now? What's the culture right now? What did they go through? Did they go from Oracle three months ago? Where they're born? Did they go through the \[00:42:49.08] **\_** experience, cloud-native startup, day zero data-driven? It's how we win, so many ways to win or lose with data, so many practices to follow or not, but amazing to hear it. Benjamin, you see, there is a reason to wake up every morning. There is a reason to bend the data. Benjamin asks me sometimes, why do we need to build the database.
Doron: Why are we doing this?
Liran: Yeah.
Eldad: Do you see the pain?
Benjamin: It's so hard
Doron: It is the hardest thing in the world. It's too hard. What's it for?
Liran: Exactly, exactly.
Doron: Say the world. Pinky.
Benjamin: Yeah. I appreciate you giving me back my sense of purpose.
Eldad: That was the purpose of this podcast.
Liran: What are you doing? Why are you talking to us?
Doron: Just leave everything and go build something.
Liran: Yeah. Build some nicer, neat index.
Eldad: Guys, we're frozen in time and space right now. We don't want to disconnect and go back to life.
Benjamin: All right. Liran and Doron any closing remarks from your end? Tips, tricks, anything else you wanted to say?
Liran: I don't know.
Doron: Yeah. I don't know if you have any tips or tricks, but, I just think I said it so many times, but I can repeat myself, but again I want to say that for a very long time, Liran used to laugh at me always that I'm like data infrastructure engineer that doesn't like data.
Liran: It's a cool thing. It's special.
Doron: That's my mojo. It's good for nothing though, but I think that recently and again, the motivation was cost, but I think that we really doubled down on analytics, on what we do. And I think it's fascinating and it's also about self-measurement, also product measurement for the product that we build and also for observability, which I talked about before. I think it's fascinating. It makes our job much more efficient and, and it really helps to bridge with our internal customers.
That's my tip, I think.
Liran: I think if I also going to add to what you just said, those interfaces are what interests me right now. So the analysts called decision scientists, and we have BI and we have data engineers, we have data scientists, we have ML engineers, all of those need to work together and they're all using the same platform. By the way, in some companies, don't use the same platform. There is the Snowflake area, that's only for the analyst and then you have a data platform based on it, that's only for engineers.
I would like to have a world and I think that's what happens in both of our companies where we work really well together and we are using the same platform, the same tools, and we are all enjoying because I think that's how it should work. And I think having those organizations disconnect, and by the way, that's another thing, organization, why are they disconnected and sitting under the same roof? They all need data somehow.
Eldad: Different database licenses are bought by different managers, mostly.
Liran: I guess that's good for you. Not good for the world. But yeah, I think I would like to see us help each other better, all those different departments and I think that's what fascinates me now and I hope to I have a better future.
Eldad: Boom! Amazing. Love it!
Benjamin: Awesome. Thank you so much for joining today. It was a total pleasure and kind good luck with the super ambitious projects you guys have over next year.
Liran: Thank you.
Doron: You too. You have the biggest ambition.
Eldad: Thank you so much.
Doron: Thank you.
Liran: Thank you.
Eldad: Thank you for joining us.
Benjamin: Bye.
Liran: Bye-bye.
# How ZoomInfo transitioned from data graveyards to ROI-driven data projects (/blog/how-zoominfo-transitioned-from-data-graveyards-to-roi-driven-data-projects)
Too often expensive resources and manhours are spent on dashboards no one uses, resulting in zero ROI. Philip Zelitchenko, VP of Data & Analytics at ZoomInfo met the bros to talk about adopting product management principles to ensure data projects have value, and provide an unfiltered peak into ZoomInfo's data stack and unique tech culture.
Listen on [Spotify](https://open.spotify.com/episode/2b56ZcsJioF1FR9hXlEo3b) or [Apple Podcasts](https://podcasts.apple.com/us/podcast/how-zoominfo-transitioned-from-data-graveyards-to-roi/id1561927688?i=1000652603546)
Transcript:
Benjamin (00:01.518) All right. Hi, everyone, and welcome back to the Data Engineering Show. We're super fortunate to have Filip Telychenko join us today on the show. He's the Vice President of Data Analytics and Zoom Info. Welcome. So great to have you.
Eldad (00:17.698) So, yeah.
Philip Zelitchenko (00:19.067) Yeah, thanks for having you Benjamin and Nildad. It's nice to meet you both and happy to be here.
Benjamin (00:23.566) Awesome. Cool. Do you want to tell us a bit about yourself, right? Kind of your background, how you got into data, your current role, and just introduce yourself to the audience.
Eldad (00:24.13) Thanks for being.
Philip Zelitchenko (00:35.515) Sure, yeah. So my name is Philip Zelitchenko. I'm the VP of Data and Analytics here at ZoomInfo. My career journey started in the world of stats. Most of my career is in the data science and machine learning world. That's where I built most of my career. And then in the last, I would say, six, seven years, I've started to expand to other areas, so look into the data platform side, which I understood that is a... big dependency for data scientists on the data platform. So it started from MLOps, then expanded to other areas. Then it started looking into data governance and data quality is important. How can we build things on the data science side with understanding the quality of the data? How do we ensure it's at a high quality? From there, it went to the data engineering fronts, because at the end of the day, as a data scientist or an ML engineer, you need to do things in batch or in stream. So it goes into those areas.
And basically, I started exploring. And the last seven years, I'm in the exploration mode. I'm still exploring and learning about platform, product management, data science, and so on. And how do you apply? And I talk about it a lot because I think it's a very good mental model, which I'm a big fan of mental models. How do you apply Marty Cagan's Inspire and Empower frameworks onto data product management and building data products, which I think is crucial because it's
If you look at the world, I think we spent a lot of resources and time on things that go into the graveyard, the Tableau graveyard, the Snowflake graveyard, you define the graveyard you want to look at. And there's a lot of time that was spent on things that in a lot of cases were not needed. And we paid a lot of money on it. So that's the world I live in and what I try to focus on in the last few years.
Benjamin (02:25.966) Nice. We really look forward to hearing more about this today. So at Zoom Info, do you maybe want to give us a high level overview of your data stack? What's the key technology you guys use? What are the data sizes? Just a one minute rundown before we all in the grave.
Eldad (02:42.434) It's all in the graveyard. It's all in the graveyard. It's all in the graveyard. Sorry.
Philip Zelitchenko (02:47.515) Exactly. No, no, yeah, it's a good question. So Zoom Info is a data company. I feel very fortunate to work at a company like this for many reasons. Everyone does data. Our CHRO writes Python. So just to give you an understanding of how technical the teams are, and it's embedded everywhere. So you're not the SME. SMEs are across the board. We have different data teams. And
Philip Zelitchenko (03:16.731) Generally, we split the data teams into two groups. There is my side of the house, which I deal with everything related to corporate. So if you think about how do we make our business more efficient, gathering information from different data sources, both on the go -to -market side, on the product side, and how do we make sure that we're building the right thing moving forward. And then we have all the data teams that support our products that are built on top of data.
And there's a set of leaders in the org that basically support those initiatives. Within my org, we have the enterprise data platform. That data platform is built out of multiple components. We have things that we build. We have things that we buy. And we have all kinds of hybrid solutions that we encounter. And that platform goes from warehouse. We're a warehouse -first company. We utilize Snowflake to.
Philip Zelitchenko (04:13.243) serverless ML ops solutions, so Databricks and some other solutions that we use in parallel, and many things in between. So MWA on AWS, how do we do orchestration using Airflow. We have something similar on GCP. The stack is pretty wide and has a lot of components to it. From the data observability side, we look at Monte Carlo. We use Atlan for data cataloging.
It's a pretty mature, I would say, serverless stack that is more modern than, I would say, most of the companies that I worked at or consulted for, where they're still in different areas, are still trying to figure out what they're doing. So we can go deeply into the different areas here, if you'd like, but I try to give you a quick overview of what we do.
Benjamin (05:01.838) Yeah, it was a great overview. So one thing I'm kind of particularly curious about is you said in the beginning, you're like your data company, you offer at the end kind of data analytics to your customers at a super wide scale, but you also have internal BI and all of these things, right? Are these like two largely separate data stacks or is it all served on the same underlying data infrastructure?
Philip Zelitchenko (05:13.627) Mm -hmm.
Philip Zelitchenko (05:24.443) So I think it's a perspective question. The way I see it is it's a unified stack. At the end of the day, the fact we are, I think we're sort of being a data company and being leaders as a data company, we have the privilege of dealing with a lot of data challenges across the board. And we are working on big initiatives. ZDP is one of them, basically the Zoom in for Data platform that's being led by a few very talented folks that come with a lot of experience of doing these things. And then the question becomes, OK, so we have ZDP. We have our EDP, which is the Enterprise Data Platform. How do we marry those together? And I think the vision is that we take this EDP as being our main platform that we're going to be utilizing and expanding it with EDP capabilities. So for example, if you need this type of tool for this type of use case, for this type of latency, this type of freshness, there's no reason why.
Philip Zelitchenko (06:23.419) EDP shouldn't be just embedded into ZDP. So in the long term, what EDP will become as part of ZDP and all the great work that the team has done to build the different components will be embedded into that great offering that will be utilized for many purposes, both external and internal.
Eldad (06:39.426) The lines are blurred and it's a good thing. It's a good thing.
Philip Zelitchenko (06:47.931) Yeah, I'm a big fan of adversity in general and being Israeli and Russian probably bad marketing, but that's who I am. And I think adversity grows interesting things and we don't have adversity here. And I think we partner very well with all these teams and we work very closely and we help each other out. But I think these different views on how to build things correctly give us the ability to bring to birth outcomes that are much better than most organizations that I worked for in the past. In the past, I've been in other organizations where there wasn't a lot of communication or a lot of adversity, if you'd like, about how to build things correctly. And the outcomes were usually suboptimal versus here. I feel like the fact that we have so many data experts, back to my point about the CHRO writing Python scripts or other people across the org doing data.
It allows us to make sure that the path we're going is the right path and it's a democratic state so people can contribute and make sure that we're not missing anything.
Benjamin (07:51.79) Nice. So let's talk a bit about data as a product, right? And you mentioned it earlier, like this kind of graveyard of components. Tell us a bit more about your thoughts on that, right? Kind of things being built or never being used and so on.
Philip Zelitchenko (08:05.435) Yeah, the way I see the world, at work at least, is that there's people, process, product, and tech stack. People, and there's a reason why I say those in that order, is because I think that's the way I think about the priority around the company and how I think about how we do things. So the people have, if you invest 10 % in the people, you're going to have a return of 50%, 80 % on that investment. You put 10 % in process. you'll probably have 30 % to 40 % return. You put invest in the product, you'll probably have 10 % to 20 % return. You put it in the tech stack, depends on the tech stack. Usually it has almost no return unless you fix the things up top with the people in the process. Because all products have a great prop or value prop, but it almost always fails on the people in the process side. That said, if you think about.
Data products in general. And we can define data products in a sec. Let's call that for now. Data products are basically a presentation of time and effort that was being put by humans in an organization. So if you sit for, and I'll give you the simplest example. Let's talk about a dashboard that you've built. And I see the dashboard as a data product. If it took three data analysts and two data engineers to build something
Philip Zelitchenko (09:33.755) for someone to consume and the Dow or the Mao or the Wow of these data products is zero, that means that you take the salaries of the people that we talked about, multiply it by the amount of hours and you get the amount of dollars that were invested and ROI of zero.
Eldad (09:54.754) There's a lot of effort put into PowerPoint presentations coordination meetings project management Decision -making leadership everything to get that dashboard up and running and be successful Just to get to what you say. So
Philip Zelitchenko (09:54.811) So if you take that.
Philip Zelitchenko (10:10.139) Exactly. So take that phenomena, multiply it by the amount of dashboards in the Tableau graveyard into the tables in the warehouse graveyard, into the ML models that are running or bad jobs that are running. Take that and multiply it and you get a dollar value. That dollar value is going to be very high. And that's something that we ignore because usually we have the fortune to work.
Philip Zelitchenko (10:39.323) at tech companies, which I'm so happy that I was born at this time in history and not 300 years ago where I'm sure I'd be unemployed or not sure what I would be doing. We have that benefit and because we move fast and we try to move forward, things, we don't have a back -view view. We don't do a lot of postmortems. So we continue throwing things behind the back and continue moving forward. But I think what's interesting about it is if you step for a second and look at the back, you understand that something isn't working well in general. And what's not working well is how do you invest the time in the right things and make sure that what you're building is right and is going to work. The same way you don't build products moving away from the data product world to the real product world, you go through a set of validations, you evaluate. There's more rigor about how you invest your time. But for some reason, when we talk about data products, it's sort of treated as free.
And I think that paradigm shift will happen in the next few years, where this will become something of the past. This would be an absurd, the fact that we invest so much time in things that are not being used.
Eldad (11:48.514) I've heard that about PowerPoint presentations and PDF files and I'm still getting those. I saved dashboards as PDFs by the way. So you can actually read it. But I completely, I'm with you on a lot of stuff you're saying. But sometimes, you know, people get religious over dashboards. But in reality, it's just people translating decisions into a nice presentation. Sometimes the purpose of the dashboard is really to be viewed once because of the effort that was put to behind the scenes to get the dashboard up and running, the cleansing, the modeling, the picking of the right stack. So the dashboard is really the least important thing. As you say, it says you move on, you just evolve and dashboards are just, right? Like those points in time where you did a snapshot of your data, you know, your data capabilities.
So, you know, it's okay, create wrong dashboards. As long as your model evolves, as long as your team gets smarter with the data, make faster decisions. And it's hard, it's really hard. So you're right, in many cases, the dashboard becomes the project. And I think that's when you know something is really wrong.
As you said, it's not the dashboard fault, it's the people behind it and the intentions that drive that dashboard. And I've seen some nasty things in my career, but always good intentions. So yes, I contradict myself and our theme here to be really nasty on dashboard and to go to the graveyard, pick them out, kill them again. But we don't want that. We are good with them.
Philip Zelitchenko (13:24.699) Yeah, yeah.
Eldad (13:39.842) Benji, what's your thought?
Benjamin (13:42.35) So one thing I'm curious about Philip is like, when you're talking about data product management, and you're saying, right, like for internal dashboards, for the CEO or whoever kind of, we consistently create things that have no ROI. Now you're at a kind of data product company. So in a sense, for all of the, many of the data projects you're doing, the ROI is easy to measure, right? Like kind of Zoom info is a public company, kind of you earn money, kind of off. at the dashboards and data experiences you create. Like, why is that going better? Cause in a sense, I guess you're advocating for all of the processes you have around taking data experiences to your own customers, also mattering then internally for any organization that kind of have large amounts of data.
Philip Zelitchenko (14:31.643) Yeah, and I think it's a good question. But again, I think our company is unique in many ways. Most companies in the world are not data companies. And most companies, the data function that they have is a corporate data function that serves the internal product. In some cases, you'll have some sprouts of data that is ingested in different areas of a product. But most companies in the world are in the world where the data function mainly serves internal stakeholders. And the reason like any other company, that function is something that I think needs to be stabilized and built in the right way to make sure that you invest the time of these resources in the right place. And so back to your question of why is it, I think your question was why is it so important to me? Because at the end of the day, the fact that our product team works and thinks about things as products is very dependent on the fact that
Philip Zelitchenko (15:29.435) It is a product and engineering team versus data analytics teams today are not managed as product teams. Most data analytics teams in companies that I've seen that are internal data analytics teams, they don't have a product manager that thinks about it. Mostly it's business stakeholder to engineer. That's the first relationship that you get. And that's, I think, one of the main reasons that that graveyard that we've talked about is so big is because we don't think about what we build as products in those use cases.
Philip Zelitchenko (15:59.003) We think about them as stopgap solutions. And if you look at the most of these stopgap solutions, what happens a lot of time is that we recycle things that are in the graveyard, but we are not even aware of it. And the reason why is because the requirements, thinking about how it's going to be utilized, how it's going to drive value, all these questions are usually being run by the business. And unfortunately, business people, they are amazing at a lot of things, but they're not product managers. They're not thinking about, OK, how do I build an experience? For my internal team to ensure that they can utilize data in a smart way that will help to drive outcomes, that will help drive ROI, that will help us upsell or cross -sell our customers or reduce turn on our customers or save costs around the org. These things are not part of their day -to -day thinking. It's mostly a transactional thing that they do as a side job. And usually it looks like a side job. That's why the graveyard is so big.
Benjamin (16:56.27) But calculating like your data ROI does seem much easier in a company like data product company, right? Cause like you sell the product for X amounts of dollars. Like it's easy to calculate this. If I give a dashboard to some executive and kind of she makes a better decision because of that becomes much, much harder to measure this. So how do you think that kind of transfers, right? Kind of from the world you have at zoom info to maybe some company where it's mainly internal.
Philip Zelitchenko (17:26.427) And I think that's a great question. That's part of what you do when you write a DPRD, when you write data product requirements document, you think about how do you measure the impact of what you do. And in a lot of cases, you don't. What happens in most cases, in most companies that I've worked for, have you heard of the HIPPO?
Eldad (17:38.498) Thank you.
Benjamin (17:42.958) I haven't heard of the hippo, no.
Philip Zelitchenko (17:44.603) Okay, so the HIPPO stands for the Highest Pending Person Opinion. That's HIPPO. And usually the HIPPO rules. In those companies, the HIPPO is the way things are decided on. Don't measure the HIPPO because the HIPPO is the HIPPO. And what happens is, if you don't put any measurements around the HIPPO, and you don't, and forget about the HIPPO itself, the HIPPO is just a representation of it's, yeah.
Benjamin (18:07.47) Measured a hippo, measured a hippo.
Eldad (18:09.73) Hippo!
Philip Zelitchenko (18:12.283) So the symptom, this is just a symptom of what I'm trying to say is when you work on a product, we're trying to release a product, you're going to think about the metrics, how you're going to measure the impact. You're going to think about maybe I'll run an A -B test, right? Maybe I'll see if I'm building a way to improve my sellers, I'm going to run an A -B test between sellers that can use my insights or not and evaluate that. These practices have never been something that the go -to -market teams or the business teams in companies thought about. Because they try to move fast. And moving fast meaning creating a larger graveyard with less thoughts around, what is the framework to do this? Instead of doing 50 projects that are trying to micro -optimize each step here, is there a framework we can build that will help run these more efficiently, but also will be able to measure them, measure the impact of them, and will help us make sure that we're going the right direction?
Benjamin (19:03.918) So this role, you would basically embrace the kind of flows and processes you have at these data product companies kind of much more widely internally as well in organizations that aren't first and foremost data product companies. Like to me, that makes sense.
Eldad (19:14.082) By the way, there are many organizations that will never be data first because they're just selling something else and they will always have a different exposure to data. And most importantly, from a cultural perspective, those companies learn to make decisions.
Eldad (19:42.658) almost completely with humans. So when they go and sell a car, it's the agent that decides the discount and how based on the, I don't know how much the customer blinks. And that's hard. And that's kind of goes way back. But when you're building a data company, you have, you remove so many limitations and you can rethink a lot of the go -to market.
Philip Zelitchenko (19:58.235) Okay.
Eldad (20:11.362) that you would apply, right? Like so many things, little things that you've mentioned when you talked, like, yes, you just change the product you're selling. And of course, anyone, you know, anyone in the services business would say, you can't just have everyone being in the service business. Someone needs to drive those people to work. And, but I think, yes, looking forward and how information workers transition.
Benjamin (20:33.39) Okay.
Eldad (20:41.314) from being consumers to making decisions which they're not with dashboard and this is what you're saying having a dashboard is it's not a trophy okay it's like a spreadsheet it's nothing more it's like whatever um but transitioning information worker it's basically your same information worker got it wrong and this all the stack we've did to serve decision makers right like what is bi what is self -serve?
Eldad (21:09.826) It's like a thousand different ways of doing the same thing, making less mistakes. And we're getting to square one. So data companies and data is a product, is the future across many industries and we're seeing it. So you better work or plan or be somewhere in that ecosystem. What else do you think will happen?
Eldad (21:36.002) As we move forward as we'll have less interfaces to do with right? I mean what you're saying is there is no looker interface in a few years from now Nobody's doing that. So what do you think will happen? How will you sell your data products? Dashboards last for 20 years. Believe me. I know
Benjamin (21:36.686) Remind, remind me, three years is look at that.
Philip Zelitchenko (21:56.827) I think, so I agree with a lot of what you said. I think the big thing that I think will happen is dashboards currently are transactional products that are not meant to stay in the form that we think about them. So that takes me back to the Dow, the Mao, the Wow. You look at these things on these metrics, on these assets, and you see that they're not meant to stay.
Eldad (22:06.178) But what if your AI call pilot generates that dashboard which
Philip Zelitchenko (22:25.115) because it always declines with time. Well, it...
Eldad (22:32.29) between us, right? Like this is kind of first feature for a co -pilot. Isn't that just yet an animated PowerPoint with the right data? So the dashboard itself, as we said, is not the point. It's the journey, the model, the cleansing, everything that happened to get there, but then generating the dashboard is free. It's like a PowerPoint. You get it for free, but it's everything else that matters. So.
Philip Zelitchenko (22:52.571) That's a good question. And I think that what is the time horizon we're looking at? Are we looking at 50 years from today or five years from today?
Eldad (22:59.426) Will you change your perspective on dashboards? And that's my question. Will you change your perspective on dashboards when they become yet another file format? So on Wired, unfortunately, it's 12 months in reality, I guess 15 years.
Philip Zelitchenko (23:22.331) Yeah. Yeah. And if you think Elon Musk would say it's going to be ready by end of year, right? So I think at the end of the day, I think the timeline is much longer than that. And I think in the meantime, what will happen is instead of creating tens or hundreds of dashboards that are going to be consumed by people at the organization.
Philip Zelitchenko (23:51.707) Yeah, no worries. So I think what will happen is that in the short, medium term, we'll move from dashboards to data apps, which I think is going to be the new world. Because when you think of building a data app, you're going to start thinking about the experience of the people who are going to access the data app. And now it's getting closer to a product rather than a transactional object that is there to serve you for 30 minutes for a meeting. And when you start thinking about it, you will start thinking of frameworks.
Eldad (24:18.562) and I'm going to be talking about that in a minute. Thank you.
Philip Zelitchenko (24:38.524) So that will be a centralized place, one central place that people can consume. But that's not the only one, because from an activation perspective, some of the data will flow into a data app. But some of the information will flow into your systems of engagement, where you are trying to drive a certain behavior, where there it's going to be not just another field in Salesforce or another field in whatever system you'd like, but it's going to be driving a workflow in that experience that will drive that workflow for the
Philip Zelitchenko (25:06.748) for online practitioners using those systems of engagement. And I think the activity.
Benjamin (25:08.174) What's the, I never heard the term system of engagement. Like what's the actual like textbook definition on that for someone who doesn't know.
Eldad (25:10.882) Absolutely. This is.
Philip Zelitchenko (25:18.588) So there's two systems that usually people talk to about, system of record and system of engagement. System of record is basically a place where you maintain information. So if you think about Salesforce today, removal of the Salesforce marketplace and all the things that connect to it, it is a system of record. People go there and put in notes or put in details and so forth. A system of engagement is a system where you log in and you engage with it.
Philip Zelitchenko (25:47.324) to make actions. So it's not just a place where you drop data in, it's a place where you activate different initiatives or you send outreach as an email approach that SDRs use to send out emails in bulk, define workflow. It basically helps automate a lot of the work. Slack is a system of engagement exactly in some sense. Some things can be both, right? So it depends on what basically the product is.
Eldad (26:05.282) Slack.
Philip Zelitchenko (26:16.06) but usually refers to some product, not data product, product, that has qualities of either writing, or writing and reading, or writing, reading, and acting upon whatever is happening in that system.
Eldad (26:29.666) And it's smart, you know, Salesforce, being Salesforce, being smart and being a great company. Like you just described like outladed for everyone, right? Like build a system of record and then we translated it into a system of engagement. So they acquired Slack, they acquired Tableau. They give any way to engage data through the dashboard. Um, that didn't go as well as planned and they got Slack.
Eldad (26:58.594) to have the other kind of engagement, which could they got really well, way beyond what they expected up to the point where this is kind of becoming their operating system of engagement. Thanks for playing us like those two things are crispy for us.
Philip Zelitchenko (27:16.892) Yeah, and I would also add that basically if you think about products in general, some of them are a pool, some of them are a pool and like reactive, proactive, pull and push, whatever you want to call it. I think the data apps will become a pool method where you go and you consume. And the activation layer or the system engagement layer would be the push where you set up automations that are going to be assisting you to help productivity across the different functions in your org.
Philip Zelitchenko (27:45.596) to make sure that you're doing your job in a good way. Because at the end of the day, with all of the things that we can do as a single front -line practitioner at a company, from a seller to a marketing person to a CSM, you can't really do your job in an amazing way without the assistance of some automated processing in the background. Because currently what happens is the method is you call someone, you write notes, you call someone, you write notes, you call someone, you write notes. That is not an effective operating model.
Philip Zelitchenko (28:15.228) And it's probably the productivity, if we had a measure on productivity of human beings and companies, that productivity metric would be pretty low because of of. Yeah, because low interest money is free, so you can throw bodies at any problem and then that's how you solve it in a world where interest is high. You need to find ways to be more creative in some sense.
Eldad (28:26.114) But it worked well with low interest. It worked well with low interest.
Eldad (28:42.594) Exactly. Good to be living in high interest rate times.
Philip Zelitchenko (28:48.572) Yeah, rise efficiency.
Philip Zelitchenko (28:56.316) The Industrial Revolution would probably come much faster in a world where we had a lot of high interest times.
Eldad (29:11.458) Benjamin, I can see you're circulating some thoughts with yourself. So feel free to share it with us.
Benjamin (29:15.822) And I'm still like...
Benjamin (29:20.494) No, so I'm still kind of trying to, to wrap my head around kind of your overall perspective on, on kind of the, these like data pipelines and so on, Philip, right. So first of all, I need to understand what system of engagements are and figure out how this fits in my mental model. I think my question kind of in the bigger scheme is right. Like when you're talking about, okay, people writing notes into Salesforce, kind of keeping track of all of their interactions with customers and so on, and then needing more efficient data products and data pipelines in order to become more effective.
Does this to you also then fall under projects actually owned by the data team within the company where you want to embrace all of this kind of then product workflow around figuring out ROI, kind of figuring out kind of requirements and so on, or is this more like, okay, there's going to be specific products emerging around this that actually make this easier? Like, do you feel like this kind of business function will be built more in -house in the future by data teams? Because it's very specific to the business or people will just buy it from like Salesforce 2 .0.
Philip Zelitchenko (30:31.388) It's a good question. I think it'll be probably a hybrid. There's going to be more tools. And we'll have a new infographic coming out every month showing how populated the whole industry is. And these tools will solve some use cases. But the biggest problem, or one of the big problems, is that as these set of tools continue to grow, the more discrepancy you have in your operating model, for personas that are working on a certain problem. At the end of the day, the number of tools per FLP, per frontline practitioner, has grown, I don't know, 300x over the past 10 years. So when I use, not me, but on an average seller used to sell 20 years ago, they had one system. Today they have 40. And that's true not just for a seller. It's true for an SDR, an MDR, an AE, marketing manager, a data engineer, a software engineer. You choose. Yeah. Yeah.
Eldad (31:35.746) phones, your phone suddenly, you can download as many apps as you want to them. So yeah, I mean, it's interesting to see how that will end and how far that will actually go because it was driving us really nicely, right? All of us kind of having that tool productivity mindset going from using one SAP with 50 apps in it with negative productivity to having that interconnected, well -behaved ecosystem with humans to connect the dots, which I think is much better than a glued single tag stack. But I don't know, right? Times change, things look at snowflake now, right? Like I remember people used to laugh at Oracle for trying to do more than just building a database.
So now it's okay to have a CRM. It's back okay to sell CRM with your database. So it's fashion, you know, it goes in cycles. You never know. You just need to focus, as you said, on great products, finesse and have teams that love the products they build and iterate fast, especially if you're in data. No egos. It's not needed because mistakes are almost practically for free.
Philip Zelitchenko (32:53.276) Yeah, I agree. Yeah.
Eldad (33:01.794) And so you don't need that ego in the room to protect the project, to have defense balance, nothing. No, it's like a query. It's like a model. It's a right. I think listening to you at the beginning, that is what I took. It's like, it gives me flexibility and freedom. So I can approach my team. I can lead my team differently. Maybe we can all build better stuff. Boom!
Benjamin (33:31.822) So maybe pivoting a bit and then kind of also generative AI, right? Like obviously kind of that's on everyone's mind. And I did want to kind of bring it up today because I think you have a really interesting perspective on that because one, you're using data infrastructure so heavily, right? And like systems like Snowflake have Cortex now and all of these generative AI features to deal better with semi -structured data and so on. At the same time, you're also thinking a lot about.
How do paying customers interact with data? Because this is what your business does in the end. Where do you see generative AI fit in? Both in terms of the internal data stack as a company, but then also in terms of how customers and users interact with data as a data product.
Philip Zelitchenko (34:19.868) It's a good question. So I'll split it into two. I think let's talk about ZoomInfo as a company, because I think there is some exciting work that is being done. We're working on releasing something we call Co -Pilot. ZoomInfo Co -Pilot is going to be basically a way to interact with your data and act on it in a more of a natural language processing way. And I think there's some exciting work that is being done currently in the R\&D and product org that will be exciting for a lot of our customers in a few months. And in that world, basically, the approach of write versus writing or reading, all that will be a bit easier, or I think much more easier from a productivity standpoint. If you think about Microsoft Copilot, I think there is research showing 15 % to 20 % to 25 % improvement in productivity of writing code.
I think a similar thing is coming from ZoomInfo for our ICP, our ideal customer profile. So our AEs, our SDRs, MDRs, they're going to have a very high benefit from interacting with a co -pilot that is going to basically help automate a lot of the things that currently they spend a lot of time doing. And there's some interesting research coming both from Forster and Gardner about how much time on average people spend on maintaining tasks that can be basically automated using GenNI. So I think there's great work that's coming from there. On the other side, I think for the.
Eldad (35:55.586) Oh man, what's gonna happen when everyone stops moving stuff around and start thinking? Because you don't need to move stuff around anymore.
Benjamin (36:02.67) So now that Eldad interrupted you already, I might as well follow up with another question. Having experience now taking these products, at least on their way to production, what's actually the hardest part? What's the biggest engineering challenges you have there? Because I think a year ago, everyone was just talking about these LLM companies and, oh, which LLM will win. That debate is kind of.
Philip Zelitchenko (36:04.316) Yeah, I think it's exciting. I agree with you.
Eldad (36:08.418) Mmm!
Benjamin (36:32.174) Over like as someone taking such a data product to production now, like what's the hard engineering about it?
Philip Zelitchenko (36:39.708) I think there's a few challenges there. I'm not sure I'm the right person to answer that question here, just because that's not my realm of responsibility. But I think, in general, that if I had to classify it into categories, I would say A, which is something that we're all probably aware of, the hallucination problems that are happening, how do you solve them, and B, I think, scale. So I'd say these are the two things. And the third one, which is hidden under scale, is cost. So combination of those three factors, but again, I wouldn't call myself the expert here on those areas.
Benjamin (37:11.022) Sounds good. So then kind of backtracking to my original question around kind of these, like you wanted to split your answer in two and talk about the second dimension now as well.
Philip Zelitchenko (37:21.98) Yeah, I think the other piece is how do we not make Gen .AI the new graveyard pool for similar to Snowflake and Tableau and other areas, right? Because at the end of the day,
Benjamin (37:31.854) The data engineering show season 55.
Eldad (37:31.874) You know what's more important? How do we make sure dashboards don't become the best thing that everyone does because everything else is just too expensive and just people go and say, let's just create a dashboard and run it for a few dollars and that's it.
Benjamin (37:47.982) Yeah. 20 years from now, we'll be sitting here again and Philip, you'll tell us about the graveyard of generative AI kind of products you've seen.
Eldad (37:55.938) I'm sorry.
Philip Zelitchenko (37:58.556) It's already being populated by bodies of things that are trying to be built. Every time I see an announcement of Microsoft or OpenAI, I basically see a whole industry just getting wiped. It's like a Marvel movie where basically someone...
Eldad (38:00.546) Who's?
Benjamin (38:03.598) It's filling up.
Eldad (38:05.634) of lost souls.
Eldad (38:18.306) It's crazy.
Philip Zelitchenko (38:25.5) snaps the fingers, and a third of the population disappears. But joking aside, I think in a similar sense, what will happen is that now there's this hype around what can generative AI do. And I think the Kruger effect, which I don't know if you guys heard of it, but you probably have, basically a very famous curve where there's the first hype, and then it goes down, and then it continues growing in a more, I would say, in a less steep way. And what happens in those areas, I think we're still on the first hype cycle where people are saying, what is your AI strategy? How are you going to solve world peace with AI? And all of the buzzwords around that. And I think soon enough, the tide will go down and we'll see who's swimming without any underwear.
These people walk out of the room and we'll stay with things that are really beneficial and help us do things better. So I think that would be the... Exactly. So and we're also...
Eldad (39:29.698) I don't want to compete. I don't want to compete with you guys on the data product on your domain. But I love, I love the, I love the attitude. Like really I love everyone who, every engineer who love what they did and love their peers and wake up every morning and love doing it. So thank you for.
Being an inspiration on that for sure. And there's the graveyard. Yes, there is that slide. And, and in our domain, it's changing actually much faster because there are no constraints. So yes, we might wake up in a few days, weeks or months. And that will be really a restart for, for, for data stack. And when that happened, it's good. That's what we do. But you're still in the data warehouse always, always we need a data warehouse.
Philip Zelitchenko (40:24.156) Yeah, no, I agree.
Eldad (40:29.346) Other than that, I don't know. Other than that, I don't know.
Philip Zelitchenko (40:32.54) I agree 100%. And again, what I'm trying to say is that I'm not trying to aim for a world where there's no graveyard. The graveyard is important. I'm just saying that I think we currently don't measure how big the graveyard is or what is the ratio between people who are alive versus people who are dead. Like humanity is trying to improve life expectancy, lifespan, health span, and how we're doing things better. I think in the same way, we need to be more thoughtful about what we build.
And how do we ensure that the ratio of things that go to the graveyard versus stay in live are being thought through? And the way to do it is to start understanding what exactly, what is in that graveyard and why is it there? How do we ensure to reduce the amount of things that go into the graveyard?
Benjamin (41:15.534) I think that was the most inspirational outro we ever had on this show, Philip. That was great. It was so good having you on today. Really, like I learned a lot. It was so good hearing your perspectives. Thanks a bunch. Any closing words from you other than your perfect words just now.
Philip Zelitchenko (41:38.268) No, just thank you again for hosting me. And again, I think as Eldad mentioned, which I want to echo, no one really knows what's right or wrong. We can be, we're almost around us. There's a lot of smart people. And I think remembering that and just coming open to conversation to hear other thoughts and argue and make sure the right decision are being made, I think that's the key for success of all of us. So yeah.
Benjamin (42:06.862) Thank you so much, Philip. Awesome. Cool.
# Implementing Explicit Multi-Statement Transactions in a Stateless, Cloud-Native Architecture (/blog/implementing-explicit-multi-statement-transactions-in-a-stateless-cloud-native-architecture)
### TL;DR [#tldr]
Firebolt supports explicit, multi-statement transactions using the familiar BEGIN, COMMIT, and ROLLBACK syntax. You can now group DML, DDL, and DCL statements into a single, atomic operation that maintains full ACID compliance while preserving the value of Firebolt's stateless architecture.
These transactions can span different compute clusters, provide snapshot isolation, and continue running even during compute cluster replacements or upgrades, with no downtime. This article explains the architectural decisions that make this possible, focusing on how Firebolt's centralized metadata service enables this capability
Syntax:
```sql
BEGIN TRANSACTION;
-- Read data to inform next steps
-- SELECT ...
-- Perform data modifications (DML)
-- INSERT, UPDATE, DELETE, MERGE
-- TRUNCATE, DROP PARTITION
-- Modify the database schema (DDL)
-- CREATE / ALTER / DROP TABLE/DATABASE
-- Manage permissions (DCL)
-- GRANT / REVOKE ...
COMMIT TRANSACTION;
```
### Why You Need Multi-Statement Transactions [#why-you-need-multi-statement-transactions]
Grouping multiple INSERT, UPDATE, and DELETE statements into a single atomic unit is fundamental to data integrity. Without this capability, partial failures can leave your data in an inconsistent state.
Consider a common ETL/ELT pattern where you update a fact table and its related dimension tables simultaneously. All updates must succeed or fail together to maintain consistency across your data model.
An even more demanding scenario is blue-green deployment: replacing a production database or table with a new version from staging. This involves a sequence of DROP and ALTER RENAME TO commands that must execute atomically. Partial success could leave your production environment broken.
In production-critical data analytics, atomicity isn't optional, it's essential. The challenge is providing these guarantees within a stateless, distributed architecture.
### Decoupling transactions from compute clusters [#decoupling-transactions-from-compute-clusters]
Traditional data platforms often tie ACID transactions to a single, stateful compute node. While familiar, this approach creates rigidity that can't handle modern, distributed workloads. You sacrifice elasticity and resilience because a single point of failure can disrupt your entire workflow.
Firebolt decouples transactions from the compute layer. Transaction state is managed centrally by the metadata service, allowing transactions to be long-running and resilient to infrastructure changes.
A single transaction can seamlessly span compute cluster replacements, scaling events, or [online upgrades](https://www.firebolt.io/blog/engines-online-scaling-and-upgrades) without interruption or downtime. You can perform complex, multi-step transactions without worrying about the underlying compute cluster infrastructure.
This is a unique capability in cloud data platforms, providing integrity and control that works with, rather than against, a stateless, cloud-native architecture.
### Why Firebolt uses centralized metadata architecture [#why-firebolt-uses-centralized-metadata-architecture]
This table compares different approaches to transaction management in distributed systems:
### Stateless execution with metadata-coordinated transactions [#stateless-execution-with-metadata-coordinated-transactions]
Firebolt's approach to explicit transactions leverages a centralized metadata service that is decoupled from compute clusters. This service acts as the single source of truth for all transaction states and content.
When you issue a BEGIN command, the metadata service provides a unique transaction ID. This ID serves as the identifier for an isolated workspace in the metadata service where all metadata changes, followed by INSERT or UPDATE operations, are staged.
These changes are initially available only to your session using that specific transaction ID. Once you commit the transaction, the metadata service atomically validates and marks the transaction as complete, making the staged data visible to all future requests.
This design ensures that compute clusters remain stateless and interchangeable, allowing transactions to maintain full ACID compliance even as they span different compuete clusters and resources. You never need sticky sessions or compute cluster affinity.
### Metadata service architecture and design [#metadata-service-architecture-and-design]
The metadata service stores all system information that isn't user data. This includes table and database structure definitions, access control permissions, and other system-level information. Your actual data is stored separately in a data lake. The metadata service only stores metadata pointers called "tablets" that serve as a map to where your data resides.
The metadata service is built on FoundationDB, a distributed, transactional key-value store renowned for its high performance, strong consistency, and ability to handle large-scale distributed workloads with millisecond latencies.
This service maintains all metadata transactionally in the order it was committed, allowing Firebolt to construct a complete picture of the database state at any given point. To ensure strict, chronological ordering between transactions, Firebolt uses a Log Sequence Number (LSN) that acts as a global, monotonically increasing timestamp for the entire database.
### Understanding the Log Sequence Number (LSN) [#understanding-the-log-sequence-number-lsn]
Think of the LSN as a precise timestamp that marks when each transaction begins and commits. It ensures transactions execute in a strict, predictable order.
Firebolt generates this LSN using FoundationDB's Versionstamp feature, which embeds the unique transaction's commit version into a key or value. This produces a textually sortable 20-hex-character LSN (e.g., 00000000f03ac8b10000) used extensively for transaction identification and for storing cache entries on the database compute cluster.
When a transaction begins, it receives an LSN (called begin\_lsn) that defines the "snapshot" of the database the transaction will see. This ensures your transaction starts with the most up-to-date data, including all data from transactions that committed before it.
When the transaction commits, it receives a new LSN (called commit\_lsn), which is guaranteed to be greater than any previous LSN. This commit\_lsn marks the point at which changes become visible to other transactions.
This process guarantees that commit\_lsn > begin\_lsn for any given transaction. More importantly, the begin\_lsn of the next transaction to start will be greater than the commit\_lsn of the previous one. This strict, chronological ordering prevents transactions from seeing or interfering with each other's in-progress changes, maintaining the isolation and consistency of the database.
**Building the transaction log**
To ensure atomicity and isolation, Firebolt creates a dedicated transactional workspace for every transaction, whether explicit or implicit. All changes, from a single row update to a new table creation, are logged to a Write-Ahead Log (WAL) identified by the unique begin\_lsn.
This log-based approach ensures data durability and acts as the building block for the entire transactional system. Changes written in a transaction are only accessible via the unique begin\_lsn, guaranteeing they remain completely isolated from other transactions until they are officially committed.
The commit process makes these changes visible to the rest of the system. Instead of rewriting or moving the data, the transaction simply receives a new, unique commit\_lsn. This commit\_lsn serves as a pointer to the begin\_lsn, making the changes permanently visible and part of the official transaction history.
This allows Firebolt to replay the transaction log in the correct commit order to reconstruct a snapshot of the database at any given point in time across all compute clusters.
To read the transaction log in the correct order, Firebolt first reads the list of ordered commit\_lsn values. Each commit\_lsn acts as a redirection to the corresponding begin\_lsn that holds the data, allowing the system to accumulate the content needed to construct a consistent snapshot.
**Key concepts:**
* The begin\_lsn defines a transaction's consistent snapshot, and its order reflects when transactions started.
* The commit\_lsn is a globally ordered and sequential number that defines the final, true order of transactions, used to replay the transaction log and reconstruct a consistent system state.
**Example:** A transaction with begin\_lsn 10010 can see and delete a table because it began after the table's creation was committed at commit\_lsn 10070.
**Another example**: A transaction with begin\_lsn 10005 can insert data into a table because it started after the table was committed. However, if that transaction is not committed, its data remains invisible to others.
### Indexing WAL for Optimized Read Patterns [#indexing-wal-for-optimized-read-patterns]
To avoid scanning the entire transaction log for every query, Firebolt maintains a crucial indexing layer on top of the WAL. These indexes quickly differentiate which committed changes are relevant to a given query, dramatically improving read performance.
The indexes are created by storing metadata changes, such as new or modified tablets and objects, in a separate, structured way. They are organized based on common access patterns (e.g., per-database or per-table) and sorted by their commit\_lsn.
This approach provides two main benefits:
**Single range read:** The underlying key-value store is highly optimized for reading a range of ordered data instead of many small reads. By using these indexes, a query can access all relevant log entries in one operation. This avoids the inefficiency of many small, scattered reads and reduces I/O and latency.
**Targeted access:** The indexes enable reading only what you need, allowing a query to request only the portion of the log relevant for specific tablets or objects, within a specific time frame. This means you can read only the incremental changes from a recent snapshot the compute cluster already holds.
These indexes transform the WAL from a simple append-only log into a highly performant and queryable data structure.
**WAL index structure**:
The WAL index sorts data first by the read pattern (e.g., database for reading schema, table for reading table data), then by commit\_lsn.
The index allows for a single, efficient read of a specific table, even within a given commit\_lsn range. For example, if an compute cluster already knows the table's state up to commit\_lsn 10112, it can request only the incremental changes in the range (10112, 10122] and apply those changes.
### Compacting the Transaction Log into Snapshots [#compacting-the-transaction-log-into-snapshots]
To manage the growth of the transaction log, Firebolt periodically compacts the recent transaction tail into a single, concise snapshot. This process creates a deterministic picture of the system at a given point in time, permanently applying all committed deletions and modifications and eliminating them from the log.
This compaction serves a crucial purpose: it allows Firebolt to safely remove old records from the transaction log and its indexes. After a snapshot is created and has passed a defined retention period, the log entries it replaced can be safely deleted.
This retention period enables features like time-travel, allowing you to reconstruct a system snapshot from a requested time in the past. The oldest snapshot you can use to reconstruct the system's state is determined by the retention period (also called the time horizon).
Since Firebolt regularly compacts older transaction logs into snapshots and then deletes the logs, you cannot construct a view of the system before this time horizon. This is why the time-travel feature is limited to the defined retention period.
A long-running explicit transaction is not a problem even if it runs for hours and tries to construct an "old" system state, because the data retention period is much larger than the transaction's lifetime.
## Isolation levels and conflict handling [#isolation-levels-and-conflict-handling]
In a system with parallel transactions, a conflict occurs when two or more operations logically contradict each other, potentially leaving the database in an inconsistent state. To handle this, Firebolt employs Optimistic Concurrency Control (OCC).
This approach assumes conflicts are rare, allowing transactions to proceed without locks. Conflicts are detected during a final validation step before a transaction commits. The key difference between isolation levels is the specific types of conflicts they prevent using this OCC mechanism.
### Isolation levels [#isolation-levels]
**Strict serializability:** For critical metadata operations (e.g., creating or dropping a table), Firebolt applies strict serializability, the strongest isolation level. The OCC system checks for both read-write and write-write conflicts to ensure all structural changes are globally ordered.
For example, a transaction attempting an INSERT will fail if the target table was concurrently dropped, as this is a read-write conflict on the metadata.
**Snapshot isolation:** For data operations (e.g., SELECT, INSERT, UPDATE, DELETE), Firebolt uses snapshot isolation. A transaction is given a consistent view of the data from when it began, regardless of concurrent changes, which eliminates read-write conflicts.
Therefore, the OCC mechanism only needs to check for write-write conflicts, aborting the transaction only if it tries to modify data that another concurrent transaction has already committed.
### How Firebolt enforces isolation [#how-firebolt-enforces-isolation]
To enforce the specific isolation levels described above, Firebolt tracks every transaction's activity using a system of objects called dependency tags. These tags are the key to detecting conflicts during the commit phase.
Here's how it works:
**Tracking during the transaction:** Following an operation such as INSERT, when a transaction performs a WRITE or READ of metadata, the compute cluster creates a simple record, a dependency tag. This tag notes the type of operation (read or write) and the specific data involved.
**Validation check:** When a transaction is ready to commit, the metadata service examines all the tags created by other transactions that have committed since this transaction began.
**Conflict detection:** The metadata service then checks for conflicts. For example, to prevent a read-write conflict, it checks if any of the data this transaction READ has been written to by another committed transaction. If it finds a match, the transaction is immediately prevented from committing.
This process provides a flexible and precise way to enforce isolation. Firebolt simply configures the system to check for the specific types of conflicts required by each isolation level, preventing write-write conflicts for snapshot isolation and both read-write and write-write conflicts for strict serializability.
### Example of read-write conflict validation [#example-of-read-write-conflict-validation]
Consider two concurrent transactions, T1 and T2, interacting with a table named sales\_data.
T1 begins an UPDATE operation on a row. It first reads the sales\_data metadata to confirm the table's structure. The compute cluster registers a dependency tag for a READ on the sales\_data metadata key.
T2 begins a DROP TABLE operation on sales\_data, which is a structural change. The compute cluster registers a WRITE dependency tag on the sales\_data metadata key.
T2 commits successfully. T2's WRITE dependency tag on the metadata key is recorded in the committed log.
T1 attempts to commit. The system checks T1's READ dependency tag against committed tags since T1 began. It finds a conflict with T2's WRITE tag on the same key.
T1 is aborted. Because T1's operation depended on a state that was concurrently changed, it cannot commit. This mechanism ensures the system's structural integrity by enforcing strict serializability for metadata operations.
Example:
In an explicit multi-statement transaction, you don't need to wait until COMMIT to realize there is a conflict. To improve your experience, Firebolt performs this validation after every statement rather than only at commit. This "fail-fast" behavior gives you earlier feedback on conflicts, saving time and resources.
### Controlling Sequential Execution Within a Transaction [#controlling-sequential-execution-within-a-transaction]
In a distributed system where any statement can be routed to any compute resource, ensuring sequential execution within a single transaction is crucial. A transaction depends on a consistent snapshot to operate, so allowing multiple write operations to run in parallel within the same isolated transactional space could introduce inconsistencies before the transaction is verified and committed.
For this reason, Firebolt explicitly restricts parallelism within a single transaction, rather than allowing undefined behavior like some other database systems.
Firebolt achieves this by generating a sequence\_id along with the transaction\_id. On every statement, the sequence\_id is atomically checked and updated in the centralized metadata service and then sent back to the client.
The client, when submitting a subsequent statement in the same transaction, must attach the correct sequence\_id in the HTTP headers. If the provided sequence\_id does not match the expected value, the transaction is immediately aborted.
This mechanism guarantees that statements within a transaction execute sequentially, maintaining integrity even in a highly distributed environment.
### Security Considerations [#security-considerations]
Security enforcement is real-time: Every statement in a transaction is individually authorized based on role permissions.
RBAC (Role-Based Access Control) is always checked against the most recent state of permissions, regardless of when the transaction began. This ensures that if your privileges are revoked mid-transaction, your next statement is immediately blocked. Access control is never stale.
#### Real-time Security Enforcement [#real-time-security-enforcement]
The crucial point is that permission checks use the current, up-to-date state rather than the snapshot state at the beginning of the transaction. While data operations within a transaction see a consistent snapshot defined by the begin\_lsn, security checks must always query the most up-to-date, globally committed state of the system's permissions.
This means that a separate read is performed on the permissions metadata for every statement to ensure up-to-date state. Even if a transaction has been running for a long time, its access is validated against the very latest security policies, preventing any staleness.
#### Atomic DCL Within Transactions [#atomic-dcl-within-transactions]
A powerful and unique feature of Firebolt's security model is its full support for Data Control Language (DCL) operations within transactions. You can create a new table and atomically GRANT permissions on it in the same transaction. This guarantees that security policies and schema changes are applied simultaneously, preventing any inconsistencies between access controls and database structure.
Just like other metadata changes (e.g., CREATE TABLE), DCL operations like GRANT and REVOKE are also written to the isolated transactional space. They are represented as specific objects in the transaction's WAL (Write-Ahead Log).
When the transaction successfully commits, both the new table and the corresponding GRANT are simultaneously made visible to the rest of the system using the commit\_lsn. This prevents a race condition where a table is created and visible, but its intended permissions haven't been applied yet, ensuring the system is never in an inconsistent state.
## Conclusion [#conclusion]
Firebolt delivers robust multi-statement transactions without introducing a stateful compute layer. The centralized metadata service, powered by an append-only Write-Ahead Log, ensures data durability and consistency.
This architecture allows for the atomic execution of DML, DDL, and DCL operations within a single transaction, a capability that sets Firebolt apart from platforms like Snowflake and Databricks. As a result, you can:
* Run transactional statements across different compute clusters
* Replace compute nodes mid-transaction without losing state
* Atomically update both data and schema in a single transaction
This approach preserves the core benefits of Firebolt's cloud-native design that optimises elasticity, resilience, and statelessness, while delivering one of the most powerful and highly requested database features.
# Implementing Firebolt MERGE Statement (/blog/implementing-firebolt-merge-statement)
The MERGE statement is a powerful SQL command that allows you to simultaneously perform multiple INSERT, UPDATE, and DELETE operations on a single table. It's a classic PostgreSQL feature that simplifies data synchronization tasks by
1. supporting complex conditional business logic, and
2. moving the entire data mutation operation into a single ACID transaction, thereby natively protecting against stale reads from frequently updated data sources
This blog goes deep on what it takes to build a scale-out merge operation that can run over terabytes of data.
## MERGE Statement Overview [#merge-statement-overview]
As an example, if you had a large table of students that you wanted to sync after the course registration period completed, thus making recent enrollment changes visible publicly (e.g. students that transferred in, that dropped out, or that changed their planned graduation year, etc), you could use MERGE to do so. Here's a toy example.
```sql
-- dated 990 A.D. upon school founding
CREATE TABLE hogwarts_students (
id INT,
full_name TEXT,
dob DATE,
graduation_date DATE,
owl_exams ARRAY(TEXT)
);
-- dated 1017 A.D. alongside new privacy and ethical recruitement reforms
CREATE TABLE public_hogwarts_students (
id INT,
full_name TEXT,
graduation_date DATE
);
-- running the sync from the staged source data to the public target table
MERGE INTO public_hogwarts_students AS t USING hogwarts_students AS s
ON t.id = s.id
WHEN NOT MATCHED THEN INSERT (id, full_name, graduation_date) VALUES (s.id, s.full_name, s.graduation_date)
WHEN MATCHED THEN UPDATE SET full_name = s.full_name, graduation_date = s.graduation_date
WHEN NOT MATCHED BY SOURCE THEN DELETE;
```
The MERGE statement works by comparing rows from a source (which can be a table, view, or subquery) with rows in a target table. Based on whether rows match or not, you can specify different actions:
* **WHEN MATCHED**: This clause defines actions to take when a row in the source matches a row in the target. You can specify UPDATE or DELETE operations here.
* **WHEN NOT MATCHED (e.g. WHEN NOT MATCHED BY TARGET)**: This clause defines actions to take when a row in the source does *not* have a corresponding match in the target. It is used for INSERT operations, in order to backfill the target table with rows from the source.
* **WHEN NOT MATCHED BY SOURCE**: This clause defines actions to take when a row in the target does *not* have a corresponding match in the source. This is used to UPDATE or DELETE rows from the target table, to clean up rows that no longer exist in the source.
Row "matching" is evaluated by an explicit JOIN condition. Traditionally, this is done over a set of primary key columns. However, any subset of columns from the source and target tables can be used, or any transformations therein.
You can find common use cases and examples of the syntax in our [documentation](https://docs.firebolt.io/reference-sql/commands/data-management/merge). For the remainder of this blog, however, we proceed with the low-level details of how we implemented MERGE.
## What MERGE Requires Beyond Existing DML [#what-merge-requires-beyond-existing-dml]
At a high level, MERGE needed to wrap together functionality that we already had. We already knew how to modify tablets using the logical representations of INSERT, UPDATE, and DELETE requests (physical tablet management is explained below). We already knew how to transparently modify aggregating indexes, alongside any affected table (these are the consistent materialized views that the planner uses to optimize aggregation queries: [documentation](https://docs.firebolt.io/overview/indexes/aggregating-index#aggregating-index)). We already knew, even, how to run multiple DML operations under a single transaction, since the UPDATE command reduces to a DELETE and an INSERT op. And, quite significantly, we already had battle tested distributed executions of massive joins.
So what was missing? Mainly, adding the parsing/planning layer to translate a user's MERGE query into our proven, scalable building blocks. While not breaking any of the existing code paths 😂. Let's dig into that translation.
## Ordering User-Provided Clauses [#ordering-user-provided-clauses]
While these terms are personal colloquialisms, they are useful for describing nuances of the MERGE syntax that have to make it into the implementation.
**Match category**: A set of clauses within which order matters. E.g. the *category* of MATCHED clauses, or the category of NOT MATCHED clauses, or the category of NOT MATCHED BY SOURCE clauses.
**Match clause**: A tuple of match category, optional conditional, and action. Several unrelated examples:
```sql
-- Delete action with the (default) catch-all condition
WHEN MATCHED THEN DELETE
-- Update action with a data transformation
-- Note the nontrivial condition
WHEN MATCHED AND t.cond > 5 THEN UPDATE foo = s.foo + 1
-- A different match category, with a no-op action
-- Note that conditions don't have to be meaningful 🤔
WHEN NOT MATCHED and FALSE THEN DO NOTHING
```
As defined by the syntax, ordering within each category provides **short-circuiting** (e.g. a row matching an earlier MATCHED clause condition will be sent to that earlier action, and will not be considered for any further MATCHED clauses). So within a single category, the order of clauses that the user defined is important.
But what if we look across different categories? Conceptually, it cannot be, for example, that a target and source row pair both MATCH and do NOT MATCH BY SOURCE. **Match categories act on non-overlapping sets of target-source row pairs**. Put another way, given a well-defined JOIN condition, every target-source row pair will fall into exactly one of the match categories, or into zero of them (in case the JOIN condition evaluates to false or NULL). Therefore, clauses from different categories will have no effect on each other's row sets. This means that it is irrelevant in which order the user defines the different categories, and makes no difference whether the user intermixed clauses across categories.
An aside: If we add to these observations the thoughtfully-limited number of category/action pairings that the MERGE syntax allows, and the errors that are thrown when attempting to update the same row multiple times, we get the following lemma: The MERGE syntax **guarantees that each row from the target table will be modified (updated/deleted) at most once and each row from the source will be inserted into the target at most once**. It is a nice cap on the maximal number of operations.
## Degenerate No-Op Scenario [#degenerate-no-op-scenario]
Let's discuss the base case before the inductive case: In the most basic scenario, a MERGE statement represents a DML no-op. It must pass parsing and be registered as a successfully run DML, however it adds or affects exactly zero rows. Whereas this may be discovered at runtime due to data distribution (e.g. the source and target were already in sync and there was nothing new to handle), there are certain degenerate MERGE queries that represent global no-op's and actually require special handling by the query planner.
These consist of all DO NOTHING clause actions. Consider this example, where the syntax is used in a valid though futile way. The user is literally not asking for any actions to be taken:
```sql
MERGE INTO target t
USING source AS s
ON t.a = s.a
-- single DO NOTHING branch
WHEN MATCHED THEN DO NOTHING;
```
So how do you translate a DML statement with no requests for action? Instead of defining a runtime primitive for a "non-action", we chose to catch such cases at the earliest convenience and provide a well defined DML plan. For simplicity sake, it is: INSERT INTO target SELECT \* from target LIMIT 0. And here's what the planner simplifies a LIMIT 0 down to: a local read from an empty static table.
The query plan generated:
```shell
[0] [TableModify]
\_[1] [Insert] target table: "target"
\_[2] [Projection] c_0: NULL, c_1: NULL
\_[3] [StaticTable] Columns: [][]
```
Neat! Note that distributing DO NOTHING actions across multiple categories or conditions still hits this degenerate scenario.
```sql
MERGE INTO target t
USING source AS s
ON t.a = s.a
-- Match category I:
WHEN NOT MATCHED BY SOURCE and s.name in ('Harry', 'Ron', 'Hermione') THEN DO NOTHING;
-- Match category II:
WHEN NOT MATCHED and s.color = 'red' THEN DO NOTHING
WHEN NOT MATCHED and s.iteration > 10 THEN DO NOTHING;
```
As an aside, you can also get "zero effect" queries via always-false conditionals. However, it is not the parser's job to catch those. So the following plan will still have a DELETE node in it, but note that the planner has collapsed any real scan of the source and target tables into, once again, a local "read" from an empty static table.
The query:
```sql
MERGE INTO target t USING source s ON t.a = s.a
WHEN NOT MATCHED BY SOURCE AND 7 < 5 THEN DELETE;
```
The plan:
```shell
[0] [TableModify]
\_[1] [Delete] target table: "target", tablet_row_number_column: 0, tablet_txid_column: 1, tablet_id_column: 2, source_node_id_column: 3
\_[2] [Projection] c_0: 0, c_1: '', c_2: '', c_3: 0
\_[3] [StaticTable] Columns: [][]
```
## Single Node Query Plan [#single-node-query-plan]
So what does the query plan look like for a MERGE statement with at least one interesting action? Let's try an INSERT action, where two local tables are simply generated and filled with 10 and 20 rows, respectively.
\[EXAMPLE 1] The script:
```sql
CREATE TABLE target (a int, b int);
CREATE TABLE source (a int, b int);
-- Populate 10 entries with keys 1,11,21,etc
INSERT INTO target SELECT i, i%17 FROM generate_series(1,100,10) i;
-- populate 20 entries with keys 1,6,11,16,etc
INSERT INTO source SELECT i, i%17 FROM generate_series(1,100,5) i;
-- Note that the source contains all the keys already in the target, and then some (specifically, 10 other rows)
-- Merge in new entries from the source, with an extra conditional based on the mod of the key space, and a transformation of some of the source values.
-- The condition is picked purely for demonstrative purposes. It happens to capture 6 of the 10 possible rows in the target-source row diff.
MERGE INTO target t USING source s ON t.a = s.a
WHEN NOT MATCHED AND s.b < 11 THEN
INSERT VALUES (s.a, 100 * s.b);
```
The executed plan:
```shell
[0] [TableModify]
\_[1] [Insert] target table: "target"
\_[2] [Projection] source.a, multiply_checked_0: (100 * source.b)
\_[3] [Filter] ((CASE WHEN (c_0 IS NULL and (source.b < 11)) THEN 0 ELSE NULL END) = 0)
\_[4] [Projection] c_0, source.a, source.b
\_[5] [Join] Mode: Left [(source.a = target.a)]
\_[6] [StoredTable] Name: "source"
\_[7] [Projection] target.a, c_0: TRUE
\_[8] [StoredTable] Name: "target"
```
What are we looking at? Going bottom up:
\[6, 8] At the bottom are the scans of the source and target tables.
\[7] Before we get to joining rows from the source and target tables, we introduce a constant boolean projection onto the target side, mimicking a **mark join**. Mark joins are an optimization technique that propagate, alongside every source row, the result of some condition in a temporary "mark" column. Sometimes, this row-wise evaluation, passed along to a JOIN, is actually enough to compute some subquery like EXISTS or IN, without continuing to propagate the full data set itself.
Firebolt doesn't have native support for mark joins, but we can utilize the technique by manually inserting this 'magic' constant column. The point is that we are about to compute the set of source rows that do not have matches in the target table, i.e. we really want a LEFT ANTI JOIN. However our join infra will return the full set of source rows, extended by either the matching target rows or by all NULL's. To distinguish a NULL value set because "there was no matching target row" from a NULL value set from "a target row consisting of all NULL's", we need an extra marker that will be TRUE alongside all target rows (even a completely NULL one), and NULL anytime there is no matching target row.
This constant (dubbed c\_0 in the plan) will be filtered against further up.
\[5] A left side outer join, which will return all rows in source, matched either with a row from target or with all NULL's.
\[4] A pass-through of all the source columns required for insertion, plus the target mark
\[3] A filter that encapsulates the NOT MATCHED BY TARGET clause and the additional condition that we passed. Note the use of c\_0 from before, to isolate only source rows without matches. It may look odd that the CASE statement in the filter is returning 0 and NULL, rather than true and false, but given more merge clauses, it will be classifying each row as belonging to clause index 0, 1, 2, etc, up to the number of clauses. More on that in the CASE WHEN section below.
\[2] The computation of the transform we requested on the data to-be-inserted.
\[0,1] The top level goal: we're modifying the table target by inserting some rows into it.
Ready to dive in deeper?
### Choosing the JOIN Type [#choosing-the-join-type]
Plain WHEN MATCHED clauses require simply an INNER JOIN between the source and target tables. Target rows can be updated or deleted based on conditions on the source row values, and only row-pairs with direct matches are candidates for consideration.
In the prior example, the NOT MATCHED BY TARGET clause requests a LEFT OUTER JOIN because it's looking to operate over source rows (and the JOIN inputs happen to be ordered as source table scan on the left, target table scan on the right)
More generally, NOT MATCHED BY SOURCE clauses will require the flipped RIGHT OUTER JOIN in order to be able to update/delete target rows.
If both the NOT MATCHED categories are requested - the JOIN type will be expanded into a FULL OUTER JOIN.
The JOIN type is dynamically chosen to the strictest one possible, given which subset of merge categories are requested.
### CASE WHEN: Stacking Conditions [#case-when-stacking-conditions]
Let's look at the CASE statement from that example's filter step \[3].
The MERGE syntax mandates short circuiting between different match clauses of a single category, and a CASE statement is exactly the primitive to describe that behavior. This is a PostgreSQL operator that enables describing a series of if-then-else statements. Once a row matches one of the CASE filter conditions, it will not be evaluated again. So in the previous example, we assign the value zero to rows matching the INSERT clause condition, and then filter for exactly those rows that were evaluated to zero. Not too complex. However, consider the case where there are multiple match clauses within a category, with different filters and even potentially different actions.
\[EXAMPLE 2] Given this query:
```sql
MERGE INTO target t USING source s ON t.a = s.a
WHEN NOT MATCHED AND s.b % 2 == 0 THEN DO NOTHING
WHEN NOT MATCHED AND s.b < 11 THEN INSERT VALUES (s.a, 100 * s.b)
WHEN NOT MATCHED AND s.b >= 11 THEN INSERT VALUES (s.a, 1 + s.b);
```
We want to route row pairs to four possible outcomes (fine for rows to fall through and land in the fourth and default "ELSE do nothing" outcome. At this point, the filter step will be elevated up (to sit below each of the INSERT operators), and the CASE statement will be clear within its own projection step. Note the assignment of a unique index to each of the possible outcomes, including the first DO NOTHING group:
```shell
[Projection]
source.a, source.b, multiIf_0:
(CASE
WHEN (c_0 IS NULL and ((source.b % 2) = 0)) THEN 0
WHEN (c_0 IS NULL and (source.b < 11)) THEN 1
WHEN (c_0 IS NULL and (source.b >= 11)) THEN 2
ELSE NULL
END)
```
Although previously mentioned as the recipe for "degenerate" merge queries, DO NOTHING clauses are typically quite useful. They are used to take out a subset of rows from consideration. Since match clauses are evaluated in order, having a DO NOTHING clause lifts a conditional from having to be repeated in every later clause.
This overall projection step shows that the result of the CASE is forwarded alongside the source data (dubbed multiIf\_0 in the plan). This allows DML nodes further up in the plan to filter on precisely the subset of rows that they are meant to handle.
### Loopback Shuffle [#loopback-shuffle]
So what does the plan look like for \[EXAMPLE 2]? With multiple different row subsets getting inserted (each with its own custom filter and transformation), it's getting a bit thornier to read. I have greyed out the sections that have not been changed - namely the topmost INSERT node and the base JOIN.
Observe that we are now working with a DAG-shaped plan.
```shell
[0] [TableModify]
\_[1] [Insert] target table: "target"
\_[2] [Union]
\_[3] [Projection] source.a, multiply_checked_0: (100 * source.b)
| \_[4] [Filter] (multiIf_0 = 1)
| \_[5] [Shuffle] Loopback with disjoint readers
| \_[6] [Projection] source.a, source.b, multiIf_0:
| \ (CASE
| | WHEN (c_0 IS NULL and ((source.b % 2) = 0)) THEN 0
| | WHEN (c_0 IS NULL and (source.b < 11)) THEN 1
| | WHEN (c_0 IS NULL and (source.b >= 11)) THEN 2
| | ELSE NULL
| | END)
| \_[7] [Join] Mode: Left [(source.a = target.a)]
| \_[8] [StoredTable] Name: "source"
| \_[9] [Projection] target.a, c_0: TRUE
| \_[10] [StoredTable] Name: "target"
\_[11] [Projection] source.a, add_checked_0: (1 + source.b)
\_[12] [Filter] (multiIf_0 = 2)
\_Recurring Node --> [5]
```
Going bottom up once again, beyond the base join of source and target tables:
\[6] is the projection discussed in the CASE WHEN section above, which is classifying every row-pair in terms of which merge clause it should be handled by, and forwarding that classification in the new column dubbed multiIf\_0. All the c\_0 IS NULL checks are coming from the fact that all three clauses are of the NOT MATCHED BY TARGET category.
\[5] Introduces what Firebolt calls a **loopback shuffle**, a building block that enables multiple operators to reuse the same input stream. Conceptually, this can be viewed as a form of "fork" operator which allows both \[4] and \[14] to consume the data produced by \[6]. In this case specifically, there are two Filters that are going to split the classified stream of row-pairs into sections, and pass only subsets of the stream up further to the Insert node.
Note that the "shuffle" keyword is usually reserved for distributed join or aggregation processing in which nodes need to send data to each other and hence data is "shuffled" across the network. In contrast, the loopback shuffle does not move any data off the node and simply broadcasts data from one local input stream to multiple local output streams. It keeps track of which blocks have been consumed by which output stream, and applies backpressure onto the input stream so that no consumer gets "too far ahead" or falls "too far behind". (It would lead to a memory usage explosion if one consumer was reading very slowly, whereas another was speeding ahead and asking to load the whole input stream into memory.)
At Firebolt, we use loopback shuffle for other DAG shaped plans, such as join pruning using a filtered primary index key set that is passed sideways from one side of the join to the other. More on this in a separate blog.
So now you can see why and how branches \[3,4] and \[13,14] are reusing node \[5].
\[3,4] Is filtering out the subset of row-pairs that match the second WHEN condition, and then preparing the data for insertion: WHEN NOT MATCHED AND s.b \< 11 THEN INSERT VALUES (s.a, 100 \* s.b). These are the rows that were assigned classification 1 by the CASE statement \[6].
\[11,12] Is filtering out the subset of row-pairs that match the third WHEN condition, and then preparing the data for insertion: WHEN NOT MATCHED AND s.b >= 11 THEN INSERT VALUES (s.a, 1 + s.b). These are the rows that were assigned classification 2 by the CASE statement \[6].
Note that there is no action branch to handle rows matching (multiIf\_0 = 0). These would be the rows caught by the first match clause of the query. As a DO NOTHING action, its purpose was precisely to classify some subset of rows for non-action. And so its execution is satisfied by the CASE statement assigning those row-pairs a unique classification which will not get picked up by any of the downstream filters. There is "nothing more" to do for those row-pairs.
\[2] The Union operator joins the two transformed streams into a single input for the Insert step. Handing the Insert operator the full data stream will let it best distribute the data across all newly generated tablets, thereby improving tablet quality.
\[0,1] Are the same top-level operators as before.
In a nutshell, while performing a table synchronization or a data deduplication task, the introduction of the loopback shuffle is the *key performance boost* that MERGE offers over running multiple separate DML statements. The fact that the base join can be done once and then reused by all the data mutation branches is crucial.
Now, let's see what the reuse looks like when there's even more action types requested. But before that, let's step to the side for a moment to refresh what it actually means to INSERT, UPDATE, or DELETE from a managed (internally persisted) Firebolt table.
### Tablets: What INSERT, UPDATE & DELETE effect [#tablets-what-insert-update--delete-effect]
At Firebolt, tables are persisted in units of "tablets", and this is what all DML operations ultimately operate over. A tablet is a collection of data and metadata files that are (for the most part) immutable. As data is ingested into a table, one or more new tablets are written and stored somewhere in cloud. At the conclusion of the insert transaction, our metadata service attaches those new tablets to the tablet list that defines the table.
The sole caveat for tablet immutability comes in the form of deletion vectors. Either through UPDATE or DELETE, a user may ask to delete some subset of rows from a pre-existing tablet (say all rows where column foo is divisible by 5). While we could immediately construct a new tablet that has only the rows to-be-kept-alive, that is often overkill, and we punt on that heavy copy operation until a VACUUM op is run. Instead, a deletion vector is attached to the tablet files (conceptually a bitset of which rows are still alive; implementation wise it is a RoaringBitmap). This bitset is then referenced prior to any future reads of tablet data, ensuring old rows are ignored.
If multiple mutation statements are applied to a single tablet, the previous deletion vector will be read each time, and a new one will be generated to reflect the net sum of the changes. The fact that there is only one trusted deletion version at any one time actually explains why it's important to have at most one Deletion operator per table per transaction. Just planting the seed here, but this turns out to be the root cause behind adding the UNION operators to the DML nodes in the MERGE plan in the first place.
To reiterate: other than this mutable deletion bitset concept, all files in a tablet are immutable. And DML operations operate over tablets. INSERT creates new tablets; DELETE marks subsections of a tablet as deleted, or drops tablets entirely; UPDATE applies a DELETE on all rows that are to be changed, and then inserts new tablets with precisely the affected rows and their new values.
### Multiple Action Types in MERGE [#multiple-action-types-in-merge]
Let's go all in. What if a user requests the classic "please sync my tables" MERGE statement? Insert the new rows, delete the old rows, and overwrite any duplicate rows. In other words, the source table is more up-to-date and we would like to trust it.
In the general MERGE query, of course, there could be multiple match clauses per category, and every clause could have further conditionals. We present the simplest case of one clause per category and zero extra conditionals, purely to demonstrate how the Insert/Delete nodes roll up.
\[EXAMPLE 3] Query:
```sql
MERGE INTO target t USING source s ON t.a = s.a
WHEN NOT MATCHED BY TARGET THEN INSERT VALUES (s.a, s.b)
WHEN NOT MATCHED BY SOURCE THEN DELETE
WHEN MATCHED THEN UPDATE SET b = s.b;
```
The executed plan, for readability having extracted the CASE statement like so:
```sql
multiIf :=
(CASE
WHEN (c_1 IS NOT NULL and c_0 IS NOT NULL) THEN 0 -- when matched
WHEN c_1 IS NULL THEN 1 -- when not matched by target
WHEN c_0 IS NULL THEN 2 -- when not matched by source
ELSE NULL
END)
```
```shell
[0] [TableModify]
\_[1] [Delete] target table: "target"
| \_[2] [Projection] target.$tablet_row_number, target.$tablet_txid, target.$tablet_id, target.$source_node_id
| \_[3] [Union]
| \_[4] [Projection]
| | \_[5] [Filter] (multiIf = 0)
| | \_[6] [Projection]
| | \_[7] [Shuffle] Loopback with disjoint readers
| | \_[8] [Join] Mode: Full [(source.a = target.a)]
| | \_[9] [Projection] source.a, source.b, c_0: TRUE
| | \_[10] [StoredTable] Name: "source"
| | \_[11] [Projection] target.a, target.$source_node_id, target.$tablet_id, target.$tablet_txid, target.$tablet_row_number, c_1: TRUE
| | \_[12] [StoredTable] Name: "target"
| \_[13] [Projection]
| \_[14] [Filter] (multiIf = 2)
| \_[15] [Projection]
| \_Recurring Node --> [7]
\_[16] [Insert] target table: "target"
\_[17] [Union]
\_[18] [Projection] target.a, source.b
| \_[19] [Filter] (multiIf = 0)
| \_[20] [Projection] source.b, c_0, target.a, c_1
| \_Recurring Node --> [7]
\_[21] [Projection] source.a, source.b
\_[22] [Filter] (multiIf = 1)
\_[23] [Projection] source.a, source.b, c_0, c_1
\_Recurring Node --> [7]
```
Having again greyed (\[8] - \[12] above) out the bottom most Join, as well as the topmost TableModify operator, I'd like to point out the high level hierarchy.
At the top, multiple types of DML operations (Delete and Insert), are now inputs to a single TableModify node (\[1], \[16]). This is what is enabling multiple types of data mutation to be run under the same transaction. The final query will succeed only if all children of the TableModify node succeed.
Both the Delete and Insert nodes happen to have multiple row-pair classifications sent to them, and hence are receiving Unions of substreams (\[3], \[17]).
Why are they both receiving multiple classifications, given the three original match clause actions? Because the subset multiIf = 0 , expressing the WHEN MATCHED clause, is an UPDATE action, which splits into a Delete and an Insert component. The subset multiIf = 1 has only an Insert component. And the subset multiIf = 2 has only a Delete component.
All these filters and projections are reading from a single base loopback shuffle \[7], and the base join is now a FULL OUTER (marked) join, rather than a LEFT OUTER (marked) join, because both target and source row values are required in order to execute all later branches.
Note also that there are now markers c\_0, c\_1 applied to both source and target rows, respectively, such that we can tell apart a source row of all null's, from a target row without a match in the source table, and vice versa.
Phew. There you have the composability of this query plan:
* Complex conditionals will make it into the branch filters.
* Complex data transformations (upon Insert or Update) will make it into the projections before Insert.
* As many clauses as requested will get unioned and piped up to the DML processing nodes.
What are the differences introduced by making this plan distributed?
## Distributed (Multi Node) Query Plan [#distributed-multi-node-query-plan]
We've mentioned a type of shuffle already: the loopback shuffle. Now we introduce the Hash and Key-Identity (or Homecoming) shuffles that will actually move data between the nodes of a distributed engine, at two key steps.
Let's continue using the same full-sync query \[EXAMPLE 3].
### Difference 1: Hash Shuffle of JOIN inputs [#difference-1-hash-shuffle-of-join-inputs]
```shell
\_[Join] Mode: Full [(source.a = target.a)]
\_[Shuffle] Hash by [source.a]
| \_[Projection] source.a, source.b, c_0: TRUE
| \_[StoredTable] Name: "source"
\_[Shuffle] Hash by [target.a]
\_[Projection] target.a, target.$source_node_id, target.$tablet_id, target.$tablet_txid, target.$tablet_row_number, c_1: TRUE
\_[StoredTable] Name: "target"
```
The source and target tables (being default FACT, rather than DIMENSION tables) are now distributed, and each node can read only a portion of them locally. On a multi-node engine then, given the join condition of target.a = source.a, each table's data will get shuffled across the engine, partitioned by the hash of each of those columns. More complex join conditions will hash all relevant columns from each base table.
Note that once the full data partitions land on every node, the join can proceed node-locally. Nothing about the node-local reuse of this base join (via loopback shuffle), nor the filter branches higher up, is affected.
Fundamentally speaking, the scalability of this basemost join is what enabled MERGE to be scalable out-of-the-box. Convenient!
### Difference 2: Key-Identity (or Homecoming) Shuffle of Delete inputs [#difference-2-key-identity-or-homecoming-shuffle-of-delete-inputs]
```shell
[0] [TableModify]
\_[1] [Delete] target table: "target"
| \_[2] [Shuffle] KeyIdentity by target.$source_node_id
| \_[3] [Projection] target.$tablet_row_number, target.$tablet_txid, target.$tablet_id, target.$source_node_id
| \_[4] [Union]
| \_ ... multiple branches
```
For this difference, I have to more deeply explain how delete works. For starters, a Key-Identity Shuffle sends rows to a specific node ordinal (rather than hashing row values to decide the receiver node at runtime). But why would you need to shuffle the stream of (tablet, row\_number) pairs sent to a Delete node? And it looks like we're sending each pair back to its $source\_node\_id?
Here's why. On a multi-node engine, the entire query pipeline is duplicated on every node. So every node will have its own active Delete operator that can receive row sets, write and serialize RoaringBitmap deletion vectors, etc. But a limitation mentioned earlier is that at the end of any given transaction, a persisted tablet must have exactly one deletion vector specified. Which means you cannot allow multiple nodes to delete rows from the same tablet - in this case they would upload multiple independent (and partial) deletion vectors, and all but one of those would fail to be committed to the tablet's metadata. It is like some deletions would have never taken place.
So it's great to have an active Delete operator per table per node. However each of them must handle rows from distinct sets of tablets.
The workaround is to assign to every tablet a "parent" node, here called the "source". That's the node responsible for initially scanning the tablet and then for uploading a single and complete deletion vector for that tablet at the end of every DML query. (Note that it behooves us that this node will already have the previous version of the deletion vector cached.) Applying the Key-Identity Shuffle using this "source" node id lets us guarantee the correctness of the distributed Delete step. I've always thought it a bit sweeter to call the mechanism a "Homecoming Shuffle" instead 🐑, for all the wandering lil' rows to be sent back to where they came from.
Note that this shuffle is introduced only if at least one prior layer of the pipeline ever shuffled data away from its source node. (In our case, it is the distributed join at the bottom of the MERGE plan which added the initial hash shuffles.)
And now a neat aside - to be fully transparent, implementing MERGE did not require building out or even newly instantiating this Key-Identity Shuffle layer. It was an existing building block, already used to support distributed UPDATE and DELETE queries. What MERGE introduced was the first ever time that more than one data stream could contribute to a distributed DELETE of the same table, all within a single query plan. (Consider multiple match clauses, with different conditionals, that all request the DO DELETE action.) So whereas the shuffle layer was already in place - this is actually the true reason that we started adding UNION operators to all the branches funneling into DML nodes. I mentioned earlier that unioning INSERT streams improves tablet quality. But unioning DELETE streams is truly a basic correctness requirement to ensure that all rows-to-be-deleted make it through the same shuffle infrastructure, and get marked exactly once per tablet.
## INSERT ON CONFLICT: Syntactic Sugar for MERGE [#insert-on-conflict-syntactic-sugar-for-merge]
The final cherry on top of building out this new syntax was enabling a syntactic "candy wrapper" around the merge plan to support INSERT ON CONFLICT syntax. See the [documentation](https://docs.firebolt.io/reference-sql/commands/data-management/insert#insert-on-conflict) for the full syntax spec and its current limitations.
The MERGE spec provides full flexibility on the number and order of merge clauses that you choose to interleave. However many common "upsert" patterns can be expressed more concisely. Specifically - say that you want some tuples to be inserted into a target table without creating duplicates. Once a duplicate is encountered, the only decision is whether to keep the original row or to overwrite it. PostgreSQL defines this as an ON CONFLICT DO \
The primitives for addition with overflow checking can most likely be tuned further: we could try to manually vectorize them, be smarter about avoiding overflow checks in some cases, or not throw exceptions in the hot loop. But the naive implementation nicely shows that we're paying a hefty price for the extra safeguards: both using \_\_builtin\_add\_overflow() and the manual PG-style check is about 4x slower than the version without overflow checking. The microbenchmark also nicely shows why it makes sense to go for the builtin when available: it's about 1,7 times faster than the manual check.
Because of this, avoiding these checks might make sense for some systems: if you're focused on serving already cleaned-up data, optimizing to squeeze out the last bit of performance can be a good choice. This moves more responsibility to the user who needs to use the system in a safe way, but implementing a very fast system becomes easier.
We believe that for a data warehouse, having defensive function implementations that raise clear errors to users is a must. Aligning with Postgres here was not negotiable for us, as we believe that it's the right thing to do for our users. The good news is that it's still possible to build an extremely fast engine with defensive function implementations: for most queries only very little time is spent in scalar functions as other operators such as joins or aggregations are computationally more expensive. Even for a query where a lot of time is spent in the scalar functions, it's possible to tune the function implementations to provide very good performance.
We decided to spend a lot of time performance tuning our runtime to provide a customer experience that's both safe and extremely fast. We'll drill into this in the next sections.
### Firebolt's roots in clickhouse [#firebolts-roots-in-clickhouse]
Alright, with the background on modern SQL dialects behind us, let's start diving into technical details of how we built Firebolt. This section gives an overview of why and how we forked our runtime off ClickHouse.
Building a database system is a daunting task that requires a very large engineering investment. This runs counter to many of the goals you have as a startup: going to market early, finding first design partners and customers, and iterating with them to build a really great product.
To go to market quickly and iterate with real companies, we wanted to fork off an existing open-source system. For us, ClickHouse was the only natural choice. We had three core criteria when it came to the runtime:
1. **High-performance vectorized engine:** we wanted the query engine to be a state-of-the art, low-latency OLAP runtime. If you've never thought about what makes such a runtime special, the next section will tell you more about it.
2. **Battle-tested:** we wanted the runtime to be battle tested and widely used for production use-cases.
3. **Scale-out engine:** we wanted the runtime to have basic support for distributed query processing. As a data warehouse we need to be able to handle ELT queries on massive data volume within a fixed time budget, and scale-out support is a must for this. Nowadays, we've completely rebuilt our distributed query processing layer, but the ClickHouse runtime was great for us in the early days as it allowed for basic support for scale-out processing.
If you want to dig deeper, take a look at our CDMS\@VLDB'22 paper on just this topic \[1].
At this point, we want to give a huge shoutout to all ClickHouse contributors: **ClickHouse has been an exceptional system to innovate upon**. It's incredibly fast, layered in a clean way, and elegantly makes use of the building blocks required for modern high-performance OLAP engines such as vectorized processing, a multithreaded and push-based query engine, and efficient columnar storage.
While the rest of the blog post will talk about things we've changed in significant ways, this has only been possible because ClickHouse provided a great architecture that allowed us to make these changes. We made these changes because we believe that it's the right thing to do for the workloads that matter to Firebolt customers, but different choices make sense in the context of different systems and workloads.
By forking off ClickHouse, we managed to achieve our goal of quickly having a functioning, high-performance system. However, we also realized that we had a very long way to go to actually make our query engine Postgres compliant.
Historically, ClickHouse was built for massive-scale serving & reporting workloads at Yandex. Because of this, they've made the conscious choice that they want to optimize their dialect for allowing peak performance for these specific workloads.
As we've seen in the previous section, we decided that this wasn't our desired path for Firebolt's dialect: as we want users to run large data pipelines with complex transformations, raising errors early was extremely important for us. This meant that we had to rebuild large parts of the runtime in order to become Postgres compliant. The next sections will discuss this journey in detail.
### Modern OLAP Query engines [#modern-olap-query-engines]
Why are some query engines faster than others? While two systems might speak a similar SQL dialect, they can be built in completely different ways when you take a look "under the hood".
Historically, most systems implemented a row-at-a-time interpreter for relational algebra \[3]. This is also called "Volcano"-style query execution. Many OLTP systems such as Postgres are still built that way. In a Volcano engine, individual rows are moved from one operator in the query plan to the next. They flow "upwards" from scan operators through filters, projections, aggregations, etc. Once rows arrive at the root of the query plan they can be returned to the user.
At a technical level, these systems implement relational operators such as filters, aggregations, or joins using a simple getNextRow() interface. When calling that function on a filter for example, the filter repeatedly calls getNextRow() on its child operator until a row matches the filter. This row is then passed to the parent. Your query optimizer builds the relational algebra tree for the query, and the result can be retrieved by calling getNextRow() on the root operator until the tuple stream is exhausted.
For OLTP systems that don't process massive datasets in the context of a single query, such an approach performs well. Other things like the buffer manager or indices are much more important for high performance in these systems. For an OLAP engine that does analytics on terabytes of data however, a row-at-a-time interpreter leads to poor performance: it has bad code locality, lots of virtual function calls, and isn't tailored to modern CPU architectures.
The MonetDB and MonetDB/X100 projects from CWI redefined how to build a high-performance analytical query engine \[4, 5]. Instead of implementing a row-based interpreter, MonetDB performs in-memory columnar query execution with specialized primitives. X100 went beyond that by passing batches of around one thousand rows through the query engine. Such query engines are called "vectorized" query engines.
The data flow in vectorized engines is similar to the one in a Volcano engine. Rows also flow "upwards" the query plan from scan operators all the way to the root of the query plan. Compared to the Volcano model where you just pass a single row from one operator to the next however, you now pass a whole batch of rows between them. These rows are usually stored in a columnar layout, meaning that a single column in a batch occupies a dense memory region.
Modern vectorized engines are much more efficient than traditional Volcano engines: the working set fits into the CPU caches, code locality is improved and virtual function calls are amortized over multiple rows. Most importantly however: CPUs love operating on batches of data. The code in the engine is much more friendly towards modern CPU architectures. It can leverage prefetchers, branch predictors, and superscalar pipelined CPUs much more efficiently.
The code in such a query engine looks similar to the microbenchmark we showed earlier. A scalar function for example receives a batch containing a few thousand input rows, and computes the function result for the entire batch:
These primitives are the basic building blocks of your vectorized query engine. You need them for all your supported scalar and aggregate expressions. More complex relational operators such as joins and aggregations should also always operate on batches of tuples. This can be especially beneficial when operating on large hash tables: by working on a batch of rows you can issue lots of independent memory loads and saturate your memory bandwidth.
Most modern OLAP engines implement vectorized query execution. ClickHouse, the system Firebolt forked from, is built in exactly this way. This is one of the core reasons why ClickHouse is so fast. There's a great paper about their engine at this year's VLDB \[2]!
Many other modern OLAP systems such as DuckDB \[6] (also from CWI, the group where vectorized execution was invented), Snowflake \[7], and Databricks' Photon \[8] follow this approach.
Note that there is a second approach to build modern high-performance query engines: some systems such as Hyper \[9] or Umbra \[10] just-in-time (JIT) compile a declarative SQL query into machine code \[11]. While this is super cool technology, it doesn't really matter for this blog. In practice, vectorized and compiling engines tend to perform similarly \[12].
### A PG compliant runtime - scalar functions [#a-pg-compliant-runtime---scalar-functions]
You now know how modern vectorized query engines work: they have simple primitives operating on batches of a few thousand rows, allowing them to efficiently leverage modern CPUs. You also know that Firebolt forked from ClickHouse, which is a modern OLAP engine that implements vectorized execution. In this section, we will dig into how we evolved our runtime to become PostgreSQL compliant.
### Keeping users safe - more exceptions [#keeping-users-safe---more-exceptions]
We mentioned earlier that we decided to align our dialect with PostgreSQL because it's widely used, close to ANSI SQL, and protects the users by throwing errors early. This meant that we had to take the parts of ClickHouse that aren't PostgreSQL compliant, and rewrite them to align with our target dialect.
ClickHouse decided to optimize their dialect for pure performance. In the context of ClickHouse workloads, this makes sense. This means that many functions do not throw exceptions when PostgreSQL does. Let's look at a few examples. You can easily try these out yourself using tools like [ClickHouse Fiddle](https://fiddle.clickhouse.com) or [OneCompiler](https://onecompiler.com/postgresql) for Postgres.
All of these examples are quite simple, but imagine subtle conversion errors happening deep inside of an ELT job. This makes debugging hard or – even worse – you might not even notice that things went wrong. This is why we believe that a data warehouse should throw errors in all of the above examples.
## Postgres compliance - one function at a time [#postgres-compliance---one-function-at-a-time]
This meant we had to carefully evaluate [for every function in our dialect](https://docs.firebolt.io/godocs/sql_reference/functions-reference/functions-glossary.html) whether it's compliant with PostgreSQL. For functions that are not supported by PostgreSQL, life is a bit easier, but we still wanted to make sure that our implementation matches PostgreSQL "in spirit" and raises errors early.
It's hard to come up with all the subtle ways functions might behave differently. As a result, for any function that we added to our SQL dialect, we followed a checklist to ensure that we align with Postgres:
1. Port SQL correctness tests from the PostgreSQL test suite
2. Port SQL correctness tests from other open-source SQL test collections. In our case we always checked at least DuckDB, Sqlite, MySQL, SparkSQL, and ZetaSQL
3. Write custom Firebolt tests, explicitly checking for NULLs, corner cases, etc
This effort ultimately made us completely overhaul the scalar and aggregate functions in the Firebolt runtime. For most functions in our dialect, we either (1) completely reimplemented them, or (2) took the original ClickHouse implementation and changed it significantly to change error handling.
This was a lot of work, and we built some tools to make it easier for us to port tests and ensure that Firebolt really matches Postgres behaviour.
#### SQL Test driver [#sql-test-driver]
We have a simple SQL-based test driver that's hooked up in our CI. Using this test driver, you can very easily define SQL tests. You can write arbitrary SQL queries (DDLs, DMLs, DQLs), and then check that the query's result set matches the expected result. Here's an example test that our CTO Mosha wrote that I especially love:
Our SQL tests can run in different environments:
* A "local" mode that mocks away dependencies such as S3/MinIO and our metadata services. This mode allows for extremely fast iteration times when working on the runtime.
* A "remote" mode that can run against a version of "Firebolt light" running on a local developer machine. This makes full use of MinIO and our metadata services. Compared to the local mode, this allows us to test much more of our system: interacting with object storage, SSD caching, and transaction processing.
* A "cloud" mode that runs against a [Firebolt engine](https://www.firebolt.io/resources/firebolt-elasticity-technical-whitepaper) running on AWS. This also tests our cloud infrastructure, routing layers, and authentication flows.
The "local" and "remote" modes run in the CI on every PR that gets merged to main. We nowadays have more than 100.000 SQL queries stressing every part of our dialect that run on every commit. The "cloud" version of the SQL tests runs nightly and before every release.
### Test Transpilers [#test-transpilers]
While almost all database systems implement text-based SQL tests as the ones mentioned above, many systems use slightly different formats. DuckDB uses a similar format to Sqlite. Here's an example of the [DuckDB tests for length](https://github.com/duckdb/duckdb/blob/main/test/sql/function/string/test_length.test):
To make it easy to port tests from other systems, we wrote custom transpilers that take text-based tests from ZetaSQL, DuckDB, etc. and automatically turn them into Firebolt's SQL test format. This made it much easier to quickly cover Firebolt's SQL dialect with tests from other systems.
### The Postgres executor [#the-postgres-executor]
When porting tests from other systems, or writing our own tests, we wanted to make it easy to ensure that Firebolt actually matches the Postgres behaviour. For this, we wrote an additional execution mode for our SQL tests: the Postgres executor.
This mode uses the Firebolt SQL tests, but doesn't run against Firebolt at all. Instead, it sends the queries to a locally running Postgres server. We can then take the Postgres output and render it in the same way as our text-based format requires.
When adding a function, this makes it super easy to actually ensure that Postgres returns the same results at Firebolt. Of course, this execution mode doesn't help with custom Firebolt dialect extensions such as our [array functions](https://docs.firebolt.io/godocs/sql_reference/functions-reference/array/).
### Overhauling our runtime primitives [#overhauling-our-runtime-primitives]
In addition to writing a lot of tools for testing, we also overhauled how we write scalar functions in our C++ runtime. As you can see in the SQL examples above, ClickHouse doesn't raise exceptions for many functions. This leads to some interesting runtime implementation details when handling NULL values.
In the section on "Modern OLAP query Engines" we've seen that vectorized engines such as ClickHouse always operate on batches of rows. These batches are stored in a columnar format. For primitive data types such as integers, the columns are represented as contiguous memory regions (vectors) packing one integer after the next. More complex types such as TEXTs or ARRAYs usually store an additional vector for TEXT/ARRAY lengths.
For columns that can be nullable, vectorized engines usually store which values are NULL in an additional vector. There are different ways to represent these NULLs \[13], Firebolt implements a byte mask indicating which indices in a column of a batch are NULL.
Let's understand how the following data would be represented in an in-memory batch:
The five integer values for Id are stored in an integer column that has four bytes for every entry. The balance values are stored in a double column that has eight bytes per entry. Separate one byte null maps store one byte for every value in the source column. If the null map entry is set to one, this indicates that the value is NULL.
Note that for any value that's NULL, there's still an actual value stored in the data column "behind" the NULL mask. In the above example, this value is set to 0 for the two NULLs in the balance column.
Most scalar functions propagate NULLs: an input NULL also results in an output NULL. At first glance, this seems to yield a very simple implementation for vectorized primitives that contain NULLs. For some scalar function\_implementation, the primitive can just look as follows:
You take the logical OR of the input NULLs, and call the function implementation on all values. The above code snippet is great from a performance standpoint: working with NULLs does not require any extra branches, just the extremely efficient logical OR of two byte maps. Splitting the primitive into separate for loops for the null map and the function implementation improves performance. This is how ClickHouse implements many of their vectorized primitives. **However, this only works if function\_implementation does not throw errors.**
In the example above, imagine we wanted to run a query such as SELECT log(in\_game\_balance) as res FROM players. A primitive that looks like the one above is only safe in the context of the ClickHouse log implementation. As the log of 0 returns -inf rather than raising an error, we can safely perform the computation on all nested values. The output block simply looks as follows:
We keep the original NULL map (we only need OR for n-ary functions), and evaluate the logarithm on all nested arguments. The output column now has -inf stored "behind" the NULL values.
This breaks down completely once you start throwing frequent errors in your runtime. **We cannot evaluate the function on values that are "behind" NULL, as they might cause errors.** In Firebolt, log(0) throws an error, and if our primitives were implemented in the same way, we would throw an error in the above query computing log(balance), even though all balances are positive or NULL.
When implementing the above primitive for a function that can throw errors, you need to be more careful:
This code is less friendly for modern CPUs, as there's a branch (!out\_nulls) in the second nested for loop. However, it's safe if function\_implementation can throw errors. We could also write the primitive in a single nested for loop. However, the above version is more efficient. [Let's write another Microbenchmark](https://quick-bench.com/q/F8cbQk8BnX_21ifzNZowRk1RbEA) on QuickBench to compare the performance. Here, we compare (1) an unsafe primitive that performs addition without overflow checking on all values, irrespective of if they are null, (2) a primitive that changes the control flow to not perform operations behind null values, but still does not perform overflow checking, and (3) a primitive with changed control flow and overflow checking.
Comparing (1) to (2) allows us to see how expensive just the change in control flow is. Comparing (1) to (3) allows us to compare end-to-end performance of the primitives. We run the benchmark with a 10% null ratio.
We can see that the change in control flow with the extra branches creates a \~4.3x overhead. Overflow checking causes another 20% slowdown. When you look at the generated assembly, it becomes quite clear why the version without overflow checking is so much faster: it can unroll the loop more effectively and leverage SIMD registers. We've added some comments to show you what is happening where. Note you won't see the null map computation here. That happens in a separate loop before:
Meanwhile, the generated assembly for the safe primitive with overflow checking doesn't do any unrolling:
This shows that when implementing scalar functions that throw exceptions, it's quite easy to end up in a state where the performance of your vectorized primitives suffers.
It might be comfortable to ignore this problem: when executing a query in a system, there's usually a lot more going on: table scans might read data from SSD, you might perform more expensive relational operations such as aggregations, and you need to serialize query results. In many cases, even a 4x slowdown in your addition primitive won't have a large effect on end-to-end query runtimes.
However, the performance difference of the primitives can become noticeable for some query patterns: complex expression trees, selective filters early on in your query, or small scans where data is stored in main-memory. Firebolt is all about high-performance, cost-effective SQL processing. As a result, just living with a 4x slowdown for some of our core primitives was not acceptable. The next section will provide a deep-dive into how we tuned our scalar functions further to make them as fast as possible.
### Keeping firebolt fast - function dispatching [#keeping-firebolt-fast---function-dispatching]
The previous section taught us that it's harder than it might seem to keep a query engine very fast while still throwing exceptions early. The extra argument and result checks in the inner hot loop of the vectorized primitive make it harder to generate very efficient code for modern CPUs. This section shows how we tuned our runtime primitives to still provide the fastest possible performance in as many cases as possible.
#### Dispatching for numeric functions [#dispatching-for-numeric-functions]
Usually, people run SQL queries that don't throw exceptions. Let's stick with the example of an ELT job. Throwing exceptions matters a lot when getting an initial version of an ETL query running. But once you have that query running, you want it to run every day/hour/minute and ingest fresh data. Exceptions might still happen here if e.g. your data distribution changed, in this case they are helpful and alert you that you should take a look rather than serving wrong results. But exceptions should be the exception, not the norm.
Can we use this property to actually improve the performance of our system? If we know that it's safe to do addition without overflow checking on a block of data, we can save the cost of doing overflow checks. We can then execute a ClickHouse-style, super efficient primitive, and be sure that we don't miss over- and underflows.
One thing that's great about vectorized engines is that since they operate on at most a few thousand rows at a time, the working set of an expression fits fully into cache. Maybe we can do a first pass over the function, do a cheap check with one-sided error that ensures that the addition can definitely not overflow, and then run addition without overflow checking whenever possible.
We know that adding two four byte integers can definitely not overflow if they are in the range of -1 billion and 1 billion. We can use this property to rewrite our vectorized primitive as follows:
Time for [another microbenchmark](https://quick-bench.com/q/EAV5XgdTCfasKVUjP7nF8guy0TU). We're now comparing (1) the super fast unsafe primitive, (2) the "safe" primitive that always does overflow checking, and (3) the attempted optimization above. There are no rows that overflow, and 10% NULLs.
This is a pretty good improvement. We went from being \~5x slower than the original ClickHouse primitive, to being just \~2.7x slower.
What happens if we can't choose the fast specialization? We can also test this in a simple [microbenchmark](https://quick-bench.com/q/GtWkI7at0ylYyKpsqYezbssCw1o).
We can see that in those cases, the extra checks cost about 30%. As these cases should be rare, being \~2x faster in the common case seems worth it. There are ways to improve the situation further: if we see that we often choose the specialization with overflow checks, we could at query runtime decide to just always call the safe specialization and not do the extra range checks. We currently don't do that in Firebolt, but plan on doing so in the future.
Addition is one of the cheapest possible functions you can calculate. So getting within \~2.7x of the original primitive is already quite amazing. For more expensive functions, the performance of dispatching like this will become even closer to the performance of the original primitive.
For different functions, you can use different tricks to maximize performance. In many cases, first doing an efficient loop over the arguments to check for invalid values is a good choice. Then you can implement a very efficient loop afterwards that doesn't require nested control flow for argument checks.
#### Dispatching for text functions [#dispatching-for-text-functions]
Smart dispatching such as above can help a lot for squeezing even more performance out of primitives for numeric functions. However, numeric functions tend to be quite cheap. Text functions in a database system are often much more expensive. Firebolt uses a similar dispatching concept as above to speed up common operations on text columns.
We've seen before how we encode simple columns in our vectorized engine. TEXT columns are stored in two dense memory regions: one containing the text byte sequence, and one containing the offsets of where the individual TEXT values start. There is one more offset than there are TEXT values in the column. The following example column contains three TEXT values: "HELLO, WORLD", "FIREBOLT", and "BLOG POST":
While the text data is variable-sized, the offsets are fixed-size. The offsets act as pointers into the byte sequence and make it easy to work with a string at a specific index.
In Firebolt, TEXT data is always represented as a [UTF-8](https://en.wikipedia.org/wiki/UTF-8) byte sequence. This means that a single text character can have multiple bytes in the binary representation. The [🔥](https://apps.timwhitlock.info/emoji/tables/unicode#emoji-modal) emoji for example becomes 0xF09F94A5, which is a four byte code point.
The offsets act as byte indexes and not character indexes in the column. This means if we replace "Firebolt" with the 🔥 emoji the column would be encoded as-follows:
Even though 🔥 is just a single UTF-8 character, the byte-based offsets show that four bytes storage are required.
Most TEXT functions operate on the UTF-8 characters. When calling length(\
The improvement here is super impressive. By checking whether there are only single-byte UTF-8 characters and then calling into the faster specialization, length() becomes 11x faster than before! When it turns out that there are multi-byte code points, the above numbers also show that the overhead will be \<10%. This is because the fast version is an upper bound for the overhead of the code-point checking.
We use this trick for other text functions as well. Further nice examples where this trick can be applied are [lpad(\• Dedicated compute resources available via Provisioned Capacity
• VPC endpoint connections supported | • Isolated tenant & resources • Runs in your VPC | | Control vs abstraction of compute | • Serverless by default with no infrastructure control
• Optional Provisioned Capacity allows dedicated DPU allocation (minimum 24 DPUs)
• Two pricing models: on-demand ($5/TB scanned) or provisioned ($0.30/DPU-hour) | • Configurable cluster size
• Configurable compute types | | Self-hosted and hybrid deployment options | No self-hosted options – serverless only | Limited hybrid options with Redshift Serverless | | ACID Compliance and Transactions | No ACID compliance – eventual consistency model | ACID compliant at table level with some limitations on concurrent operations | **Athena** is serverless and built on a decoupled storage and compute architecture that queries data directly in S3, without the need to ingest/copy the data. It runs in multi-tenancy with shared resources. Users do not have control over the compute resources Athena chooses to allocate per query from the shared resource pool. For folks requiring additional or dedicated resources, they can reserve dedicated processing capacity in the form of Data Processing Units (DPU), with each DPU providing 4 vCPU and 16 GB RAM. RPU allocation ranges from 24 - 1000 per region. **Redshift** has the oldest architecture, being the first Cloud DW in the group. Its architecture wasn't designed to separate storage & compute. While it now has RA3 nodes which allow you to scale compute and only cache the data you need locally, all compute still operates together. You cannot separate and isolate different workloads over the same data, which puts it behind other decoupled storage/compute architectures. Redshift runs as an isolated tenant per customer, and unlike other cloud data warehouses, it is deployed in your VPC. Redshift offers a serverless option which is based on an abstracted unit called Redshift Processing Unit (RPU) ranging from 8 to 512 in increments of 8. Each RPU provides 2 vCPU and 16GB RAM. Thus, 8 RPU is equivalent to 16 vCPU / 128GB RAM. The minimum RPU is 8. ## Scalability [#scalability] There are three big differences among data warehouses and query engines that limit scalability: decoupled storage and compute, dedicated resources, and continuous ingestion. | Feature | Athena | Redshift | | --------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | | Elasticity – Scaling for larger data volumes and faster queries | • Fully abstracted on-demand scaling • Provisioned Capacity allows manual scaling of DPUs for predictable performance • Capacity reservations can be adjusted with minimum 1-hour billing periods | Available via Elastic Resize – slow and limited, downtime required | | Elasticity – Scaling for higher concurrency | • Default limit of 25 concurrent DML queries and 20 DDL queries (adjustable via service quotas) • Provisioned Capacity enables higher concurrency with dedicated DPUs • Query queuing available when capacity is exceeded | • 5 concurrent queries per WLM queue by default (up to 8 queues) • Concurrency Scaling enables thousands of concurrent queries | **Athena** is a shared multi-tenant resource, with no guarantees on the amount or availability of the resources allocated for your queries. From a data volume perspective, it can scale to large volumes, but large data volumes can suffer from very long run times and frequent time outs. Query concurrency is maxed at 20. If scalability is a top priority, Athena is probably not the best choice. **Redshift** is limited in scale because even with RA3, it cannot distribute different workloads across clusters. While it can scale to up to 10 clusters automatically to support query concurrency, it can only handle a maximum of 50 queued queries across all clusters by default. ## Performance [#performance] Performance is the biggest challenge with most data warehouses today. While decoupled storage and compute architectures improved scalability and simplified administration, for most data warehouses it introduced two bottlenecks; storage, and compute. Most modern cloud data warehouses fetch entire partitions over the network instead of just fetching the specific data needed for each query. While many invest in caching, most do not invest heavily in query optimization. Most vendors also have not improved continuous ingestion or semi-structured data analytics performance, both of which are needed for operational and customer-facing use cases. | Feature | Athena | Redshift | | ------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Indexes | No traditional indexes – relies on partition pruning and data organization in S3. Uses columnar formats and compression for optimization | None | | Compute tuning | • No compute tuning in on-demand mode • Provisioned Capacity allows DPU allocation control (4 vCPU and 16GB RAM per DPU) • Minimum 24 DPUs with scaling in 4-DPU increments | Choice over number of nodes and their type | | Storage format | Supports multiple formats: Parquet, ORC, Avro, JSON, CSV, TSV on S3. Native support for open table formats including Apache Iceberg, Apache Hudi, and Delta Lake | Columnar & compressed storage (RA3 nodes) | | Table-level partition & pruning techniques | • User-defined table-level partitions with Hive-style partitioning • Pruning at partition level • Partition projection for advanced performance optimization • Supports open table formats with built-in partitioning | • No table partitions • User-defined distribution & sort keys are used to optimize for speed | | Result cache | Query result caching for up to 30 days with configurable retention. Results reuse supported across workgroups | Yes | | Warm cache (SSD) | No local caching – queries data directly from S3. Relies on S3's performance characteristics and intelligent tiering | Only with RA3 nodes at partition-level granularity | | Support for semi-structured data & JSON functions within SQL | Yes, comprehensive JSON support including Lambda expressions, array functions, and native nested data handling | Limited | | Vector Search and AI Capabilities | No native AI or vector search capabilities | Limited AI capabilities – primarily through integrations | | Query Optimizations | • Cost-based optimizer (CBO) in Athena engine v3 • Query result caching (up to 30 days) • Partition projection for advanced optimization • CTAS for precomputed queries • Join reordering and aggregation pushdown • Automatic parallel query execution • Support for columnar formats (Parquet, ORC) • Integration with AWS Glue Data Catalog | • Basic query optimizer • Materialized views • Result caching • ANALYZE for table statistics • Workload management (WLM) • Automated materialized views (AutoMV) • AI-driven scaling (Serverless preview) | **Athena**, (and Presto) are designed to query data where it is, sacrificing storage-compute optimizations. This makes it very convenient for easy and immediate querying but at the expense of performance. This typically puts Athena behind cloud data warehouses in terms of performance. But Athena still does relatively well in performance benchmarks, especially when external storage is managed by experts. While it supports partitions, there is no support for indexing, and together with the fact that resources are pooled from a shared multi-tenant service, low-latency and consistent performance are not Athena's sweet spot. A cloud data warehouse be more performant better than Athena in most cases. **Redshift** does provide a result cache for accelerating repetitive query workloads and also has more tuning options than some others. But it does not deliver much faster compute performance than other cloud data warehouses in benchmarks. Sort keys can be used to optimize performance, but their contribution is limited. There is no support for indexes, and low-latency analytics at large data volumes is hard to achieve. Because Redshift decoupling of storage & compute is limited compared to other cloud data warehouses, it doesn't support isolating workloads, which means performance can degrade under pressure and competition for resources ## Use cases [#use-cases] There are a host of different analytics use cases that can be supported by a data warehouse. Look at your legacy technologies and their workloads, as well as the new possible use cases, and figure out which ones you will need to support in the next few years. | Feature | Athena | Redshift | | ---------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Low-latency dashboards | • Seconds to minutes response times for interactive dashboards • Performance varies based on data partitioning, file formats, and query optimization • Provisioned Capacity can improve consistency for dashboard workloads • Best suited for analytical dashboards rather than sub-second operational dashboards | • Seconds to tens of seconds load times at 100s of GB scale • Can achieve faster performance with Concurrency Scaling and proper tuning | | Enterprise BI | • Good integration with AWS ecosystem BI tools (QuickSight, etc.) • Standard SQL compatibility enables most BI tool connections • Cost-effective for variable workloads and ad-hoc analytics • JDBC/ODBC drivers support enterprise BI tools • Limited advanced BI features compared to dedicated data warehouses | • Mature and comprehensive Enterprise DW feature set • Extensive integrations with Enterprise BI ecosystem • Strong AWS ecosystem integration | | Data Apps and AI Applications (Customer-facing low-latency high concurrency) | • Default concurrency limits (25 DML/20 DDL queries) may require service quota increases • Provisioned Capacity enables higher concurrency with dedicated resources • Seconds-level response times typical • Cost-effective for customer-facing analytics with proper optimization • Best suited for analytical rather than operational workloads • No native AI capabilities | • 5 concurrent queries per WLM queue by default (up to 8 queues) • Concurrency Scaling enables thousands of concurrent queries • Seconds-level response times typical • Automatic scaling for burst workloads • Limited AI application support | | Ad hoc | • Purpose-built for ad-hoc analytics on data lakes • Serverless with zero infrastructure management • Direct querying of S3 data without ETL • Cost-effective pay-per-query model ideal for exploratory analysis • Strong support for multiple data formats and federated queries • Apache Spark integration for advanced analytics | • Performance dependent on predefined distribution & sort keys • Elastic Resize enables adding compute resources • Typically subset of data loaded for ad-hoc analysis | **Athena** is a great choice for Ad-Hoc analytics. You can keep the data where it is, and start querying without worrying about hardware or pretty much anything else, given that Athena is serverless and takes care of everything behind the scenes. However, it is not a great fit when you need consistent and fast query performance, and/or high concurrency. This is why it is typically not the best choice for operational and customer-facing applications. It can be also easily and flexibly used for batch processing, which is often leveraged for ML use cases. **Redshift** was originally designed to support traditional internal BI reporting and dashboard use cases for analysts. As such, it is typically used as a general-purpose Enterprise data warehouse. With deep integrations into the AWS ecosystem, it can also leverage AWS ML service, making it also useful for ML projects. However, given the coupling of storage & compute, and the difficulty in delivering low-latency analytics at scale, it is less suited for operational use cases and customer-facing use cases like Data Apps. The coupling of storage and compute, together with the need to predefine sort & dist keys for optimal performance, make it challenging to use for Ad-Hoc analytics. # BigQuery vs Athena (/comparison/bigquery-vs-athena) ## Architecture [#architecture] The biggest difference among cloud data warehouses are whether they separate storage and compute, how much they isolate data and compute, and what clouds they can run on. | Feature | BigQuery | Athena | | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Separation of storage and compute | Fully serverless with complete separation of compute (Dremel) and storage (Colossus), powered by Jupiter network and Borg orchestration | Yes, serverless with optional provisioned capacity. Workloads can be isolated through Workgroups and Capacity Reservations | | Supported cloud infrastructure | Google Cloud only | AWS only | | Isolated tenancy – option for dedicated resources | • Multi-tenant pooled resources
• VPC Service Controls provide enhanced security and connectivity isolation to customer VPCs
• Cross-region disaster recovery for enterprise workloads | • Multi-tenant pooled resources by default
• Dedicated compute resources available via Provisioned Capacity
• VPC endpoint connections supported | | Control vs abstraction of compute | Fully serverless with no control over compute resources – BigQuery automatically allocates computing resources as needed with intelligent workload management and dynamic slot allocation | • Serverless by default with no infrastructure control
• Optional Provisioned Capacity allows dedicated DPU allocation (minimum 24 DPUs)
• Two pricing models: on-demand ($5/TB scanned) or provisioned ($0.30/DPU-hour) | | Self-hosted and hybrid deployment options | No self-hosted options – fully managed service only | No self-hosted options – serverless only | | ACID Compliance and Transactions | Limited ACID support – eventual consistency model with some transactional capabilities | No ACID compliance – eventual consistency model | **BigQuery** was one of the first decoupled storage and compute architectures. It is a unique piece of engineering and not a typical data warehouse in part because it started as an on-demand serverless query engine. It runs in multi-tenancy with shared resources, allocated as "slots" which represent a virtual CPU that executes SQL. BigQuery determines how many slots a query requires, without the ability of the user to control it. BigQuery can be priced on a $/TB scanned basis or through slot reservations. A slot in BigQuery is logically equivalent to 0.5 vCPU and 0.5GB of RAM. There are multiple models to allocate slots in BigQuery. **Athena** is serverless and built on a decoupled storage and compute architecture that queries data directly in S3, without the need to ingest/copy the data. It runs in multi-tenancy with shared resources. Users do not have control over the compute resources Athena chooses to allocate per query from the shared resource pool. For folks requiring additional or dedicated resources, they can reserve dedicated processing capacity in the form of Data Processing Units (DPU), with each DPU providing 4 vCPU and 16 GB RAM. RPU allocation ranges from 24 - 1000 per region. ## Scalability [#scalability] There are three big differences among data warehouses and query engines that limit scalability: decoupled storage and compute, dedicated resources, and continuous ingestion. | Feature | BigQuery | Athena | | --------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Elasticity – Scaling for larger data volumes and faster queries | Fully automated serverless scaling – BigQuery automatically determines resource allocation and scales to petabytes without user intervention. Can dynamically burst beyond baseline slot allocations for performance optimization | • Fully abstracted on-demand scaling • Provisioned Capacity allows manual scaling of DPUs for predictable performance • Capacity reservations can be adjusted with minimum 1-hour billing periods | | Elasticity – Scaling for higher concurrency | Dynamic concurrency management with query queueing supporting up to 1,000 interactive queries and 20,000 batch queries per project per region. Automatic fair scheduling and slot distribution across workloads | • Default limit of 25 concurrent DML queries and 20 DDL queries (adjustable via service quotas) • Provisioned Capacity enables higher concurrency with dedicated DPUs • Query queuing available when capacity is exceeded | **BigQuery** scales very well to large data volumes, and automatically assigns more compute resources when needed behind the scenes, in the form of "slots". BigQuery works either in an "on-demand pricing model", where slot assignment is completely in the hands of BigQuery and the state of the shared resource pool, or in "flat-rate pricing model" where slots are reserved in advanced. With reserved slots there is more control over compute resources, thus making scaling more predictable. Concurrency is limited to 100 users by default. **Athena** is a shared multi-tenant resource, with no guarantees on the amount or availability of the resources allocated for your queries. From a data volume perspective, it can scale to large volumes, but large data volumes can suffer from very long run times and frequent time outs. Query concurrency is maxed at 20. If scalability is a top priority, Athena is probably not the best choice. ## Performance [#performance] Performance is the biggest challenge with most data warehouses today. While decoupled storage and compute architectures improved scalability and simplified administration, for most data warehouses it introduced two bottlenecks; storage, and compute. Most modern cloud data warehouses fetch entire partitions over the network instead of just fetching the specific data needed for each query. While many invest in caching, most do not invest heavily in query optimization. Most vendors also have not improved continuous ingestion or semi-structured data analytics performance, both of which are needed for operational and customer-facing use cases. | Feature | BigQuery | Athena | | ------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Indexes | Search indexes (GA) for efficient text search optimization on STRING, JSON, and array columns. Support for LOG\_ANALYZER, NO\_OP\_ANALYZER, and PATTERN\_ANALYZER with column-level granularity for improved query performance and cost efficiency | No traditional indexes – relies on partition pruning and data organization in S3. Uses columnar formats and compression for optimization | | Compute tuning | Serverless architecture with automatic resource optimization – no manual tuning required. Intelligent workload management with AI-powered resource allocation and dynamic slot distribution | • No compute tuning in on-demand mode • Provisioned Capacity allows DPU allocation control (4 vCPU and 16GB RAM per DPU) • Minimum 24 DPUs with scaling in 4-DPU increments | | Storage format | Columnar & compressed storage (Capacitor format) with support for open table formats including Apache Iceberg, Delta Lake, and Hudi. Intelligent tiering with automatic long-term storage cost reduction after 90 days | Supports multiple formats: Parquet, ORC, Avro, JSON, CSV, TSV on S3. Native support for open table formats including Apache Iceberg, Apache Hudi, and Delta Lake | | Table-level partition & pruning techniques | • Automatic table organization with intelligent micro-partitioning • Clustering keys for data organization • Automatic partition pruning optimization • Supports time-based and custom partitioning strategies | • User-defined table-level partitions with Hive-style partitioning • Pruning at partition level • Partition projection for advanced performance optimization • Supports open table formats with built-in partitioning | | Result cache | Yes, with cross-user result caching and intelligent cache management for up to 24 hours | Query result caching for up to 30 days with configurable retention. Results reuse supported across workgroups | | Warm cache (SSD) | BI Engine provides in-memory caching and acceleration for frequently accessed data and dashboards | No local caching – queries data directly from S3. Relies on S3's performance characteristics and intelligent tiering | | Support for semi-structured data & JSON functions within SQL | Yes, including advanced JSON functions, Lambda expressions, and native support for nested and repeated fields | Yes, comprehensive JSON support including Lambda expressions, array functions, and native nested data handling | | Vector Search and AI Capabilities | • BigQuery ML for in-database machine learning • Vertex AI integration • Natural language querying with Gemini AI • Limited vector search capabilities | No native AI or vector search capabilities | | Query Optimizations | • Advanced query optimizer with Dremel engine • Search indexes with column-level granularity • Materialized views with smart refresh and automatic query rewriting • BI Engine in-memory acceleration • Gemini AI-powered query optimization and natural language querying • Cross-user result caching • Automatic partitioning and clustering optimization • Cost-based optimization with intelligent workload management | • Cost-based optimizer (CBO) in Athena engine v3 • Query result caching (up to 30 days) • Partition projection for advanced optimization • CTAS for precomputed queries • Join reordering and aggregation pushdown • Automatic parallel query execution • Support for columnar formats (Parquet, ORC) • Integration with AWS Glue Data Catalog | **BigQuery** lines up in benchmarks in the same ballpark as other cloud data warehouses but does come in consistently last in most queries. Beyond implementing according to best practices, there is little you can do to accelerate BigQuery performance, as it determines the amount of resources (slots) the query needs for you. BigQuery can be used together with "BigQuery BI Engine" for lower latency analytics. However, BI Engine is limited in terms of scale because it runs in memory. Its maximum capacity is 100GB. **Athena** (and Presto) are designed to query data where it is, sacrificing storage-compute optimizations. This makes it very convenient for easy and immediate querying but at the expense of performance. This typically puts Athena behind cloud data warehouses in terms of performance. But Athena still does relatively well in performance benchmarks, especially when external storage is managed by experts. While it supports partitions, there is no support for indexing, and together with the fact that resources are pooled from a shared multi-tenant service, low-latency and consistent performance are not Athena's sweet spot. A cloud data warehouse be more performant better than Athena in most cases. ## Use cases [#use-cases] There are a host of different analytics use cases that can be supported by a data warehouse. Look at your legacy technologies and their workloads, as well as the new possible use cases, and figure out which ones you will need to support in the next few years. | Feature | BigQuery | Athena | | ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Low-latency dashboards | • Sub-second to seconds response times at TB+ scale with BI Engine acceleration • Search indexes and materialized views provide significant performance improvements for dashboard queries • Intelligent caching reduces query costs for repeated dashboard access • Dynamic concurrency management supports high user loads | • Seconds to minutes response times for interactive dashboards • Performance varies based on data partitioning, file formats, and query optimization • Provisioned Capacity can improve consistency for dashboard workloads • Best suited for analytical dashboards rather than sub-second operational dashboards | | Enterprise BI | • Mature and comprehensive Enterprise DW feature set with native Google Cloud ecosystem integration • Strong integration with Looker, Looker Studio, and major BI tools • Gemini AI integration for natural language insights • Cross-cloud analytics capabilities • Advanced governance with Dataplex Universal Catalog | • Good integration with AWS ecosystem BI tools (QuickSight, etc.) • Standard SQL compatibility enables most BI tool connections • Cost-effective for variable workloads and ad-hoc analytics • JDBC/ODBC drivers support enterprise BI tools • Limited advanced BI features compared to dedicated data warehouses | | Data Apps and AI Applications (Customer-facing low-latency high concurrency) | • Dynamic concurrency supporting 1,000+ interactive queries with intelligent queuing and fair scheduling • Sub-second response times with BI Engine acceleration and search indexes • Serverless architecture eliminates infrastructure management overhead • Advanced caching and materialized views optimize repeated queries • AI-powered optimization for customer-facing applications • BigQuery ML for AI workloads | • Default concurrency limits (25 DML/20 DDL queries) may require service quota increases • Provisioned Capacity enables higher concurrency with dedicated resources • Seconds-level response times typical • Cost-effective for customer-facing analytics with proper optimization • Best suited for analytical rather than operational workloads • No native AI capabilities | | Ad hoc | • Excellent for ad-hoc analytics with serverless architecture requiring zero infrastructure management • Intelligent query optimization handles unpredictable workloads automatically • Gemini AI integration enables natural language querying for business users • Advanced JSON support and schema inference enable flexible data exploration • Cross-cloud analytics capabilities for federated queries | • Purpose-built for ad-hoc analytics on data lakes • Serverless with zero infrastructure management • Direct querying of S3 data without ETL • Cost-effective pay-per-query model ideal for exploratory analysis • Strong support for multiple data formats and federated queries • Apache Spark integration for advanced analytics | **BigQuery** is a mature general-purpose data warehouse, which lends itself well to internal BI & reporting. The fact that it's serverless in nature and tightly integrated with GCP, makes it very convenient for Ad-Hoc analytics and ML use cases on GCP. On the other hand, because BigQuery makes resource allocation decisions for you, it is not always the best fit for operational use cases and Data Apps where performance needs to be consistent and predictable. **Athena** is a great choice for Ad-Hoc analytics. You can keep the data where it is, and start querying without worrying about hardware or pretty much anything else, given that Athena is serverless and takes care of everything behind the scenes. However, it is not a great fit when you need consistent and fast query performance, and/or high concurrency. This is why it is typically not the best choice for operational and customer-facing applications. It can be also easily and flexibly used for batch processing, which is often leveraged for ML use cases. # BigQuery vs ClickHouse (2025) (/comparison/bigquery-vs-clickhouse) ## Architecture [#architecture] The biggest difference among cloud data warehouses are whether they separate storage and compute, how much they isolate data and compute, and what clouds they can run on. | Feature | BigQuery | ClickHouse | | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Separation of storage and compute | Fully serverless with complete separation of compute (Dremel) and storage (Colossus), powered by Jupiter network and Borg orchestration | Yes – SharedMergeTree engine in ClickHouse Cloud enables full separation of storage and compute, with compute-compute separation through Warehouses feature (introduced 2025) allowing multiple isolated compute services sharing the same data | | Supported cloud infrastructure | Google Cloud only | AWS, GCP, Azure, cloud service and on-premises | | Isolated tenancy – option for dedicated resources | • Multi-tenant pooled resources
• VPC Service Controls provide enhanced security and connectivity isolation to customer VPCs
• Cross-region disaster recovery for enterprise workloads | • Multi-tenant metadata layer
• Isolated tenancy for compute & storage per client in cloud | | Control vs abstraction of compute | Fully serverless with no control over compute resources – BigQuery automatically allocates computing resources as needed with intelligent workload management and dynamic slot allocation | Configurable cluster size and compute types in ClickHouse Cloud with granular control over nodes (1-128 nodes) and node characteristics. Warehouses feature enables multiple isolated read-only compute environments. | | Self-hosted and hybrid deployment options | No self-hosted options – fully managed service only | Self-managed deployments available with full control over infrastructure | | ACID Compliance and Transactions | Limited ACID support – eventual consistency model with some transactional capabilities | Limited ACID compliance with MergeTree engine family. | **BigQuery** was one of the first decoupled storage and compute architectures. It is a unique piece of engineering and not a typical data warehouse in part because it started as an on-demand serverless query engine. It runs in multi-tenancy with shared resources, allocated as "slots" which represent a virtual CPU that executes SQL. BigQuery determines how many slots a query requires, without the ability of the user to control it. BigQuery can be priced on a $/TB scanned basis or through slot reservations. A slot in BigQuery is logically equivalent to 0.5 vCPU and 0.5GB of RAM. There are multiple models to allocate slots in BigQuery. **ClickHouse** was originally developed at Yandex, the Russian search engine, as an OLAP engine for low latency analytics. It was built as an on-premise solution with coupled storage & compute, and a large variety of tuning options in the form of indexes and and merge trees. ClickHouse's architecture is famous for its focus on performance and low-latency queries. The tradeoff is that it is considered very difficult to work with. SQL support is very limited, and tuning/running it requires significant engineering resources. ## Scalability [#scalability] There are three big differences among data warehouses and query engines that limit scalability: decoupled storage and compute, dedicated resources, and continuous ingestion. | Feature | BigQuery | ClickHouse | | --------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Elasticity – Scaling for larger data volumes and faster queries | Fully automated serverless scaling – BigQuery automatically determines resource allocation and scales to petabytes without user intervention. Can dynamically burst beyond baseline slot allocations for performance optimization | Automatic horizontal and vertical scaling in ClickHouse Cloud with SharedMergeTree architecture. Manual scaling for self-managed deployments with cluster rebalancing capabilities | | Elasticity – Scaling for higher concurrency | Dynamic concurrency management with query queueing supporting up to 1,000 interactive queries and 20,000 batch queries per project per region. Automatic fair scheduling and slot distribution across workloads | Supports high concurrency with proper resource allocation and configuration. Vertical auto-scaling and horizontal manual scaling. Additional warehouses can idle to zero billing. Primary service always on in multi-warehouse configurations. | **BigQuery** scales very well to large data volumes, and automatically assigns more compute resources when needed behind the scenes, in the form of "slots". BigQuery works either in an "on-demand pricing model", where slot assignment is completely in the hands of BigQuery and the state of the shared resource pool, or in "flat-rate pricing model" where slots are reserved in advanced. With reserved slots there is more control over compute resources, thus making scaling more predictable. Concurrency is limited to 100 users by default. **ClickHouse** doesn't offer any dedicated scaling features or mechanisms. While it can deliver linearly scalable performance for some types of queries, scaling itself has to be done manually. Hardware is self-managed in ClickHouse. This means that to scale you would have to provision a cluster and migrate. ## Performance [#performance] Performance is the biggest challenge with most data warehouses today. While decoupled storage and compute architectures improved scalability and simplified administration, for most data warehouses it introduced two bottlenecks; storage, and compute. Most modern cloud data warehouses fetch entire partitions over the network instead of just fetching the specific data needed for each query. While many invest in caching, most do not invest heavily in query optimization. Most vendors also have not improved continuous ingestion or semi-structured data analytics performance, both of which are needed for operational and customer-facing use cases. | Feature | BigQuery | ClickHouse | | ------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Indexes | Search indexes (GA) for efficient text search optimization on STRING, JSON, and array columns. Support for LOG\_ANALYZER, NO\_OP\_ANALYZER, and PATTERN\_ANALYZER with column-level granularity for improved query performance and cost efficiency | • Primary indexes • Skipping indexes (minmax, set, bloom filters, ngrambf\_v1, tokenbf\_v1)
• MergeTree indexes
• Incremental Materialized views | | Compute tuning | Serverless architecture with automatic resource optimization – no manual tuning required. Intelligent workload management with AI-powered resource allocation and dynamic slot distribution | Configurable compute resources in cloud offering | | Storage format | Columnar & compressed storage (Capacitor format) with support for open table formats including Apache Iceberg, Delta Lake, and Hudi. Intelligent tiering with automatic long-term storage cost reduction after 90 days | Columnar, supports sorted, compressed, encoded & sparsely indexed files with native Apache Iceberg support. | | Table-level partition & pruning techniques | • Automatic table organization with intelligent micro-partitioning • Clustering keys for data organization • Automatic partition pruning optimization • Supports time-based and custom partitioning strategies | Partitioning by date/time and custom partitions with MergeTree indexes. | | Result cache | Yes, with cross-user result caching and intelligent cache management for up to 24 hours | Yes, results cache with TTL and query condition cache. | | Warm cache (SSD) | BI Engine provides in-memory caching and acceleration for frequently accessed data and dashboards | Yes, at indexed data-range level granularity | | Support for semi-structured data & JSON functions within SQL | Yes, including advanced JSON functions, Lambda expressions, and native support for nested and repeated fields | Yes, including Lambda expressions and native JSON data type (GA in v25.3) | | Vector Search and AI Capabilities | • BigQuery ML for in-database machine learning • Vertex AI integration • Natural language querying with Gemini AI • Limited vector search capabilities | • Native vector search capabilities and embeddings
• MCP Server for AI driven analytics
• Natural Language to SQL
• SQL based Inference | | Query Optimizations | • Advanced query optimizer with Dremel engine • Search indexes with column-level granularity • Materialized views with smart refresh and automatic query rewriting • BI Engine in-memory acceleration • Gemini AI-powered query optimization and natural language querying • Cross-user result caching • Automatic partitioning and clustering optimization • Cost-based optimization with intelligent workload management | • Primary indexes (ORDER BY)
• Data skipping indexes (minmax, set, bloom filters, ngrambf\_v1, tokenbf\_v1)
• Materialized views
• Projections
• PREWHERE optimization
• Query analysis tools
• Automatic global join reordering (v25.9)
• Enhanced JSON query optimization
• Streaming secondary indices | **BigQuery** lines up in benchmarks in the same ballpark as other cloud data warehouses but does come in consistently last in most queries. Beyond implementing according to best practices, there is little you can do to accelerate BigQuery performance, as it determines the amount of resources (slots) the query needs for you. BigQuery can be used together with "BigQuery BI Engine" for lower latency analytics. However, BI Engine is limited in terms of scale because it runs in memory. Its maximum capacity is 100GB. **ClickHouse** is famous for being one of the fastest local runtimes ever built for OLAP workloads. Its columnar storage, compression and indexing capabilities make it a consistent leader in benchmarks. Its lack of support for standard SQL and lack of query optimizer means that it's less suitable for traditional BI workloads, and more suitable for engineering managed workloads. While fast, it requires a lot of tuning and optimization. ## Use cases [#use-cases] There are a host of different analytics use cases that can be supported by a data warehouse. Look at your legacy technologies and their workloads, as well as the new possible use cases, and figure out which ones you will need to support in the next few years. | Feature | BigQuery | ClickHouse | | ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Low-latency dashboards | • Sub-second to seconds response times at TB+ scale with BI Engine acceleration • Search indexes and materialized views provide significant performance improvements for dashboard queries • Intelligent caching reduces query costs for repeated dashboard access • Dynamic concurrency management supports high user loads | • Sub-second load times at TB+ scale with proper indexing
• ClickHouse Cloud reduces engineering overhead with managed service • Proven low-latency performance (120ms at 2500 QPS in benchmarks)
• Purpose-built for low-latency OLAP and real-time analytics | | Enterprise BI | • Mature and comprehensive Enterprise DW feature set with native Google Cloud ecosystem integration • Strong integration with Looker, Looker Studio, and major BI tools • Gemini AI integration for natural language insights • Cross-cloud analytics capabilities • Advanced governance with Dataplex Universal Catalog | • Growing ecosystem with 50+ integrations including major BI tools
• Native MySQL protocol support enables broad BI tool compatibility • Strong SQL compliance with PostgreSQL compatibility
• Best suited for modern analytical workloads and engineering-managed use cases | | Data Apps and AI Applications (Customer-facing low-latency high concurrency) | • Dynamic concurrency supporting 1,000+ interactive queries with intelligent queuing and fair scheduling • Sub-second response times with BI Engine acceleration and search indexes • Serverless architecture eliminates infrastructure management overhead • Advanced caching and materialized views optimize repeated queries • AI-powered optimization for customer-facing applications • BigQuery ML for AI workloads | • Sub-second response times at TB+ scale
• Supports 1000 concurrent users per replica
• Strong price-performance on customer-facing applications
• Native vector search and embeddings | | Ad hoc | • Excellent for ad-hoc analytics with serverless architecture requiring zero infrastructure management • Intelligent query optimization handles unpredictable workloads automatically • Gemini AI integration enables natural language querying for business users • Advanced JSON support and schema inference enable flexible data exploration • Cross-cloud analytics capabilities for federated queries | • Good for ad-hoc queries with ClickHouse Cloud’s separated storage/compute architecture
• Join optimizations enable more query complexity
• Strong sampling capabilities (TABLESAMPLE) for exploratory analysis
• Resource management through user quotas prevents query interference
• Materialized views offer performance improvements for common aggregation patterns, ad-hoc users specify directly in SQL | **BigQuery** is a mature general-purpose data warehouse, which lends itself well to internal BI & reporting. The fact that it's serverless in nature and tightly integrated with GCP, makes it very convenient for Ad-Hoc analytics and ML use cases on GCP. On the other hand, because BigQuery makes resource allocation decisions for you, it is not always the best fit for operational use cases and Data Apps where performance needs to be consistent and predictable. **ClickHouse** was not designed to be a data warehouse, but rather a low-latency query execution runtime. Managing it typically requires significant engineering overhead. Hence, it's a good fit for engineering managed operational use cases and customer-facing data apps, where low latency matters. It is not a good fit for a general purpose data warehouse, nor for Ad-Hoc analytics or ELT. # BigQuery vs Databricks (2025) (/comparison/bigquery-vs-databricks) ## Architecture [#architecture] The biggest difference among cloud data warehouses are whether they separate storage and compute, how much they isolate data and compute, and what clouds they can run on. | Feature | BigQuery | Databricks | | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Separation of storage and compute | Fully serverless with complete separation of compute (Dremel) and storage (Colossus), powered by Jupiter network and Borg orchestration | Yes | | Supported cloud infrastructure | Google Cloud only | AWS, Azure, GCP. Marketplaces and BYOC | | Isolated tenancy – option for dedicated resources | • Multi-tenant pooled resources
• VPC Service Controls provide enhanced security and connectivity isolation to customer VPCs
• Cross-region disaster recovery for enterprise workloads | • Control plane in Databricks account
• Data plane in customer VPC (optional)
• Storage in customer VPC
• Serverless SQL runs in Databricks account with private connectivity | | Control vs abstraction of compute | Fully serverless with no control over compute resources – BigQuery automatically allocates computing resources as needed with intelligent workload management and dynamic slot allocation | • Configurable clusters and instance types
• Serverless SQL warehouses (GA 2025) run in Databricks account with private connectivity, no public IPs
• Pro/Classic warehouses run in customer VPC | | Self-hosted and hybrid deployment options | No self-hosted options – fully managed service only | • Databricks on customer cloud accounts
• Unity Catalog for hybrid governance | | ACID Compliance and Transactions | Limited ACID support – eventual consistency model with some transactional capabilities | • ACID transactions with Delta Lake
• Time travel and versioning
• Concurrent read/write operations | **BigQuery** was one of the first decoupled storage and compute architectures. It is a unique piece of engineering and not a typical data warehouse in part because it started as an on-demand serverless query engine. It runs in multi-tenancy with shared resources, allocated as “slots” which represent a virtual CPU that executes SQL. BigQuery determines how many slots a query requires, without the ability of the user to control it. BigQuery can be priced on a $/TB scanned basis or through slot reservations. A slot in BigQuery is logically equivalent to 0.5 vCPU and 0.5GB of RAM. There are multiple models to allocate slots in BigQuery. **Databricks** was built by the founders of Spark as an analytics platform to support machine learning use cases. It leverages the Spark framework to process data residing in a data lake and is supported on AWS, GCP and Azure. Databricks coined the marketing term “Lakehouse '' architecture to illustrate the unification of data lake and data warehouse use cases. Customers still manage Spark clusters that process data residing in a Delta lake. Conversion of data to Delta Lake format is required to leverage the functionality of Delta Lake. Databricks Sql is a relatively new addition to simplify access to data stored in a data lake. ## Scalability [#scalability] There are three big differences among data warehouses and query engines that limit scalability: decoupled storage and compute, dedicated resources, and continuous ingestion. | Feature | BigQuery | Databricks | | --------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Elasticity – Scaling for larger data volumes and faster queries | Fully automated serverless scaling – BigQuery automatically determines resource allocation and scales to petabytes without user intervention. Can dynamically burst beyond baseline slot allocations for performance optimization | Autoscaling clusters based on workload demand. Serverless SQL warehouses provide near-instant scaling (2-6 seconds startup) | | Elasticity – Scaling for higher concurrency | Dynamic concurrency management with query queueing supporting up to 1,000 interactive queries and 20,000 batch queries per project per region. Automatic fair scheduling and slot distribution across workloads | • 10 concurrent queries per cluster limit • Scales up to 40 clusters per warehouse (400 total concurrent queries) • Serverless SQL warehouses provide near-instant autoscaling • Pro/Classic warehouses take several minutes to provision new clusters • Real-world performance degradation typically occurs at 50-150 concurrent queries depending on complexity | **BigQuery** scales very well to large data volumes, and automatically assigns more compute resources when needed behind the scenes, in the form of “slots”. BigQuery works either in an “on-demand pricing model”, where slot assignment is completely in the hands of BigQuery and the state of the shared resource pool, or in “flat-rate pricing model” where slots are reserved in advance. With reserved slots there is more control over compute resources, thus making scaling more predictable. Concurrency is limited to 100 users by default. **Databricks** allow for autoscaling of clusters based on utilization. Additionally, increasing concurrency associated with a sql endpoint can be accomplished through the addition of clusters. Query concurrency per cluster is maxed at 10. However, scaling with additional clusters for concurrency is possible. Databricks provides a choice of instance types. ## Performance [#performance] Performance is the biggest challenge with most data warehouses today. While decoupled storage and compute architectures improved scalability and simplified administration, for most data warehouses it introduced two bottlenecks; storage, and compute. Most modern cloud data warehouses fetch entire partitions over the network instead of just fetching the specific data needed for each query. While many invest in caching, most do not invest heavily in query optimization. Most vendors also have not improved continuous ingestion or semi-structured data analytics performance, both of which are needed for operational and customer-facing use cases. | Feature | BigQuery | Databricks | | ------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Indexes | Search indexes (GA) for efficient text search optimization on STRING, JSON, and array columns. Support for LOG\_ANALYZER, NO\_OP\_ANALYZER, and PATTERN\_ANALYZER with column-level granularity for improved query performance and cost efficiency | None | | Compute tuning | Serverless architecture with automatic resource optimization – no manual tuning required. Intelligent workload management with AI-powered resource allocation and dynamic slot distribution | Choice of cluster type, node types including SSD-optimized instances. Serverless provides automatic resource allocation with Intelligent Workload Management (IWM) | | Storage format | Columnar & compressed storage (Capacitor format) with support for open table formats including Apache Iceberg, Delta Lake, and Hudi. Intelligent tiering with automatic long-term storage cost reduction after 90 days | • Delta Lake format with Liquid Clustering (February 2025 – replaces Z-ordering and traditional partitioning) • Cannot use Liquid Clustering alongside Z-ordering on same table • Allows for sorted data in Delta Lake • Requires Optimize to maintain ordering | | Table-level partition & pruning techniques | • Automatic table organization with intelligent micro-partitioning • Clustering keys for data organization • Automatic partition pruning optimization • Supports time-based and custom partitioning strategies | • Table level partitioning • Liquid Clustering for improved query performance and reduced data skew (February 2025) • Z-ordering (legacy, replaced by Liquid Clustering) • Periodic optimization of storage required | | Result cache | Yes, with cross-user result caching and intelligent cache management for up to 24 hours | Multi-layered caching: local in-memory cache per cluster plus remote result cache (serverless only) that persists across all warehouses in workspace | | Warm cache (SSD) | BI Engine provides in-memory caching and acceleration for frequently accessed data and dashboards | Yes. Delta cache for data read by queries at file level granularity | | Support for semi-structured data & JSON functions within SQL | Yes, including advanced JSON functions, Lambda expressions, and native support for nested and repeated fields | Yes | | Vector Search and AI Capabilities | • BigQuery ML for in-database machine learning • Vertex AI integration • Natural language querying with Gemini AI • Limited vector search capabilities | • MLflow integration and Databricks ML platform • Native vector search in Delta Lake (Vector Search) • AI and ML workloads optimized | | Query Optimizations | • Advanced query optimizer with Dremel engine • Search indexes with column-level granularity • Materialized views with smart refresh and automatic query rewriting • BI Engine in-memory acceleration • Gemini AI-powered query optimization and natural language querying • Cross-user result caching • Automatic partitioning and clustering optimization • Cost-based optimization with intelligent workload management | • Photon engine (C++ vectorized engine providing 3-8x average speedups, maximum speedups over 10x) • Automated stats collection (January 2025) enables cost-based optimization • Predictive I/O for faster point lookups and data updates • Liquid Clustering (February 2025) • Intelligent Workload Management (IWM) with AI-powered resource allocation • Delta cache • Materialized views support | **BigQuery** lines up in benchmarks in the same ballpark as other cloud data warehouses but does come in consistently last in most queries. Beyond implementing according to best practices, there is little you can do to accelerate BigQuery performance, as it determines the amount of resources (slots) the query needs for you. BigQuery can be used together with the “BigQuery BI Engine” for lower latency analytics. However, BI Engine is limited in terms of scale because it runs in memory. Its maximum capacity is 100GB. **Databricks** is designed to leverage the Spark framework for processing large volumes of data. It leverages compressed Parquet files in a Delta Lake. To reduce the amount of data processed, it uses data pruning on partitions and Parquet file metadata. Databricks does not provide any indexes. ## Use cases [#use-cases] There are a host of different analytics use cases that can be supported by a data warehouse. Look at your legacy technologies and their workloads, as well as the new possible use cases, and figure out which ones you will need to support in the next few years. | Feature | BigQuery | Databricks | | ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Low-latency dashboards | • Sub-second to seconds response times at TB+ scale with BI Engine acceleration • Search indexes and materialized views provide significant performance improvements for dashboard queries • Intelligent caching reduces query costs for repeated dashboard access • Dynamic concurrency management supports high user loads | • Sub-second to seconds load times at TB+ scale • Enhanced by Photon engine (3-8x average speedups) and Delta cache • Serverless SQL warehouses provide rapid startup (2-6 seconds) • Performance depends on cluster configuration | | Enterprise BI | • Mature and comprehensive Enterprise DW feature set with native Google Cloud ecosystem integration • Strong integration with Looker, Looker Studio, and major BI tools • Gemini AI integration for natural language insights • Cross-cloud analytics capabilities • Advanced governance with Dataplex Universal Catalog | • Strong for data science and ML workloads • Unified analytics platform approach • Growing traditional BI integrations • Serverless SQL warehouses improve accessibility • Delta sharing capabilities | | Data Apps and AI Applications (Customer-facing low-latency high concurrency) | • Dynamic concurrency supporting 1,000+ interactive queries with intelligent queuing and fair scheduling • Sub-second response times with BI Engine acceleration and search indexes • Serverless architecture eliminates infrastructure management overhead • Advanced caching and materialized views optimize repeated queries • AI-powered optimization for customer-facing applications • BigQuery ML for AI workloads | • 10 concurrent queries per cluster, scaling to 400 total concurrent queries per warehouse • Real-world performance degradation typically occurs at 50-150 concurrent queries depending on workload complexity • Serverless provides near-instant autoscaling • Photon engine delivers 3-8x performance improvements • Strong ML and AI platform integration | | Ad hoc | • Excellent for ad-hoc analytics with serverless architecture requiring zero infrastructure management • Intelligent query optimization handles unpredictable workloads automatically • Gemini AI integration enables natural language querying for business users • Advanced JSON support and schema inference enable flexible data exploration • Cross-cloud analytics capabilities for federated queries | • Excellent for ad-hoc with decoupled storage/compute • Serverless SQL warehouses provide instant provisioning • Intelligent Workload Management handles unpredictable workloads automatically • Strong for exploratory data analysis and ML workloads • Automated stats collection improves query planning | **BigQuery** is a mature general-purpose data warehouse, which lends itself well to internal BI & reporting. The fact that it’s serverless in nature and tightly integrated with GCP, makes it very convenient for Ad-Hoc analytics and ML use cases on GCP. On the other hand, because BigQuery makes resource allocation decisions for you, it is not always the best fit for operational use cases and Data Apps where performance needs to be consistent and predictable. **Databricks** is a mature Spark based platform proven for processing streaming data. It is widely used for Machine Learning use cases by data scientists through the use of integrated notebooks. From a low latency query perspective, while it offers features like Delta Cache, it does not provide specialized indexes that can deliver low latency queries. # BigQuery vs Druid (2025) (/comparison/bigquery-vs-druid) The H2 headings, prose, and tables converted to clean MDX body: ## Architecture [#architecture] The biggest difference among cloud data warehouses are whether they separate storage and compute, how much they isolate data and compute, and what clouds they can run on. | Feature | BigQuery | Druid | | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Separation of storage and compute | Fully serverless with complete separation of compute (Dremel) and storage (Colossus), powered by Jupiter network and Borg orchestration | No | | Supported cloud infrastructure | Google Cloud only | Can be installed anywhere | | Isolated tenancy – option for dedicated resources | • Multi-tenant pooled resources
• VPC Service Controls provide enhanced security and connectivity isolation to customer VPCs
• Cross-region disaster recovery for enterprise workloads | Single tenant | | Control vs abstraction of compute | Fully serverless with no control over compute resources – BigQuery automatically allocates computing resources as needed with intelligent workload management and dynamic slot allocation | • Complex configuration of compute tier with multiple role-specific nodes
• Configurable node count
• Configurable compute types (virtual machines or kubernetes) | | Self-hosted and hybrid deployment options | No self-hosted options – fully managed service only | Self-managed deployment required | | ACID Compliance and Transactions | Limited ACID support – eventual consistency model with some transactional capabilities | Limited ACID support with eventual consistency | **BigQuery** was one of the first decoupled storage and compute architectures. It is a unique piece of engineering and not a typical data warehouse in part because it started as an on-demand serverless query engine. It runs in multi-tenancy with shared resources, allocated as “slots” which represent a virtual CPU that executes SQL. BigQuery determines how many slots a query requires, without the ability of the user to control it. BigQuery can be priced on a $/TB scanned basis or through slot reservations. A slot in BigQuery is logically equivalent to 0.5 vCPU and 0.5GB of RAM. There are multiple models to allocate slots in BigQuery. **Druid** is an OLAP engine designed to provide fast real time analytics. Druid adopts a clustered architecture with servers that host various role specific processes. These processes address real time and batch ingestion, indexing, querying of historical and real time data. Apache Druid can be deployed as a virtual machine or a Kubernetes based cluster. Druid does not support a decoupled compute & storage architecture. Deep storage in the form of object storage is used to replicate data to. ## Scalability [#scalability] There are three big differences among data warehouses and query engines that limit scalability: decoupled storage and compute, dedicated resources, and continuous ingestion. | Feature | BigQuery | Druid | | --------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- | | Elasticity – Scaling for larger data volumes and faster queries | Fully automated serverless scaling – BigQuery automatically determines resource allocation and scales to petabytes without user intervention. Can dynamically burst beyond baseline slot allocations for performance optimization | Scale-up of nodes requires careful planning and downtime. Addition of new nodes for scale-out is possible | | Elasticity – Scaling for higher concurrency | Dynamic concurrency management with query queueing supporting up to 1,000 interactive queries and 20,000 batch queries per project per region. Automatic fair scheduling and slot distribution across workloads | Supports 100s to 100,000s queries per second (1000+ QPS) with proper configuration and scaling | **BigQuery** scales very well to large data volumes, and automatically assigns more compute resources when needed behind the scenes, in the form of “slots”. BigQuery works either in an “on-demand pricing model”, where slot assignment is completely in the hands of BigQuery and the state of the shared resource pool, or in “flat-rate pricing model” where slots are reserved in advance. With reserved slots there is more control over compute resources, thus making scaling more predictable. Concurrency is limited to 100 users by default. **Druid** provides the ability to handle fast ingest and high concurrency. Custom sizing and cluster tuning are required to balance the compute, memory, storage needs of each process within Druid and to provide high concurrency. Druid clusters can be grown by adding nodes with automatic rebalancing of storage segments assigned to nodes. Self hosted Druid on Kubernetes is an option that users leverage to simplify scaling. Additionally, Cloud based managed Druid offerings are being rolled out. However, these managed offerings are limited in scale and scaling is not granular. ## Performance [#performance] Performance is the biggest challenge with most data warehouses today. While decoupled storage and compute architectures improved scalability and simplified administration, for most data warehouses it introduced two bottlenecks; storage, and compute. Most modern cloud data warehouses fetch entire partitions over the network instead of just fetching the specific data needed for each query. While many invest in caching, most do not invest heavily in query optimization. Most vendors also have not improved continuous ingestion or semi-structured data analytics performance, both of which are needed for operational and customer-facing use cases. | Feature | BigQuery | Druid | | ------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | | Indexes | Search indexes (GA) for efficient text search optimization on STRING, JSON, and array columns. Support for LOG\_ANALYZER, NO\_OP\_ANALYZER, and PATTERN\_ANALYZER with column-level granularity for improved query performance and cost efficiency | Compressed bitmap indexes for data access and roll-ups to manage aggregations | | Compute tuning | Serverless architecture with automatic resource optimization – no manual tuning required. Intelligent workload management with AI-powered resource allocation and dynamic slot distribution | On-premises, self-managed hardware. Druid requires infrastructure management and leverages commonly available instance types | | Storage format | Columnar & compressed storage (Capacitor format) with support for open table formats including Apache Iceberg, Delta Lake, and Hudi. Intelligent tiering with automatic long-term storage cost reduction after 90 days | Columnar storage format with time-based sorting | | Table-level partition & pruning techniques | • Automatic table organization with intelligent micro-partitioning • Clustering keys for data organization • Automatic partition pruning optimization • Supports time-based and custom partitioning strategies | Restrictive time-based partitioning. Can partition based on other secondary columns | | Result cache | Yes, with cross-user result caching and intelligent cache management for up to 24 hours | Ability to support caching on broker (set to off by default) | | Warm cache (SSD) | BI Engine provides in-memory caching and acceleration for frequently accessed data and dashboards | Yes, at much larger segment level granularity | | Support for semi-structured data & JSON functions within SQL | Yes, including advanced JSON functions, Lambda expressions, and native support for nested and repeated fields | Recommend flattening JSON or translate to array prior to loading. No support for JSON parsing at query runtime | | Vector Search and AI Capabilities | • BigQuery ML for in-database machine learning • Vertex AI integration • Natural language querying with Gemini AI • Limited vector search capabilities | No native AI or vector search capabilities | | Query Optimizations | • Advanced query optimizer with Dremel engine • Search indexes with column-level granularity • Materialized views with smart refresh and automatic query rewriting • BI Engine in-memory acceleration • Gemini AI-powered query optimization and natural language querying • Cross-user result caching • Automatic partitioning and clustering optimization • Cost-based optimization with intelligent workload management | • Compressed bitmap indexes • Roll-up aggregations • Time-based optimization • Query optimization requires manual tuning | **BigQuery** lines up in benchmarks in the same ballpark as other cloud data warehouses but does come in consistently last in most queries. Beyond implementing according to best practices, there is little you can do to accelerate BigQuery performance, as it determines the amount of resources (slots) the query needs for you. BigQuery can be used together with the “BigQuery BI Engine” for lower latency analytics. However, BI Engine is limited in terms of scale because it runs in memory. Its maximum capacity is 100GB. **Druid** provides high performance through columnar storage format, parallel processing, bitmap indexes and roll-ups. Druid, however, recommends a denormalized data model for performance needs. Join operations in Druid are a relatively new feature with various limitations, especially if there is a need to join large datasets. ## Use cases [#use-cases] There are a host of different analytics use cases that can be supported by a data warehouse. Look at your legacy technologies and their workloads, as well as the new possible use cases, and figure out which ones you will need to support in the next few years. | Feature | BigQuery | Druid | | ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Low-latency dashboards | • Sub-second to seconds response times at TB+ scale with BI Engine acceleration • Search indexes and materialized views provide significant performance improvements for dashboard queries • Intelligent caching reduces query costs for repeated dashboard access • Dynamic concurrency management supports high user loads | • Sub-second load times optimized for time-series and real-time analytics • Built for high-concurrency interactive dashboards • Requires denormalized data model | | Enterprise BI | • Mature and comprehensive Enterprise DW feature set with native Google Cloud ecosystem integration • Strong integration with Looker, Looker Studio, and major BI tools • Gemini AI integration for natural language insights • Cross-cloud analytics capabilities • Advanced governance with Dataplex Universal Catalog | • Limited integrations with traditional Enterprise BI tools • Strong for real-time operational dashboards • Requires specialized visualization tools | | Data Apps and AI Applications (Customer-facing low-latency high concurrency) | • Dynamic concurrency supporting 1,000+ interactive queries with intelligent queuing and fair scheduling • Sub-second response times with BI Engine acceleration and search indexes • Serverless architecture eliminates infrastructure management overhead • Advanced caching and materialized views optimize repeated queries • AI-powered optimization for customer-facing applications • BigQuery ML for AI workloads | • Built for high concurrency (1000+ QPS) with distributed architecture • Sub-second response times for time-series data • Optimized for real-time operational applications • No AI capabilities | | Ad hoc | • Excellent for ad-hoc analytics with serverless architecture requiring zero infrastructure management • Intelligent query optimization handles unpredictable workloads automatically • Gemini AI integration enables natural language querying for business users • Advanced JSON support and schema inference enable flexible data exploration • Cross-cloud analytics capabilities for federated queries | • Not optimized for ad-hoc queries • Requires predefined roll-ups and data modeling • Limited flexibility for exploratory analysis | **BigQuery** is a mature general-purpose data warehouse, which lends itself well to internal BI & reporting. The fact that it’s serverless in nature and tightly integrated with GCP, makes it very convenient for Ad-Hoc analytics and ML use cases on GCP. On the other hand, because BigQuery makes resource allocation decisions for you, it is not always the best fit for operational use cases and Data Apps where performance needs to be consistent and predictable. **Druid** is designed as an OLAP engine to provide fast access to aggregations that are run against large volumes of data. Druid is typically used for customer facing analytics and streaming data processing. Druid is used as an add-on with other data warehousing products that are efficient at scaling, joining, and filtering large volumes of data. It is not a suitable option for data warehouse replacement. # BigQuery vs Redshift (/comparison/bigquery-vs-redshift) ## Architecture [#architecture] The biggest difference among cloud data warehouses are whether they separate storage and compute, how much they isolate data and compute, and what clouds they can run on. | Feature | BigQuery | Redshift | | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------ | | Separation of storage and compute | Fully serverless with complete separation of compute (Dremel) and storage (Colossus), powered by Jupiter network and Borg orchestration | RA3 instances enable separation of compute and storage, but limited workload isolation compared to other platforms | | Supported cloud infrastructure | Google Cloud only | AWS only | | Isolated tenancy – option for dedicated resources | • Multi-tenant pooled resources
• VPC Service Controls provide enhanced security and connectivity isolation to customer VPCs
• Cross-region disaster recovery for enterprise workloads | • Isolated tenant & resources • Runs in your VPC | | Control vs abstraction of compute | Fully serverless with no control over compute resources – BigQuery automatically allocates computing resources as needed with intelligent workload management and dynamic slot allocation | • Configurable cluster size
• Configurable compute types | | Self-hosted and hybrid deployment options | No self-hosted options – fully managed service only | Limited hybrid options with Redshift Serverless | | ACID Compliance and Transactions | Limited ACID support – eventual consistency model with some transactional capabilities | ACID compliant at table level with some limitations on concurrent operations | **BigQuery** was one of the first decoupled storage and compute architectures. It is a unique piece of engineering and not a typical data warehouse in part because it started as an on-demand serverless query engine. It runs in multi-tenancy with shared resources, allocated as "slots" which represent a virtual CPU that executes SQL. BigQuery determines how many slots a query requires, without the ability of the user to control it. BigQuery can be priced on a $/TB scanned basis or through slot reservations. A slot in BigQuery is logically equivalent to 0.5 vCPU and 0.5GB of RAM. There are multiple models to allocate slots in BigQuery. **Redshift** has the oldest architecture, being the first Cloud DW in the group. Its architecture wasn't designed to separate storage & compute. While it now has RA3 nodes which allow you to scale compute and only cache the data you need locally, all compute still operates together. You cannot separate and isolate different workloads over the same data, which puts it behind other decoupled storage/compute architectures. Redshift runs as an isolated tenant per customer, and unlike other cloud data warehouses, it is deployed in your VPC. Redshift offers a serverless option which is based on an abstracted unit called Redshift Processing Unit (RPU) ranging from 8 to 512 in increments of 8. Each RPU provides 2 vCPU and 16GB RAM. Thus, 8 RPU is equivalent to 16 vCPU / 128GB RAM. The minimum RPU is 8. ## Scalability [#scalability] There are three big differences among data warehouses and query engines that limit scalability: decoupled storage and compute, dedicated resources, and continuous ingestion. | Feature | BigQuery | Redshift | | --------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | | Elasticity – Scaling for larger data volumes and faster queries | Fully automated serverless scaling – BigQuery automatically determines resource allocation and scales to petabytes without user intervention. Can dynamically burst beyond baseline slot allocations for performance optimization | Available via Elastic Resize – slow and limited, downtime required | | Elasticity – Scaling for higher concurrency | Dynamic concurrency management with query queueing supporting up to 1,000 interactive queries and 20,000 batch queries per project per region. Automatic fair scheduling and slot distribution across workloads | • 5 concurrent queries per WLM queue by default (up to 8 queues) • Concurrency Scaling enables thousands of concurrent queries | **BigQuery** scales very well to large data volumes, and automatically assigns more compute resources when needed behind the scenes, in the form of "slots". BigQuery works either in an "on-demand pricing model", where slot assignment is completely in the hands of BigQuery and the state of the shared resource pool, or in "flat-rate pricing model" where slots are reserved in advanced. With reserved slots there is more control over compute resources, thus making scaling more predictable. Concurrency is limited to 100 users by default. **Redshift** is limited in scale because even with RA3, it cannot distribute different workloads across clusters. While it can scale to up to 10 clusters automatically to support query concurrency, it can only handle a maximum of 50 queued queries across all clusters by default. ## Performance [#performance] Performance is the biggest challenge with most data warehouses today. While decoupled storage and compute architectures improved scalability and simplified administration, for most data warehouses it introduced two bottlenecks; storage, and compute. Most modern cloud data warehouses fetch entire partitions over the network instead of just fetching the specific data needed for each query. While many invest in caching, most do not invest heavily in query optimization. Most vendors also have not improved continuous ingestion or semi-structured data analytics performance, both of which are needed for operational and customer-facing use cases. | Feature | BigQuery | Redshift | | ------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Indexes | Search indexes (GA) for efficient text search optimization on STRING, JSON, and array columns. Support for LOG\_ANALYZER, NO\_OP\_ANALYZER, and PATTERN\_ANALYZER with column-level granularity for improved query performance and cost efficiency | None | | Compute tuning | Serverless architecture with automatic resource optimization – no manual tuning required. Intelligent workload management with AI-powered resource allocation and dynamic slot distribution | Choice over number of nodes and their type | | Storage format | Columnar & compressed storage (Capacitor format) with support for open table formats including Apache Iceberg, Delta Lake, and Hudi. Intelligent tiering with automatic long-term storage cost reduction after 90 days | Columnar & compressed storage (RA3 nodes) | | Table-level partition & pruning techniques | • Automatic table organization with intelligent micro-partitioning • Clustering keys for data organization • Automatic partition pruning optimization • Supports time-based and custom partitioning strategies | • No table partitions • User-defined distribution & sort keys are used to optimize for speed | | Result cache | Yes, with cross-user result caching and intelligent cache management for up to 24 hours | Yes | | Warm cache (SSD) | BI Engine provides in-memory caching and acceleration for frequently accessed data and dashboards | Only with RA3 nodes at partition-level granularity | | Support for semi-structured data & JSON functions within SQL | Yes, including advanced JSON functions, Lambda expressions, and native support for nested and repeated fields | Limited | | Vector Search and AI Capabilities | • BigQuery ML for in-database machine learning • Vertex AI integration • Natural language querying with Gemini AI • Limited vector search capabilities | Limited AI capabilities – primarily through integrations | | Query Optimizations | • Advanced query optimizer with Dremel engine • Search indexes with column-level granularity • Materialized views with smart refresh and automatic query rewriting • BI Engine in-memory acceleration • Gemini AI-powered query optimization and natural language querying • Cross-user result caching • Automatic partitioning and clustering optimization • Cost-based optimization with intelligent workload management | • Basic query optimizer • Materialized views • Result caching • ANALYZE for table statistics • Workload management (WLM) • Automated materialized views (AutoMV) • AI-driven scaling (Serverless preview) | **BigQuery** lines up in benchmarks in the same ballpark as other cloud data warehouses but does come in consistently last in most queries. Beyond implementing according to best practices, there is little you can do to accelerate BigQuery performance, as it determines the amount of resources (slots) the query needs for you. BigQuery can be used together with "BigQuery BI Engine" for lower latency analytics. However, BI Engine is limited in terms of scale because it runs in memory. Its maximum capacity is 100GB. **Redshift** does provide a result cache for accelerating repetitive query workloads and also has more tuning options than some others. But it does not deliver much faster compute performance than other cloud data warehouses in benchmarks. Sort keys can be used to optimize performance, but their contribution is limited. There is no support for indexes, and low-latency analytics at large data volumes is hard to achieve. Because Redshift decoupling of storage & compute is limited compared to other cloud data warehouses, it doesn't support isolating workloads, which means performance can degrade under pressure and competition for resources. ## Use cases [#use-cases] There are a host of different analytics use cases that can be supported by a data warehouse. Look at your legacy technologies and their workloads, as well as the new possible use cases, and figure out which ones you will need to support in the next few years. | Feature | BigQuery | Redshift | | ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Low-latency dashboards | • Sub-second to seconds response times at TB+ scale with BI Engine acceleration • Search indexes and materialized views provide significant performance improvements for dashboard queries • Intelligent caching reduces query costs for repeated dashboard access • Dynamic concurrency management supports high user loads | • Seconds to tens of seconds load times at 100s of GB scale • Can achieve faster performance with Concurrency Scaling and proper tuning | | Enterprise BI | • Mature and comprehensive Enterprise DW feature set with native Google Cloud ecosystem integration • Strong integration with Looker, Looker Studio, and major BI tools • Gemini AI integration for natural language insights • Cross-cloud analytics capabilities • Advanced governance with Dataplex Universal Catalog | • Mature and comprehensive Enterprise DW feature set • Extensive integrations with Enterprise BI ecosystem • Strong AWS ecosystem integration | | Data Apps and AI Applications (Customer-facing low-latency high concurrency) | • Dynamic concurrency supporting 1,000+ interactive queries with intelligent queuing and fair scheduling • Sub-second response times with BI Engine acceleration and search indexes • Serverless architecture eliminates infrastructure management overhead • Advanced caching and materialized views optimize repeated queries • AI-powered optimization for customer-facing applications • BigQuery ML for AI workloads | • 5 concurrent queries per WLM queue by default (up to 8 queues) • Concurrency Scaling enables thousands of concurrent queries • Seconds-level response times typical • Automatic scaling for burst workloads • Limited AI application support | | Ad hoc | • Excellent for ad-hoc analytics with serverless architecture requiring zero infrastructure management • Intelligent query optimization handles unpredictable workloads automatically • Gemini AI integration enables natural language querying for business users • Advanced JSON support and schema inference enable flexible data exploration • Cross-cloud analytics capabilities for federated queries | • Performance dependent on predefined distribution & sort keys • Elastic Resize enables adding compute resources • Typically subset of data loaded for ad-hoc analysis | **BigQuery** is a mature general-purpose data warehouse, which lends itself well to internal BI & reporting. The fact that it's serverless in nature and tightly integrated with GCP, makes it very convenient for Ad-Hoc analytics and ML use cases on GCP. On the other hand, because BigQuery makes resource allocation decisions for you, it is not always the best fit for operational use cases and Data Apps where performance needs to be consistent and predictable. **Redshift** was originally designed to support traditional internal BI reporting and dashboard use cases for analysts. As such, it is typically used as a general-purpose Enterprise data warehouse. With deep integrations into the AWS ecosystem, it can also leverage AWS ML service, making it also useful for ML projects. However, given the coupling of storage & compute, and the difficulty in delivering low-latency analytics at scale, it is less suited for operational use cases and customer-facing use cases like Data Apps. The coupling of storage and compute, together with the need to predefine sort & dist keys for optimal performance, make it challenging to use for Ad-Hoc analytics. # ClickHouse vs Athena (2025) (/comparison/clickhouse-vs-athena) ## Architecture [#architecture] The biggest difference among cloud data warehouses are whether they separate storage and compute, how much they isolate data and compute, and what clouds they can run on. | Feature | ClickHouse | Athena | | ------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Separation of storage and compute | Yes – SharedMergeTree engine in ClickHouse Cloud enables full separation of storage and compute, with compute-compute separation through Warehouses feature (introduced 2025) allowing multiple isolated compute services sharing the same data | Yes, serverless with optional provisioned capacity. Workloads can be isolated through Workgroups and Capacity Reservations | | Supported cloud infrastructure | AWS, GCP, Azure, cloud service and on-premises | AWS only | | Isolated tenancy – option for dedicated resources | • Multi-tenant metadata layer
• Isolated tenancy for compute & storage per client in cloud | • Multi-tenant pooled resources by default
• Dedicated compute resources available via Provisioned Capacity
• VPC endpoint connections supported | | Control vs abstraction of compute | Configurable cluster size and compute types in ClickHouse Cloud with granular control over nodes (1-128 nodes) and node characteristics. Warehouses feature enables multiple isolated read-only compute environments. | • Serverless by default with no infrastructure control
• Optional Provisioned Capacity allows dedicated DPU allocation (minimum 24 DPUs)
• Two pricing models: on-demand ($5/TB scanned) or provisioned ($0.30/DPU-hour) | | Self-hosted and hybrid deployment options | Self-managed deployments available with full control over infrastructure | No self-hosted options – serverless only | | ACID Compliance and Transactions | Limited ACID compliance with MergeTree engine family. | No ACID compliance – eventual consistency model | **ClickHouse** was originally developed at Yandex, the Russian search engine, as an OLAP engine for low latency analytics. It was built as an on-premise solution with coupled storage & compute, and a large variety of tuning options in the form of indexes and and merge trees. ClickHouse's architecture is famous for its focus on performance and low-latency queries. The tradeoff is that it is considered very difficult to work with. SQL support is very limited, and tuning/running it requires significant engineering resources. **Athena** is serverless and built on a decoupled storage and compute architecture that queries data directly in S3, without the need to ingest/copy the data. It runs in multi-tenancy with shared resources. Users do not have control over the compute resources Athena chooses to allocate per query from the shared resource pool. For folks requiring additional or dedicated resources, they can reserve dedicated processing capacity in the form of Data Processing Units (DPU), with each DPU providing 4 vCPU and 16 GB RAM. RPU allocation ranges from 24 - 1000 per region. ## Scalability [#scalability] There are three big differences among data warehouses and query engines that limit scalability: decoupled storage and compute, dedicated resources, and continuous ingestion. | Feature | ClickHouse | Athena | | --------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Elasticity – Scaling for larger data volumes and faster queries | Automatic horizontal and vertical scaling in ClickHouse Cloud with SharedMergeTree architecture. Manual scaling for self-managed deployments with cluster rebalancing capabilities | • Fully abstracted on-demand scaling • Provisioned Capacity allows manual scaling of DPUs for predictable performance • Capacity reservations can be adjusted with minimum 1-hour billing periods | | Elasticity – Scaling for higher concurrency | Supports high concurrency with proper resource allocation and configuration. Vertical auto-scaling and horizontal manual scaling. Additional warehouses can idle to zero billing. Primary service always on in multi-warehouse configurations. | • Default limit of 25 concurrent DML queries and 20 DDL queries (adjustable via service quotas) • Provisioned Capacity enables higher concurrency with dedicated DPUs • Query queuing available when capacity is exceeded | **ClickHouse** doesn't offer any dedicated scaling features or mechanisms. While it can deliver linearly scalable performance for some types of queries, scaling itself has to be done manually. Hardware is self-managed in ClickHouse. This means that to scale you would have to provision a cluster and migrate. **Athena** is a shared multi-tenant resource, with no guarantees on the amount or availability of the resources allocated for your queries. From a data volume perspective, it can scale to large volumes, but large data volumes can suffer from very long run times and frequent time outs. Query concurrency is maxed at 20. If scalability is a top priority, Athena is probably not the best choice. ## Performance [#performance] Performance is the biggest challenge with most data warehouses today. While decoupled storage and compute architectures improved scalability and simplified administration, for most data warehouses it introduced two bottlenecks; storage, and compute. Most modern cloud data warehouses fetch entire partitions over the network instead of just fetching the specific data needed for each query. While many invest in caching, most do not invest heavily in query optimization. Most vendors also have not improved continuous ingestion or semi-structured data analytics performance, both of which are needed for operational and customer-facing use cases. | Feature | ClickHouse | Athena | | ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Indexes | • Primary indexes • Skipping indexes (minmax, set, bloom filters, ngrambf\_v1, tokenbf\_v1)
• MergeTree indexes
• Incremental Materialized views | No traditional indexes – relies on partition pruning and data organization in S3. Uses columnar formats and compression for optimization | | Compute tuning | Configurable compute resources in cloud offering | • No compute tuning in on-demand mode • Provisioned Capacity allows DPU allocation control (4 vCPU and 16GB RAM per DPU) • Minimum 24 DPUs with scaling in 4-DPU increments | | Storage format | Columnar, supports sorted, compressed, encoded & sparsely indexed files with native Apache Iceberg support. | Supports multiple formats: Parquet, ORC, Avro, JSON, CSV, TSV on S3. Native support for open table formats including Apache Iceberg, Apache Hudi, and Delta Lake | | Table-level partition & pruning techniques | Partitioning by date/time and custom partitions with MergeTree indexes. | • User-defined table-level partitions with Hive-style partitioning • Pruning at partition level • Partition projection for advanced performance optimization • Supports open table formats with built-in partitioning | | Result cache | Yes, results cache with TTL and query condition cache. | Query result caching for up to 30 days with configurable retention. Results reuse supported across workgroups | | Warm cache (SSD) | Yes, at indexed data-range level granularity | No local caching – queries data directly from S3. Relies on S3's performance characteristics and intelligent tiering | | Support for semi-structured data & JSON functions within SQL | Yes, including Lambda expressions and native JSON data type (GA in v25.3) | Yes, comprehensive JSON support including Lambda expressions, array functions, and native nested data handling | | Vector Search and AI Capabilities | • Native vector search capabilities and embeddings
• MCP Server for AI driven analytics
• Natural Language to SQL
• SQL based Inference | No native AI or vector search capabilities | | Query Optimizations | • Primary indexes (ORDER BY)
• Data skipping indexes (minmax, set, bloom filters, ngrambf\_v1, tokenbf\_v1)
• Materialized views
• Projections
• PREWHERE optimization
• Query analysis tools
• Automatic global join reordering (v25.9)
• Enhanced JSON query optimization
• Streaming secondary indices | • Cost-based optimizer (CBO) in Athena engine v3 • Query result caching (up to 30 days) • Partition projection for advanced optimization • CTAS for precomputed queries • Join reordering and aggregation pushdown • Automatic parallel query execution • Support for columnar formats (Parquet, ORC) • Integration with AWS Glue Data Catalog | **ClickHouse** is famous for being one of the fastest local runtimes ever built for OLAP workloads. Its columnar storage, compression and indexing capabilities make it a consistent leader in benchmarks. Its lack of support for standard SQL and lack of query optimizer means that it's less suitable for traditional BI workloads, and more suitable for engineering managed workloads. While fast, it requires a lot of tuning and optimization. **Athena** (and Presto) are designed to query data where it is, sacrificing storage-compute optimizations. This makes it very convenient for easy and immediate querying but at the expense of performance. This typically puts Athena behind cloud data warehouses in terms of performance. But Athena still does relatively well in performance benchmarks, especially when external storage is managed by experts. While it supports partitions, there is no support for indexing, and together with the fact that resources are pooled from a shared multi-tenant service, low-latency and consistent performance are not Athena's sweet spot. A cloud data warehouse be more performant better than Athena in most cases. ## Use cases [#use-cases] There are a host of different analytics use cases that can be supported by a data warehouse. Look at your legacy technologies and their workloads, as well as the new possible use cases, and figure out which ones you will need to support in the next few years. | Feature | ClickHouse | Athena | | ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Low-latency dashboards | • Sub-second load times at TB+ scale with proper indexing
• ClickHouse Cloud reduces engineering overhead with managed service • Proven low-latency performance (120ms at 2500 QPS in benchmarks)
• Purpose-built for low-latency OLAP and real-time analytics | • Seconds to minutes response times for interactive dashboards • Performance varies based on data partitioning, file formats, and query optimization • Provisioned Capacity can improve consistency for dashboard workloads • Best suited for analytical dashboards rather than sub-second operational dashboards | | Enterprise BI | • Growing ecosystem with 50+ integrations including major BI tools
• Native MySQL protocol support enables broad BI tool compatibility • Strong SQL compliance with PostgreSQL compatibility
• Best suited for modern analytical workloads and engineering-managed use cases | • Good integration with AWS ecosystem BI tools (QuickSight, etc.) • Standard SQL compatibility enables most BI tool connections • Cost-effective for variable workloads and ad-hoc analytics • JDBC/ODBC drivers support enterprise BI tools • Limited advanced BI features compared to dedicated data warehouses | | Data Apps and AI Applications (Customer-facing low-latency high concurrency) | • Sub-second response times at TB+ scale
• Supports 1000 concurrent users per replica
• Strong price-performance on customer-facing applications
• Native vector search and embeddings | • Default concurrency limits (25 DML/20 DDL queries) may require service quota increases • Provisioned Capacity enables higher concurrency with dedicated resources • Seconds-level response times typical • Cost-effective for customer-facing analytics with proper optimization • Best suited for analytical rather than operational workloads • No native AI capabilities | | Ad hoc | • Good for ad-hoc queries with ClickHouse Cloud's separated storage/compute architecture
• Join optimizations enable more query complexity
• Strong sampling capabilities (TABLESAMPLE) for exploratory analysis
• Resource management through user quotas prevents query interference
• Materialized views offer performance improvements for common aggregation patterns, ad-hoc users specify directly in SQL | • Purpose-built for ad-hoc analytics on data lakes • Serverless with zero infrastructure management • Direct querying of S3 data without ETL • Cost-effective pay-per-query model ideal for exploratory analysis • Strong support for multiple data formats and federated queries • Apache Spark integration for advanced analytics | **ClickHouse** was not designed to be a data warehouse, but rather a low-latency query execution runtime. Managing it typically requires significant engineering overhead. Hence, it's a good fit for engineering managed operational use cases and customer-facing data apps, where low latency matters. It is not a good fit for a general purpose data warehouse, nor for Ad-Hoc analytics or ELT. **Athena** is a great choice for Ad-Hoc analytics. You can keep the data where it is, and start querying without worrying about hardware or pretty much anything else, given that Athena is serverless and takes care of everything behind the scenes. However, it is not a great fit when you need consistent and fast query performance, and/or high concurrency. This is why it is typically not the best choice for operational and customer-facing applications. It can be also easily and flexibly used for batch processing, which is often leveraged for ML use cases. # ClickHouse vs Databricks (2025) (/comparison/clickhouse-vs-databricks) ## Architecture [#architecture] The biggest difference among cloud data warehouses are whether they separate storage and compute, how much they isolate data and compute, and what clouds they can run on. | Feature | ClickHouse | Databricks | | ------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Separation of storage and compute | Yes – SharedMergeTree engine in ClickHouse Cloud enables full separation of storage and compute, with compute-compute separation through Warehouses feature (introduced 2025) allowing multiple isolated compute services sharing the same data | Yes | | Supported cloud infrastructure | AWS, GCP, Azure, cloud service and on-premises | AWS, Azure, GCP. Marketplaces and BYOC | | Isolated tenancy – option for dedicated resources | • Multi-tenant metadata layer
• Isolated tenancy for compute & storage per client in cloud | • Control plane in Databricks account
• Data plane in customer VPC (optional)
• Storage in customer VPC
• Serverless SQL runs in Databricks account with private connectivity | | Control vs abstraction of compute | Configurable cluster size and compute types in ClickHouse Cloud with granular control over nodes (1-128 nodes) and node characteristics. Warehouses feature enables multiple isolated read-only compute environments. | • Configurable clusters and instance types
• Serverless SQL warehouses (GA 2025) run in Databricks account with private connectivity, no public IPs
• Pro/Classic warehouses run in customer VPC | | Self-hosted and hybrid deployment options | Self-managed deployments available with full control over infrastructure | • Databricks on customer cloud accounts
• Unity Catalog for hybrid governance | | ACID Compliance and Transactions | Limited ACID compliance with MergeTree engine family. | • ACID transactions with Delta Lake
• Time travel and versioning
• Concurrent read/write operations | **ClickHouse** was originally developed at Yandex, the Russian search engine, as an OLAP engine for low latency analytics. It was built as an on-premise solution with coupled storage & compute, and a large variety of tuning options in the form of indexes and merge trees. ClickHouse's architecture is famous for its focus on performance and low-latency queries. The tradeoff is that it is considered very difficult to work with. SQL support is very limited, and tuning/running it requires significant engineering resources. **Databricks** was built by the founders of Spark as an analytics platform to support machine learning use cases. It leverages the Spark framework to process data residing in a data lake and is supported on AWS, GCP and Azure. Databricks coined the marketing term "Lakehouse '' architecture to illustrate the unification of data lake and data warehouse use cases. Customers still manage Spark clusters that process data residing in a Delta lake. Conversion of data to Delta Lake format is required to leverage the functionality of Delta Lake. Databricks Sql is a relatively new addition to simplify access to data stored in a data lake. ## Scalability [#scalability] There are three big differences among data warehouses and query engines that limit scalability: decoupled storage and compute, dedicated resources, and continuous ingestion. | Feature | ClickHouse | Databricks | | --------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Elasticity – Scaling for larger data volumes and faster queries | Automatic horizontal and vertical scaling in ClickHouse Cloud with SharedMergeTree architecture. Manual scaling for self-managed deployments with cluster rebalancing capabilities | Autoscaling clusters based on workload demand. Serverless SQL warehouses provide near-instant scaling (2-6 seconds startup) | | Elasticity – Scaling for higher concurrency | Supports high concurrency with proper resource allocation and configuration. Vertical auto-scaling and horizontal manual scaling. Additional warehouses can idle to zero billing. Primary service always on in multi-warehouse configurations. | • 10 concurrent queries per cluster limit • Scales up to 40 clusters per warehouse (400 total concurrent queries) • Serverless SQL warehouses provide near-instant autoscaling • Pro/Classic warehouses take several minutes to provision new clusters • Real-world performance degradation typically occurs at 50-150 concurrent queries depending on complexity | **ClickHouse** doesn't offer any dedicated scaling features or mechanisms. While it can deliver linearly scalable performance for some types of queries, scaling itself has to be done manually. Hardware is self-managed in ClickHouse. This means that to scale you would have to provision a cluster and migrate. **Databricks** allow for autoscaling of clusters based on utilization. Additionally, increasing concurrency associated with a sql endpoint can be accomplished through the addition of clusters. Query concurrency per cluster is maxed at 10. However, scaling with additional clusters for concurrency is possible. Databricks provides a choice of instance types. ## Performance [#performance] Performance is the biggest challenge with most data warehouses today. While decoupled storage and compute architectures improved scalability and simplified administration, for most data warehouses it introduced two bottlenecks; storage, and compute. Most modern cloud data warehouses fetch entire partitions over the network instead of just fetching the specific data needed for each query. While many invest in caching, most do not invest heavily in query optimization. Most vendors also have not improved continuous ingestion or semi-structured data analytics performance, both of which are needed for operational and customer-facing use cases. | Feature | ClickHouse | Databricks | | ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Indexes | • Primary indexes • Skipping indexes (minmax, set, bloom filters, ngrambf\_v1, tokenbf\_v1)
• MergeTree indexes
• Incremental Materialized views | None | | Compute tuning | Configurable compute resources in cloud offering | Choice of cluster type, node types including SSD-optimized instances. Serverless provides automatic resource allocation with Intelligent Workload Management (IWM) | | Storage format | Columnar, supports sorted, compressed, encoded & sparsely indexed files with native Apache Iceberg support. | • Delta Lake format with Liquid Clustering (February 2025 – replaces Z-ordering and traditional partitioning) • Cannot use Liquid Clustering alongside Z-ordering on same table • Allows for sorted data in Delta Lake • Requires Optimize to maintain ordering | | Table-level partition & pruning techniques | Partitioning by date/time and custom partitions with MergeTree indexes. | • Table level partitioning • Liquid Clustering for improved query performance and reduced data skew (February 2025) • Z-ordering (legacy, replaced by Liquid Clustering) • Periodic optimization of storage required | | Result cache | Yes, results cache with TTL and query condition cache. | Multi-layered caching: local in-memory cache per cluster plus remote result cache (serverless only) that persists across all warehouses in workspace | | Warm cache (SSD) | Yes, at indexed data-range level granularity | Yes. Delta cache for data read by queries at file level granularity | | Support for semi-structured data & JSON functions within SQL | Yes, including Lambda expressions and native JSON data type (GA in v25.3) | Yes | | Vector Search and AI Capabilities | • Native vector search capabilities and embeddings
• MCP Server for AI driven analytics
• Natural Language to SQL
• SQL based Inference | • MLflow integration and Databricks ML platform • Native vector search in Delta Lake (Vector Search) • AI and ML workloads optimized | | Query Optimizations | • Primary indexes (ORDER BY)
• Data skipping indexes (minmax, set, bloom filters, ngrambf\_v1, tokenbf\_v1)
• Materialized views
• Projections
• PREWHERE optimization
• Query analysis tools
• Automatic global join reordering (v25.9)
• Enhanced JSON query optimization
• Streaming secondary indices | • Photon engine (C++ vectorized engine providing 3-8x average speedups, maximum speedups over 10x) • Automated stats collection (January 2025) enables cost-based optimization • Predictive I/O for faster point lookups and data updates • Liquid Clustering (February 2025) • Intelligent Workload Management (IWM) with AI-powered resource allocation • Delta cache • Materialized views support | **ClickHouse** is famous for being one of the fastest local runtimes ever built for OLAP workloads. Its columnar storage, compression and indexing capabilities make it a consistent leader in benchmarks. Its lack of support for standard SQL and lack of query optimizer means that it's less suitable for traditional BI workloads, and more suitable for engineering managed workloads. While fast, it requires a lot of tuning and optimization. **Databricks** is designed to leverage the Spark framework for processing large volumes of data. It leverages compressed Parquet files in a Delta Lake. To reduce the amount of data processed, it uses data pruning on partitions and Parquet file metadata. Databricks does not provide any indexes. ## Use cases [#use-cases] There are a host of different analytics use cases that can be supported by a data warehouse. Look at your legacy technologies and their workloads, as well as the new possible use cases, and figure out which ones you will need to support in the next few years. | Feature | ClickHouse | Databricks | | ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Low-latency dashboards | • Sub-second load times at TB+ scale with proper indexing
• ClickHouse Cloud reduces engineering overhead with managed service • Proven low-latency performance (120ms at 2500 QPS in benchmarks)
• Purpose-built for low-latency OLAP and real-time analytics | • Sub-second to seconds load times at TB+ scale • Enhanced by Photon engine (3-8x average speedups) and Delta cache • Serverless SQL warehouses provide rapid startup (2-6 seconds) • Performance depends on cluster configuration | | Enterprise BI | • Growing ecosystem with 50+ integrations including major BI tools
• Native MySQL protocol support enables broad BI tool compatibility • Strong SQL compliance with PostgreSQL compatibility
• Best suited for modern analytical workloads and engineering-managed use cases | • Strong for data science and ML workloads • Unified analytics platform approach • Growing traditional BI integrations • Serverless SQL warehouses improve accessibility • Delta sharing capabilities | | Data Apps and AI Applications (Customer-facing low-latency high concurrency) | • Sub-second response times at TB+ scale
• Supports 1000 concurrent users per replica
• Strong price-performance on customer-facing applications
• Native vector search and embeddings | • 10 concurrent queries per cluster, scaling to 400 total concurrent queries per warehouse • Real-world performance degradation typically occurs at 50-150 concurrent queries depending on workload complexity • Serverless provides near-instant autoscaling • Photon engine delivers 3-8x performance improvements • Strong ML and AI platform integration | | Ad hoc | • Good for ad-hoc queries with ClickHouse Cloud's separated storage/compute architecture
• Join optimizations enable more query complexity
• Strong sampling capabilities (TABLESAMPLE) for exploratory analysis
• Resource management through user quotas prevents query interference
• Materialized views offer performance improvements for common aggregation patterns, ad-hoc users specify directly in SQL | • Excellent for ad-hoc with decoupled storage/compute • Serverless SQL warehouses provide instant provisioning • Intelligent Workload Management handles unpredictable workloads automatically • Strong for exploratory data analysis and ML workloads • Automated stats collection improves query planning | **ClickHouse** was not designed to be a data warehouse, but rather a low-latency query execution runtime. Managing it typically requires significant engineering overhead. Hence, it's a good fit for engineering managed operational use cases and customer-facing data apps, where low latency matters. It is not a good fit for a general purpose data warehouse, nor for Ad-Hoc analytics or ELT. **Databricks** is a mature Spark based platform proven for processing streaming data. It is widely used for Machine Learning use cases by data scientists through the use of integrated notebooks. From a low latency query perspective, while it offers features like Delta Cache, it does not provide specialized indexes that can deliver low latency queries. # Databricks vs Athena (/comparison/databricks-vs-athena) ## Architecture [#architecture] The biggest difference among cloud data warehouses are whether they separate storage and compute, how much they isolate data and compute, and what clouds they can run on. | Feature | Databricks | Athena | | ------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Separation of storage and compute | Yes | Yes, serverless with optional provisioned capacity. Workloads can be isolated through Workgroups and Capacity Reservations | | Supported cloud infrastructure | AWS, Azure, GCP. Marketplaces and BYOC | AWS only | | Isolated tenancy – option for dedicated resources | • Control plane in Databricks account
• Data plane in customer VPC (optional)
• Storage in customer VPC
• Serverless SQL runs in Databricks account with private connectivity | • Multi-tenant pooled resources by default
• Dedicated compute resources available via Provisioned Capacity
• VPC endpoint connections supported | | Control vs abstraction of compute | • Configurable clusters and instance types
• Serverless SQL warehouses (GA 2025) run in Databricks account with private connectivity, no public IPs
• Pro/Classic warehouses run in customer VPC | • Serverless by default with no infrastructure control
• Optional Provisioned Capacity allows dedicated DPU allocation (minimum 24 DPUs)
• Two pricing models: on-demand ($5/TB scanned) or provisioned ($0.30/DPU-hour) | | Self-hosted and hybrid deployment options | • Databricks on customer cloud accounts
• Unity Catalog for hybrid governance | No self-hosted options – serverless only | | ACID Compliance and Transactions | • ACID transactions with Delta Lake
• Time travel and versioning
• Concurrent read/write operations | No ACID compliance – eventual consistency model | **Databricks** was built by the founders of Spark as an analytics platform to support machine learning use cases. It leverages the Spark framework to process data residing in a data lake and is supported on AWS, GCP and Azure. Databricks coined the marketing term "Lakehouse '' architecture to illustrate the unification of data lake and data warehouse use cases. Customers still manage Spark clusters that process data residing in a Delta lake. Conversion of data to Delta Lake format is required to leverage the functionality of Delta Lake. Databricks Sql is a relatively new addition to simplify access to data stored in a data lake. **Athena** is serverless and built on a decoupled storage and compute architecture that queries data directly in S3, without the need to ingest/copy the data. It runs in multi-tenancy with shared resources. Users do not have control over the compute resources Athena chooses to allocate per query from the shared resource pool. For folks requiring additional or dedicated resources, they can reserve dedicated processing capacity in the form of Data Processing Units (DPU), with each DPU providing 4 vCPU and 16 GB RAM. RPU allocation ranges from 24 - 1000 per region. ## Scalability [#scalability] There are three big differences among data warehouses and query engines that limit scalability: decoupled storage and compute, dedicated resources, and continuous ingestion. | Feature | Databricks | Athena | | --------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Elasticity – Scaling for larger data volumes and faster queries | Autoscaling clusters based on workload demand. Serverless SQL warehouses provide near-instant scaling (2-6 seconds startup) | • Fully abstracted on-demand scaling • Provisioned Capacity allows manual scaling of DPUs for predictable performance • Capacity reservations can be adjusted with minimum 1-hour billing periods | | Elasticity – Scaling for higher concurrency | • 10 concurrent queries per cluster limit • Scales up to 40 clusters per warehouse (400 total concurrent queries) • Serverless SQL warehouses provide near-instant autoscaling • Pro/Classic warehouses take several minutes to provision new clusters • Real-world performance degradation typically occurs at 50-150 concurrent queries depending on complexity | • Default limit of 25 concurrent DML queries and 20 DDL queries (adjustable via service quotas) • Provisioned Capacity enables higher concurrency with dedicated DPUs • Query queuing available when capacity is exceeded | \*\*Databricks \*\*allow for autoscaling of clusters based on utilization. Additionally, increasing concurrency associated with a sql endpoint can be accomplished through the addition of clusters. Query concurrency per cluster is maxed at 10. However, scaling with additional clusters for concurrency is possible. Databricks provides a choice of instance types. **Athena** is a shared multi-tenant resource, with no guarantees on the amount or availability of the resources allocated for your queries. From a data volume perspective, it can scale to large volumes, but large data volumes can suffer from very long run times and frequent timeouts. Query concurrency is maxed at 20. If scalability is a top priority, Athena is probably not the best choice. ## Performance [#performance] Performance is the biggest challenge with most data warehouses today. While decoupled storage and compute architectures improved scalability and simplified administration, for most data warehouses it introduced two bottlenecks; storage, and compute. Most modern cloud data warehouses fetch entire partitions over the network instead of just fetching the specific data needed for each query. While many invest in caching, most do not invest heavily in query optimization. Most vendors also have not improved continuous ingestion or semi-structured data analytics performance, both of which are needed for operational and customer-facing use cases. | Feature | Databricks | Athena | | ------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Indexes | None | No traditional indexes – relies on partition pruning and data organization in S3. Uses columnar formats and compression for optimization | | Compute tuning | Choice of cluster type, node types including SSD-optimized instances. Serverless provides automatic resource allocation with Intelligent Workload Management (IWM) | • No compute tuning in on-demand mode • Provisioned Capacity allows DPU allocation control (4 vCPU and 16GB RAM per DPU) • Minimum 24 DPUs with scaling in 4-DPU increments | | Storage format | • Delta Lake format with Liquid Clustering (February 2025 – replaces Z-ordering and traditional partitioning) • Cannot use Liquid Clustering alongside Z-ordering on same table • Allows for sorted data in Delta Lake • Requires Optimize to maintain ordering | Supports multiple formats: Parquet, ORC, Avro, JSON, CSV, TSV on S3. Native support for open table formats including Apache Iceberg, Apache Hudi, and Delta Lake | | Table-level partition & pruning techniques | • Table level partitioning • Liquid Clustering for improved query performance and reduced data skew (February 2025) • Z-ordering (legacy, replaced by Liquid Clustering) • Periodic optimization of storage required | • User-defined table-level partitions with Hive-style partitioning • Pruning at partition level • Partition projection for advanced performance optimization • Supports open table formats with built-in partitioning | | Result cache | Multi-layered caching: local in-memory cache per cluster plus remote result cache (serverless only) that persists across all warehouses in workspace | Query result caching for up to 30 days with configurable retention. Results reuse supported across workgroups | | Warm cache (SSD) | Yes. Delta cache for data read by queries at file level granularity | No local caching – queries data directly from S3. Relies on S3’s performance characteristics and intelligent tiering | | Support for semi-structured data & JSON functions within SQL | Yes | Yes, comprehensive JSON support including Lambda expressions, array functions, and native nested data handling | | Vector Search and AI Capabilities | • MLflow integration and Databricks ML platform • Native vector search in Delta Lake (Vector Search) • AI and ML workloads optimized | No native AI or vector search capabilities | | Query Optimizations | • Photon engine (C++ vectorized engine providing 3-8x average speedups, maximum speedups over 10x) • Automated stats collection (January 2025) enables cost-based optimization • Predictive I/O for faster point lookups and data updates • Liquid Clustering (February 2025) • Intelligent Workload Management (IWM) with AI-powered resource allocation • Delta cache • Materialized views support | • Cost-based optimizer (CBO) in Athena engine v3 • Query result caching (up to 30 days) • Partition projection for advanced optimization • CTAS for precomputed queries • Join reordering and aggregation pushdown • Automatic parallel query execution • Support for columnar formats (Parquet, ORC) • Integration with AWS Glue Data Catalog | **Databricks** is designed to leverage the Spark framework for processing large volumes of data. It leverages compressed Parquet files in a Delta Lake. To reduce the amount of data processed, it uses data pruning on partitions and Parquet file metadata. Databricks does not provide any indexes. **Athena** (and Presto) are designed to query data where it is, sacrificing storage-compute optimizations. This makes it very convenient for easy and immediate querying but at the expense of performance. This typically puts Athena behind cloud data warehouses in terms of performance. But Athena still does relatively well in performance benchmarks, especially when external storage is managed by experts. While it supports partitions, there is no support for indexing, and together with the fact that resources are pooled from a shared multi-tenant service, low-latency and consistent performance are not Athena’s sweet spot. A cloud data warehouse is more performant than Athena in most cases. ## Use cases [#use-cases] There are a host of different analytics use cases that can be supported by a data warehouse. Look at your legacy technologies and their workloads, as well as the new possible use cases, and figure out which ones you will need to support in the next few years. | Feature | Databricks | Athena | | ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Low-latency dashboards | • Sub-second to seconds load times at TB+ scale • Enhanced by Photon engine (3-8x average speedups) and Delta cache • Serverless SQL warehouses provide rapid startup (2-6 seconds) • Performance depends on cluster configuration | • Seconds to minutes response times for interactive dashboards • Performance varies based on data partitioning, file formats, and query optimization • Provisioned Capacity can improve consistency for dashboard workloads • Best suited for analytical dashboards rather than sub-second operational dashboards | | Enterprise BI | • Strong for data science and ML workloads • Unified analytics platform approach • Growing traditional BI integrations • Serverless SQL warehouses improve accessibility • Delta sharing capabilities | • Good integration with AWS ecosystem BI tools (QuickSight, etc.) • Standard SQL compatibility enables most BI tool connections • Cost-effective for variable workloads and ad-hoc analytics • JDBC/ODBC drivers support enterprise BI tools • Limited advanced BI features compared to dedicated data warehouses | | Data Apps and AI Applications (Customer-facing low-latency high concurrency) | • 10 concurrent queries per cluster, scaling to 400 total concurrent queries per warehouse • Real-world performance degradation typically occurs at 50-150 concurrent queries depending on workload complexity • Serverless provides near-instant autoscaling • Photon engine delivers 3-8x performance improvements • Strong ML and AI platform integration | • Default concurrency limits (25 DML/20 DDL queries) may require service quota increases • Provisioned Capacity enables higher concurrency with dedicated resources • Seconds-level response times typical • Cost-effective for customer-facing analytics with proper optimization • Best suited for analytical rather than operational workloads • No native AI capabilities | | Ad hoc | • Excellent for ad-hoc with decoupled storage/compute • Serverless SQL warehouses provide instant provisioning • Intelligent Workload Management handles unpredictable workloads automatically • Strong for exploratory data analysis and ML workloads • Automated stats collection improves query planning | • Purpose-built for ad-hoc analytics on data lakes • Serverless with zero infrastructure management • Direct querying of S3 data without ETL • Cost-effective pay-per-query model ideal for exploratory analysis • Strong support for multiple data formats and federated queries • Apache Spark integration for advanced analytics | \*\*Databricks \*\*is a mature Spark based platform proven for processing streaming data. It is widely used for Machine Learning use cases by data scientists through the use of integrated notebooks. From a low latency query perspective, while it offers features like Delta Cache, it does not provide specialized indexes that can deliver low latency queries. **Athena** is a great choice for Ad-Hoc analytics. You can keep the data where it is, and start querying without worrying about hardware or pretty much anything else, given that Athena is serverless and takes care of everything behind the scenes. However, it is not a great fit when you need consistent and fast query performance, and/or high concurrency. This is why it is typically not the best choice for operational and customer-facing applications. It can be also easily and flexibly used for batch processing, which is often leveraged for ML use cases. # Databricks vs Snowflake (/comparison/databricks-vs-snowflake) ## Architecture [#architecture] The biggest difference among cloud data warehouses are whether they separate storage and compute, how much they isolate data and compute, and what clouds they can run on. | Feature | Databricks | Snowflake | Firebolt | | ------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Separation of storage and compute | Yes | Yes | Yes, separation of storage and metadata as well as compute from compute with full workload isolation. | | Supported cloud infrastructure | AWS, Azure, GCP. Marketplaces and BYOC | AWS, Azure, GCP with full feature parity across all three major clouds | AWS (GCP coming soon) & anywhere (Firebolt Core) | | Isolated tenancy – option for dedicated resources | • Control plane in Databricks account
• Data plane in customer VPC (optional)
• Storage in customer VPC
• Serverless SQL runs in Databricks account with private connectivity | • Multi-tenant pooled resources
• Isolated tenancy available via VPS tier | • Multi-tenant metadata layer
• Isolated tenancy for compute & storage per client | | Control vs abstraction of compute | • Configurable clusters and instance types
• Serverless SQL warehouses (GA 2025) run in Databricks account with private connectivity, no public IPs
• Pro/Classic warehouses run in customer VPC | • Configurable warehouse sizes (XS to 6XL)
• Multi-cluster warehouses with auto-scaling
• Choice between Generation 1 and Generation 2 standard warehouses
• MAX\_CONCURRENCY\_LEVEL parameter for resource allocation | Uses engine abstraction:
• Each engine has configurable cluster size (1-128 nodes) for horizontal scaling.
• Configurable compute family (compute vs storage optimized) and type (XS, S, M, L, XL) for vertical scaling
• Number of clusters for concurrency (auto)scaling.
Provides full workload isolation across engines. | | Self-hosted and hybrid deployment options | • Databricks on customer cloud accounts
• Unity Catalog for hybrid governance | Snowflake for Government Cloud and private cloud options available | • Firebolt Core: Forever free, self-hosted edition with full query engine capabilities
• Same performance and features as managed service
• Deploy anywhere: local laptop, cloud, datacenter, Kubernetes
• Production-grade distributed architecture
• No usage restrictions except building competing SaaS | | ACID Compliance and Transactions | • ACID transactions with Delta Lake
• Time travel and versioning
• Concurrent read/write operations | Full ACID compliance with Time Travel and zero-copy cloning capabilities | • Full ACID compliance with snapshot isolation
• Multi-statement transactions supported
• Strong consistency across all operations
• Supports concurrent reads and writes
• Transactional integrity for data applications | **Databricks** was built by the founders of Spark as an analytics platform to support machine learning use cases. It leverages the Spark framework to process data residing in a data lake and is supported on AWS, GCP and Azure. Databricks coined the marketing term “Lakehouse '' architecture to illustrate the unification of data lake and data warehouse use cases. Customers still manage Spark clusters that process data residing in a Delta lake. Conversion of data to Delta Lake format is required to leverage the functionality of Delta Lake. Databricks Sql is a relatively new addition to simplify access to data stored in a data lake. **Snowflake** was one of the first decoupled storage and compute architectures, making it the first to have nearly unlimited compute scale and workload isolation, and horizontal user scalability. It runs on AWS, Azure and GCP. It is multi-tenant over shared resources in nature and requires you to move data out of your VPC and into the Snowflake cloud. “Virtual Private Snowflake” (VPS) is its highest-priced tier, and can run a dedicated isolated version of Snowflake. Its virtual warehouses can be T-shirt sized along an XS/S/M…/4XL axis, where each discrete T-shirt size is bundled with fixed HW properties that are abstracted from the users. Snowflake has recently added support for Snowflake managed Iceberg tables. **Firebolt** is built on a natively decoupled storage & compute architecture, on AWS only. Data has to be copied outside of your VPC into the Firebolt, where both your compute and data run in a dedicated and isolated tenant. A “Firebolt Engine” can be granularly configured across # of nodes and different CPU/RAM/SSD combinations. ## Scalability [#scalability] There are three big differences among data warehouses and query engines that limit scalability: decoupled storage and compute, dedicated resources, and continuous ingestion. | Feature | Databricks | Snowflake | Firebolt | | --------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Elasticity – Scaling for larger data volumes and faster queries | Autoscaling clusters based on workload demand. Serverless SQL warehouses provide near-instant scaling (2-6 seconds startup) | • Instant warehouse resize (XS to 6XL) with no downtime • Multi-cluster auto-scaling • Generation 2 warehouses provide \~2x performance improvement over Generation 1 | Granular cluster resize with node types, number of nodes and number of clusters. Zero downtime. | | Elasticity – Scaling for higher concurrency | • 10 concurrent queries per cluster limit • Scales up to 40 clusters per warehouse (400 total concurrent queries) • Serverless SQL warehouses provide near-instant autoscaling • Pro/Classic warehouses take several minutes to provision new clusters • Real-world performance degradation typically occurs at 50-150 concurrent queries depending on complexity | • Single warehouse supports many concurrent queries (MAX\_CONCURRENCY\_LEVEL=8 controls resource allocation per query, not query limit) • Multi-cluster warehouses enable thousands of concurrent queries with auto-scaling • Unlimited virtual warehouses can be created | A single engine can handle hundreds of concurrent queries. Engines auto-scale the number of clusters up and down base on resource usage thresholds. Idle engines scale down to zero billing. | **Databricks** allow for autoscaling of clusters based on utilization. Additionally, increasing concurrency associated with a sql endpoint can be accomplished through the addition of clusters. Query concurrency per cluster is maxed at 10. However, scaling with additional clusters for concurrency is possible. Databricks provides a choice of instance types. **Snowflake** scales very well both for data volumes and query concurrency. The decoupled storage/compute architecture supports resizing clusters without downtime, and in addition, supports auto-scaling horizontally for higher query concurrency during peak hours. **Firebolt** can handle the largest data volumes and concurrency on a single comparable cluster size, thanks to its superior hardware efficiency. Thanks to its decoupled storage & compute architecture it scales very well to large data volumes. However, resizing an engine size isn’t instant and requires orchestration if avoiding downtime is necessary. A single Firebolt engine can support hundreds of concurrent queries, avoiding the need to scale out for most use cases. Scaling horizontally for even higher concurrency is manual. ## Performance [#performance] Performance is the biggest challenge with most data warehouses today. While decoupled storage and compute architectures improved scalability and simplified administration, for most data warehouses it introduced two bottlenecks; storage, and compute. Most modern cloud data warehouses fetch entire partitions over the network instead of just fetching the specific data needed for each query. While many invest in caching, most do not invest heavily in query optimization. Most vendors also have not improved continuous ingestion or semi-structured data analytics performance, both of which are needed for operational and customer-facing use cases. | Feature | Databricks | Snowflake | Firebolt | | ------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Indexes | None | • Search Optimization Service for point lookups and selective queries (additional cost) • Clustering keys for data organization and automatic clustering • Materialized views • Snowflake Optima automatic indexing on Generation 2 warehouses (no additional cost) • No traditional database indexes | • Sparse primary indexes
• Aggregating indexes
• Join indexes
• Optimizer driven index usage | | Compute tuning | Choice of cluster type, node types including SSD-optimized instances. Serverless provides automatic resource allocation with Intelligent Workload Management (IWM) | • Warehouse T-shirt sizing (XS to 6XL) • Multi-cluster configuration and scaling policies • Generation 1 vs Generation 2 warehouse selection • MAX\_CONCURRENCY\_LEVEL parameter tuning • Query Acceleration Service for long-running queries | SQL defined engines. Control number of nodes, node family and type per cluster, with one or more clusters per engine. Multiple engines isolate workloads. | | Storage format | • Delta Lake format with Liquid Clustering (February 2025 – replaces Z-ordering and traditional partitioning) • Cannot use Liquid Clustering alongside Z-ordering on same table • Allows for sorted data in Delta Lake • Requires Optimize to maintain ordering | Columnar micro-partitioned & compressed storage | Columnar, sorted & compressed & sparsely indexed storage (F3 – Firebolt File Format) with native Apache Iceberg support | | Table-level partition & pruning techniques | • Table level partitioning • Liquid Clustering for improved query performance and reduced data skew (February 2025) • Z-ordering (legacy, replaced by Liquid Clustering) • Periodic optimization of storage required | • Data automatically divided into micro-partitions • Automatic pruning at micro-partition level • Clustering keys for data organization with automatic clustering • Snowflake Optima provides additional automatic pruning optimization on Gen2 warehouses | • User-defined table-level partitions are optional.
• Data is automatically sorted, compressed and indexed into F3 format.
• Pruning at indexed data-range level. | | Result cache | Multi-layered caching: local in-memory cache per cluster plus remote result cache (serverless only) that persists across all warehouses in workspace | Yes | Yes, results and sub-results cache with transactional spoiling. | | Warm cache (SSD) | Yes. Delta cache for data read by queries at file level granularity | Yes, at micro-partition level granularity | Yes, at indexed data-range level granularity | | Support for semi-structured data & JSON functions within SQL | Yes | Yes | Yes, including Lambda expressions and native nested array structures | | Vector Search and AI Capabilities | • MLflow integration and Databricks ML platform • Native vector search in Delta Lake (Vector Search) • AI and ML workloads optimized | AI integration through Cortex AI and Snowpark ML | • Native vector search capabilities and embeddings
• MCP Server for AI driven analytics
• Natural Language to SQL
• SQL based Inference | | Query Optimizations | • Photon engine (C++ vectorized engine providing 3-8x average speedups, maximum speedups over 10x) • Automated stats collection (January 2025) enables cost-based optimization • Predictive I/O for faster point lookups and data updates • Liquid Clustering (February 2025) • Intelligent Workload Management (IWM) with AI-powered resource allocation • Delta cache • Materialized views support | • Search Optimization Service for point lookups (additional cost)
• Query Acceleration Service (QAS) for long-running and unpredictable workloads • Snowflake Optima automatic optimization on Generation 2 warehouses (no additional cost) • Automatic clustering with background maintenance • Materialized views with automatic refresh • Result cache (24hrs) • Cost-based optimization with dynamic query rewriting | • Primary indexes, aggregating indexes, join indexes, sparse indexes
• Sub-plan result caching
• F3 storage format optimization
• Automatic query optimizer with aggressive pruning
• Late column materialization
• Query analysis tools based on execution telemetry | **Databricks** is designed to leverage the Spark framework for processing large volumes of data. It leverages compressed Parquet files in a Delta Lake. To reduce the amount of data processed, it uses data pruning on partitions and Parquet file metadata. Databricks does not provide any indexes. **Snowflake** typically comes on top for most queries when it comes to performance in public TPC-based benchmarks when compared to BigQuery and Redshift, but only marginally. Its micro partition storage approach effectively scans less data compared to larger partitions. The ability to isolate workloads over the decoupled storage & compute architecture lets you avoid competition for resources compared to multi-tenant shared resource solutions, and the ability to increase warehouse sizes can often enhance performance (for a higher price), but not always linearly. Snowflake’s recently released “Search optimization service” delivers index-like behavior for point queries, but comes at an additional cost. **Firebolt** is the fastest when it comes to query performance when compared to cloud data warehouses and services like Athena. Its unique approach to storage and indexing results in highly aggressive data pruning that scans dramatically less data compared to other technologies. While other technologies scan partitions or micro-partitions, Firebolt works with indexed data ranges that are significantly smaller. In addition, Firebolt lets users accelerate queries further with multiple index types (Aggregating index, Join index), and using its decoupled storage & compute architecture workloads can be easily isolated to guarantee consistent performance. ## Use cases [#use-cases] There are a host of different analytics use cases that can be supported by a data warehouse. Look at your legacy technologies and their workloads, as well as the new possible use cases, and figure out which ones you will need to support in the next few years. | Feature | Databricks | Snowflake | Firebolt | | ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Low-latency dashboards | • Sub-second to seconds load times at TB+ scale • Enhanced by Photon engine (3-8x average speedups) and Delta cache • Serverless SQL warehouses provide rapid startup (2-6 seconds) • Performance depends on cluster configuration | • Sub-second to seconds response times at TB+ scale with proper clustering and optimization • Enhanced by Query Acceleration Service and Search Optimization Service • Generation 2 warehouses provide \~2x performance improvement over Generation 1 • Snowflake Optima provides automatic optimization | • 120ms query latency at 4000 QPS (FireScale benchmark 2025)
• Sub-second performance at TB+ scale with proper indexing
• Built for AI-driven analytics, dashboards, and real-time analytic applications | | Enterprise BI | • Strong for data science and ML workloads • Unified analytics platform approach • Growing traditional BI integrations • Serverless SQL warehouses improve accessibility • Delta sharing capabilities | • Mature and comprehensive Enterprise DW feature set • Extensive integrations with Enterprise BI ecosystem • Multi-cloud deployment options with consistent experience • Strong SQL compliance and wide ecosystem support • Zero-copy data sharing capabilities | • Growing ecosystem with focus on modern BI tools
• Strong SQL compliance with PostgreSQL
• Wire level compatibility drives expansion to PostgreSQL BI and ETL ecosystem | | Data Apps and AI Applications (Customer-facing low-latency high concurrency) | • 10 concurrent queries per cluster, scaling to 400 total concurrent queries per warehouse • Real-world performance degradation typically occurs at 50-150 concurrent queries depending on workload complexity • Serverless provides near-instant autoscaling • Photon engine delivers 3-8x performance improvements • Strong ML and AI platform integration | • Multi-cluster warehouses support thousands of concurrent users with auto-scaling • Individual warehouses support many concurrent queries (not limited to 8 concurrent queries) • Sub-second to seconds response times with proper optimization • Generation 2 warehouses provide significant performance improvements for high-concurrency workloads • AI integration through Cortex AI | • 120ms latency at 4000+ QPS proven performance at TB+ scale
• Supports hundreds to thousands of concurrent queries on single engine
• Price-performance leader (8x better than Snowflake, 18x vs Redshift) • Purpose-built for AI agents and data-intensive applications
• Native vector search and embeddings | | Ad hoc | • Excellent for ad-hoc with decoupled storage/compute • Serverless SQL warehouses provide instant provisioning • Intelligent Workload Management handles unpredictable workloads automatically • Strong for exploratory data analysis and ML workloads • Automated stats collection improves query planning | • Excellent for ad-hoc with decoupled storage/compute • Auto-scaling and instant compute provisioning • Minimal predefined optimization required • Query Acceleration Service handles unpredictable workloads automatically • Snowflake Optima provides automatic optimization for recurring patterns | • Excellent performance out-of-the-box with engine optimized for star and snowflake joins and aggregations
• Self learning query plan optimizer
• Full workload isolation prevents ad-hoc complexity from affecting real-time workloads
• Aggregating indexes are automatically used by optimizer | **Databricks** is a mature Spark based platform proven for processing streaming data. It is widely used for Machine Learning use cases by data scientists through the use of integrated notebooks. From a low latency query perspective, while it offers features like Delta Cache, it does not provide specialized indexes that can deliver low latency queries. **Snowflake** is a well rounded general purpose cloud data warehouse, that can also span beyond traditional BI & Analytics use cases into Ad-Hoc and ML use cases. Thanks to the flexible decoupeld storage & compute architecture that allows you to isolate and control the amount of compute per workload, it’s possible to tackle a broad spectrum of workloads. However, like its close siblings Redshift & BigQuery, it struggles to deliver low-latency query performance at scale, making it a lesser fit for operational use cases and customer-facing data apps. **Firebolt** stands out by being the fastest cloud data warehouse when compared to Snowflake, Redshift, BigQuery and Athena. It’s great for delivering sub-second analytics at scale, while remaining hardware efficient and high concurrency friendly. This makes it a great choice for operational use cases and customer-facing data apps. Given that it is not as feature-rich and integration rich as the more mature data warehouses makes it a lesser fit for a general-purpose Enterprise data warehouse. It is also not the best fit for ad-hoc use cases, because of the need to predefine indexing at the table level. # Druid vs Athena (2025) (/comparison/druid-vs-athena) ## Architecture [#architecture] The biggest difference among cloud data warehouses are whether they separate storage and compute, how much they isolate data and compute, and what clouds they can run on. | Feature | Druid | Athena | | ------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Separation of storage and compute | No | Yes, serverless with optional provisioned capacity. Workloads can be isolated through Workgroups and Capacity Reservations | | Supported cloud infrastructure | Can be installed anywhere | AWS only | | Isolated tenancy – option for dedicated resources | Single tenant | • Multi-tenant pooled resources by default
• Dedicated compute resources available via Provisioned Capacity
• VPC endpoint connections supported | | Control vs abstraction of compute | • Complex configuration of compute tier with multiple role-specific nodes
• Configurable node count
• Configurable compute types (virtual machines or kubernetes) | • Serverless by default with no infrastructure control
• Optional Provisioned Capacity allows dedicated DPU allocation (minimum 24 DPUs)
• Two pricing models: on-demand ($5/TB scanned) or provisioned ($0.30/DPU-hour) | | Self-hosted and hybrid deployment options | Self-managed deployment required | No self-hosted options – serverless only | | ACID Compliance and Transactions | Limited ACID support with eventual consistency | No ACID compliance – eventual consistency model | **Druid** is an OLAP engine designed to provide fast real time analytics. Druid adopts a clustered architecture with servers that host various role specific processes. These processes address real time and batch ingestion, indexing, querying of historical and real time data. Apache Druid can be deployed as a virtual machine or a Kubernetes based cluster. Druid does not support a decoupled compute & storage architecture. Deep storage in the form of object storage is used to replicate data to. **Athena** is serverless and built on a decoupled storage and compute architecture that queries data directly in S3, without the need to ingest/copy the data. It runs in multi-tenancy with shared resources. Users do not have control over the compute resources Athena chooses to allocate per query from the shared resource pool. For folks requiring additional or dedicated resources, they can reserve dedicated processing capacity in the form of Data Processing Units (DPU), with each DPU providing 4 vCPU and 16 GB RAM. RPU allocation ranges from 24 - 1000 per region. ## Scalability [#scalability] There are three big differences among data warehouses and query engines that limit scalability: decoupled storage and compute, dedicated resources, and continuous ingestion. | Feature | Druid | Athena | | --------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Elasticity – Scaling for larger data volumes and faster queries | Scale-up of nodes requires careful planning and downtime. Addition of new nodes for scale-out is possible | • Fully abstracted on-demand scaling • Provisioned Capacity allows manual scaling of DPUs for predictable performance • Capacity reservations can be adjusted with minimum 1-hour billing periods | | Elasticity – Scaling for higher concurrency | Supports 100s to 100,000s queries per second (1000+ QPS) with proper configuration and scaling | • Default limit of 25 concurrent DML queries and 20 DDL queries (adjustable via service quotas) • Provisioned Capacity enables higher concurrency with dedicated DPUs • Query queuing available when capacity is exceeded | **Druid** provides the ability to handle fast ingest and high concurrency. Custom sizing and cluster tuning are required to balance the compute, memory, storage needs of each process within Druid and to provide high concurrency. Druid clusters can be grown by adding nodes with automatic rebalancing of storage segments assigned to nodes. Self hosted Druid on Kubernetes is an option that users leverage to simplify scaling. Additionally, Cloud based managed Druid offerings are being rolled out. However, these managed offerings are limited in scale and scaling is not granular. **Athena** is a shared multi-tenant resource, with no guarantees on the amount or availability of the resources allocated for your queries. From a data volume perspective, it can scale to large volumes, but large data volumes can suffer from very long run times and frequent timeouts. Query concurrency is maxed at 20. If scalability is a top priority, Athena is probably not the best choice. ## Performance [#performance] Performance is the biggest challenge with most data warehouses today. While decoupled storage and compute architectures improved scalability and simplified administration, for most data warehouses it introduced two bottlenecks; storage, and compute. Most modern cloud data warehouses fetch entire partitions over the network instead of just fetching the specific data needed for each query. While many invest in caching, most do not invest heavily in query optimization. Most vendors also have not improved continuous ingestion or semi-structured data analytics performance, both of which are needed for operational and customer-facing use cases. | Feature | Druid | Athena | | ------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Indexes | Compressed bitmap indexes for data access and roll-ups to manage aggregations | No traditional indexes – relies on partition pruning and data organization in S3. Uses columnar formats and compression for optimization | | Compute tuning | On-premises, self-managed hardware. Druid requires infrastructure management and leverages commonly available instance types | • No compute tuning in on-demand mode • Provisioned Capacity allows DPU allocation control (4 vCPU and 16GB RAM per DPU) • Minimum 24 DPUs with scaling in 4-DPU increments | | Storage format | Columnar storage format with time-based sorting | Supports multiple formats: Parquet, ORC, Avro, JSON, CSV, TSV on S3. Native support for open table formats including Apache Iceberg, Apache Hudi, and Delta Lake | | Table-level partition & pruning techniques | Restrictive time-based partitioning. Can partition based on other secondary columns | • User-defined table-level partitions with Hive-style partitioning • Pruning at partition level • Partition projection for advanced performance optimization • Supports open table formats with built-in partitioning | | Result cache | Ability to support caching on broker (set to off by default) | Query result caching for up to 30 days with configurable retention. Results reuse supported across workgroups | | Warm cache (SSD) | Yes, at much larger segment level granularity | No local caching – queries data directly from S3. Relies on S3’s performance characteristics and intelligent tiering | | Support for semi-structured data & JSON functions within SQL | Recommend flattening JSON or translate to array prior to loading. No support for JSON parsing at query runtime | Yes, comprehensive JSON support including Lambda expressions, array functions, and native nested data handling | | Vector Search and AI Capabilities | No native AI or vector search capabilities | No native AI or vector search capabilities | | Query Optimizations | • Compressed bitmap indexes • Roll-up aggregations • Time-based optimization • Query optimization requires manual tuning | • Cost-based optimizer (CBO) in Athena engine v3 • Query result caching (up to 30 days) • Partition projection for advanced optimization • CTAS for precomputed queries • Join reordering and aggregation pushdown • Automatic parallel query execution • Support for columnar formats (Parquet, ORC) • Integration with AWS Glue Data Catalog | **Druid** provides high performance through columnar storage format, parallel processing, bitmap indexes and roll-ups. Druid, however, recommends a denormalized data model for performance needs. Join operations in Druid are a relatively new feature with various limitations, especially if there is a need to join large datasets. **Athena** (and Presto) are designed to query data where it is, sacrificing storage-compute optimizations. This makes it very convenient for easy and immediate querying but at the expense of performance. This typically puts Athena behind cloud data warehouses in terms of performance. But Athena still does relatively well in performance benchmarks, especially when external storage is managed by experts. While it supports partitions, there is no support for indexing, and together with the fact that resources are pooled from a shared multi-tenant service, low-latency and consistent performance are not Athena’s sweet spot. A cloud data warehouse is more performant than Athena in most cases. ## Use cases [#use-cases] There are a host of different analytics use cases that can be supported by a data warehouse. Look at your legacy technologies and their workloads, as well as the new possible use cases, and figure out which ones you will need to support in the next few years. | Feature | Druid | Athena | | ---------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Low-latency dashboards | • Sub-second load times optimized for time-series and real-time analytics • Built for high-concurrency interactive dashboards • Requires denormalized data model | • Seconds to minutes response times for interactive dashboards • Performance varies based on data partitioning, file formats, and query optimization • Provisioned Capacity can improve consistency for dashboard workloads • Best suited for analytical dashboards rather than sub-second operational dashboards | | Enterprise BI | • Limited integrations with traditional Enterprise BI tools • Strong for real-time operational dashboards • Requires specialized visualization tools | • Good integration with AWS ecosystem BI tools (QuickSight, etc.) • Standard SQL compatibility enables most BI tool connections • Cost-effective for variable workloads and ad-hoc analytics • JDBC/ODBC drivers support enterprise BI tools • Limited advanced BI features compared to dedicated data warehouses | | Data Apps and AI Applications (Customer-facing low-latency high concurrency) | • Built for high concurrency (1000+ QPS) with distributed architecture • Sub-second response times for time-series data • Optimized for real-time operational applications • No AI capabilities | • Default concurrency limits (25 DML/20 DDL queries) may require service quota increases • Provisioned Capacity enables higher concurrency with dedicated resources • Seconds-level response times typical • Cost-effective for customer-facing analytics with proper optimization • Best suited for analytical rather than operational workloads • No native AI capabilities | | Ad hoc | • Not optimized for ad-hoc queries • Requires predefined roll-ups and data modeling • Limited flexibility for exploratory analysis | • Purpose-built for ad-hoc analytics on data lakes • Serverless with zero infrastructure management • Direct querying of S3 data without ETL • Cost-effective pay-per-query model ideal for exploratory analysis • Strong support for multiple data formats and federated queries • Apache Spark integration for advanced analytics | **Druid** is designed as an OLAP engine to provide fast access to aggregations that are run against large volumes of data. Druid is typically used for customer facing analytics and streaming data processing. Druid is used as an add-on with other data warehousing products that are efficient at scaling, joining, and filtering large volumes of data. It is not a suitable option for data warehouse replacement. **Athena** is a great choice for Ad-Hoc analytics. You can keep the data where it is, and start querying without worrying about hardware or pretty much anything else, given that Athena is serverless and takes care of everything behind the scenes. However, it is not a great fit when you need consistent and fast query performance, and/or high concurrency. This is why it is typically not the best choice for operational and customer-facing applications. It can be also easily and flexibly used for batch processing, which is often leveraged for ML use cases. # Druid vs ClickHouse (2025) (/comparison/druid-vs-clickhouse) ## Architecture [#architecture] The biggest difference among cloud data warehouses are whether they separate storage and compute, how much they isolate data and compute, and what clouds they can run on. | Feature | Druid | ClickHouse | | ------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Separation of storage and compute | No | Yes – SharedMergeTree engine in ClickHouse Cloud enables full separation of storage and compute, with compute-compute separation through Warehouses feature (introduced 2025) allowing multiple isolated compute services sharing the same data | | Supported cloud infrastructure | Can be installed anywhere | AWS, GCP, Azure, cloud service and on-premises | | Isolated tenancy – option for dedicated resources | Single tenant | • Multi-tenant metadata layer
• Isolated tenancy for compute & storage per client in cloud | | Control vs abstraction of compute | • Complex configuration of compute tier with multiple role-specific nodes
• Configurable node count
• Configurable compute types (virtual machines or kubernetes) | Configurable cluster size and compute types in ClickHouse Cloud with granular control over nodes (1-128 nodes) and node characteristics. Warehouses feature enables multiple isolated read-only compute environments. | | Self-hosted and hybrid deployment options | Self-managed deployment required | Self-managed deployments available with full control over infrastructure | | ACID Compliance and Transactions | Limited ACID support with eventual consistency | Limited ACID compliance with MergeTree engine family. | **Druid** is an OLAP engine designed to provide fast real time analytics. Druid adopts a clustered architecture with servers that host various role specific processes. These processes address real time and batch ingestion, indexing, querying of historical and real time data. Apache Druid can be deployed as a virtual machine or a Kubernetes based cluster. Druid does not support a decoupled compute & storage architecture. Deep storage in the form of object storage is used to replicate data to. **ClickHouse** was originally developed at Yandex, the Russian search engine, as an OLAP engine for low latency analytics. It was built as an on-premise solution with coupled compute & storage, and a large variety of tuning options in the form of indexes and merge trees. ClickHouse's architecture is famous for its focus on performance and low-latency queries. The tradeoff is that it is considered very difficult to work with. SQL support is very limited, and tuning/running it requires significant engineering resources. ## Scalability [#scalability] There are three big differences among data warehouses and query engines that limit scalability: decoupled storage and compute, dedicated resources, and continuous ingestion. | Feature | Druid | ClickHouse | | --------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Elasticity – Scaling for larger data volumes and faster queries | Scale-up of nodes requires careful planning and downtime. Addition of new nodes for scale-out is possible | Automatic horizontal and vertical scaling in ClickHouse Cloud with SharedMergeTree architecture. Manual scaling for self-managed deployments with cluster rebalancing capabilities | | Elasticity – Scaling for higher concurrency | Supports 100s to 100,000s queries per second (1000+ QPS) with proper configuration and scaling | Supports high concurrency with proper resource allocation and configuration. Vertical auto-scaling and horizontal manual scaling. Additional warehouses can idle to zero billing. Primary service always on in multi-warehouse configurations. | **Druid** provides the ability to handle fast ingest and high concurrency. Custom sizing and cluster tuning are required to balance the compute, memory, storage needs of each process within Druid and to provide high concurrency. Druid clusters can be grown by adding nodes with automatic rebalancing of storage segments assigned to nodes. Self hosted Druid on Kubernetes is an option that users leverage to simplify scaling. Additionally, Cloud based managed Druid offerings are being rolled out. However, these managed offerings are limited in scale and scaling is not granular. **ClickHouse** doesn't offer any dedicated scaling features or mechanisms. While it can deliver linearly scalable performance for some types of queries, scaling itself has to be done manually. Hardware is self-managed in ClickHouse. This means that to scale you would have to provision a cluster and migrate. ## Performance [#performance] Performance is the biggest challenge with most data warehouses today. While decoupled storage and compute architectures improved scalability and simplified administration, for most data warehouses it introduced two bottlenecks; storage, and compute. Most modern cloud data warehouses fetch entire partitions over the network instead of just fetching the specific data needed for each query. While many invest in caching, most do not invest heavily in query optimization. Most vendors also have not improved continuous ingestion or semi-structured data analytics performance, both of which are needed for operational and customer-facing use cases. | Feature | Druid | ClickHouse | | ------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Indexes | Compressed bitmap indexes for data access and roll-ups to manage aggregations | • Primary indexes • Skipping indexes (minmax, set, bloom filters, ngrambf\_v1, tokenbf\_v1)
• MergeTree indexes
• Incremental Materialized views | | Compute tuning | On-premises, self-managed hardware. Druid requires infrastructure management and leverages commonly available instance types | Configurable compute resources in cloud offering | | Storage format | Columnar storage format with time-based sorting | Columnar, supports sorted, compressed, encoded & sparsely indexed files with native Apache Iceberg support. | | Table-level partition & pruning techniques | Restrictive time-based partitioning. Can partition based on other secondary columns | Partitioning by date/time and custom partitions with MergeTree indexes. | | Result cache | Ability to support caching on broker (set to off by default) | Yes, results cache with TTL and query condition cache. | | Warm cache (SSD) | Yes, at much larger segment level granularity | Yes, at indexed data-range level granularity | | Support for semi-structured data & JSON functions within SQL | Recommend flattening JSON or translate to array prior to loading. No support for JSON parsing at query runtime | Yes, including Lambda expressions and native JSON data type (GA in v25.3) | | Vector Search and AI Capabilities | No native AI or vector search capabilities | • Native vector search capabilities and embeddings
• MCP Server for AI driven analytics
• Natural Language to SQL
• SQL based Inference | | Query Optimizations | • Compressed bitmap indexes • Roll-up aggregations • Time-based optimization • Query optimization requires manual tuning | • Primary indexes (ORDER BY)
• Data skipping indexes (minmax, set, bloom filters, ngrambf\_v1, tokenbf\_v1)
• Materialized views
• Projections
• PREWHERE optimization
• Query analysis tools
• Automatic global join reordering (v25.9)
• Enhanced JSON query optimization
• Streaming secondary indices | **Druid** provides high performance through columnar storage format, parallel processing, bitmap indexes and roll-ups. Druid, however, recommends a denormalized data model for performance needs. Join operations in Druid are a relatively new feature with various limitations, especially if there is a need to join large datasets. **ClickHouse** is famous for being one of the fastest local runtimes ever built for OLAP workloads. Its columnar storage, compression and indexing capabilities make it a consistent leader in benchmarks. Its lack of support for standard SQL and lack of query optimizer means that it's less suitable for traditional BI workloads, and more suitable for engineering managed workloads. While fast, it requires a lot of tuning and optimization. ## Use cases [#use-cases] There are a host of different analytics use cases that can be supported by a data warehouse. Look at your legacy technologies and their workloads, as well as the new possible use cases, and figure out which ones you will need to support in the next few years. | Feature | Druid | ClickHouse | | ---------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Low-latency dashboards | • Sub-second load times optimized for time-series and real-time analytics • Built for high-concurrency interactive dashboards • Requires denormalized data model | • Sub-second load times at TB+ scale with proper indexing
• ClickHouse Cloud reduces engineering overhead with managed service • Proven low-latency performance (120ms at 2500 QPS in benchmarks)
• Purpose-built for low-latency OLAP and real-time analytics | | Enterprise BI | • Limited integrations with traditional Enterprise BI tools • Strong for real-time operational dashboards • Requires specialized visualization tools | • Growing ecosystem with 50+ integrations including major BI tools
• Native MySQL protocol support enables broad BI tool compatibility • Strong SQL compliance with PostgreSQL compatibility
• Best suited for modern analytical workloads and engineering-managed use cases | | Data Apps and AI Applications (Customer-facing low-latency high concurrency) | • Built for high concurrency (1000+ QPS) with distributed architecture • Sub-second response times for time-series data • Optimized for real-time operational applications • No AI capabilities | • Sub-second response times at TB+ scale
• Supports 1000 concurrent users per replica
• Strong price-performance on customer-facing applications
• Native vector search and embeddings | | Ad hoc | • Not optimized for ad-hoc queries • Requires predefined roll-ups and data modeling • Limited flexibility for exploratory analysis | • Good for ad-hoc queries with ClickHouse Cloud's separated storage/compute architecture
• Join optimizations enable more query complexity
• Strong sampling capabilities (TABLESAMPLE) for exploratory analysis
• Resource management through user quotas prevents query interference
• Materialized views offer performance improvements for common aggregation patterns, ad-hoc users specify directly in SQL | **Druid** is designed as an OLAP engine to provide fast access to aggregations that are run against large volumes of data. Druid is typically used for customer facing analytics and streaming data processing. Druid is used as an add-on with other data warehousing products that are efficient at scaling, joining, and filtering large volumes of data. It is not a suitable option for data warehouse replacement. **ClickHouse** was not designed to be a data warehouse, but rather a low-latency query execution runtime. Managing it typically requires significant engineering overhead. Hence, it's a good fit for engineering managed operational use cases and customer-facing data apps, where low latency matters. It is not a good fit for a general purpose data warehouse, nor for Ad-Hoc analytics or ELT. # Druid vs Databricks (2025) (/comparison/druid-vs-databricks) ## Architecture [#architecture] The biggest difference among cloud data warehouses are whether they separate storage and compute, how much they isolate data and compute, and what clouds they can run on. | Feature | Druid | Databricks | | ------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Separation of storage and compute | No | Yes | | Supported cloud infrastructure | Can be installed anywhere | AWS, Azure, GCP. Marketplaces and BYOC | | Isolated tenancy – option for dedicated resources | Single tenant | • Control plane in Databricks account
• Data plane in customer VPC (optional)
• Storage in customer VPC
• Serverless SQL runs in Databricks account with private connectivity | | Control vs abstraction of compute | • Complex configuration of compute tier with multiple role-specific nodes
• Configurable node count
• Configurable compute types (virtual machines or kubernetes) | • Configurable clusters and instance types
• Serverless SQL warehouses (GA 2025) run in Databricks account with private connectivity, no public IPs
• Pro/Classic warehouses run in customer VPC | | Self-hosted and hybrid deployment options | Self-managed deployment required | • Databricks on customer cloud accounts
• Unity Catalog for hybrid governance | | ACID Compliance and Transactions | Limited ACID support with eventual consistency | • ACID transactions with Delta Lake
• Time travel and versioning
• Concurrent read/write operations | **Druid** is an OLAP engine designed to provide fast real time analytics. Druid adopts a clustered architecture with servers that host various role specific processes. These processes address real time and batch ingestion, indexing, querying of historical and real time data. Apache Druid can be deployed as a virtual machine or a Kubernetes based cluster. Druid does not support a decoupled compute & storage architecture. Deep storage in the form of object storage is used to replicate data to. **Databricks** was built by the founders of Spark as an analytics platform to support machine learning use cases. It leverages the Spark framework to process data residing in a data lake and is supported on AWS, GCP and Azure. Databricks coined the marketing term "Lakehouse '' architecture to illustrate the unification of data lake and data warehouse use cases. Customers still manage Spark clusters that process data residing in a Delta lake. Conversion of data to Delta Lake format is required to leverage the functionality of Delta Lake. Databricks Sql is a relatively new addition to simplify access to data stored in a data lake. ## Scalability [#scalability] There are three big differences among data warehouses and query engines that limit scalability: decoupled storage and compute, dedicated resources, and continuous ingestion. | Feature | Druid | Databricks | | --------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Elasticity – Scaling for larger data volumes and faster queries | Scale-up of nodes requires careful planning and downtime. Addition of new nodes for scale-out is possible | Autoscaling clusters based on workload demand. Serverless SQL warehouses provide near-instant scaling (2-6 seconds startup) | | Elasticity – Scaling for higher concurrency | Supports 100s to 100,000s queries per second (1000+ QPS) with proper configuration and scaling | • 10 concurrent queries per cluster limit • Scales up to 40 clusters per warehouse (400 total concurrent queries) • Serverless SQL warehouses provide near-instant autoscaling • Pro/Classic warehouses take several minutes to provision new clusters • Real-world performance degradation typically occurs at 50-150 concurrent queries depending on complexity | **Druid** provides the ability to handle fast ingest and high concurrency. Custom sizing and cluster tuning are required to balance the compute, memory, storage needs of each process within Druid and to provide high concurrency. Druid clusters can be grown by adding nodes with automatic rebalancing of storage segments assigned to nodes. Self hosted Druid on Kubernetes is an option that users leverage to simplify scaling. Additionally, Cloud based managed Druid offerings are being rolled out. However, these managed offerings are limited in scale and scaling is not granular. **Databricks** allow for autoscaling of clusters based on utilization. Additionally, increasing concurrency associated with a sql endpoint can be accomplished through the addition of clusters. Query concurrency per cluster is maxed at 10. However, scaling with additional clusters for concurrency is possible. Databricks provides a choice of instance types. ## Performance [#performance] Performance is the biggest challenge with most data warehouses today. While decoupled storage and compute architectures improved scalability and simplified administration, for most data warehouses it introduced two bottlenecks; storage, and compute. Most modern cloud data warehouses fetch entire partitions over the network instead of just fetching the specific data needed for each query. While many invest in caching, most do not invest heavily in query optimization. Most vendors also have not improved continuous ingestion or semi-structured data analytics performance, both of which are needed for operational and customer-facing use cases. | Feature | Druid | Databricks | | ------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Indexes | Compressed bitmap indexes for data access and roll-ups to manage aggregations | None | | Compute tuning | On-premises, self-managed hardware. Druid requires infrastructure management and leverages commonly available instance types | Choice of cluster type, node types including SSD-optimized instances. Serverless provides automatic resource allocation with Intelligent Workload Management (IWM) | | Storage format | Columnar storage format with time-based sorting | • Delta Lake format with Liquid Clustering (February 2025 – replaces Z-ordering and traditional partitioning) • Cannot use Liquid Clustering alongside Z-ordering on same table • Allows for sorted data in Delta Lake • Requires Optimize to maintain ordering | | Table-level partition & pruning techniques | Restrictive time-based partitioning. Can partition based on other secondary columns | • Table level partitioning • Liquid Clustering for improved query performance and reduced data skew (February 2025) • Z-ordering (legacy, replaced by Liquid Clustering) • Periodic optimization of storage required | | Result cache | Ability to support caching on broker (set to off by default) | Multi-layered caching: local in-memory cache per cluster plus remote result cache (serverless only) that persists across all warehouses in workspace | | Warm cache (SSD) | Yes, at much larger segment level granularity | Yes. Delta cache for data read by queries at file level granularity | | Support for semi-structured data & JSON functions within SQL | Recommend flattening JSON or translate to array prior to loading. No support for JSON parsing at query runtime | Yes | | Vector Search and AI Capabilities | No native AI or vector search capabilities | • MLflow integration and Databricks ML platform • Native vector search in Delta Lake (Vector Search) • AI and ML workloads optimized | | Query Optimizations | • Compressed bitmap indexes • Roll-up aggregations • Time-based optimization • Query optimization requires manual tuning | • Photon engine (C++ vectorized engine providing 3-8x average speedups, maximum speedups over 10x) • Automated stats collection (January 2025) enables cost-based optimization • Predictive I/O for faster point lookups and data updates • Liquid Clustering (February 2025) • Intelligent Workload Management (IWM) with AI-powered resource allocation • Delta cache • Materialized views support | **Druid** provides high performance through columnar storage format, parallel processing, bitmap indexes and roll-ups. Druid, however, recommends a denormalized data model for performance needs. Join operations in Druid are a relatively new feature with various limitations, especially if there is a need to join large datasets. **Databricks** is designed to leverage the Spark framework for processing large volumes of data. It leverages compressed Parquet files in a Delta Lake. To reduce the amount of data processed, it uses data pruning on partitions and Parquet file metadata. Databricks does not provide any indexes. ## Use cases [#use-cases] There are a host of different analytics use cases that can be supported by a data warehouse. Look at your legacy technologies and their workloads, as well as the new possible use cases, and figure out which ones you will need to support in the next few years. | Feature | Druid | Databricks | | ---------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Low-latency dashboards | • Sub-second load times optimized for time-series and real-time analytics • Built for high-concurrency interactive dashboards • Requires denormalized data model | • Sub-second to seconds load times at TB+ scale • Enhanced by Photon engine (3-8x average speedups) and Delta cache • Serverless SQL warehouses provide rapid startup (2-6 seconds) • Performance depends on cluster configuration | | Enterprise BI | • Limited integrations with traditional Enterprise BI tools • Strong for real-time operational dashboards • Requires specialized visualization tools | • Strong for data science and ML workloads • Unified analytics platform approach • Growing traditional BI integrations • Serverless SQL warehouses improve accessibility • Delta sharing capabilities | | Data Apps and AI Applications (Customer-facing low-latency high concurrency) | • Built for high concurrency (1000+ QPS) with distributed architecture • Sub-second response times for time-series data • Optimized for real-time operational applications • No AI capabilities | • 10 concurrent queries per cluster, scaling to 400 total concurrent queries per warehouse • Real-world performance degradation typically occurs at 50-150 concurrent queries depending on workload complexity • Serverless provides near-instant autoscaling • Photon engine delivers 3-8x performance improvements • Strong ML and AI platform integration | | Ad hoc | • Not optimized for ad-hoc queries • Requires predefined roll-ups and data modeling • Limited flexibility for exploratory analysis | • Excellent for ad-hoc with decoupled storage/compute • Serverless SQL warehouses provide instant provisioning • Intelligent Workload Management handles unpredictable workloads automatically • Strong for exploratory data analysis and ML workloads • Automated stats collection improves query planning | **Druid** is designed as an OLAP engine to provide fast access to aggregations that are run against large volumes of data. Druid is typically used for customer facing analytics and streaming data processing. Druid is used as an add-on with other data warehousing products that are efficient at scaling, joining, and filtering large volumes of data. It is not a suitable option for data warehouse replacement. **Databricks** is a mature Spark based platform proven for processing streaming data. It is widely used for Machine Learning use cases by data scientists through the use of integrated notebooks. From a low latency query perspective, while it offers features like Delta Cache, it does not provide specialized indexes that can deliver low latency queries. # Druid vs Snowflake (2025) (/comparison/druid-vs-snowflake) ## Architecture [#architecture] The biggest difference among cloud data warehouses are whether they separate storage and compute, how much they isolate data and compute, and what clouds they can run on. | Feature | Druid | Snowflake | | ------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Separation of storage and compute | No | Yes | | Supported cloud infrastructure | Can be installed anywhere | AWS, Azure, GCP with full feature parity across all three major clouds | | Isolated tenancy – option for dedicated resources | Single tenant | • Multi-tenant pooled resources
• Isolated tenancy available via VPS tier | | Control vs abstraction of compute | • Complex configuration of compute tier with multiple role-specific nodes
• Configurable node count
• Configurable compute types (virtual machines or kubernetes) | • Configurable warehouse sizes (XS to 6XL)
• Multi-cluster warehouses with auto-scaling
• Choice between Generation 1 and Generation 2 standard warehouses
• MAX\_CONCURRENCY\_LEVEL parameter for resource allocation | | Self-hosted and hybrid deployment options | Self-managed deployment required | Snowflake for Government Cloud and private cloud options available | | ACID Compliance and Transactions | Limited ACID support with eventual consistency | Full ACID compliance with Time Travel and zero-copy cloning capabilities | **Druid** provides the ability to handle fast ingest and high concurrency. Custom sizing and cluster tuning are required to balance the compute, memory, storage needs of each process within Druid and to provide high concurrency. Druid clusters can be grown by adding nodes with automatic rebalancing of storage segments assigned to nodes. Self hosted Druid on Kubernetes is an option that users leverage to simplify scaling. Additionally, Cloud based managed Druid offerings are being rolled out. However, these managed offerings are limited in scale and scaling is not granular. **Snowflake** was one of the first decoupled storage and compute architectures, making it the first to have nearly unlimited compute scale and workload isolation, and horizontal user scalability. It runs on AWS, Azure and GCP. It is multi-tenant over shared resources in nature and requires you to move data out of your VPC and into the Snowflake cloud. “Virtual Private Snowflake” (VPS) is its highest-priced tier, and can run a dedicated isolated version of Snowflake. Its virtual warehouses can be T-shirt sized along an XS/S/M…/4XL axis, where each discrete T-shirt size is bundled with fixed HW properties that are abstracted from the users. Snowflake has recently added support for Snowflake managed Iceberg tables. ## Scalability [#scalability] There are three big differences among data warehouses and query engines that limit scalability: decoupled storage and compute, dedicated resources, and continuous ingestion. | Feature | Druid | Snowflake | | --------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Elasticity – Scaling for larger data volumes and faster queries | Scale-up of nodes requires careful planning and downtime. Addition of new nodes for scale-out is possible | • Instant warehouse resize (XS to 6XL) with no downtime • Multi-cluster auto-scaling • Generation 2 warehouses provide \~2x performance improvement over Generation 1 | | Elasticity – Scaling for higher concurrency | Supports 100s to 100,000s queries per second (1000+ QPS) with proper configuration and scaling | • Single warehouse supports many concurrent queries (MAX\_CONCURRENCY\_LEVEL=8 controls resource allocation per query, not query limit) • Multi-cluster warehouses enable thousands of concurrent queries with auto-scaling • Unlimited virtual warehouses can be created | **Druid** provides the ability to handle fast ingest and high concurrency. Custom sizing and cluster tuning are required to balance the compute, memory, storage needs of each process within Druid and to provide high concurrency. Druid clusters can be grown by adding nodes with automatic rebalancing of storage segments assigned to nodes. **Snowflake** scales very well both for data volumes and query concurrency. The decoupled storage/compute architecture supports resizing clusters without downtime, and in addition, supports auto-scaling horizontally for higher query concurrency during peak hours. ## Performance [#performance] Performance is the biggest challenge with most data warehouses today. While decoupled storage and compute architectures improved scalability and simplified administration, for most data warehouses it introduced two bottlenecks; storage, and compute. Most modern cloud data warehouses fetch entire partitions over the network instead of just fetching the specific data needed for each query. While many invest in caching, most do not invest heavily in query optimization. Most vendors also have not improved continuous ingestion or semi-structured data analytics performance, both of which are needed for operational and customer-facing use cases. | Feature | Druid | Snowflake | | ------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Indexes | Compressed bitmap indexes for data access and roll-ups to manage aggregations | • Search Optimization Service for point lookups and selective queries (additional cost) • Clustering keys for data organization and automatic clustering • Materialized views • Snowflake Optima automatic indexing on Generation 2 warehouses (no additional cost) • No traditional database indexes | | Compute tuning | On-premises, self-managed hardware. Druid requires infrastructure management and leverages commonly available instance types | • Warehouse T-shirt sizing (XS to 6XL) • Multi-cluster configuration and scaling policies • Generation 1 vs Generation 2 warehouse selection • MAX\_CONCURRENCY\_LEVEL parameter tuning • Query Acceleration Service for long-running queries | | Storage format | Columnar storage format with time-based sorting | Columnar micro-partitioned & compressed storage | | Table-level partition & pruning techniques | Restrictive time-based partitioning. Can partition based on other secondary columns | • Data automatically divided into micro-partitions • Automatic pruning at micro-partition level • Clustering keys for data organization with automatic clustering • Snowflake Optima provides additional automatic pruning optimization on Gen2 warehouses | | Result cache | Ability to support caching on broker (set to off by default) | Yes | | Warm cache (SSD) | Yes, at much larger segment level granularity | Yes, at micro-partition level granularity | | Support for semi-structured data & JSON functions within SQL | Recommend flattening JSON or translate to array prior to loading. No support for JSON parsing at query runtime | Yes | | Vector Search and AI Capabilities | No native AI or vector search capabilities | AI integration through Cortex AI and Snowpark ML | | Query Optimizations | • Compressed bitmap indexes • Roll-up aggregations • Time-based optimization • Query optimization requires manual tuning | • Search Optimization Service for point lookups (additional cost)
• Query Acceleration Service (QAS) for long-running and unpredictable workloads • Snowflake Optima automatic optimization on Generation 2 warehouses (no additional cost) • Automatic clustering with background maintenance • Materialized views with automatic refresh • Result cache (24hrs) • Cost-based optimization with dynamic query rewriting | **Druid** provides high performance through columnar storage format, parallel processing, bitmap indexes and roll-ups. Druid, however, recommends a denormalized data model for performance needs. Join operations in Druid are a relatively new feature with various limitations, especially if there is a need to join large datasets. **Snowflake** typically comes on top for most queries when it comes to performance in public TPC-based benchmarks when compared to BigQuery and Redshift, but only marginally. Its micro partition storage approach effectively scans less data compared to larger partitions. The ability to isolate workloads over the decoupled storage & compute architecture lets you avoid competition for resources compared to multi-tenant shared resource solutions, and the ability to increase warehouse sizes can often enhance performance (for a higher price), but not always linearly. Snowflake’s recently released “Search optimization service” delivers index-like behavior for point queries, but comes at an additional cost. ## Use cases [#use-cases] There are a host of different analytics use cases that can be supported by a data warehouse. Look at your legacy technologies and their workloads, as well as the new possible use cases, and figure out which ones you will need to support in the next few years. | Feature | Druid | Snowflake | | ---------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Low-latency dashboards | • Sub-second load times optimized for time-series and real-time analytics • Built for high-concurrency interactive dashboards • Requires denormalized data model | • Sub-second to seconds response times at TB+ scale with proper clustering and optimization • Enhanced by Query Acceleration Service and Search Optimization Service • Generation 2 warehouses provide \~2x performance improvement over Generation 1 • Snowflake Optima provides automatic optimization | | Enterprise BI | • Limited integrations with traditional Enterprise BI tools • Strong for real-time operational dashboards • Requires specialized visualization tools | • Mature and comprehensive Enterprise DW feature set • Extensive integrations with Enterprise BI ecosystem • Multi-cloud deployment options with consistent experience • Strong SQL compliance and wide ecosystem support • Zero-copy data sharing capabilities | | Data Apps and AI Applications (Customer-facing low-latency high concurrency) | • Built for high concurrency (1000+ QPS) with distributed architecture • Sub-second response times for time-series data • Optimized for real-time operational applications • No AI capabilities | • Multi-cluster warehouses support thousands of concurrent users with auto-scaling • Individual warehouses support many concurrent queries (not limited to 8 concurrent queries) • Sub-second to seconds response times with proper optimization • Generation 2 warehouses provide significant performance improvements for high-concurrency workloads • AI integration through Cortex AI | | Ad hoc | • Not optimized for ad-hoc queries • Requires predefined roll-ups and data modeling • Limited flexibility for exploratory analysis | • Excellent for ad-hoc with decoupled storage/compute • Auto-scaling and instant compute provisioning • Minimal predefined optimization required • Query Acceleration Service handles unpredictable workloads automatically • Snowflake Optima provides automatic optimization for recurring patterns | **Druid** is designed as an OLAP engine to provide fast access to aggregations that are run against large volumes of data. Druid is typically used for customer facing analytics and streaming data processing. Druid is used as an add-on with other data warehousing products that are efficient at scaling, joining, and filtering large volumes of data. It is not a suitable option for data warehouse replacement. **Snowflake** is a well rounded general purpose cloud data warehouse, that can also span beyond traditional BI & Analytics use cases into Ad-Hoc and ML use cases. Thanks to the flexible decoupeld storage & compute architecture that allows you to isolate and control the amount of compute per workload, it’s possible to tackle a broad spectrum of workloads. However, like its close siblings Redshift & BigQuery, it struggles to deliver low-latency query performance at scale, making it a lesser fit for operational use cases and customer-facing data apps. # Firebolt vs Athena (2025) (/comparison/firebolt-vs-athena) ## Architecture [#architecture] The biggest difference among cloud data warehouses are whether they separate storage and compute, how much they isolate data and compute, and what clouds they can run on. | Feature | Firebolt | Athena | | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Separation of storage and compute | Yes, separation of storage and metadata as well as compute from compute with full workload isolation. | Yes, serverless with optional provisioned capacity. Workloads can be isolated through Workgroups and Capacity Reservations | | Supported cloud infrastructure | AWS (GCP coming soon) & anywhere (Firebolt Core) | AWS only | | Isolated tenancy – option for dedicated resources | • Multi-tenant metadata layer
• Isolated tenancy for compute & storage per client | • Multi-tenant pooled resources by default
• Dedicated compute resources available via Provisioned Capacity
• VPC endpoint connections supported | | Control vs abstraction of compute | Uses engine abstraction:
• Each engine has configurable cluster size (1-128 nodes) for horizontal scaling.
• Configurable compute family (compute vs storage optimized) and type (XS, S, M, L, XL) for vertical scaling
• Number of clusters for concurrency (auto)scaling.
Provides full workload isolation across engines. | • Serverless by default with no infrastructure control
• Optional Provisioned Capacity allows dedicated DPU allocation (minimum 24 DPUs)
• Two pricing models: on-demand ($5/TB scanned) or provisioned ($0.30/DPU-hour) | | Self-hosted and hybrid deployment options | • Firebolt Core: Forever free, self-hosted edition with full query engine capabilities
• Same performance and features as managed service
• Deploy anywhere: local laptop, cloud, datacenter, Kubernetes
• Production-grade distributed architecture
• No usage restrictions except building competing SaaS | No self-hosted options – serverless only | | ACID Compliance and Transactions | • Full ACID compliance with snapshot isolation
• Multi-statement transactions supported
• Strong consistency across all operations
• Supports concurrent reads and writes
• Transactional integrity for data applications | No ACID compliance – eventual consistency model | **Firebolt** is built on a natively decoupled storage & compute architecture, on AWS only. Data has to be copied outside of your VPC into the Firebolt, where both your compute and data run in a dedicated and isolated tenant. A "Firebolt Engine" can be granularly configured across # of nodes and different CPU/RAM/SSD combinations. **Athena** is serverless and built on a decoupled storage and compute architecture that queries data directly in S3, without the need to ingest/copy the data. It runs in multi-tenancy with shared resources. Users do not have control over the compute resources Athena chooses to allocate per query from the shared resource pool. For folks requiring additional or dedicated resources, they can reserve dedicated processing capacity in the form of Data Processing Units (DPU), with each DPU providing 4 vCPU and 16 GB RAM. RPU allocation ranges from 24 - 1000 per region. ## Scalability [#scalability] There are three big differences among data warehouses and query engines that limit scalability: decoupled storage and compute, dedicated resources, and continuous ingestion. | Feature | Firebolt | Athena | | --------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Elasticity – Scaling for larger data volumes and faster queries | Granular cluster resize with node types, number of nodes and number of clusters. Zero downtime. | • Fully abstracted on-demand scaling • Provisioned Capacity allows manual scaling of DPUs for predictable performance • Capacity reservations can be adjusted with minimum 1-hour billing periods | | Elasticity – Scaling for higher concurrency | A single engine can handle hundreds of concurrent queries. Engines auto-scale the number of clusters up and down base on resource usage thresholds. Idle engines scale down to zero billing. | • Default limit of 25 concurrent DML queries and 20 DDL queries (adjustable via service quotas) • Provisioned Capacity enables higher concurrency with dedicated DPUs • Query queuing available when capacity is exceeded | **Firebolt** can handle the largest data volumes and concurrency on a single comparable cluster size, thanks to its superior hardware efficiency. Thanks to its decoupled storage & compute architecture it scales very well to large data volumes. However, resizing an engine size isn't instant and requires orchestration if avoiding downtime is necessary. A single Firebolt engine can support hundreds of concurrent queries, avoiding the need to scale out for most use cases. Scaling horizontally for even higher concurrency is manual. **Athena** is a shared multi-tenant resource, with no guarantees on the amount or availability of the resources allocated for your queries. From a data volume perspective, it can scale to large volumes, but large data volumes can suffer from very long run times and frequent time outs. Query concurrency is maxed at 20. If scalability is a top priority, Athena is probably not the best choice. ## Performance [#performance] Performance is the biggest challenge with most data warehouses today. While decoupled storage and compute architectures improved scalability and simplified administration, for most data warehouses it introduced two bottlenecks; storage, and compute. Most modern cloud data warehouses fetch entire partitions over the network instead of just fetching the specific data needed for each query. While many invest in caching, most do not invest heavily in query optimization. Most vendors also have not improved continuous ingestion or semi-structured data analytics performance, both of which are needed for operational and customer-facing use cases. | Feature | Firebolt | Athena | | ------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Indexes | • Sparse primary indexes
• Aggregating indexes
• Join indexes
• Optimizer driven index usage | No traditional indexes – relies on partition pruning and data organization in S3. Uses columnar formats and compression for optimization | | Compute tuning | SQL defined engines. Control number of nodes, node family and type per cluster, with one or more clusters per engine. Multiple engines isolate workloads. | • No compute tuning in on-demand mode • Provisioned Capacity allows DPU allocation control (4 vCPU and 16GB RAM per DPU) • Minimum 24 DPUs with scaling in 4-DPU increments | | Storage format | Columnar, sorted & compressed & sparsely indexed storage (F3 – Firebolt File Format) with native Apache Iceberg support | Supports multiple formats: Parquet, ORC, Avro, JSON, CSV, TSV on S3. Native support for open table formats including Apache Iceberg, Apache Hudi, and Delta Lake | | Table-level partition & pruning techniques | • User-defined table-level partitions are optional.
• Data is automatically sorted, compressed and indexed into F3 format.
• Pruning at indexed data-range level. | • User-defined table-level partitions with Hive-style partitioning • Pruning at partition level • Partition projection for advanced performance optimization • Supports open table formats with built-in partitioning | | Result cache | Yes, results and sub-results cache with transactional spoiling. | Query result caching for up to 30 days with configurable retention. Results reuse supported across workgroups | | Warm cache (SSD) | Yes, at indexed data-range level granularity | No local caching – queries data directly from S3. Relies on S3's performance characteristics and intelligent tiering | | Support for semi-structured data & JSON functions within SQL | Yes, including Lambda expressions and native nested array structures | Yes, comprehensive JSON support including Lambda expressions, array functions, and native nested data handling | | Vector Search and AI Capabilities | • Native vector search capabilities and embeddings
• MCP Server for AI driven analytics
• Natural Language to SQL
• SQL based Inference | No native AI or vector search capabilities | | Query Optimizations | • Primary indexes, aggregating indexes, join indexes, sparse indexes
• Sub-plan result caching
• F3 storage format optimization
• Automatic query optimizer with aggressive pruning
• Late column materialization
• Query analysis tools based on execution telemetry | • Cost-based optimizer (CBO) in Athena engine v3 • Query result caching (up to 30 days) • Partition projection for advanced optimization • CTAS for precomputed queries • Join reordering and aggregation pushdown • Automatic parallel query execution • Support for columnar formats (Parquet, ORC) • Integration with AWS Glue Data Catalog | **Firebolt** is the fastest when it comes to query performance when compared to cloud data warehouses and services like Athena. Its unique approach to storage and indexing results in highly aggressive data pruning that scans dramatically less data compared to other technologies. While other technologies scan partitions or micro-partitions, Firebolt works with indexed data ranges, that are significantly smaller. In addition, Firebolt lets user accelerate queries further with multiple index types (Aggregating index, Join index), and using its decoupled storage & compute architecture workloads can be easily isolated to guarantee consistent performance. **Athena** (and Presto) are designed to query data where it is, sacrificing storage-compute optimizations. This makes it very convenient for easy and immediate querying but at the expense of performance. This typically puts Athena behind cloud data warehouses in terms of performance. But Athena still does relatively well in performance benchmarks, especially when external storage is managed by experts. While it supports partitions, there is no support for indexing, and together with the fact that resources are pooled from a shared multi-tenant service, low-latency and consistent performance are not Athena's sweet spot. A cloud data warehouse be more performant better than Athena in most cases. ## Use cases [#use-cases] There are a host of different analytics use cases that can be supported by a data warehouse. Look at your legacy technologies and their workloads, as well as the new possible use cases, and figure out which ones you will need to support in the next few years. | Feature | Firebolt | Athena | | ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Low-latency dashboards | • 120ms query latency at 4000 QPS (FireScale benchmark 2025)
• Sub-second performance at TB+ scale with proper indexing
• Built for AI-driven analytics, dashboards, and real-time analytic applications | • Seconds to minutes response times for interactive dashboards • Performance varies based on data partitioning, file formats, and query optimization • Provisioned Capacity can improve consistency for dashboard workloads • Best suited for analytical dashboards rather than sub-second operational dashboards | | Enterprise BI | • Growing ecosystem with focus on modern BI tools
• Strong SQL compliance with PostgreSQL
• Wire level compatibility drives expansion to PostgreSQL BI and ETL ecosystem | • Good integration with AWS ecosystem BI tools (QuickSight, etc.) • Standard SQL compatibility enables most BI tool connections • Cost-effective for variable workloads and ad-hoc analytics • JDBC/ODBC drivers support enterprise BI tools • Limited advanced BI features compared to dedicated data warehouses | | Data Apps and AI Applications (Customer-facing low-latency high concurrency) | • 120ms latency at 4000+ QPS proven performance at TB+ scale
• Supports hundreds to thousands of concurrent queries on single engine
• Price-performance leader (8x better than Snowflake, 18x vs Redshift) • Purpose-built for AI agents and data-intensive applications
• Native vector search and embeddings | • Default concurrency limits (25 DML/20 DDL queries) may require service quota increases • Provisioned Capacity enables higher concurrency with dedicated resources • Seconds-level response times typical • Cost-effective for customer-facing analytics with proper optimization • Best suited for analytical rather than operational workloads • No native AI capabilities | | Ad hoc | • Excellent performance out-of-the-box with engine optimized for star and snowflake joins and aggregations
• Self learning query plan optimizer
• Full workload isolation prevents ad-hoc complexity from affecting real-time workloads
• Aggregating indexes are automatically used by optimizer | • Purpose-built for ad-hoc analytics on data lakes • Serverless with zero infrastructure management • Direct querying of S3 data without ETL • Cost-effective pay-per-query model ideal for exploratory analysis • Strong support for multiple data formats and federated queries • Apache Spark integration for advanced analytics | **Firebolt** stands out by being the fastest cloud data warehouse when compared to Snowflake, Redshift, BigQuery and Athena. It's great for delivering sub-second analytics at scale, while remaining hardware efficient and high concurrency friendly. This makes it a great choice for operational use cases and customer-facing data apps. Given that it is not as feature-rich and integration rich as the more mature data warehouses makes it a lesser fit for a general-purpose Enterprise data warehouse. It is also not the best fit for ad-hoc use cases, because of the need to predefine indexing at the table level. **Athena** is a great choice for Ad-Hoc analytics. You can keep the data where it is, and start querying without worrying about hardware or pretty much anything else, given that Athena is serverless and takes care of everything behind the scenes. However, it is not a great fit when you need consistent and fast query performance, and/or high concurrency. This is why it is typically not the best choice for operational and customer-facing applications. It can be also easily and flexibly used for batch processing, which is often leveraged for ML use cases. # Firebolt vs BigQuery (2025) (/comparison/firebolt-vs-bigquery) ## Architecture [#architecture] The biggest difference among cloud data warehouses are whether they separate storage and compute, how much they isolate data and compute, and what clouds they can run on. | Feature | Firebolt | BigQuery | | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Separation of storage and compute | Yes, separation of storage and metadata as well as compute from compute with full workload isolation. | Fully serverless with complete separation of compute (Dremel) and storage (Colossus), powered by Jupiter network and Borg orchestration | | Supported cloud infrastructure | AWS (GCP coming soon) & anywhere (Firebolt Core) | Google Cloud only | | Isolated tenancy – option for dedicated resources | • Multi-tenant metadata layer
• Isolated tenancy for compute & storage per client | • Multi-tenant pooled resources
• VPC Service Controls provide enhanced security and connectivity isolation to customer VPCs
• Cross-region disaster recovery for enterprise workloads | | Control vs abstraction of compute | Uses engine abstraction:
• Each engine has configurable cluster size (1-128 nodes) for horizontal scaling.
• Configurable compute family (compute vs storage optimized) and type (XS, S, M, L, XL) for vertical scaling
• Number of clusters for concurrency (auto)scaling.
Provides full workload isolation across engines. | Fully serverless with no control over compute resources – BigQuery automatically allocates computing resources as needed with intelligent workload management and dynamic slot allocation | | Self-hosted and hybrid deployment options | • Firebolt Core: Forever free, self-hosted edition with full query engine capabilities
• Same performance and features as managed service
• Deploy anywhere: local laptop, cloud, datacenter, Kubernetes
• Production-grade distributed architecture
• No usage restrictions except building competing SaaS | No self-hosted options – fully managed service only | | ACID Compliance and Transactions | • Full ACID compliance with snapshot isolation
• Multi-statement transactions supported
• Strong consistency across all operations
• Supports concurrent reads and writes
• Transactional integrity for data applications | Limited ACID support – eventual consistency model with some transactional capabilities | **Firebolt** is built on a natively decoupled storage & compute architecture, on AWS only. Data has to be copied outside of your VPC into the Firebolt, where both your compute and data run in a dedicated and isolated tenant. A "Firebolt Engine" can be granularly configured across # of nodes and different CPU/RAM/SSD combinations **BigQuery** was one of the first decoupled storage and compute architectures. It is a unique piece of engineering and not a typical data warehouse in part because it started as an on-demand serverless query engine. It runs in multi-tenancy with shared resources, allocated as "slots" which represent a virtual CPU that executes SQL. BigQuery determines how many slots a query requires, without the ability of the user to control it. BigQuery can be priced on a $/TB scanned basis or through slot reservations. A slot in BigQuery is logically equivalent to 0.5 vCPU and 0.5GB of RAM. There are multiple models to allocate slots in BigQuery. ## Scalability [#scalability] There are three big differences among data warehouses and query engines that limit scalability: decoupled storage and compute, dedicated resources, and continuous ingestion. | Feature | Firebolt | BigQuery | | --------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Elasticity – Scaling for larger data volumes and faster queries | Granular cluster resize with node types, number of nodes and number of clusters. Zero downtime. | Fully automated serverless scaling – BigQuery automatically determines resource allocation and scales to petabytes without user intervention. Can dynamically burst beyond baseline slot allocations for performance optimization | | Elasticity – Scaling for higher concurrency | A single engine can handle hundreds of concurrent queries. Engines auto-scale the number of clusters up and down base on resource usage thresholds. Idle engines scale down to zero billing. | Dynamic concurrency management with query queueing supporting up to 1,000 interactive queries and 20,000 batch queries per project per region. Automatic fair scheduling and slot distribution across workloads | **Firebolt** can handle the largest data volumes and concurrency on a single comparable cluster size, thanks to its superior hardware efficiency. Thanks to its decoupled storage & compute architecture it scales very well to large data volumes. However, resizing an engine size isn't instant and requires orchestration if avoiding downtime is necessary. A single Firebolt engine can support hundreds of concurrent queries, avoiding the need to scale out for most use cases. Scaling horizontally for even higher concurrency is manual. **BigQuery** scales very well to large data volumes, and automatically assigns more compute resources when needed behind the scenes, in the form of "slots". BigQuery works either in an "on-demand pricing model", where slot assignment is completely in the hands of BigQuery and the state of the shared resource pool, or in "flat-rate pricing model" where slots are reserved in advance. With reserved slots there is more control over compute resources, thus making scaling more predictable. Concurrency is limited to 100 users by default. ## Performance [#performance] Performance is the biggest challenge with most data warehouses today. While decoupled storage and compute architectures improved scalability and simplified administration, for most data warehouses it introduced two bottlenecks; storage, and compute. Most modern cloud data warehouses fetch entire partitions over the network instead of just fetching the specific data needed for each query. While many invest in caching, most do not invest heavily in query optimization. Most vendors also have not improved continuous ingestion or semi-structured data analytics performance, both of which are needed for operational and customer-facing use cases. | Feature | Firebolt | BigQuery | | ------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Indexes | • Sparse primary indexes
• Aggregating indexes
• Join indexes
• Optimizer driven index usage | Search indexes (GA) for efficient text search optimization on STRING, JSON, and array columns. Support for LOG\_ANALYZER, NO\_OP\_ANALYZER, and PATTERN\_ANALYZER with column-level granularity for improved query performance and cost efficiency | | Compute tuning | SQL defined engines. Control number of nodes, node family and type per cluster, with one or more clusters per engine. Multiple engines isolate workloads. | Serverless architecture with automatic resource optimization – no manual tuning required. Intelligent workload management with AI-powered resource allocation and dynamic slot distribution | | Storage format | Columnar, sorted & compressed & sparsely indexed storage (F3 – Firebolt File Format) with native Apache Iceberg support | Columnar & compressed storage (Capacitor format) with support for open table formats including Apache Iceberg, Delta Lake, and Hudi. Intelligent tiering with automatic long-term storage cost reduction after 90 days | | Table-level partition & pruning techniques | • User-defined table-level partitions are optional.
• Data is automatically sorted, compressed and indexed into F3 format.
• Pruning at indexed data-range level. | • Automatic table organization with intelligent micro-partitioning • Clustering keys for data organization • Automatic partition pruning optimization • Supports time-based and custom partitioning strategies | | Result cache | Yes, results and sub-results cache with transactional spoiling. | Yes, with cross-user result caching and intelligent cache management for up to 24 hours | | Warm cache (SSD) | Yes, at indexed data-range level granularity | BI Engine provides in-memory caching and acceleration for frequently accessed data and dashboards | | Support for semi-structured data & JSON functions within SQL | Yes, including Lambda expressions and native nested array structures | Yes, including advanced JSON functions, Lambda expressions, and native support for nested and repeated fields | | Vector Search and AI Capabilities | • Native vector search capabilities and embeddings
• MCP Server for AI driven analytics
• Natural Language to SQL
• SQL based Inference | • BigQuery ML for in-database machine learning • Vertex AI integration • Natural language querying with Gemini AI • Limited vector search capabilities | | Query Optimizations | • Primary indexes, aggregating indexes, join indexes, sparse indexes
• Sub-plan result caching
• F3 storage format optimization
• Automatic query optimizer with aggressive pruning
• Late column materialization
• Query analysis tools based on execution telemetry | • Advanced query optimizer with Dremel engine • Search indexes with column-level granularity • Materialized views with smart refresh and automatic query rewriting • BI Engine in-memory acceleration • Gemini AI-powered query optimization and natural language querying • Cross-user result caching • Automatic partitioning and clustering optimization • Cost-based optimization with intelligent workload management | **Firebolt** is the fastest when it comes to query performance when compared to cloud data warehouses and services like Athena. Its unique approach to storage and indexing results in highly aggressive data pruning that scans dramatically less data compared to other technologies. While other technologies scan partitions or micro-partitions, Firebolt works with indexed data ranges, that are significantly smaller. In addition, Firebolt lets user accelerate queries further with multiple index types (Aggregating index, Join index), and using its decoupled storage & compute architecture workloads can be easily isolated to guarantee consistent performance. **BigQuery** lines up in benchmarks in the same ballpark as other cloud data warehouses but does come in consistently last in most queries. Beyond implementing according to best practices, there is little you can do to accelerate BigQuery performance, as it determines the amount of resources (slots) the query needs for you. BigQuery can be used together with "BigQuery BI Engine" for lower latency analytics. However, BI Engine is limited in terms of scale because it runs in memory. Its maximum capacity is 100GB. ## Use cases [#use-cases] There are a host of different analytics use cases that can be supported by a data warehouse. Look at your legacy technologies and their workloads, as well as the new possible use cases, and figure out which ones you will need to support in the next few years. | Feature | Firebolt | BigQuery | | ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Low-latency dashboards | • 120ms query latency at 4000 QPS (FireScale benchmark 2025)
• Sub-second performance at TB+ scale with proper indexing
• Built for AI-driven analytics, dashboards, and real-time analytic applications | • Sub-second to seconds response times at TB+ scale with BI Engine acceleration • Search indexes and materialized views provide significant performance improvements for dashboard queries • Intelligent caching reduces query costs for repeated dashboard access • Dynamic concurrency management supports high user loads | | Enterprise BI | • Growing ecosystem with focus on modern BI tools
• Strong SQL compliance with PostgreSQL
• Wire level compatibility drives expansion to PostgreSQL BI and ETL ecosystem | • Mature and comprehensive Enterprise DW feature set with native Google Cloud ecosystem integration • Strong integration with Looker, Looker Studio, and major BI tools • Gemini AI integration for natural language insights • Cross-cloud analytics capabilities • Advanced governance with Dataplex Universal Catalog | | Data Apps and AI Applications (Customer-facing low-latency high concurrency) | • 120ms latency at 4000+ QPS proven performance at TB+ scale
• Supports hundreds to thousands of concurrent queries on single engine
• Price-performance leader (8x better than Snowflake, 18x vs Redshift) • Purpose-built for AI agents and data-intensive applications
• Native vector search and embeddings | • Dynamic concurrency supporting 1,000+ interactive queries with intelligent queuing and fair scheduling • Sub-second response times with BI Engine acceleration and search indexes • Serverless architecture eliminates infrastructure management overhead • Advanced caching and materialized views optimize repeated queries • AI-powered optimization for customer-facing applications • BigQuery ML for AI workloads | | Ad hoc | • Excellent performance out-of-the-box with engine optimized for star and snowflake joins and aggregations
• Self learning query plan optimizer
• Full workload isolation prevents ad-hoc complexity from affecting real-time workloads
• Aggregating indexes are automatically used by optimizer | • Excellent for ad-hoc analytics with serverless architecture requiring zero infrastructure management • Intelligent query optimization handles unpredictable workloads automatically • Gemini AI integration enables natural language querying for business users • Advanced JSON support and schema inference enable flexible data exploration • Cross-cloud analytics capabilities for federated queries | **Firebolt** stands out by being the fastest cloud data warehouse when compared to Snowflake, Redshift, BigQuery and Athena. It's great for delivering sub-second analytics at scale, while remaining hardware efficient and high concurrency friendly. This makes it a great choice for operational use cases and customer-facing data apps. Given that it is not as feature-rich and integration rich as the more mature data warehouses makes it a lesser fit for a general-purpose Enterprise data warehouse. It is also not the best fit for ad-hoc use cases, because of the need to predefine indexing at the table level. **BigQuery** is a mature general-purpose data warehouse, which lends itself well to internal BI & reporting. The fact that it's serverless in nature and tightly integrated with GCP, makes it very convenient for Ad-Hoc analytics and ML use cases on GCP. On the other hand, because BigQuery makes resource allocation decisions for you, it is not always the best fit for operational use cases and Data Apps where performance needs to be consistent and predictable. # Firebolt vs ClickHouse (2025) (/comparison/firebolt-vs-clickhouse) ## Architecture [#architecture] The biggest difference among cloud data warehouses are whether they separate storage and compute, how much they isolate data and compute, and what clouds they can run on. | Feature | Firebolt | ClickHouse | | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Separation of storage and compute | Yes, separation of storage and metadata as well as compute from compute with full workload isolation. | Yes – SharedMergeTree engine in ClickHouse Cloud enables full separation of storage and compute, with compute-compute separation through Warehouses feature (introduced 2025) allowing multiple isolated compute services sharing the same data | | Supported cloud infrastructure | AWS (GCP coming soon) & anywhere (Firebolt Core) | AWS, GCP, Azure, cloud service and on-premises | | Isolated tenancy – option for dedicated resources | • Multi-tenant metadata layer
• Isolated tenancy for compute & storage per client | • Multi-tenant metadata layer
• Isolated tenancy for compute & storage per client in cloud | | Control vs abstraction of compute | Uses engine abstraction:
• Each engine has configurable cluster size (1-128 nodes) for horizontal scaling.
• Configurable compute family (compute vs storage optimized) and type (XS, S, M, L, XL) for vertical scaling
• Number of clusters for concurrency (auto)scaling.
Provides full workload isolation across engines. | Configurable cluster size and compute types in ClickHouse Cloud with granular control over nodes (1-128 nodes) and node characteristics. Warehouses feature enables multiple isolated read-only compute environments. | | Self-hosted and hybrid deployment options | • Firebolt Core: Forever free, self-hosted edition with full query engine capabilities
• Same performance and features as managed service
• Deploy anywhere: local laptop, cloud, datacenter, Kubernetes
• Production-grade distributed architecture
• No usage restrictions except building competing SaaS | Self-managed deployments available with full control over infrastructure | | ACID Compliance and Transactions | • Full ACID compliance with snapshot isolation
• Multi-statement transactions supported
• Strong consistency across all operations
• Supports concurrent reads and writes
• Transactional integrity for data applications | Limited ACID compliance with MergeTree engine family. | **Firebolt** is built around three architectural bets that compound: disaggregated storage and compute with sub-second cold start, a vectorized query engine with SIMD-optimized kernels, and sparse indexes that make full scans almost obsolete at scale. When you combine partition pruning with sparse indexes, you're often touching \<1% of the data for typical analytical queries. Disaggregated storage isn't just a cost story, it fundamentally changes what's possible for multi-cluster concurrency without the data movement overhead that kills P99 latency in shuffle-heavy systems. **ClickHouse** is a column-oriented OLAP database built around a shared-nothing architecture where each node owns its storage and compute, scaling horizontally through explicit sharding and replication. The MergeTree engine family is the core primitive - sorted, sparse-indexed storage with variants (ReplacingMergeTree, AggregatingMergeTree) handling different update and aggregation patterns. Query execution is vectorized with LLVM JIT compilation for hot paths. The tight coupling of storage and compute that makes single-node performance exceptional becomes operationally non-trivial at very large scale. ## Scalability [#scalability] There are three big differences among data warehouses and query engines that limit scalability: decoupled storage and compute, dedicated resources, and continuous ingestion. | Feature | Firebolt | ClickHouse | | --------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Elasticity – Scaling for larger data volumes and faster queries | Granular cluster resize with node types, number of nodes and number of clusters. Zero downtime. | Automatic horizontal and vertical scaling in ClickHouse Cloud with SharedMergeTree architecture. Manual scaling for self-managed deployments with cluster rebalancing capabilities | | Elasticity – Scaling for higher concurrency | A single engine can handle hundreds of concurrent queries. Engines auto-scale the number of clusters up and down base on resource usage thresholds. Idle engines scale down to zero billing. | Supports high concurrency with proper resource allocation and configuration. Vertical auto-scaling and horizontal manual scaling. Additional warehouses can idle to zero billing. Primary service always on in multi-warehouse configurations. | **Firebolt** separates storage and compute entirely. Data lives in object storage, engines are stateless and spin up in seconds. Multiple engine clusters can read and write from the same storage simultaneously, isolating workloads without contention or data movement. Scaling out is just provisioning; no rebalancing, no shuffle overhead, no distributed consensus. Uses simple SQL commands to scale vertically, horizontally and in cluster count with zero down-time. **ClickHouse** Cloud runs on SharedMergeTree, a closed-source engine where data lives in object storage and compute nodes are fully stateless. No sharding needed; you scale by adding compute nodes against shared storage, with vertical autoscaling and scale-to-zero built in. Compute-compute separation lets multiple isolated node groups share the same data without extra copies, useful for isolating reads from writes. The main caveats: metadata coordination through ClickHouse Keeper introduces concurrency limits. ## Performance [#performance] Performance is the biggest challenge with most data warehouses today. While decoupled storage and compute architectures improved scalability and simplified administration, for most data warehouses it introduced two bottlenecks; storage, and compute. Most modern cloud data warehouses fetch entire partitions over the network instead of just fetching the specific data needed for each query. While many invest in caching, most do not invest heavily in query optimization. Most vendors also have not improved continuous ingestion or semi-structured data analytics performance, both of which are needed for operational and customer-facing use cases. | Feature | Firebolt | ClickHouse | | ------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Indexes | • Sparse primary indexes
• Aggregating indexes
• Join indexes
• Optimizer driven index usage | • Primary indexes • Skipping indexes (minmax, set, bloom filters, ngrambf\_v1, tokenbf\_v1)
• MergeTree indexes
• Incremental Materialized views | | Compute tuning | SQL defined engines. Control number of nodes, node family and type per cluster, with one or more clusters per engine. Multiple engines isolate workloads. | Configurable compute resources in cloud offering | | Storage format | Columnar, sorted & compressed & sparsely indexed storage (F3 – Firebolt File Format) with native Apache Iceberg support | Columnar, supports sorted, compressed, encoded & sparsely indexed files with native Apache Iceberg support. | | Table-level partition & pruning techniques | • User-defined table-level partitions are optional.
• Data is automatically sorted, compressed and indexed into F3 format.
• Pruning at indexed data-range level. | Partitioning by date/time and custom partitions with MergeTree indexes. | | Result cache | Yes, results and sub-results cache with transactional spoiling. | Yes, results cache with TTL and query condition cache. | | Warm cache (SSD) | Yes, at indexed data-range level granularity | Yes, at indexed data-range level granularity | | Support for semi-structured data & JSON functions within SQL | Yes, including Lambda expressions and native nested array structures | Yes, including Lambda expressions and native JSON data type (GA in v25.3) | | Vector Search and AI Capabilities | • Native vector search capabilities and embeddings
• MCP Server for AI driven analytics
• Natural Language to SQL
• SQL based Inference | • Native vector search capabilities and embeddings
• MCP Server for AI driven analytics
• Natural Language to SQL
• SQL based Inference | | Query Optimizations | • Primary indexes, aggregating indexes, join indexes, sparse indexes
• Sub-plan result caching
• F3 storage format optimization
• Automatic query optimizer with aggressive pruning
• Late column materialization
• Query analysis tools based on execution telemetry | • Primary indexes (ORDER BY)
• Data skipping indexes (minmax, set, bloom filters, ngrambf\_v1, tokenbf\_v1)
• Materialized views
• Projections
• PREWHERE optimization
• Query analysis tools
• Automatic global join reordering (v25.9)
• Enhanced JSON query optimization
• Streaming secondary indices | **Firebolt**'s performance advantage starts with how it reads data. While most data warehouses fetch entire partitions over the network, Firebolt works with indexed data ranges that are dramatically smaller, resulting in aggressive pruning that scans a fraction of what other systems touch. Multiple index types (aggregating indexes, join indexes) let users push this further for specific query patterns. Decoupled storage and compute means workloads can be isolated to guarantee consistent latency. A heavy analytical job doesn't degrade concurrent dashboard queries. A single engine handles hundreds of concurrent queries without needing to scale out, which makes it particularly well-suited for operational and customer-facing analytics where sub-second response times are non-negotiable. **ClickHouse's** performance story is built on its columnar storage, compression, and indexing capabilities, which make it a consistent benchmark leader for raw query execution speed. The MergeTree engine family and sparse primary indexes are highly effective at minimizing I/O for queries that align well with the sort key. Where it gets complicated is that this performance is not automatic, it requires significant engineering investment to tune table engines, indexes, and merge strategies for each workload. The lack of a cost-based query optimizer means query performance is sensitive to how SQL is written, and standard BI tooling that generates arbitrary SQL will often underperform. For engineering-managed workloads where the query patterns are known and controlled, ClickHouse is extremely fast. ## Use cases [#use-cases] There are a host of different analytics use cases that can be supported by a data warehouse. Look at your legacy technologies and their workloads, as well as the new possible use cases, and figure out which ones you will need to support in the next few years. | Feature | Firebolt | ClickHouse | | ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Low-latency dashboards | • 120ms query latency at 4000 QPS (FireScale benchmark 2025)
• Sub-second performance at TB+ scale with proper indexing
• Built for AI-driven analytics, dashboards, and real-time analytic applications | • Sub-second load times at TB+ scale with proper indexing
• ClickHouse Cloud reduces engineering overhead with managed service • Proven low-latency performance (120ms at 2500 QPS in benchmarks)
• Purpose-built for low-latency OLAP and real-time analytics | | Enterprise BI | • Growing ecosystem with focus on modern BI tools
• Strong SQL compliance with PostgreSQL
• Wire level compatibility drives expansion to PostgreSQL BI and ETL ecosystem | • Growing ecosystem with 50+ integrations including major BI tools
• Native MySQL protocol support enables broad BI tool compatibility • Strong SQL compliance with PostgreSQL compatibility
• Best suited for modern analytical workloads and engineering-managed use cases | | Data Apps and AI Applications (Customer-facing low-latency high concurrency) | • 120ms latency at 4000+ QPS proven performance at TB+ scale
• Supports hundreds to thousands of concurrent queries on single engine
• Price-performance leader (8x better than Snowflake, 18x vs Redshift) • Purpose-built for AI agents and data-intensive applications
• Native vector search and embeddings | • Sub-second response times at TB+ scale
• Supports 1000 concurrent users per replica
• Strong price-performance on customer-facing applications
• Native vector search and embeddings | | Ad hoc | • Excellent performance out-of-the-box with engine optimized for star and snowflake joins and aggregations
• Self learning query plan optimizer
• Full workload isolation prevents ad-hoc complexity from affecting real-time workloads
• Aggregating indexes are automatically used by optimizer | • Good for ad-hoc queries with ClickHouse Cloud’s separated storage/compute architecture
• Join optimizations enable more query complexity
• Strong sampling capabilities (TABLESAMPLE) for exploratory analysis
• Resource management through user quotas prevents query interference
• Materialized views offer performance improvements for common aggregation patterns, ad-hoc users specify directly in SQL | **Firebolt** is purpose-built for operational and customer-facing analytics where consistent, sub-second query latency is a hard requirement at scale. The combination of aggressive data pruning, multiple index types, and workload isolation makes it the strongest fit for data apps and AI applications serving end users directly, scenarios where P99 latency matters as much as P50, and where concurrent workloads need to be isolated to guarantee SLA consistency. It's less suited for ad-hoc analytics or general-purpose enterprise BI, where ecosystem maturity matter more than raw performance. **ClickHouse** is best suited for engineering-managed, high-throughput analytical workloads where query patterns are known in advance and teams have the expertise to tune schema design accordingly. It has a strong track record in observability, telemetry, event analytics, and time-series use cases - scenarios where data volumes are massive, ingestion rates are high, and the queries are well-understood. The tradeoff is operational overhead: getting the most out of ClickHouse requires deliberate investment in sort keys, projections, and materialized views. For teams willing to make that investment, it delivers exceptional raw performance. For teams expecting a more managed, self-optimizing experience, that overhead becomes a liability. # Firebolt vs Databricks (2025) (/comparison/firebolt-vs-databricks) ## Architecture [#architecture] The biggest difference among cloud data warehouses are whether they separate storage and compute, how much they isolate data and compute, and what clouds they can run on. | Feature | Firebolt | Databricks | | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Separation of storage and compute | Yes, separation of storage and metadata as well as compute from compute with full workload isolation. | Yes | | Supported cloud infrastructure | AWS (GCP coming soon) & anywhere (Firebolt Core) | AWS, Azure, GCP. Marketplaces and BYOC | | Isolated tenancy – option for dedicated resources | • Multi-tenant metadata layer
• Isolated tenancy for compute & storage per client | • Control plane in Databricks account
• Data plane in customer VPC (optional)
• Storage in customer VPC
• Serverless SQL runs in Databricks account with private connectivity | | Control vs abstraction of compute | Uses engine abstraction:
• Each engine has configurable cluster size (1-128 nodes) for horizontal scaling.
• Configurable compute family (compute vs storage optimized) and type (XS, S, M, L, XL) for vertical scaling
• Number of clusters for concurrency (auto)scaling.
Provides full workload isolation across engines. | • Configurable clusters and instance types
• Serverless SQL warehouses (GA 2025) run in Databricks account with private connectivity, no public IPs
• Pro/Classic warehouses run in customer VPC | | Self-hosted and hybrid deployment options | • Firebolt Core: Forever free, self-hosted edition with full query engine capabilities
• Same performance and features as managed service
• Deploy anywhere: local laptop, cloud, datacenter, Kubernetes
• Production-grade distributed architecture
• No usage restrictions except building competing SaaS | • Databricks on customer cloud accounts
• Unity Catalog for hybrid governance | | ACID Compliance and Transactions | • Full ACID compliance with snapshot isolation
• Multi-statement transactions supported
• Strong consistency across all operations
• Supports concurrent reads and writes
• Transactional integrity for data applications | • ACID transactions with Delta Lake
• Time travel and versioning
• Concurrent read/write operations | **Firebolt** is built on a natively decoupled storage & compute architecture, on AWS only. Data has to be copied outside of your VPC into the Firebolt, where both your compute and data run in a dedicated and isolated tenant. A "Firebolt Engine" can be granularly configured across # of nodes and different CPU/RAM/SSD combinations. **Databricks** was built by the founders of Spark as an analytics platform to support machine learning use cases. It leverages the Spark framework to process data residing in a data lake and is supported on AWS, GCP and Azure. Databricks coined the marketing term "Lakehouse '' architecture to illustrate the unification of data lake and data warehouse use cases. Customers still manage Spark clusters that process data residing in a Delta lake. Conversion of data to Delta Lake format is required to leverage the functionality of Delta Lake. Databricks Sql is a relatively new addition to simplify access to data stored in a data lake. ## Scalability [#scalability] There are three big differences among data warehouses and query engines that limit scalability: decoupled storage and compute, dedicated resources, and continuous ingestion. | Feature | Firebolt | Databricks | | --------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Elasticity – Scaling for larger data volumes and faster queries | Granular cluster resize with node types, number of nodes and number of clusters. Zero downtime. | Autoscaling clusters based on workload demand. Serverless SQL warehouses provide near-instant scaling (2-6 seconds startup) | | Elasticity – Scaling for higher concurrency | A single engine can handle hundreds of concurrent queries. Engines auto-scale the number of clusters up and down base on resource usage thresholds. Idle engines scale down to zero billing. | • 10 concurrent queries per cluster limit • Scales up to 40 clusters per warehouse (400 total concurrent queries) • Serverless SQL warehouses provide near-instant autoscaling • Pro/Classic warehouses take several minutes to provision new clusters • Real-world performance degradation typically occurs at 50-150 concurrent queries depending on complexity | **Firebolt** can handle the largest data volumes and concurrency on a single comparable cluster size, thanks to its superior hardware efficiency. Thanks to its decoupled storage & compute architecture it scales very well to large data volumes. However, resizing an engine size isn't instant and requires orchestration if avoiding downtime is necessary. A single Firebolt engine can support hundreds of concurrent queries, avoiding the need to scale out for most use cases. Scaling horizontally for even higher concurrency is manual. **Databricks** allow for autoscaling of clusters based on utilization. Additionally, increasing concurrency associated with a sql endpoint can be accomplished through the addition of clusters. Query concurrency per cluster is maxed at 10. However, scaling with additional clusters for concurrency is possible. Databricks provides a choice of instance types. ## Performance [#performance] Performance is the biggest challenge with most data warehouses today. While decoupled storage and compute architectures improved scalability and simplified administration, for most data warehouses it introduced two bottlenecks; storage, and compute. Most modern cloud data warehouses fetch entire partitions over the network instead of just fetching the specific data needed for each query. While many invest in caching, most do not invest heavily in query optimization. Most vendors also have not improved continuous ingestion or semi-structured data analytics performance, both of which are needed for operational and customer-facing use cases. | Feature | Firebolt | Databricks | | ------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Indexes | • Sparse primary indexes
• Aggregating indexes
• Join indexes
• Optimizer driven index usage | None | | Compute tuning | SQL defined engines. Control number of nodes, node family and type per cluster, with one or more clusters per engine. Multiple engines isolate workloads. | Choice of cluster type, node types including SSD-optimized instances. Serverless provides automatic resource allocation with Intelligent Workload Management (IWM) | | Storage format | Columnar, sorted & compressed & sparsely indexed storage (F3 – Firebolt File Format) with native Apache Iceberg support | • Delta Lake format with Liquid Clustering (February 2025 – replaces Z-ordering and traditional partitioning) • Cannot use Liquid Clustering alongside Z-ordering on same table • Allows for sorted data in Delta Lake • Requires Optimize to maintain ordering | | Table-level partition & pruning techniques | • User-defined table-level partitions are optional.
• Data is automatically sorted, compressed and indexed into F3 format.
• Pruning at indexed data-range level. | • Table level partitioning • Liquid Clustering for improved query performance and reduced data skew (February 2025) • Z-ordering (legacy, replaced by Liquid Clustering) • Periodic optimization of storage required | | Result cache | Yes, results and sub-results cache with transactional spoiling. | Multi-layered caching: local in-memory cache per cluster plus remote result cache (serverless only) that persists across all warehouses in workspace | | Warm cache (SSD) | Yes, at indexed data-range level granularity | Yes. Delta cache for data read by queries at file level granularity | | Support for semi-structured data & JSON functions within SQL | Yes, including Lambda expressions and native nested array structures | Yes | | Vector Search and AI Capabilities | • Native vector search capabilities and embeddings
• MCP Server for AI driven analytics
• Natural Language to SQL
• SQL based Inference | • MLflow integration and Databricks ML platform • Native vector search in Delta Lake (Vector Search) • AI and ML workloads optimized | | Query Optimizations | • Primary indexes, aggregating indexes, join indexes, sparse indexes
• Sub-plan result caching
• F3 storage format optimization
• Automatic query optimizer with aggressive pruning
• Late column materialization
• Query analysis tools based on execution telemetry | • Photon engine (C++ vectorized engine providing 3-8x average speedups, maximum speedups over 10x) • Automated stats collection (January 2025) enables cost-based optimization • Predictive I/O for faster point lookups and data updates • Liquid Clustering (February 2025) • Intelligent Workload Management (IWM) with AI-powered resource allocation • Delta cache • Materialized views support | **Firebolt** is the fastest when it comes to query performance when compared to cloud data warehouses and services like Athena. Its unique approach to storage and indexing results in highly aggressive data pruning that scans dramatically less data compared to other technologies. While other technologies scan partitions or micro-partitions, Firebolt works with indexed data ranges, that are significantly smaller. In addition, Firebolt lets users accelerate queries further with multiple index types (Aggregating index, Join index), and using its decoupled storage & compute architecture workloads can be easily isolated to guarantee consistent performance. **Databricks** is designed to leverage the Spark framework for processing large volumes of data. It leverages compressed Parquet files in a Delta Lake. To reduce the amount of data processed, it uses data pruning on partitions and Parquet file metadata. Databricks does not provide any indexes. ## Use cases [#use-cases] There are a host of different analytics use cases that can be supported by a data warehouse. Look at your legacy technologies and their workloads, as well as the new possible use cases, and figure out which ones you will need to support in the next few years. | Feature | Firebolt | Databricks | | ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Low-latency dashboards | • 120ms query latency at 4000 QPS (FireScale benchmark 2025)
• Sub-second performance at TB+ scale with proper indexing
• Built for AI-driven analytics, dashboards, and real-time analytic applications | • Sub-second to seconds load times at TB+ scale • Enhanced by Photon engine (3-8x average speedups) and Delta cache • Serverless SQL warehouses provide rapid startup (2-6 seconds) • Performance depends on cluster configuration | | Enterprise BI | • Growing ecosystem with focus on modern BI tools
• Strong SQL compliance with PostgreSQL
• Wire level compatibility drives expansion to PostgreSQL BI and ETL ecosystem | • Strong for data science and ML workloads • Unified analytics platform approach • Growing traditional BI integrations • Serverless SQL warehouses improve accessibility • Delta sharing capabilities | | Data Apps and AI Applications (Customer-facing low-latency high concurrency) | • 120ms latency at 4000+ QPS proven performance at TB+ scale
• Supports hundreds to thousands of concurrent queries on single engine
• Price-performance leader (8x better than Snowflake, 18x vs Redshift) • Purpose-built for AI agents and data-intensive applications
• Native vector search and embeddings | • 10 concurrent queries per cluster, scaling to 400 total concurrent queries per warehouse • Real-world performance degradation typically occurs at 50-150 concurrent queries depending on workload complexity • Serverless provides near-instant autoscaling • Photon engine delivers 3-8x performance improvements • Strong ML and AI platform integration | | Ad hoc | • Excellent performance out-of-the-box with engine optimized for star and snowflake joins and aggregations
• Self learning query plan optimizer
• Full workload isolation prevents ad-hoc complexity from affecting real-time workloads
• Aggregating indexes are automatically used by optimizer | • Excellent for ad-hoc with decoupled storage/compute • Serverless SQL warehouses provide instant provisioning • Intelligent Workload Management handles unpredictable workloads automatically • Strong for exploratory data analysis and ML workloads • Automated stats collection improves query planning | **Firebolt** stands out by being the fastest cloud data warehouse when compared to Snowflake, Redshift, BigQuery and Athena. It's great for delivering sub-second analytics at scale, while remaining hardware efficient and high concurrency friendly. This makes it a great choice for operational use cases and customer-facing data apps. Given that it is not as feature-rich and integration rich as the more mature data warehouses makes it a lesser fit for a general-purpose Enterprise data warehouse. It is also not the best fit for ad-hoc use cases, because of the need to predefine indexing at the table level. **Databricks** is a mature Spark based platform proven for processing streaming data. It is widely used for Machine Learning use cases by data scientists through the use of integrated notebooks. From a low latency query perspective, while it offers features like Delta Cache, it does not provide specialized indexes that can deliver low latency queries. # Firebolt vs Druid (2025) (/comparison/firebolt-vs-druid) ## Architecture [#architecture] The biggest difference among cloud data warehouses are whether they separate storage and compute, how much they isolate data and compute, and what clouds they can run on. | Feature | Firebolt | Druid | | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Separation of storage and compute | Yes, separation of storage and metadata as well as compute from compute with full workload isolation. | No | | Supported cloud infrastructure | AWS (GCP coming soon) & anywhere (Firebolt Core) | Can be installed anywhere | | Isolated tenancy – option for dedicated resources | • Multi-tenant metadata layer
• Isolated tenancy for compute & storage per client | Single tenant | | Control vs abstraction of compute | Uses engine abstraction:
• Each engine has configurable cluster size (1-128 nodes) for horizontal scaling.
• Configurable compute family (compute vs storage optimized) and type (XS, S, M, L, XL) for vertical scaling
• Number of clusters for concurrency (auto)scaling.
Provides full workload isolation across engines. | • Complex configuration of compute tier with multiple role-specific nodes
• Configurable node count
• Configurable compute types (virtual machines or kubernetes) | | Self-hosted and hybrid deployment options | • Firebolt Core: Forever free, self-hosted edition with full query engine capabilities
• Same performance and features as managed service
• Deploy anywhere: local laptop, cloud, datacenter, Kubernetes
• Production-grade distributed architecture
• No usage restrictions except building competing SaaS | Self-managed deployment required | | ACID Compliance and Transactions | • Full ACID compliance with snapshot isolation
• Multi-statement transactions supported
• Strong consistency across all operations
• Supports concurrent reads and writes
• Transactional integrity for data applications | Limited ACID support with eventual consistency | **Firebolt** is built on a natively decoupled compute & storage architecture, on AWS only. Data has to be copied outside of your VPC into the Firebolt, where both your compute and data run in a dedicated and isolated tenant. A "Firebolt Engine" can be granularly configured across # of nodes and different CPU/RAM/SSD combinations. **Druid** is an OLAP engine designed to provide fast real time analytics. Druid adopts a clustered architecture with servers that host various role specific processes. These processes address real time and batch ingestion, indexing, querying of historical and real time data. Apache Druid can be deployed as a virtual machine or a Kubernetes based cluster. Druid does not support a decoupled compute & storage architecture. Deep storage in the form of object storage is used to replicate data to. ## Scalability [#scalability] There are three big differences among data warehouses and query engines that limit scalability: decoupled storage and compute, dedicated resources, and continuous ingestion. | Feature | Firebolt | Druid | | --------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- | | Elasticity – Scaling for larger data volumes and faster queries | Granular cluster resize with node types, number of nodes and number of clusters. Zero downtime. | Scale-up of nodes requires careful planning and downtime. Addition of new nodes for scale-out is possible | | Elasticity – Scaling for higher concurrency | A single engine can handle hundreds of concurrent queries. Engines auto-scale the number of clusters up and down base on resource usage thresholds. Idle engines scale down to zero billing. | Supports 100s to 100,000s queries per second (1000+ QPS) with proper configuration and scaling | **Firebolt** can handle the largest data volumes and concurrency on a single comparable cluster size, thanks to its superior hardware efficiency. Thanks to its decoupled storage & compute architecture it scales very well to large data volumes. However, resizing an engine size isn't instant and requires orchestration if avoiding downtime is necessary. A single Firebolt engine can support hundreds of concurrent queries, avoiding the need to scale out for most use cases. Scaling horizontally for even higher concurrency is manual. **Druid** provides the ability to handle fast ingest and high concurrency. Custom sizing and cluster tuning are required to balance the compute, memory, storage needs of each process within Druid and to provide high concurrency. Druid clusters can be grown by adding nodes with automatic rebalancing of storage segments assigned to nodes. Self hosted Druid on Kubernetes is an option that users leverage to simplify scaling. Additionally, Cloud based managed Druid offerings are being rolled out. However, these managed offerings are limited in scale and scaling is not granular. ## Performance [#performance] Performance is the biggest challenge with most data warehouses today. While decoupled storage and compute architectures improved scalability and simplified administration, for most data warehouses it introduced two bottlenecks; storage, and compute. Most modern cloud data warehouses fetch entire partitions over the network instead of just fetching the specific data needed for each query. While many invest in caching, most do not invest heavily in query optimization. Most vendors also have not improved continuous ingestion or semi-structured data analytics performance, both of which are needed for operational and customer-facing use cases. | Feature | Firebolt | Druid | | ------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | | Indexes | • Sparse primary indexes
• Aggregating indexes
• Join indexes
• Optimizer driven index usage | Compressed bitmap indexes for data access and roll-ups to manage aggregations | | Compute tuning | SQL defined engines. Control number of nodes, node family and type per cluster, with one or more clusters per engine. Multiple engines isolate workloads. | On-premises, self-managed hardware. Druid requires infrastructure management and leverages commonly available instance types | | Storage format | Columnar, sorted & compressed & sparsely indexed storage (F3 – Firebolt File Format) with native Apache Iceberg support | Columnar storage format with time-based sorting | | Table-level partition & pruning techniques | • User-defined table-level partitions are optional.
• Data is automatically sorted, compressed and indexed into F3 format.
• Pruning at indexed data-range level. | Restrictive time-based partitioning. Can partition based on other secondary columns | | Result cache | Yes, results and sub-results cache with transactional spoiling. | Ability to support caching on broker (set to off by default) | | Warm cache (SSD) | Yes, at indexed data-range level granularity | Yes, at much larger segment level granularity | | Support for semi-structured data & JSON functions within SQL | Yes, including Lambda expressions and native nested array structures | Recommend flattening JSON or translate to array prior to loading. No support for JSON parsing at query runtime | | Vector Search and AI Capabilities | • Native vector search capabilities and embeddings
• MCP Server for AI driven analytics
• Natural Language to SQL
• SQL based Inference | No native AI or vector search capabilities | | Query Optimizations | • Primary indexes, aggregating indexes, join indexes, sparse indexes
• Sub-plan result caching
• F3 storage format optimization
• Automatic query optimizer with aggressive pruning
• Late column materialization
• Query analysis tools based on execution telemetry | • Compressed bitmap indexes • Roll-up aggregations • Time-based optimization • Query optimization requires manual tuning | **Firebolt** is the fastest when it comes to query performance when compared to cloud data warehouses and services like Athena. Its unique approach to storage and indexing results in highly aggressive data pruning that scans dramatically less data compared to other technologies. While other technologies scan partitions or micro-partitions, Firebolt works with indexed data ranges, that are significantly smaller. In addition, Firebolt lets users accelerate queries further with multiple index types (Aggregating index, Join index), and using its decoupled storage & compute architecture workloads can be easily isolated to guarantee consistent performance. **Druid** provides high performance through columnar storage format, parallel processing, bitmap indexes and roll-ups. Druid, however, recommends a denormalized data model for performance needs. Join operations in Druid are a relatively new feature with various limitations, especially if there is a need to join large datasets. ## Use cases [#use-cases] There are a host of different analytics use cases that can be supported by a data warehouse. Look at your legacy technologies and their workloads, as well as the new possible use cases, and figure out which ones you will need to support in the next few years. | Feature | Firebolt | Druid | | ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Low-latency dashboards | • 120ms query latency at 4000 QPS (FireScale benchmark 2025)
• Sub-second performance at TB+ scale with proper indexing
• Built for AI-driven analytics, dashboards, and real-time analytic applications | • Sub-second load times optimized for time-series and real-time analytics • Built for high-concurrency interactive dashboards • Requires denormalized data model | | Enterprise BI | • Growing ecosystem with focus on modern BI tools
• Strong SQL compliance with PostgreSQL
• Wire level compatibility drives expansion to PostgreSQL BI and ETL ecosystem | • Limited integrations with traditional Enterprise BI tools • Strong for real-time operational dashboards • Requires specialized visualization tools | | Data Apps and AI Applications (Customer-facing low-latency high concurrency) | • 120ms latency at 4000+ QPS proven performance at TB+ scale
• Supports hundreds to thousands of concurrent queries on single engine
• Price-performance leader (8x better than Snowflake, 18x vs Redshift) • Purpose-built for AI agents and data-intensive applications
• Native vector search and embeddings | • Built for high concurrency (1000+ QPS) with distributed architecture • Sub-second response times for time-series data • Optimized for real-time operational applications • No AI capabilities | | Ad hoc | • Excellent performance out-of-the-box with engine optimized for star and snowflake joins and aggregations
• Self learning query plan optimizer
• Full workload isolation prevents ad-hoc complexity from affecting real-time workloads
• Aggregating indexes are automatically used by optimizer | • Not optimized for ad-hoc queries • Requires predefined roll-ups and data modeling • Limited flexibility for exploratory analysis | **Firebolt** stands out by being the fastest cloud data warehouse when compared to Snowflake, Redshift, BigQuery and Athena. It's great for delivering sub-second analytics at scale, while remaining hardware efficient and high concurrency friendly. This makes it a great choice for operational use cases and customer-facing data apps. Given that it is not as feature-rich and integration rich as the more mature data warehouses makes it a lesser fit for a general-purpose Enterprise data warehouse. It is also not the best fit for ad-hoc use cases, because of the need to predefine indexing at the table level. **Druid** is designed as an OLAP engine to provide fast access to aggregations that are run against large volumes of data. Druid is typically used for customer facing analytics and streaming data processing. Druid is used as an add-on with other data warehousing products that are efficient at scaling, joining, and filtering large volumes of data. It is not a suitable option for data warehouse replacement. # Firebolt vs Redshift (2025) (/comparison/firebolt-vs-redshift) ## Architecture [#architecture] The biggest difference among cloud data warehouses are whether they separate storage and compute, how much they isolate data and compute, and what clouds they can run on. | Feature | Firebolt | Redshift | | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------ | | Separation of storage and compute | Yes, separation of storage and metadata as well as compute from compute with full workload isolation. | RA3 instances enable separation of compute and storage, but limited workload isolation compared to other platforms | | Supported cloud infrastructure | AWS (GCP coming soon) & anywhere (Firebolt Core) | AWS only | | Isolated tenancy – option for dedicated resources | • Multi-tenant metadata layer
• Isolated tenancy for compute & storage per client | • Isolated tenant & resources • Runs in your VPC | | Control vs abstraction of compute | Uses engine abstraction:
• Each engine has configurable cluster size (1-128 nodes) for horizontal scaling.
• Configurable compute family (compute vs storage optimized) and type (XS, S, M, L, XL) for vertical scaling
• Number of clusters for concurrency (auto)scaling.
Provides full workload isolation across engines. | • Configurable cluster size
• Configurable compute types | | Self-hosted and hybrid deployment options | • Firebolt Core: Forever free, self-hosted edition with full query engine capabilities
• Same performance and features as managed service
• Deploy anywhere: local laptop, cloud, datacenter, Kubernetes
• Production-grade distributed architecture
• No usage restrictions except building competing SaaS | Limited hybrid options with Redshift Serverless | | ACID Compliance and Transactions | • Full ACID compliance with snapshot isolation
• Multi-statement transactions supported
• Strong consistency across all operations
• Supports concurrent reads and writes
• Transactional integrity for data applications | ACID compliant at table level with some limitations on concurrent operations | **Firebolt** is built on a natively decoupled storage & compute architecture, on AWS only. Data has to be copied outside of your VPC into the Firebolt, where both your compute and data run in a dedicated and isolated tenant. A "Firebolt Engine" can be granularly configured across # of nodes and different CPU/RAM/SSD combinations **Redshift** has the oldest architecture, being the first Cloud DW in the group. Its architecture wasn't designed to separate storage & compute. While it now has RA3 nodes which allow you to scale compute and only cache the data you need locally, all compute still operates together. You cannot separate and isolate different workloads over the same data, which puts it behind other decoupled storage/compute architectures. Redshift runs as an isolated tenant per customer, and unlike other cloud data warehouses, it is deployed in your VPC. Redshift offers a serverless option which is based on an abstracted unit called Redshift Processing Unit (RPU) ranging from 8 to 512 in increments of 8. Each RPU provides 2 vCPU and 16GB RAM. Thus, 8 RPU is equivalent to 16 vCPU / 128GB RAM. The minimum RPU is 8. ## Scalability [#scalability] There are three big differences among data warehouses and query engines that limit scalability: decoupled storage and compute, dedicated resources, and continuous ingestion. | Feature | Firebolt | Redshift | | --------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | | Elasticity – Scaling for larger data volumes and faster queries | Granular cluster resize with node types, number of nodes and number of clusters. Zero downtime. | Available via Elastic Resize – slow and limited, downtime required | | Elasticity – Scaling for higher concurrency | A single engine can handle hundreds of concurrent queries. Engines auto-scale the number of clusters up and down base on resource usage thresholds. Idle engines scale down to zero billing. | • 5 concurrent queries per WLM queue by default (up to 8 queues) • Concurrency Scaling enables thousands of concurrent queries | **Firebolt** can handle the largest data volumes and concurrency on a single comparable cluster size, thanks to its superior hardware efficiency. Thanks to its decoupled storage & compute architecture it scales very well to large data volumes. However, resizing an engine size isn't instant and requires orchestration if avoiding downtime is necessary. A single Firebolt engine can support hundreds of concurrent queries, avoiding the need to scale out for most use cases. Scaling horizontally for even higher concurrency is manual. **Redshift** is limited in scale because even with RA3, it cannot distribute different workloads across clusters. While it can scale to up to 10 clusters automatically to support query concurrency, it can only handle a maximum of 50 queued queries across all clusters by default. ## Performance [#performance] Performance is the biggest challenge with most data warehouses today. While decoupled storage and compute architectures improved scalability and simplified administration, for most data warehouses it introduced two bottlenecks; storage, and compute. Most modern cloud data warehouses fetch entire partitions over the network instead of just fetching the specific data needed for each query. While many invest in caching, most do not invest heavily in query optimization. Most vendors also have not improved continuous ingestion or semi-structured data analytics performance, both of which are needed for operational and customer-facing use cases. | Feature | Firebolt | Redshift | | ------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Indexes | • Sparse primary indexes
• Aggregating indexes
• Join indexes
• Optimizer driven index usage | None | | Compute tuning | SQL defined engines. Control number of nodes, node family and type per cluster, with one or more clusters per engine. Multiple engines isolate workloads. | Choice over number of nodes and their type | | Storage format | Columnar, sorted & compressed & sparsely indexed storage (F3 – Firebolt File Format) with native Apache Iceberg support | Columnar & compressed storage (RA3 nodes) | | Table-level partition & pruning techniques | • User-defined table-level partitions are optional.
• Data is automatically sorted, compressed and indexed into F3 format.
• Pruning at indexed data-range level. | • No table partitions • User-defined distribution & sort keys are used to optimize for speed | | Result cache | Yes, results and sub-results cache with transactional spoiling. | Yes | | Warm cache (SSD) | Yes, at indexed data-range level granularity | Only with RA3 nodes at partition-level granularity | | Support for semi-structured data & JSON functions within SQL | Yes, including Lambda expressions and native nested array structures | Limited | | Vector Search and AI Capabilities | • Native vector search capabilities and embeddings
• MCP Server for AI driven analytics
• Natural Language to SQL
• SQL based Inference | Limited AI capabilities – primarily through integrations | | Query Optimizations | • Primary indexes, aggregating indexes, join indexes, sparse indexes
• Sub-plan result caching
• F3 storage format optimization
• Automatic query optimizer with aggressive pruning
• Late column materialization
• Query analysis tools based on execution telemetry | • Basic query optimizer • Materialized views • Result caching • ANALYZE for table statistics • Workload management (WLM) • Automated materialized views (AutoMV) • AI-driven scaling (Serverless preview) | **Firebolt** is the fastest when it comes to query performance when compared to cloud data warehouses and services like Athena. Its unique approach to storage and indexing results in highly aggressive data pruning that scans dramatically less data compared to other technologies. While other technologies scan partitions or micro-partitions, Firebolt works with indexed data ranges, that are significantly smaller. In addition, Firebolt lets user accelerate queries further with multiple index types (Aggregating index, Join index), and using its decoupled storage & compute architecture workloads can be easily isolated to guarantee consistent performance. **Redshift** does provide a result cache for accelerating repetitive query workloads and also has more tuning options than some others. But it does not deliver much faster compute performance than other cloud data warehouses in benchmarks. Sort keys can be used to optimize performance, but their contribution is limited. There is no support for indexes, and low-latency analytics at large data volumes is hard to achieve. Because Redshift decoupling of storage & compute is limited compared to other cloud data warehouses, it doesn't support isolating workloads, which means performance can degrade under pressure and competition for resources. ## Use cases [#use-cases] There are a host of different analytics use cases that can be supported by a data warehouse. Look at your legacy technologies and their workloads, as well as the new possible use cases, and figure out which ones you will need to support in the next few years. | Feature | Firebolt | Redshift | | ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Low-latency dashboards | • 120ms query latency at 4000 QPS (FireScale benchmark 2025)
• Sub-second performance at TB+ scale with proper indexing
• Built for AI-driven analytics, dashboards, and real-time analytic applications | • Seconds to tens of seconds load times at 100s of GB scale • Can achieve faster performance with Concurrency Scaling and proper tuning | | Enterprise BI | • Growing ecosystem with focus on modern BI tools
• Strong SQL compliance with PostgreSQL
• Wire level compatibility drives expansion to PostgreSQL BI and ETL ecosystem | • Mature and comprehensive Enterprise DW feature set • Extensive integrations with Enterprise BI ecosystem • Strong AWS ecosystem integration | | Data Apps and AI Applications (Customer-facing low-latency high concurrency) | • 120ms latency at 4000+ QPS proven performance at TB+ scale
• Supports hundreds to thousands of concurrent queries on single engine
• Price-performance leader (8x better than Snowflake, 18x vs Redshift) • Purpose-built for AI agents and data-intensive applications
• Native vector search and embeddings | • 5 concurrent queries per WLM queue by default (up to 8 queues) • Concurrency Scaling enables thousands of concurrent queries • Seconds-level response times typical • Automatic scaling for burst workloads • Limited AI application support | | Ad hoc | • Excellent performance out-of-the-box with engine optimized for star and snowflake joins and aggregations
• Self learning query plan optimizer
• Full workload isolation prevents ad-hoc complexity from affecting real-time workloads
• Aggregating indexes are automatically used by optimizer | • Performance dependent on predefined distribution & sort keys • Elastic Resize enables adding compute resources • Typically subset of data loaded for ad-hoc analysis | **Firebolt** stands out by being the fastest cloud data warehouse when compared to Snowflake, Redshift, BigQuery and Athena. It's great for delivering sub-second analytics at scale, while remaining hardware efficient and high concurrency friendly. This makes it a great choice for operational use cases and customer-facing data apps. Given that it is not as feature-rich and integration rich as the more mature data warehouses makes it a lesser fit for a general-purpose Enterprise data warehouse. It is also not the best fit for ad-hoc use cases, because of the need to predefine indexing at the table level. **Redshift** was originally designed to support traditional internal BI reporting and dashboard use cases for analysts. As such, it is typically used as a general-purpose Enterprise data warehouse. With deep integrations into the AWS ecosystem, it can also leverage AWS ML service, making it also useful for ML projects. However, given the coupling of storage & compute, and the difficulty in delivering low-latency analytics at scale, it is less suited for operational use cases and customer-facing use cases like Data Apps. The coupling of storage and compute, together with the need to predefine sort & dist keys for optimal performance, make it challenging to use for Ad-Hoc analytics. # Firebolt vs Snowflake (2025) (/comparison/firebolt-vs-snowflake) ## Architecture [#architecture] The biggest difference among cloud data warehouses are whether they separate storage and compute, how much they isolate data and compute, and what clouds they can run on. | Feature | Firebolt | Snowflake | | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Separation of storage and compute | Yes, separation of storage and metadata as well as compute from compute with full workload isolation. | Yes | | Supported cloud infrastructure | AWS (GCP coming soon) & anywhere (Firebolt Core) | AWS, Azure, GCP with full feature parity across all three major clouds | | Isolated tenancy – option for dedicated resources | • Multi-tenant metadata layer
• Isolated tenancy for compute & storage per client | • Multi-tenant pooled resources
• Isolated tenancy available via VPS tier | | Control vs abstraction of compute | Uses engine abstraction:
• Each engine has configurable cluster size (1-128 nodes) for horizontal scaling.
• Configurable compute family (compute vs storage optimized) and type (XS, S, M, L, XL) for vertical scaling
• Number of clusters for concurrency (auto)scaling.
Provides full workload isolation across engines. | • Configurable warehouse sizes (XS to 6XL)
• Multi-cluster warehouses with auto-scaling
• Choice between Generation 1 and Generation 2 standard warehouses
• MAX\_CONCURRENCY\_LEVEL parameter for resource allocation | | Self-hosted and hybrid deployment options | • Firebolt Core: Forever free, self-hosted edition with full query engine capabilities
• Same performance and features as managed service
• Deploy anywhere: local laptop, cloud, datacenter, Kubernetes
• Production-grade distributed architecture
• No usage restrictions except building competing SaaS | Snowflake for Government Cloud and private cloud options available | | ACID Compliance and Transactions | • Full ACID compliance with snapshot isolation
• Multi-statement transactions supported
• Strong consistency across all operations
• Supports concurrent reads and writes
• Transactional integrity for data applications | Full ACID compliance with Time Travel and zero-copy cloning capabilities | **Firebolt** is built on a natively decoupled storage & compute architecture, on AWS only. Data has to be copied outside of your VPC into the Firebolt, where both your compute and data run in a dedicated and isolated tenant. A "Firebolt Engine" can be granularly configured across # of nodes and different CPU/RAM/SSD combinations. **Snowflake** was one of the first decoupled storage and compute architectures, making it the first to have nearly unlimited compute scale and workload isolation, and horizontal user scalability. It runs on AWS, Azure and GCP. It is multi-tenant over shared resources in nature and requires you to move data out of your VPC and into the Snowflake cloud. "Virtual Private Snowflake" (VPS) is its highest-priced tier, and can run a dedicated isolated version of Snowflake. Its virtual warehouses can be T-shirt sized along an XS/S/M…/4XL axis, where each discrete T-shirt size is bundled with fixed HW properties that are abstracted from the users. Snowflake has recently added support for Snowflake managed Iceberg tables. ## Scalability [#scalability] There are three big differences among data warehouses and query engines that limit scalability: decoupled storage and compute, dedicated resources, and continuous ingestion. | Feature | Firebolt | Snowflake | | --------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Elasticity – Scaling for larger data volumes and faster queries | Granular cluster resize with node types, number of nodes and number of clusters. Zero downtime. | • Instant warehouse resize (XS to 6XL) with no downtime • Multi-cluster auto-scaling • Generation 2 warehouses provide \~2x performance improvement over Generation 1 | | Elasticity – Scaling for higher concurrency | A single engine can handle hundreds of concurrent queries. Engines auto-scale the number of clusters up and down base on resource usage thresholds. Idle engines scale down to zero billing. | • Single warehouse supports many concurrent queries (MAX\_CONCURRENCY\_LEVEL=8 controls resource allocation per query, not query limit) • Multi-cluster warehouses enable thousands of concurrent queries with auto-scaling • Unlimited virtual warehouses can be created | **Firebolt** can handle the largest data volumes and concurrency on a single comparable cluster size, thanks to its superior hardware efficiency. Thanks to its decoupled storage & compute architecture it scales very well to large data volumes. However, resizing an engine size isn't instant and requires orchestration if avoiding downtime is necessary. A single Firebolt engine can support hundreds of concurrent queries, avoiding the need to scale out for most use cases. Scaling horizontally for even higher concurrency is manual. **Snowflake** scales very well both for data volumes and query concurrency. The decoupled storage/compute architecture supports resizing clusters without downtime, and in addition, supports auto-scaling horizontally for higher query concurrency during peak hours. ## Performance [#performance] Performance is the biggest challenge with most data warehouses today. While decoupled storage and compute architectures improved scalability and simplified administration, for most data warehouses it introduced two bottlenecks; storage, and compute. Most modern cloud data warehouses fetch entire partitions over the network instead of just fetching the specific data needed for each query. While many invest in caching, most do not invest heavily in query optimization. Most vendors also have not improved continuous ingestion or semi-structured data analytics performance, both of which are needed for operational and customer-facing use cases. | Feature | Firebolt | Snowflake | | ------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Indexes | • Sparse primary indexes
• Aggregating indexes
• Join indexes
• Optimizer driven index usage | • Search Optimization Service for point lookups and selective queries (additional cost) • Clustering keys for data organization and automatic clustering • Materialized views • Snowflake Optima automatic indexing on Generation 2 warehouses (no additional cost) • No traditional database indexes | | Compute tuning | SQL defined engines. Control number of nodes, node family and type per cluster, with one or more clusters per engine. Multiple engines isolate workloads. | • Warehouse T-shirt sizing (XS to 6XL) • Multi-cluster configuration and scaling policies • Generation 1 vs Generation 2 warehouse selection • MAX\_CONCURRENCY\_LEVEL parameter tuning • Query Acceleration Service for long-running queries | | Storage format | Columnar, sorted & compressed & sparsely indexed storage (F3 – Firebolt File Format) with native Apache Iceberg support | Columnar micro-partitioned & compressed storage | | Table-level partition & pruning techniques | • User-defined table-level partitions are optional.
• Data is automatically sorted, compressed and indexed into F3 format.
• Pruning at indexed data-range level. | • Data automatically divided into micro-partitions • Automatic pruning at micro-partition level • Clustering keys for data organization with automatic clustering • Snowflake Optima provides additional automatic pruning optimization on Gen2 warehouses | | Result cache | Yes, results and sub-results cache with transactional spoiling. | Yes | | Warm cache (SSD) | Yes, at indexed data-range level granularity | Yes, at micro-partition level granularity | | Support for semi-structured data & JSON functions within SQL | Yes, including Lambda expressions and native nested array structures | Yes | | Vector Search and AI Capabilities | • Native vector search capabilities and embeddings
• MCP Server for AI driven analytics
• Natural Language to SQL
• SQL based Inference | AI integration through Cortex AI and Snowpark ML | | Query Optimizations | • Primary indexes, aggregating indexes, join indexes, sparse indexes
• Sub-plan result caching
• F3 storage format optimization
• Automatic query optimizer with aggressive pruning
• Late column materialization
• Query analysis tools based on execution telemetry | • Search Optimization Service for point lookups (additional cost)
• Query Acceleration Service (QAS) for long-running and unpredictable workloads • Snowflake Optima automatic optimization on Generation 2 warehouses (no additional cost) • Automatic clustering with background maintenance • Materialized views with automatic refresh • Result cache (24hrs) • Cost-based optimization with dynamic query rewriting | **Firebolt** is the fastest when it comes to query performance when compared to cloud data warehouses and services like Athena. Its unique approach to storage and indexing results in highly aggressive data pruning that scans dramatically less data compared to other technologies. While other technologies scan partitions or micro-partitions, Firebolt works with indexed data ranges, that are significantly smaller. In addition, Firebolt lets user accelerate queries further with multiple index types (Aggregating index, Join index), and using its decoupled storage & compute architecture workloads can be easily isolated to guarantee consistent performance. **Snowflake** typically comes on top for most queries when it comes to performance in public TPC-based benchmarks when compared to BigQuery and Redshift, but only marginally. Its micro partition storage approach effectively scans less data compared to larger partitions. The ability to isolate workloads over the decoupled storage & compute architecture lets you avoid competition for resources compared to multi-tenant shared resource solutions, and the ability to increase warehouse sizes can often enhance performance (for a higher price), but not always linearly. Snowflake's recently released "Search optimization service" delivers index-like behavior for point queries, but comes at an additional cost. ## Use cases [#use-cases] There are a host of different analytics use cases that can be supported by a data warehouse. Look at your legacy technologies and their workloads, as well as the new possible use cases, and figure out which ones you will need to support in the next few years. | Feature | Firebolt | Snowflake | | ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Low-latency dashboards | • 120ms query latency at 4000 QPS (FireScale benchmark 2025)
• Sub-second performance at TB+ scale with proper indexing
• Built for AI-driven analytics, dashboards, and real-time analytic applications | • Sub-second to seconds response times at TB+ scale with proper clustering and optimization • Enhanced by Query Acceleration Service and Search Optimization Service • Generation 2 warehouses provide \~2x performance improvement over Generation 1 • Snowflake Optima provides automatic optimization | | Enterprise BI | • Growing ecosystem with focus on modern BI tools
• Strong SQL compliance with PostgreSQL
• Wire level compatibility drives expansion to PostgreSQL BI and ETL ecosystem | • Mature and comprehensive Enterprise DW feature set • Extensive integrations with Enterprise BI ecosystem • Multi-cloud deployment options with consistent experience • Strong SQL compliance and wide ecosystem support • Zero-copy data sharing capabilities | | Data Apps and AI Applications (Customer-facing low-latency high concurrency) | • 120ms latency at 4000+ QPS proven performance at TB+ scale
• Supports hundreds to thousands of concurrent queries on single engine
• Price-performance leader (8x better than Snowflake, 18x vs Redshift) • Purpose-built for AI agents and data-intensive applications
• Native vector search and embeddings | • Multi-cluster warehouses support thousands of concurrent users with auto-scaling • Individual warehouses support many concurrent queries (not limited to 8 concurrent queries) • Sub-second to seconds response times with proper optimization • Generation 2 warehouses provide significant performance improvements for high-concurrency workloads • AI integration through Cortex AI | | Ad hoc | • Excellent performance out-of-the-box with engine optimized for star and snowflake joins and aggregations
• Self learning query plan optimizer
• Full workload isolation prevents ad-hoc complexity from affecting real-time workloads
• Aggregating indexes are automatically used by optimizer | • Excellent for ad-hoc with decoupled storage/compute • Auto-scaling and instant compute provisioning • Minimal predefined optimization required • Query Acceleration Service handles unpredictable workloads automatically • Snowflake Optima provides automatic optimization for recurring patterns | **Firebolt** stands out by being the fastest cloud data warehouse when compared to Snowflake, Redshift, BigQuery and Athena. It's great for delivering sub-second analytics at scale, while remaining hardware efficient and high concurrency friendly. This makes it a great choice for operational use cases and customer-facing data apps. Given that it is not as feature-rich and integration rich as the more mature data warehouses makes it a lesser fit for a general-purpose Enterprise data warehouse. It is also not the best fit for ad-hoc use cases, because of the need to predefine indexing at the table level. **Snowflake** is a well rounded general purpose cloud data warehouse, that can also span beyond traditional BI & Analytics use cases into Ad-Hoc and ML use cases. Thanks to the flexible decoupeld storage & compute architecture that allows you to isolate and control the amount of compute per workload, it's possible to tackle a broad spectrum of workloads. However, like its close siblings Redshift & BigQuery, it struggles to deliver low-latency query performance at scale, making it a lesser fit for operational use cases and customer-facing data apps. # Redshift vs ClickHouse (/comparison/redshift-vs-clickhouse) ## Architecture [#architecture] The biggest difference among cloud data warehouses are whether they separate storage and compute, how much they isolate data and compute, and what clouds they can run on. | Feature | Redshift | ClickHouse | | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Separation of storage and compute | RA3 instances enable separation of compute and storage, but limited workload isolation compared to other platforms | Yes – SharedMergeTree engine in ClickHouse Cloud enables full separation of storage and compute, with compute-compute separation through Warehouses feature (introduced 2025) allowing multiple isolated compute services sharing the same data | | Supported cloud infrastructure | AWS only | AWS, GCP, Azure, cloud service and on-premises | | Isolated tenancy – option for dedicated resources | • Isolated tenant & resources • Runs in your VPC | • Multi-tenant metadata layer
• Isolated tenancy for compute & storage per client in cloud | | Control vs abstraction of compute | • Configurable cluster size
• Configurable compute types | Configurable cluster size and compute types in ClickHouse Cloud with granular control over nodes (1-128 nodes) and node characteristics. Warehouses feature enables multiple isolated read-only compute environments. | | Self-hosted and hybrid deployment options | Limited hybrid options with Redshift Serverless | Self-managed deployments available with full control over infrastructure | | ACID Compliance and Transactions | ACID compliant at table level with some limitations on concurrent operations | Limited ACID compliance with MergeTree engine family. | **Redshift** has the oldest architecture, being the first Cloud DW in the group. Its architecture wasn't designed to separate storage & compute. While it now has RA3 nodes which allow you to scale compute and only cache the data you need locally, all compute still operates together. You cannot separate and isolate different workloads over the same data, which puts it behind other decoupled storage/compute architectures. Redshift runs as an isolated tenant per customer, and unlike other cloud data warehouses, it is deployed in your VPC. Redshift offers a serverless option which is based on an abstracted unit called Redshift Processing Unit (RPU) ranging from 8 to 512 in increments of 8. Each RPU provides 2 vCPU and 16GB RAM. Thus, 8 RPU is equivalent to 16 vCPU / 128GB RAM. The minimum RPU is 8. **ClickHouse** was originally developed at Yandex, the Russian search engine, as an OLAP engine for low latency analytics. It was built as an on-premise solution with coupled storage & compute, and a large variety of tuning options in the form of indexes and and merge trees. ClickHouse's architecture is famous for its focus on performance and low-latency queries. The tradeoff is that it is considered very difficult to work with. SQL support is very limited, and tuning/running it requires significant engineering resources. ## Scalability [#scalability] There are three big differences among data warehouses and query engines that limit scalability: decoupled storage and compute, dedicated resources, and continuous ingestion. \| Feature | | \| --- | | \| Elasticity – Scaling for larger data volumes and faster queries | | \| Elasticity – Scaling for higher concurrency | | **Redshift** is limited in scale because even with RA3, it cannot distribute different workloads across clusters. While it can scale to up to 10 clusters automatically to support query concurrency, it can only handle a maximum of 50 queued queries across all clusters by default. **ClickHouse** doesn't offer any dedicated scaling features or mechanisms. While it can deliver linearly scalable performance for some types of queries, scaling itself has to be done manually. Hardware is self-managed in ClickHouse. This means that to scale you would have to provision a cluster and migrate. ## Performance [#performance] Performance is the biggest challenge with most data warehouses today. While decoupled storage and compute architectures improved scalability and simplified administration, for most data warehouses it introduced two bottlenecks; storage, and compute. Most modern cloud data warehouses fetch entire partitions over the network instead of just fetching the specific data needed for each query. While many invest in caching, most do not invest heavily in query optimization. Most vendors also have not improved continuous ingestion or semi-structured data analytics performance, both of which are needed for operational and customer-facing use cases. | Feature | Redshift | ClickHouse | | ------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Indexes | None | • Primary indexes • Skipping indexes (minmax, set, bloom filters, ngrambf\_v1, tokenbf\_v1)
• MergeTree indexes
• Incremental Materialized views | | Compute tuning | Choice over number of nodes and their type | Configurable compute resources in cloud offering | | Storage format | Columnar & compressed storage (RA3 nodes) | Columnar, supports sorted, compressed, encoded & sparsely indexed files with native Apache Iceberg support. | | Table-level partition & pruning techniques | • No table partitions • User-defined distribution & sort keys are used to optimize for speed | Partitioning by date/time and custom partitions with MergeTree indexes. | | Result cache | Yes | Yes, results cache with TTL and query condition cache. | | Warm cache (SSD) | Only with RA3 nodes at partition-level granularity | Yes, at indexed data-range level granularity | | Support for semi-structured data & JSON functions within SQL | Limited | Yes, including Lambda expressions and native JSON data type (GA in v25.3) | | Vector Search and AI Capabilities | Limited AI capabilities – primarily through integrations | • Native vector search capabilities and embeddings
• MCP Server for AI driven analytics
• Natural Language to SQL
• SQL based Inference | | Query Optimizations | • Basic query optimizer • Materialized views • Result caching • ANALYZE for table statistics • Workload management (WLM) • Automated materialized views (AutoMV) • AI-driven scaling (Serverless preview) | • Primary indexes (ORDER BY)
• Data skipping indexes (minmax, set, bloom filters, ngrambf\_v1, tokenbf\_v1)
• Materialized views
• Projections
• PREWHERE optimization
• Query analysis tools
• Automatic global join reordering (v25.9)
• Enhanced JSON query optimization
• Streaming secondary indices | **Redshift** does provide a result cache for accelerating repetitive query workloads and also has more tuning options than some others. But it does not deliver much faster compute performance than other cloud data warehouses in benchmarks. Sort keys can be used to optimize performance, but their contribution is limited. There is no support for indexes, and low-latency analytics at large data volumes is hard to achieve. Because Redshift decoupling of storage & compute is limited compared to other cloud data warehouses, it doesn't support isolating workloads, which means performance can degrade under pressure and competition for resources. **ClickHouse** is famous for being one of the fastest local runtimes ever built for OLAP workloads. Its columnar storage, compression and indexing capabilities make it a consistent leader in benchmarks. Its lack of support for standard SQL and lack of query optimizer means that it's less suitable for traditional BI workloads, and more suitable for engineering managed workloads. While fast, it requires a lot of tuning and optimization. ## Use cases [#use-cases] There are a host of different analytics use cases that can be supported by a data warehouse. Look at your legacy technologies and their workloads, as well as the new possible use cases, and figure out which ones you will need to support in the next few years. | Feature | Redshift | ClickHouse | | ---------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Low-latency dashboards | • Seconds to tens of seconds load times at 100s of GB scale • Can achieve faster performance with Concurrency Scaling and proper tuning | • Sub-second load times at TB+ scale with proper indexing
• ClickHouse Cloud reduces engineering overhead with managed service • Proven low-latency performance (120ms at 2500 QPS in benchmarks)
• Purpose-built for low-latency OLAP and real-time analytics | | Enterprise BI | • Mature and comprehensive Enterprise DW feature set • Extensive integrations with Enterprise BI ecosystem • Strong AWS ecosystem integration | • Growing ecosystem with 50+ integrations including major BI tools
• Native MySQL protocol support enables broad BI tool compatibility • Strong SQL compliance with PostgreSQL compatibility
• Best suited for modern analytical workloads and engineering-managed use cases | | Data Apps and AI Applications (Customer-facing low-latency high concurrency) | • 5 concurrent queries per WLM queue by default (up to 8 queues) • Concurrency Scaling enables thousands of concurrent queries • Seconds-level response times typical • Automatic scaling for burst workloads • Limited AI application support | • Sub-second response times at TB+ scale
• Supports 1000 concurrent users per replica
• Strong price-performance on customer-facing applications
• Native vector search and embeddings | | Ad hoc | • Performance dependent on predefined distribution & sort keys • Elastic Resize enables adding compute resources • Typically subset of data loaded for ad-hoc analysis | • Good for ad-hoc queries with ClickHouse Cloud's separated storage/compute architecture
• Join optimizations enable more query complexity
• Strong sampling capabilities (TABLESAMPLE) for exploratory analysis
• Resource management through user quotas prevents query interference
• Materialized views offer performance improvements for common aggregation patterns, ad-hoc users specify directly in SQL | **Redshift** was originally designed to support traditional internal BI reporting and dashboard use cases for analysts. As such, it is typically used as a general-purpose Enterprise data warehouse. With deep integrations into the AWS ecosystem, it can also leverage AWS ML service, making it also useful for ML projects. However, given the coupling of storage & compute, and the difficulty in delivering low-latency analytics at scale, it is less suited for operational use cases and customer-facing use cases like Data Apps. The coupling of storage and compute, together with the need to predefine sort & dist keys for optimal performance, make it challenging to use for Ad-Hoc analytics. **ClickHouse** was not designed to be a data warehouse, but rather a low-latency query execution runtime. Managing it typically requires significant engineering overhead. Hence, it's a good fit for engineering managed operational use cases and customer-facing data apps, where low latency matters. It is not a good fit for a general purpose data warehouse, nor for Ad-Hoc analytics or ELT. # Redshift vs Databricks (/comparison/redshift-vs-databricks) ## Architecture [#architecture] The biggest difference among cloud data warehouses are whether they separate storage and compute, how much they isolate data and compute, and what clouds they can run on. | Feature | Databricks | Redshift | Firebolt | | ------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Separation of storage and compute | Yes | RA3 instances enable separation of compute and storage, but limited workload isolation compared to other platforms | Yes, separation of storage and metadata as well as compute from compute with full workload isolation. | | Supported cloud infrastructure | AWS, Azure, GCP. Marketplaces and BYOC | AWS only | AWS (GCP coming soon) & anywhere (Firebolt Core) | | Isolated tenancy – option for dedicated resources | • Control plane in Databricks account
• Data plane in customer VPC (optional)
• Storage in customer VPC
• Serverless SQL runs in Databricks account with private connectivity | • Isolated tenant & resources • Runs in your VPC | • Multi-tenant metadata layer
• Isolated tenancy for compute & storage per client | | Control vs abstraction of compute | • Configurable clusters and instance types
• Serverless SQL warehouses (GA 2025) run in Databricks account with private connectivity, no public IPs
• Pro/Classic warehouses run in customer VPC | • Configurable cluster size
• Configurable compute types | Uses engine abstraction:
• Each engine has configurable cluster size (1-128 nodes) for horizontal scaling.
• Configurable compute family (compute vs storage optimized) and type (XS, S, M, L, XL) for vertical scaling
• Number of clusters for concurrency (auto)scaling.
Provides full workload isolation across engines. | | Self-hosted and hybrid deployment options | • Databricks on customer cloud accounts
• Unity Catalog for hybrid governance | Limited hybrid options with Redshift Serverless | • Firebolt Core: Forever free, self-hosted edition with full query engine capabilities
• Same performance and features as managed service
• Deploy anywhere: local laptop, cloud, datacenter, Kubernetes
• Production-grade distributed architecture
• No usage restrictions except building competing SaaS | | ACID Compliance and Transactions | • ACID transactions with Delta Lake
• Time travel and versioning
• Concurrent read/write operations | ACID compliant at table level with some limitations on concurrent operations | • Full ACID compliance with snapshot isolation
• Multi-statement transactions supported
• Strong consistency across all operations
• Supports concurrent reads and writes
• Transactional integrity for data applications | **Redshift** has the oldest architecture, being the first Cloud DW in the group. Its architecture wasn't designed to separate storage & compute. While it now has RA3 nodes which allow you to scale compute and only cache the data you need locally, all compute still operates together. You cannot separate and isolate different workloads over the same data, which puts it behind other decoupled storage/compute architectures. Redshift runs as an isolated tenant per customer, and unlike other cloud data warehouses, it is deployed in your VPC. Redshift offers a serverless option which is based on an abstracted unit called Redshift Processing Unit (RPU) ranging from 8 to 512 in increments of 8. Each RPU provides 2 vCPU and 16GB RAM. Thus, 8 RPU is equivalent to 16 vCPU / 128GB RAM. The minimum RPU is 8. **Databricks** was built by the founders of Spark as an analytics platform to support machine learning use cases. It leverages the Spark framework to process data residing in a data lake and is supported on AWS, GCP and Azure. Databricks coined the marketing term "Lakehouse '' architecture to illustrate the unification of data lake and data warehouse use cases. Customers still manage Spark clusters that process data residing in a Delta lake. Conversion of data to Delta Lake format is required to leverage the functionality of Delta Lake. Databricks Sql is a relatively new addition to simplify access to data stored in a data lake. **Firebolt** is built on a natively decoupled storage & compute architecture, on AWS only. Data has to be copied outside of your VPC into the Firebolt, where both your compute and data run in a dedicated and isolated tenant. A "Firebolt Engine" can be granularly configured across # of nodes and different CPU/RAM/SSD combinations. ## Scalability [#scalability] There are three big differences among data warehouses and query engines that limit scalability: decoupled storage and compute, dedicated resources, and continuous ingestion. | Feature | Databricks | Redshift | Firebolt | | --------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Elasticity – Scaling for larger data volumes and faster queries | Autoscaling clusters based on workload demand. Serverless SQL warehouses provide near-instant scaling (2-6 seconds startup) | Available via Elastic Resize – slow and limited, downtime required | Granular cluster resize with node types, number of nodes and number of clusters. Zero downtime. | | Elasticity – Scaling for higher concurrency | • 10 concurrent queries per cluster limit • Scales up to 40 clusters per warehouse (400 total concurrent queries) • Serverless SQL warehouses provide near-instant autoscaling • Pro/Classic warehouses take several minutes to provision new clusters • Real-world performance degradation typically occurs at 50-150 concurrent queries depending on complexity | • 5 concurrent queries per WLM queue by default (up to 8 queues) • Concurrency Scaling enables thousands of concurrent queries | A single engine can handle hundreds of concurrent queries. Engines auto-scale the number of clusters up and down base on resource usage thresholds. Idle engines scale down to zero billing. | **Redshift** is limited in scale because even with RA3, it cannot distribute different workloads across clusters. While it can scale to up to 10 clusters automatically to support query concurrency, it can only handle a maximum of 50 queued queries across all clusters by default. **Databricks** allow for autoscaling of clusters based on utilization. Additionally, increasing concurrency associated with a sql endpoint can be accomplished through the addition of clusters. Query concurrency per cluster is maxed at 10. However, scaling with additional clusters for concurrency is possible. Databricks provides a choice of instance types. **Firebolt** can handle the largest data volumes and concurrency on a single comparable cluster size, thanks to its superior hardware efficiency. Thanks to its decoupled storage & compute architecture it scales very well to large data volumes. However, resizing an engine size isn't instant and requires orchestration if avoiding downtime is necessary. A single Firebolt engine can support hundreds of concurrent queries, avoiding the need to scale out for most use cases. Scaling horizontally for even higher concurrency is manual. ## Performance [#performance] Performance is the biggest challenge with most data warehouses today. While decoupled storage and compute architectures improved scalability and simplified administration, for most data warehouses it introduced two bottlenecks; storage, and compute. Most modern cloud data warehouses fetch entire partitions over the network instead of just fetching the specific data needed for each query. While many invest in caching, most do not invest heavily in query optimization. Most vendors also have not improved continuous ingestion or semi-structured data analytics performance, both of which are needed for operational and customer-facing use cases. | Feature | Databricks | Redshift | Firebolt | | ------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Indexes | None | None | • Sparse primary indexes
• Aggregating indexes
• Join indexes
• Optimizer driven index usage | | Compute tuning | Choice of cluster type, node types including SSD-optimized instances. Serverless provides automatic resource allocation with Intelligent Workload Management (IWM) | Choice over number of nodes and their type | SQL defined engines. Control number of nodes, node family and type per cluster, with one or more clusters per engine. Multiple engines isolate workloads. | | Storage format | • Delta Lake format with Liquid Clustering (February 2025 – replaces Z-ordering and traditional partitioning) • Cannot use Liquid Clustering alongside Z-ordering on same table • Allows for sorted data in Delta Lake • Requires Optimize to maintain ordering | Columnar & compressed storage (RA3 nodes) | Columnar, sorted & compressed & sparsely indexed storage (F3 – Firebolt File Format) with native Apache Iceberg support | | Table-level partition & pruning techniques | • Table level partitioning • Liquid Clustering for improved query performance and reduced data skew (February 2025) • Z-ordering (legacy, replaced by Liquid Clustering) • Periodic optimization of storage required | • No table partitions • User-defined distribution & sort keys are used to optimize for speed | • User-defined table-level partitions are optional.
• Data is automatically sorted, compressed and indexed into F3 format.
• Pruning at indexed data-range level. | | Result cache | Multi-layered caching: local in-memory cache per cluster plus remote result cache (serverless only) that persists across all warehouses in workspace | Yes | Yes, results and sub-results cache with transactional spoiling. | | Warm cache (SSD) | Yes. Delta cache for data read by queries at file level granularity | Only with RA3 nodes at partition-level granularity | Yes, at indexed data-range level granularity | | Support for semi-structured data & JSON functions within SQL | Yes | Limited | Yes, including Lambda expressions and native nested array structures | | Vector Search and AI Capabilities | • MLflow integration and Databricks ML platform • Native vector search in Delta Lake (Vector Search) • AI and ML workloads optimized | Limited AI capabilities – primarily through integrations | • Native vector search capabilities and embeddings
• MCP Server for AI driven analytics
• Natural Language to SQL
• SQL based Inference | | Query Optimizations | • Photon engine (C++ vectorized engine providing 3-8x average speedups, maximum speedups over 10x) • Automated stats collection (January 2025) enables cost-based optimization • Predictive I/O for faster point lookups and data updates • Liquid Clustering (February 2025) • Intelligent Workload Management (IWM) with AI-powered resource allocation • Delta cache • Materialized views support | • Basic query optimizer • Materialized views • Result caching • ANALYZE for table statistics • Workload management (WLM) • Automated materialized views (AutoMV) • AI-driven scaling (Serverless preview) | • Primary indexes, aggregating indexes, join indexes, sparse indexes
• Sub-plan result caching
• F3 storage format optimization
• Automatic query optimizer with aggressive pruning
• Late column materialization
• Query analysis tools based on execution telemetry | **Redshift** does provide a result cache for accelerating repetitive query workloads and also has more tuning options than some others. But it does not deliver much faster compute performance than other cloud data warehouses in benchmarks. Sort keys can be used to optimize performance, but their contribution is limited. There is no support for indexes, and low-latency analytics at large data volumes is hard to achieve. Because Redshift decoupling of storage & compute is limited compared to other cloud data warehouses, it doesn't support isolating workloads, which means performance can degrade under pressure and competition for resources. **Databricks** is designed to leverage the Spark framework for processing large volumes of data. It leverages compressed Parquet files in a Delta Lake. To reduce the amount of data processed, it uses data pruning on partitions and Parquet file metadata. Databricks does not provide any indexes. **Firebolt** is the fastest when it comes to query performance when compared to cloud data warehouses and services like Athena. Its unique approach to storage and indexing results in highly aggressive data pruning that scans dramatically less data compared to other technologies. While other technologies scan partitions or micro-partitions, Firebolt works with indexed data ranges that are significantly smaller. In addition, Firebolt lets users accelerate queries further with multiple index types (Aggregating index, Join index), and using its decoupled storage & compute architecture workloads can be easily isolated to guarantee consistent performance. ## Use cases [#use-cases] There are a host of different analytics use cases that can be supported by a data warehouse. Look at your legacy technologies and their workloads, as well as the new possible use cases, and figure out which ones you will need to support in the next few years. | Feature | Databricks | Redshift | Firebolt | | ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Low-latency dashboards | • Sub-second to seconds load times at TB+ scale • Enhanced by Photon engine (3-8x average speedups) and Delta cache • Serverless SQL warehouses provide rapid startup (2-6 seconds) • Performance depends on cluster configuration | • Seconds to tens of seconds load times at 100s of GB scale • Can achieve faster performance with Concurrency Scaling and proper tuning | • 120ms query latency at 4000 QPS (FireScale benchmark 2025)
• Sub-second performance at TB+ scale with proper indexing
• Built for AI-driven analytics, dashboards, and real-time analytic applications | | Enterprise BI | • Strong for data science and ML workloads • Unified analytics platform approach • Growing traditional BI integrations • Serverless SQL warehouses improve accessibility • Delta sharing capabilities | • Mature and comprehensive Enterprise DW feature set • Extensive integrations with Enterprise BI ecosystem • Strong AWS ecosystem integration | • Growing ecosystem with focus on modern BI tools
• Strong SQL compliance with PostgreSQL
• Wire level compatibility drives expansion to PostgreSQL BI and ETL ecosystem | | Data Apps and AI Applications (Customer-facing low-latency high concurrency) | • 10 concurrent queries per cluster, scaling to 400 total concurrent queries per warehouse • Real-world performance degradation typically occurs at 50-150 concurrent queries depending on workload complexity • Serverless provides near-instant autoscaling • Photon engine delivers 3-8x performance improvements • Strong ML and AI platform integration | • 5 concurrent queries per WLM queue by default (up to 8 queues) • Concurrency Scaling enables thousands of concurrent queries • Seconds-level response times typical • Automatic scaling for burst workloads • Limited AI application support | • 120ms latency at 4000+ QPS proven performance at TB+ scale
• Supports hundreds to thousands of concurrent queries on single engine
• Price-performance leader (8x better than Snowflake, 18x vs Redshift) • Purpose-built for AI agents and data-intensive applications
• Native vector search and embeddings | | Ad hoc | • Excellent for ad-hoc with decoupled storage/compute • Serverless SQL warehouses provide instant provisioning • Intelligent Workload Management handles unpredictable workloads automatically • Strong for exploratory data analysis and ML workloads • Automated stats collection improves query planning | • Performance dependent on predefined distribution & sort keys • Elastic Resize enables adding compute resources • Typically subset of data loaded for ad-hoc analysis | • Excellent performance out-of-the-box with engine optimized for star and snowflake joins and aggregations
• Self learning query plan optimizer
• Full workload isolation prevents ad-hoc complexity from affecting real-time workloads
• Aggregating indexes are automatically used by optimizer | **Redshift** was originally designed to support traditional internal BI reporting and dashboard use cases for analysts. As such, it is typically used as a general-purpose Enterprise data warehouse. With deep integrations into the AWS ecosystem, it can also leverage AWS ML service, making it also useful for ML projects. However, given the coupling of storage & compute, and the difficulty in delivering low-latency analytics at scale, it is less suited for operational use cases and customer-facing use cases like Data Apps. The coupling of storage and compute, together with the need to predefine sort & dist keys for optimal performance, make it challenging to use for Ad-Hoc analytics. **Databricks** is a mature Spark based platform proven for processing streaming data. It is widely used for Machine Learning use cases by data scientists through the use of integrated notebooks. From a low latency query perspective, while it offers features like Delta Cache, it does not provide specialized indexes that can deliver low latency queries. **Firebolt** stands out by being the fastest cloud data warehouse when compared to Snowflake, Redshift, BigQuery and Athena. It's great for delivering sub-second analytics at scale, while remaining hardware efficient and high concurrency friendly. This makes it a great choice for operational use cases and customer-facing data apps. Given that it is not as feature-rich and integration rich as the more mature data warehouses makes it a lesser fit for a general-purpose Enterprise data warehouse. It is also not the best fit for ad-hoc use cases, because of the need to predefine indexing at the table level. # Redshift vs Druid (2025) (/comparison/redshift-vs-druid) ## Architecture [#architecture] The biggest difference among cloud data warehouses are whether they separate storage and compute, how much they isolate data and compute, and what clouds they can run on. | Feature | Redshift | Druid | | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Separation of storage and compute | RA3 instances enable separation of compute and storage, but limited workload isolation compared to other platforms | No | | Supported cloud infrastructure | AWS only | Can be installed anywhere | | Isolated tenancy – option for dedicated resources | • Isolated tenant & resources • Runs in your VPC | Single tenant | | Control vs abstraction of compute | • Configurable cluster size
• Configurable compute types | • Complex configuration of compute tier with multiple role-specific nodes
• Configurable node count
• Configurable compute types (virtual machines or kubernetes) | | Self-hosted and hybrid deployment options | Limited hybrid options with Redshift Serverless | Self-managed deployment required | | ACID Compliance and Transactions | ACID compliant at table level with some limitations on concurrent operations | Limited ACID support with eventual consistency | **Redshift** has the oldest architecture, being the first Cloud DW in the group. Its architecture wasn't designed to separate storage & compute. While it now has RA3 nodes which allow you to scale compute and only cache the data you need locally, all compute still operates together. You cannot separate and isolate different workloads over the same data, which puts it behind other decoupled storage/compute architectures. Redshift runs as an isolated tenant per customer, and unlike other cloud data warehouses, it is deployed in your VPC. Redshift offers a serverless option which is based on an abstracted unit called Redshift Processing Unit (RPU) ranging from 8 to 512 in increments of 8. Each RPU provides 2 vCPU and 16GB RAM. Thus, 8 RPU is equivalent to 16 vCPU / 128GB RAM. The minimum RPU is 8. **Druid** is an OLAP engine designed to provide fast real time analytics. Druid adopts a clustered architecture with servers that host various role specific processes. These processes address real time and batch ingestion, indexing, querying of historical and real time data. Apache Druid can be deployed as a virtual machine or a Kubernetes based cluster. Druid does not support a decoupled compute & storage architecture. Deep storage in the form of object storage is used to replicate data to. ## Scalability [#scalability] There are three big differences among data warehouses and query engines that limit scalability: decoupled storage and compute, dedicated resources, and continuous ingestion. | Feature | Redshift | Druid | | --------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------- | | Elasticity – Scaling for larger data volumes and faster queries | Available via Elastic Resize – slow and limited, downtime required | Scale-up of nodes requires careful planning and downtime. Addition of new nodes for scale-out is possible | | Elasticity – Scaling for higher concurrency | • 5 concurrent queries per WLM queue by default (up to 8 queues) • Concurrency Scaling enables thousands of concurrent queries | Supports 100s to 100,000s queries per second (1000+ QPS) with proper configuration and scaling | **Redshift** is limited in scale because even with RA3, it cannot distribute different workloads across clusters. While it can scale to up to 10 clusters automatically to support query concurrency, it can only handle a maximum of 50 queued queries across all clusters by default. **Druid** provides the ability to handle fast ingest and high concurrency. Custom sizing and cluster tuning are required to balance the compute, memory, storage needs of each process within Druid and to provide high concurrency. Druid clusters can be grown by adding nodes with automatic rebalancing of storage segments assigned to nodes. Self hosted Druid on Kubernetes is an option that users leverage to simplify scaling. Additionally, Cloud based managed Druid offerings are being rolled out. However, these managed offerings are limited in scale and scaling is not granular. ## Performance [#performance] Performance is the biggest challenge with most data warehouses today. While decoupled storage and compute architectures improved scalability and simplified administration, for most data warehouses it introduced two bottlenecks; storage, and compute. Most modern cloud data warehouses fetch entire partitions over the network instead of just fetching the specific data needed for each query. While many invest in caching, most do not invest heavily in query optimization. Most vendors also have not improved continuous ingestion or semi-structured data analytics performance, both of which are needed for operational and customer-facing use cases. | Feature | Redshift | Druid | | ------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | | Indexes | None | Compressed bitmap indexes for data access and roll-ups to manage aggregations | | Compute tuning | Choice over number of nodes and their type | On-premises, self-managed hardware. Druid requires infrastructure management and leverages commonly available instance types | | Storage format | Columnar & compressed storage (RA3 nodes) | Columnar storage format with time-based sorting | | Table-level partition & pruning techniques | • No table partitions • User-defined distribution & sort keys are used to optimize for speed | Restrictive time-based partitioning. Can partition based on other secondary columns | | Result cache | Yes | Ability to support caching on broker (set to off by default) | | Warm cache (SSD) | Only with RA3 nodes at partition-level granularity | Yes, at much larger segment level granularity | | Support for semi-structured data & JSON functions within SQL | Limited | Recommend flattening JSON or translate to array prior to loading. No support for JSON parsing at query runtime | | Vector Search and AI Capabilities | Limited AI capabilities – primarily through integrations | No native AI or vector search capabilities | | Query Optimizations | • Basic query optimizer • Materialized views • Result caching • ANALYZE for table statistics • Workload management (WLM) • Automated materialized views (AutoMV) • AI-driven scaling (Serverless preview) | • Compressed bitmap indexes • Roll-up aggregations • Time-based optimization • Query optimization requires manual tuning | **Redshift** does provide a result cache for accelerating repetitive query workloads and also has more tuning options than some others. But it does not deliver much faster compute performance than other cloud data warehouses in benchmarks. Sort keys can be used to optimize performance, but their contribution is limited. There is no support for indexes, and low-latency analytics at large data volumes is hard to achieve. Because Redshift decoupling of storage & compute is limited compared to other cloud data warehouses, it doesn't support isolating workloads, which means performance can degrade under pressure and competition for resources. **Druid** provides high performance through columnar storage format, parallel processing, bitmap indexes and roll-ups. Druid, however, recommends a denormalized data model for performance needs. Join operations in Druid are a relatively new feature with various limitations, especially if there is a need to join large datasets. ## Use cases [#use-cases] There are a host of different analytics use cases that can be supported by a data warehouse. Look at your legacy technologies and their workloads, as well as the new possible use cases, and figure out which ones you will need to support in the next few years. | Feature | Redshift | Druid | | ---------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Low-latency dashboards | • Seconds to tens of seconds load times at 100s of GB scale • Can achieve faster performance with Concurrency Scaling and proper tuning | • Sub-second load times optimized for time-series and real-time analytics • Built for high-concurrency interactive dashboards • Requires denormalized data model | | Enterprise BI | • Mature and comprehensive Enterprise DW feature set • Extensive integrations with Enterprise BI ecosystem • Strong AWS ecosystem integration | • Limited integrations with traditional Enterprise BI tools • Strong for real-time operational dashboards • Requires specialized visualization tools | | Data Apps and AI Applications (Customer-facing low-latency high concurrency) | • 5 concurrent queries per WLM queue by default (up to 8 queues) • Concurrency Scaling enables thousands of concurrent queries • Seconds-level response times typical • Automatic scaling for burst workloads • Limited AI application support | • Built for high concurrency (1000+ QPS) with distributed architecture • Sub-second response times for time-series data • Optimized for real-time operational applications • No AI capabilities | | Ad hoc | • Performance dependent on predefined distribution & sort keys • Elastic Resize enables adding compute resources • Typically subset of data loaded for ad-hoc analysis | • Not optimized for ad-hoc queries • Requires predefined roll-ups and data modeling • Limited flexibility for exploratory analysis | **Redshift** was originally designed to support traditional internal BI reporting and dashboard use cases for analysts. As such, it is typically used as a general-purpose Enterprise data warehouse. With deep integrations into the AWS ecosystem, it can also leverage AWS ML service, making it also useful for ML projects. However, given the coupling of storage & compute, and the difficulty in delivering low-latency analytics at scale, it is less suited for operational use cases and customer-facing use cases like Data Apps. The coupling of storage and compute, together with the need to predefine sort & dist keys for optimal performance, make it challenging to use for Ad-Hoc analytics. **Druid** is designed as an OLAP engine to provide fast access to aggregations that are run against large volumes of data. Druid is typically used for customer facing analytics and streaming data processing. Druid is used as an add-on with other data warehousing products that are efficient at scaling, joining, and filtering large volumes of data. It is not a suitable option for data warehouse replacement. # Snowflake vs Athena vs Firebolt (2025) (/comparison/snowflake-vs-athena) ## Architecture [#architecture] The biggest difference among cloud data warehouses are whether they separate storage and compute, how much they isolate data and compute, and what clouds they can run on. | Feature | Snowflake | Athena | Firebolt | | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Separation of storage and compute | Yes | Yes, serverless with optional provisioned capacity. Workloads can be isolated through Workgroups and Capacity Reservations | Yes, separation of storage and metadata as well as compute from compute with full workload isolation. | | Supported cloud infrastructure | AWS, Azure, GCP with full feature parity across all three major clouds | AWS only | AWS (GCP coming soon) & anywhere (Firebolt Core) | | Isolated tenancy – option for dedicated resources | • Multi-tenant pooled resources
• Isolated tenancy available via VPS tier | • Multi-tenant pooled resources by default
• Dedicated compute resources available via Provisioned Capacity
• VPC endpoint connections supported | • Multi-tenant metadata layer
• Isolated tenancy for compute & storage per client | | Control vs abstraction of compute | • Configurable warehouse sizes (XS to 6XL)
• Multi-cluster warehouses with auto-scaling
• Choice between Generation 1 and Generation 2 standard warehouses
• MAX\_CONCURRENCY\_LEVEL parameter for resource allocation | • Serverless by default with no infrastructure control
• Optional Provisioned Capacity allows dedicated DPU allocation (minimum 24 DPUs)
• Two pricing models: on-demand ($5/TB scanned) or provisioned ($0.30/DPU-hour) | Uses engine abstraction:
• Each engine has configurable cluster size (1-128 nodes) for horizontal scaling.
• Configurable compute family (compute vs storage optimized) and type (XS, S, M, L, XL) for vertical scaling
• Number of clusters for concurrency (auto)scaling.
Provides full workload isolation across engines. | | Self-hosted and hybrid deployment options | Snowflake for Government Cloud and private cloud options available | No self-hosted options – serverless only | • Firebolt Core: Forever free, self-hosted edition with full query engine capabilities
• Same performance and features as managed service
• Deploy anywhere: local laptop, cloud, datacenter, Kubernetes
• Production-grade distributed architecture
• No usage restrictions except building competing SaaS | | ACID Compliance and Transactions | Full ACID compliance with Time Travel and zero-copy cloning capabilities | No ACID compliance – eventual consistency model | • Full ACID compliance with snapshot isolation
• Multi-statement transactions supported
• Strong consistency across all operations
• Supports concurrent reads and writes
• Transactional integrity for data applications | **Snowflake** was one of the first decoupled storage and compute architectures, making it the first to have nearly unlimited compute scale and workload isolation, and horizontal user scalability. It runs on AWS, Azure and GCP. It is multi-tenant over shared resources in nature and requires you to move data out of your VPC and into the Snowflake cloud. "Virtual Private Snowflake" (VPS) is its highest-priced tier, and can run a dedicated isolated version of Snowflake. Its virtual warehouses can be T-shirt sized along an XS/S/M…/4XL axis, where each discrete T-shirt size is bundled with fixed HW properties that are abstracted from the users. Snowflake has recently added support for Snowflake managed Iceberg tables. **Athena** is serverless and built on a decoupled storage and compute architecture that queries data directly in S3, without the need to ingest/copy the data. It runs in multi-tenancy with shared resources. Users do not have control over the compute resources Athena chooses to allocate per query from the shared resource pool. For folks requiring additional or dedicated resources, they can reserve dedicated processing capacity in the form of Data Processing Units (DPU), with each DPU providing 4 vCPU and 16 GB RAM. RPU allocation ranges from 24 - 1000 per region. **Firebolt** is built on a natively decoupled storage & compute architecture, on AWS only. Data has to be copied outside of your VPC into the Firebolt, where both your compute and data run in a dedicated and isolated tenant. A "Firebolt Engine" can be granularly configured across # of nodes and different CPU/RAM/SSD combinations. ## Scalability [#scalability] There are three big differences among data warehouses and query engines that limit scalability: decoupled storage and compute, dedicated resources, and continuous ingestion. | Feature | Snowflake | Athena | Firebolt | | --------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Elasticity – Scaling for larger data volumes and faster queries | • Instant warehouse resize (XS to 6XL) with no downtime • Multi-cluster auto-scaling • Generation 2 warehouses provide \~2x performance improvement over Generation 1 | • Fully abstracted on-demand scaling • Provisioned Capacity allows manual scaling of DPUs for predictable performance • Capacity reservations can be adjusted with minimum 1-hour billing periods | Granular cluster resize with node types, number of nodes and number of clusters. Zero downtime. | | Elasticity – Scaling for higher concurrency | • Single warehouse supports many concurrent queries (MAX\_CONCURRENCY\_LEVEL=8 controls resource allocation per query, not query limit) • Multi-cluster warehouses enable thousands of concurrent queries with auto-scaling • Unlimited virtual warehouses can be created | • Default limit of 25 concurrent DML queries and 20 DDL queries (adjustable via service quotas) • Provisioned Capacity enables higher concurrency with dedicated DPUs • Query queuing available when capacity is exceeded | A single engine can handle hundreds of concurrent queries. Engines auto-scale the number of clusters up and down base on resource usage thresholds. Idle engines scale down to zero billing. | **Snowflake** scales very well both for data volumes and query concurrency. The decoupled storage/compute architecture supports resizing clusters without downtime, and in addition, supports auto-scaling horizontally for higher query concurrency during peak hours. **Athena** is a shared multi-tenant resource, with no guarantees on the amount or availability of the resources allocated for your queries. From a data volume perspective, it can scale to large volumes, but large data volumes can suffer from very long run times and frequent time outs. Query concurrency is maxed at 20. If scalability is a top priority, Athena is probably not the best choice. **Firebolt** can handle the largest data volumes and concurrency on a single comparable cluster size, thanks to its superior hardware efficiency. Thanks to its decoupled storage & compute architecture it scales very well to large data volumes. However, resizing an engine size isn't instant and requires orchestration if avoiding downtime is necessary. A single Firebolt engine can support hundreds of concurrent queries, avoiding the need to scale out for most use cases. Scaling horizontally for even higher concurrency is manual. ## Performance [#performance] Performance is the biggest challenge with most data warehouses today. While decoupled storage and compute architectures improved scalability and simplified administration, for most data warehouses it introduced two bottlenecks; storage, and compute. Most modern cloud data warehouses fetch entire partitions over the network instead of just fetching the specific data needed for each query. While many invest in caching, most do not invest heavily in query optimization. Most vendors also have not improved continuous ingestion or semi-structured data analytics performance, both of which are needed for operational and customer-facing use cases. | Feature | Snowflake | Athena | Firebolt | | ------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Indexes | • Search Optimization Service for point lookups and selective queries (additional cost) • Clustering keys for data organization and automatic clustering • Materialized views • Snowflake Optima automatic indexing on Generation 2 warehouses (no additional cost) • No traditional database indexes | No traditional indexes – relies on partition pruning and data organization in S3. Uses columnar formats and compression for optimization | • Sparse primary indexes
• Aggregating indexes
• Join indexes
• Optimizer driven index usage | | Compute tuning | • Warehouse T-shirt sizing (XS to 6XL) • Multi-cluster configuration and scaling policies • Generation 1 vs Generation 2 warehouse selection • MAX\_CONCURRENCY\_LEVEL parameter tuning • Query Acceleration Service for long-running queries | • No compute tuning in on-demand mode • Provisioned Capacity allows DPU allocation control (4 vCPU and 16GB RAM per DPU) • Minimum 24 DPUs with scaling in 4-DPU increments | SQL defined engines. Control number of nodes, node family and type per cluster, with one or more clusters per engine. Multiple engines isolate workloads. | | Storage format | Columnar micro-partitioned & compressed storage | Supports multiple formats: Parquet, ORC, Avro, JSON, CSV, TSV on S3. Native support for open table formats including Apache Iceberg, Apache Hudi, and Delta Lake | Columnar, sorted & compressed & sparsely indexed storage (F3 – Firebolt File Format) with native Apache Iceberg support | | Table-level partition & pruning techniques | • Data automatically divided into micro-partitions • Automatic pruning at micro-partition level • Clustering keys for data organization with automatic clustering • Snowflake Optima provides additional automatic pruning optimization on Gen2 warehouses | • User-defined table-level partitions with Hive-style partitioning • Pruning at partition level • Partition projection for advanced performance optimization • Supports open table formats with built-in partitioning | • User-defined table-level partitions are optional.
• Data is automatically sorted, compressed and indexed into F3 format.
• Pruning at indexed data-range level. | | Result cache | Yes | Query result caching for up to 30 days with configurable retention. Results reuse supported across workgroups | Yes, results and sub-results cache with transactional spoiling. | | Warm cache (SSD) | Yes, at micro-partition level granularity | No local caching – queries data directly from S3. Relies on S3's performance characteristics and intelligent tiering | Yes, at indexed data-range level granularity | | Support for semi-structured data & JSON functions within SQL | Yes | Yes, comprehensive JSON support including Lambda expressions, array functions, and native nested data handling | Yes, including Lambda expressions and native nested array structures | | Vector Search and AI Capabilities | AI integration through Cortex AI and Snowpark ML | No native AI or vector search capabilities | • Native vector search capabilities and embeddings
• MCP Server for AI driven analytics
• Natural Language to SQL
• SQL based Inference | | Query Optimizations | • Search Optimization Service for point lookups (additional cost)
• Query Acceleration Service (QAS) for long-running and unpredictable workloads • Snowflake Optima automatic optimization on Generation 2 warehouses (no additional cost) • Automatic clustering with background maintenance • Materialized views with automatic refresh • Result cache (24hrs) • Cost-based optimization with dynamic query rewriting | • Cost-based optimizer (CBO) in Athena engine v3 • Query result caching (up to 30 days) • Partition projection for advanced optimization • CTAS for precomputed queries • Join reordering and aggregation pushdown • Automatic parallel query execution • Support for columnar formats (Parquet, ORC) • Integration with AWS Glue Data Catalog | • Primary indexes, aggregating indexes, join indexes, sparse indexes
• Sub-plan result caching
• F3 storage format optimization
• Automatic query optimizer with aggressive pruning
• Late column materialization
• Query analysis tools based on execution telemetry | **Snowflake** typically comes on top for most queries when it comes to performance in public TPC-based benchmarks when compared to BigQuery and Redshift, but only marginally. Its micro partition storage approach effectively scans less data compared to larger partitions. The ability to isolate workloads over the decoupled storage & compute architecture lets you avoid competition for resources compared to multi-tenant shared resource solutions, and the ability to increase warehouse sizes can often enhance performance (for a higher price), but not always linearly. Snowflake's recently released "Search optimization service" delivers index-like behavior for point queries, but comes at an additional cost. **Athena**, (and Presto) are designed to query data where it is, sacrificing storage-compute optimizations. This makes it very convenient for easy and immediate querying but at the expense of performance. This typically puts Athena behind cloud data warehouses in terms of performance. But Athena still does relatively well in performance benchmarks, especially when external storage is managed by experts. While it supports partitions, there is no support for indexing, and together with the fact that resources are pooled from a shared multi-tenant service, low-latency and consistent performance are not Athena's sweet spot. A cloud data warehouse be more performant better than Athena in most cases. **Firebolt** is the fastest when it comes to query performance when compared to cloud data warehouses and services like Athena. Its unique approach to storage and indexing results in highly aggressive data pruning that scans dramatically less data compared to other technologies. While other technologies scan partitions or micro-partitions, Firebolt works with indexed data ranges that are significantly smaller. In addition, Firebolt lets users accelerate queries further with multiple index types (Aggregating index, Join index), and using its decoupled storage & compute architecture workloads can be easily isolated to guarantee consistent performance. ## Use cases [#use-cases] There are a host of different analytics use cases that can be supported by a data warehouse. Look at your legacy technologies and their workloads, as well as the new possible use cases, and figure out which ones you will need to support in the next few years. | Feature | Snowflake | Athena | Firebolt | | ---------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Low-latency dashboards | • Sub-second to seconds response times at TB+ scale with proper clustering and optimization • Enhanced by Query Acceleration Service and Search Optimization Service • Generation 2 warehouses provide \~2x performance improvement over Generation 1 • Snowflake Optima provides automatic optimization | • Seconds to minutes response times for interactive dashboards • Performance varies based on data partitioning, file formats, and query optimization • Provisioned Capacity can improve consistency for dashboard workloads • Best suited for analytical dashboards rather than sub-second operational dashboards | • 120ms query latency at 4000 QPS (FireScale benchmark 2025)
• Sub-second performance at TB+ scale with proper indexing
• Built for AI-driven analytics, dashboards, and real-time analytic applications | | Enterprise BI | • Mature and comprehensive Enterprise DW feature set • Extensive integrations with Enterprise BI ecosystem • Multi-cloud deployment options with consistent experience • Strong SQL compliance and wide ecosystem support • Zero-copy data sharing capabilities | • Good integration with AWS ecosystem BI tools (QuickSight, etc.) • Standard SQL compatibility enables most BI tool connections • Cost-effective for variable workloads and ad-hoc analytics • JDBC/ODBC drivers support enterprise BI tools • Limited advanced BI features compared to dedicated data warehouses | • Growing ecosystem with focus on modern BI tools
• Strong SQL compliance with PostgreSQL
• Wire level compatibility drives expansion to PostgreSQL BI and ETL ecosystem | | Data Apps and AI Applications (Customer-facing low-latency high concurrency) | • Multi-cluster warehouses support thousands of concurrent users with auto-scaling • Individual warehouses support many concurrent queries (not limited to 8 concurrent queries) • Sub-second to seconds response times with proper optimization • Generation 2 warehouses provide significant performance improvements for high-concurrency workloads • AI integration through Cortex AI | • Default concurrency limits (25 DML/20 DDL queries) may require service quota increases • Provisioned Capacity enables higher concurrency with dedicated resources • Seconds-level response times typical • Cost-effective for customer-facing analytics with proper optimization • Best suited for analytical rather than operational workloads • No native AI capabilities | • 120ms latency at 4000+ QPS proven performance at TB+ scale
• Supports hundreds to thousands of concurrent queries on single engine
• Price-performance leader (8x better than Snowflake, 18x vs Redshift) • Purpose-built for AI agents and data-intensive applications
• Native vector search and embeddings | | Ad hoc | • Excellent for ad-hoc with decoupled storage/compute • Auto-scaling and instant compute provisioning • Minimal predefined optimization required • Query Acceleration Service handles unpredictable workloads automatically • Snowflake Optima provides automatic optimization for recurring patterns | • Purpose-built for ad-hoc analytics on data lakes • Serverless with zero infrastructure management • Direct querying of S3 data without ETL • Cost-effective pay-per-query model ideal for exploratory analysis • Strong support for multiple data formats and federated queries • Apache Spark integration for advanced analytics | • Excellent performance out-of-the-box with engine optimized for star and snowflake joins and aggregations
• Self learning query plan optimizer
• Full workload isolation prevents ad-hoc complexity from affecting real-time workloads
• Aggregating indexes are automatically used by optimizer | **Snowflake** is a well rounded general purpose cloud data warehouse, that can also span beyond traditional BI & Analytics use cases into Ad-Hoc and ML use cases. Thanks to the flexible decoupeld storage & compute architecture that allows you to isolate and control the amount of compute per workload, it's possible to tackle a broad spectrum of workloads. However, like its close siblings Redshift & BigQuery, it struggles to deliver low-latency query performance at scale, making it a lesser fit for operational use cases and customer-facing data apps. **Athena** is a great choice for Ad-Hoc analytics. You can keep the data where it is, and start querying without worrying about hardware or pretty much anything else, given that Athena is serverless and takes care of everything behind the scenes. However, it is not a great fit when you need consistent and fast query performance, and/or high concurrency. This is why it is typically not the best choice for operational and customer-facing applications. It can be also easily and flexibly used for batch processing, which is often leveraged for ML use cases. **Firebolt** stands out by being the fastest cloud data warehouse when compared to Snowflake, Redshift, BigQuery and Athena. It's great for delivering sub-second analytics at scale, while remaining hardware efficient and high concurrency friendly. This makes it a great choice for operational use cases and customer-facing data apps. Given that it is not as feature-rich and integration rich as the more mature data warehouses makes it a lesser fit for a general-purpose Enterprise data warehouse. It is also not the best fit for ad-hoc use cases, because of the need to predefine indexing at the table level. # Snowflake vs BigQuery (/comparison/snowflake-vs-bigquery) ## Architecture [#architecture] The biggest difference among cloud data warehouses are whether they separate storage and compute, how much they isolate data and compute, and what clouds they can run on. | Feature | Snowflake | BigQuery | Firebolt | | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Separation of storage and compute | Yes | Fully serverless with complete separation of compute (Dremel) and storage (Colossus), powered by Jupiter network and Borg orchestration | Yes, separation of storage and metadata as well as compute from compute with full workload isolation. | | Supported cloud infrastructure | AWS, Azure, GCP with full feature parity across all three major clouds | Google Cloud only | AWS (GCP coming soon) & anywhere (Firebolt Core) | | Isolated tenancy – option for dedicated resources | • Multi-tenant pooled resources
• Isolated tenancy available via VPS tier | • Multi-tenant pooled resources
• VPC Service Controls provide enhanced security and connectivity isolation to customer VPCs
• Cross-region disaster recovery for enterprise workloads | • Multi-tenant metadata layer
• Isolated tenancy for compute & storage per client | | Control vs abstraction of compute | • Configurable warehouse sizes (XS to 6XL)
• Multi-cluster warehouses with auto-scaling
• Choice between Generation 1 and Generation 2 standard warehouses
• MAX\_CONCURRENCY\_LEVEL parameter for resource allocation | Fully serverless with no control over compute resources – BigQuery automatically allocates computing resources as needed with intelligent workload management and dynamic slot allocation | Uses engine abstraction:
• Each engine has configurable cluster size (1-128 nodes) for horizontal scaling.
• Configurable compute family (compute vs storage optimized) and type (XS, S, M, L, XL) for vertical scaling
• Number of clusters for concurrency (auto)scaling.
Provides full workload isolation across engines. | | Self-hosted and hybrid deployment options | Snowflake for Government Cloud and private cloud options available | No self-hosted options – fully managed service only | • Firebolt Core: Forever free, self-hosted edition with full query engine capabilities
• Same performance and features as managed service
• Deploy anywhere: local laptop, cloud, datacenter, Kubernetes
• Production-grade distributed architecture
• No usage restrictions except building competing SaaS | | ACID Compliance and Transactions | Full ACID compliance with Time Travel and zero-copy cloning capabilities | Limited ACID support – eventual consistency model with some transactional capabilities | • Full ACID compliance with snapshot isolation
• Multi-statement transactions supported
• Strong consistency across all operations
• Supports concurrent reads and writes
• Transactional integrity for data applications | **Snowflake** was one of the first decoupled storage and compute architectures, making it the first to have nearly unlimited compute scale and workload isolation, and horizontal user scalability. It runs on AWS, Azure and GCP. It is multi-tenant over shared resources in nature and requires you to move data out of your VPC and into the Snowflake cloud. "Virtual Private Snowflake" (VPS) is its highest-priced tier, and can run a dedicated isolated version of Snowflake. Its virtual warehouses can be T-shirt sized along an XS/S/M…/4XL axis, where each discrete T-shirt size is bundled with fixed HW properties that are abstracted from the users. Snowflake has recently added support for Snowflake managed Iceberg tables. **BigQuery** was one of the first decoupled storage and compute architectures. It is a unique piece of engineering and not a typical data warehouse in part because it started as an on-demand serverless query engine. It runs in multi-tenancy with shared resources, allocated as "slots" which represent a virtual CPU that executes SQL. BigQuery determines how many slots a query requires, without the ability of the user to control it. BigQuery can be priced on a $/TB scanned basis or through slot reservations. A slot in BigQuery is logically equivalent to 0.5 vCPU and 0.5GB of RAM. There are multiple models to allocate slots in BigQuery. **Firebolt** is built on a natively decoupled storage & compute architecture, on AWS only. Data has to be copied outside of your VPC into the Firebolt, where both your compute and data run in a dedicated and isolated tenant. A "Firebolt Engine" can be granularly configured across # of nodes and different CPU/RAM/SSD combinations. ## Scalability [#scalability] There are three big differences among data warehouses and query engines that limit scalability: decoupled storage and compute, dedicated resources, and continuous ingestion. | Feature | Snowflake | BigQuery | Firebolt | | --------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Elasticity – Scaling for larger data volumes and faster queries | • Instant warehouse resize (XS to 6XL) with no downtime • Multi-cluster auto-scaling • Generation 2 warehouses provide \~2x performance improvement over Generation 1 | Fully automated serverless scaling – BigQuery automatically determines resource allocation and scales to petabytes without user intervention. Can dynamically burst beyond baseline slot allocations for performance optimization | Granular cluster resize with node types, number of nodes and number of clusters. Zero downtime. | | Elasticity – Scaling for higher concurrency | • Single warehouse supports many concurrent queries (MAX\_CONCURRENCY\_LEVEL=8 controls resource allocation per query, not query limit) • Multi-cluster warehouses enable thousands of concurrent queries with auto-scaling • Unlimited virtual warehouses can be created | Dynamic concurrency management with query queueing supporting up to 1,000 interactive queries and 20,000 batch queries per project per region. Automatic fair scheduling and slot distribution across workloads | A single engine can handle hundreds of concurrent queries. Engines auto-scale the number of clusters up and down base on resource usage thresholds. Idle engines scale down to zero billing. | **Snowflake** scales very well both for data volumes and query concurrency. The decoupled storage/compute architecture supports resizing clusters without downtime, and in addition, supports auto-scaling horizontally for higher query concurrency during peak hours. **BigQuery** scales very well to large data volumes, and automatically assigns more compute resources when needed behind the scenes, in the form of "slots". BigQuery works either in an "on-demand pricing model", where slot assignment is completely in the hands of BigQuery and the state of the shared resource pool, or in "flat-rate pricing model" where slots are reserved in advanced. With reserved slots there is more control over compute resources, thus making scaling more predictable. Concurrency is limited to 100 users by default. **Firebolt** can handle the largest data volumes and concurrency on a single comparable cluster size, thanks to its superior hardware efficiency. Thanks to its decoupled storage & compute architecture it scales very well to large data volumes. However, resizing an engine size isn't instant and requires orchestration if avoiding downtime is necessary. A single Firebolt engine can support hundreds of concurrent queries, avoiding the need to scale out for most use cases. Scaling horizontally for even higher concurrency is manual. ## Performance [#performance] Performance is the biggest challenge with most data warehouses today. While decoupled storage and compute architectures improved scalability and simplified administration, for most data warehouses it introduced two bottlenecks; storage, and compute. Most modern cloud data warehouses fetch entire partitions over the network instead of just fetching the specific data needed for each query. While many invest in caching, most do not invest heavily in query optimization. Most vendors also have not improved continuous ingestion or semi-structured data analytics performance, both of which are needed for operational and customer-facing use cases. | Feature | Snowflake | BigQuery | Firebolt | | ------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Indexes | • Search Optimization Service for point lookups and selective queries (additional cost) • Clustering keys for data organization and automatic clustering • Materialized views • Snowflake Optima automatic indexing on Generation 2 warehouses (no additional cost) • No traditional database indexes | Search indexes (GA) for efficient text search optimization on STRING, JSON, and array columns. Support for LOG\_ANALYZER, NO\_OP\_ANALYZER, and PATTERN\_ANALYZER with column-level granularity for improved query performance and cost efficiency | • Sparse primary indexes
• Aggregating indexes
• Join indexes
• Optimizer driven index usage | | Compute tuning | • Warehouse T-shirt sizing (XS to 6XL) • Multi-cluster configuration and scaling policies • Generation 1 vs Generation 2 warehouse selection • MAX\_CONCURRENCY\_LEVEL parameter tuning • Query Acceleration Service for long-running queries | Serverless architecture with automatic resource optimization – no manual tuning required. Intelligent workload management with AI-powered resource allocation and dynamic slot distribution | SQL defined engines. Control number of nodes, node family and type per cluster, with one or more clusters per engine. Multiple engines isolate workloads. | | Storage format | Columnar micro-partitioned & compressed storage | Columnar & compressed storage (Capacitor format) with support for open table formats including Apache Iceberg, Delta Lake, and Hudi. Intelligent tiering with automatic long-term storage cost reduction after 90 days | Columnar, sorted & compressed & sparsely indexed storage (F3 – Firebolt File Format) with native Apache Iceberg support | | Table-level partition & pruning techniques | • Data automatically divided into micro-partitions • Automatic pruning at micro-partition level • Clustering keys for data organization with automatic clustering • Snowflake Optima provides additional automatic pruning optimization on Gen2 warehouses | • Automatic table organization with intelligent micro-partitioning • Clustering keys for data organization • Automatic partition pruning optimization • Supports time-based and custom partitioning strategies | • User-defined table-level partitions are optional.
• Data is automatically sorted, compressed and indexed into F3 format.
• Pruning at indexed data-range level. | | Result cache | Yes | Yes, with cross-user result caching and intelligent cache management for up to 24 hours | Yes, results and sub-results cache with transactional spoiling. | | Warm cache (SSD) | Yes, at micro-partition level granularity | BI Engine provides in-memory caching and acceleration for frequently accessed data and dashboards | Yes, at indexed data-range level granularity | | Support for semi-structured data & JSON functions within SQL | Yes | Yes, including advanced JSON functions, Lambda expressions, and native support for nested and repeated fields | Yes, including Lambda expressions and native nested array structures | | Vector Search and AI Capabilities | AI integration through Cortex AI and Snowpark ML | • BigQuery ML for in-database machine learning • Vertex AI integration • Natural language querying with Gemini AI • Limited vector search capabilities | • Native vector search capabilities and embeddings
• MCP Server for AI driven analytics
• Natural Language to SQL
• SQL based Inference | | Query Optimizations | • Search Optimization Service for point lookups (additional cost)
• Query Acceleration Service (QAS) for long-running and unpredictable workloads • Snowflake Optima automatic optimization on Generation 2 warehouses (no additional cost) • Automatic clustering with background maintenance • Materialized views with automatic refresh • Result cache (24hrs) • Cost-based optimization with dynamic query rewriting | • Advanced query optimizer with Dremel engine • Search indexes with column-level granularity • Materialized views with smart refresh and automatic query rewriting • BI Engine in-memory acceleration • Gemini AI-powered query optimization and natural language querying • Cross-user result caching • Automatic partitioning and clustering optimization • Cost-based optimization with intelligent workload management | • Primary indexes, aggregating indexes, join indexes, sparse indexes
• Sub-plan result caching
• F3 storage format optimization
• Automatic query optimizer with aggressive pruning
• Late column materialization
• Query analysis tools based on execution telemetry | **Snowflake** typically comes on top for most queries when it comes to performance in public TPC-based benchmarks when compared to BigQuery and Redshift, but only marginally. Its micro partition storage approach effectively scans less data compared to larger partitions. The ability to isolate workloads over the decoupled storage & compute architecture lets you avoid competition for resources compared to multi-tenant shared resource solutions, and the ability to increase warehouse sizes can often enhance performance (for a higher price), but not always linearly. Snowflake's recently released "Search optimization service" delivers index-like behavior for point queries, but comes at an additional cost. **BigQuery** lines up in benchmarks in the same ballpark as other cloud data warehouses but does come in consistently last in most queries. Beyond implementing according to best practices, there is little you can do to accelerate BigQuery performance, as it determines the amount of resources (slots) the query needs for you. BigQuery can be used together with "BigQuery BI Engine" for lower latency analytics. However, BI Engine is limited in terms of scale because it runs in memory. Its maximum capacity is 100GB. **Firebolt** is the fastest when it comes to query performance when compared to cloud data warehouses and services like Athena. Its unique approach to storage and indexing results in highly aggressive data pruning that scans dramatically less data compared to other technologies. While other technologies scan partitions or micro-partitions, Firebolt works with indexed data ranges that are significantly smaller. In addition, Firebolt lets users accelerate queries further with multiple index types (Aggregating index, Join index), and using its decoupled storage & compute architecture workloads can be easily isolated to guarantee consistent performance. ## Use cases [#use-cases] There are a host of different analytics use cases that can be supported by a data warehouse. Look at your legacy technologies and their workloads, as well as the new possible use cases, and figure out which ones you will need to support in the next few years. | Feature | Snowflake | BigQuery | Firebolt | | ---------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Low-latency dashboards | • Sub-second to seconds response times at TB+ scale with proper clustering and optimization • Enhanced by Query Acceleration Service and Search Optimization Service • Generation 2 warehouses provide \~2x performance improvement over Generation 1 • Snowflake Optima provides automatic optimization | • Sub-second to seconds response times at TB+ scale with BI Engine acceleration • Search indexes and materialized views provide significant performance improvements for dashboard queries • Intelligent caching reduces query costs for repeated dashboard access • Dynamic concurrency management supports high user loads | • 120ms query latency at 4000 QPS (FireScale benchmark 2025)
• Sub-second performance at TB+ scale with proper indexing
• Built for AI-driven analytics, dashboards, and real-time analytic applications | | Enterprise BI | • Mature and comprehensive Enterprise DW feature set • Extensive integrations with Enterprise BI ecosystem • Multi-cloud deployment options with consistent experience • Strong SQL compliance and wide ecosystem support • Zero-copy data sharing capabilities | • Mature and comprehensive Enterprise DW feature set with native Google Cloud ecosystem integration • Strong integration with Looker, Looker Studio, and major BI tools • Gemini AI integration for natural language insights • Cross-cloud analytics capabilities • Advanced governance with Dataplex Universal Catalog | • Growing ecosystem with focus on modern BI tools
• Strong SQL compliance with PostgreSQL
• Wire level compatibility drives expansion to PostgreSQL BI and ETL ecosystem | | Data Apps and AI Applications (Customer-facing low-latency high concurrency) | • Multi-cluster warehouses support thousands of concurrent users with auto-scaling • Individual warehouses support many concurrent queries (not limited to 8 concurrent queries) • Sub-second to seconds response times with proper optimization • Generation 2 warehouses provide significant performance improvements for high-concurrency workloads • AI integration through Cortex AI | • Dynamic concurrency supporting 1,000+ interactive queries with intelligent queuing and fair scheduling • Sub-second response times with BI Engine acceleration and search indexes • Serverless architecture eliminates infrastructure management overhead • Advanced caching and materialized views optimize repeated queries • AI-powered optimization for customer-facing applications • BigQuery ML for AI workloads | • 120ms latency at 4000+ QPS proven performance at TB+ scale
• Supports hundreds to thousands of concurrent queries on single engine
• Price-performance leader (8x better than Snowflake, 18x vs Redshift) • Purpose-built for AI agents and data-intensive applications
• Native vector search and embeddings | | Ad hoc | • Excellent for ad-hoc with decoupled storage/compute • Auto-scaling and instant compute provisioning • Minimal predefined optimization required • Query Acceleration Service handles unpredictable workloads automatically • Snowflake Optima provides automatic optimization for recurring patterns | • Excellent for ad-hoc analytics with serverless architecture requiring zero infrastructure management • Intelligent query optimization handles unpredictable workloads automatically • Gemini AI integration enables natural language querying for business users • Advanced JSON support and schema inference enable flexible data exploration • Cross-cloud analytics capabilities for federated queries | • Excellent performance out-of-the-box with engine optimized for star and snowflake joins and aggregations
• Self learning query plan optimizer
• Full workload isolation prevents ad-hoc complexity from affecting real-time workloads
• Aggregating indexes are automatically used by optimizer | **Snowflake** has broader support for use cases beyond traditional reporting and dashboards. Its decoupled storage and compute architecture enables you to isolate different workloads to meet SLAs, and it also supports high user concurrency. But Snowflake also does not provide interactive or ad hoc query performance because of inefficient data access along with a lack of extensive indexing and query optimization. Snowflake also cannot support streaming or low latency ingestion below one minute ingestion intervals. All of these limitations exclude Snowflake from many operational use cases and most customer-facing applications that require second-level performance. **BigQuery**, like Snowflake, has broader support for use cases beyond reporting and dashboards. You can isolate workloads by assigning each workload to different reserved slots. Unlike Snowflake, Redshift, or Athena, BigQuery also supports low latency streaming. But like these other three technologies. BigQuery also lacks the performance to support interactive or ad hoc queries at scale. This eliminates BigQuery from being a great option for many operational and customer-facing use cases where the users demand a few seconds of wait at worst, which translates to sub-second query times for the data warehouse. **Firebolt** stands out by being the fastest cloud data warehouse when compared to Snowflake, Redshift, BigQuery and Athena. It's great for delivering sub-second analytics at scale, while remaining hardware efficient and high concurrency friendly. This makes it a great choice for operational use cases and customer-facing data apps. Given that it is not as feature-rich and integration rich as the more mature data warehouses makes it a lesser fit for a general-purpose Enterprise data warehouse. It is also not the best fit for ad-hoc use cases, because of the need to predefine indexing at the table level. # Snowflake vs ClickHouse (/comparison/snowflake-vs-clickhouse) ## Architecture [#architecture] The biggest difference among cloud data warehouses are whether they separate storage and compute, how much they isolate data and compute, and what clouds they can run on. | Feature | Snowflake | ClickHouse | | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Separation of storage and compute | Yes | Yes – SharedMergeTree engine in ClickHouse Cloud enables full separation of storage and compute, with compute-compute separation through Warehouses feature (introduced 2025) allowing multiple isolated compute services sharing the same data | | Supported cloud infrastructure | AWS, Azure, GCP with full feature parity across all three major clouds | AWS, GCP, Azure, cloud service and on-premises | | Isolated tenancy – option for dedicated resources | • Multi-tenant pooled resources
• Isolated tenancy available via VPS tier | • Multi-tenant metadata layer
• Isolated tenancy for compute & storage per client in cloud | | Control vs abstraction of compute | • Configurable warehouse sizes (XS to 6XL)
• Multi-cluster warehouses with auto-scaling
• Choice between Generation 1 and Generation 2 standard warehouses
• MAX\_CONCURRENCY\_LEVEL parameter for resource allocation | Configurable cluster size and compute types in ClickHouse Cloud with granular control over nodes (1-128 nodes) and node characteristics. Warehouses feature enables multiple isolated read-only compute environments. | | Self-hosted and hybrid deployment options | Snowflake for Government Cloud and private cloud options available | Self-managed deployments available with full control over infrastructure | | ACID Compliance and Transactions | Full ACID compliance with Time Travel and zero-copy cloning capabilities | Limited ACID compliance with MergeTree engine family. | **Snowflake** was one of the first decoupled storage and compute architectures, making it the first to have nearly unlimited compute scale and workload isolation, and horizontal user scalability. It runs on AWS, Azure and GCP. It is multi-tenant over shared resources in nature and requires you to move data out of your VPC and into the Snowflake cloud. "Virtual Private Snowflake" (VPS) is its highest-priced tier, and can run a dedicated isolated version of Snowflake. Its virtual warehouses can be T-shirt sized along an XS/S/M…/4XL axis, where each discrete T-shirt size is bundled with fixed HW properties that are abstracted from the users. Snowflake has recently added support for Snowflake managed Iceberg tables. **ClickHouse** was originally developed at Yandex, the Russian search engine, as an OLAP engine for low latency analytics. It was built as an on-premise solution with coupled storage & compute, and a large variety of tuning options in the form of indexes and and merge trees. ClickHouse's architecture is famous for its focus on performance and low-latency queries. The tradeoff is that it is considered very difficult to work with. SQL support is very limited, and tuning/running it requires significant engineering resources. ## Scalability [#scalability] There are three big differences among data warehouses and query engines that limit scalability: decoupled storage and compute, dedicated resources, and continuous ingestion. | Feature | Snowflake | ClickHouse | | --------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Elasticity – Scaling for larger data volumes and faster queries | • Instant warehouse resize (XS to 6XL) with no downtime • Multi-cluster auto-scaling • Generation 2 warehouses provide \~2x performance improvement over Generation 1 | Automatic horizontal and vertical scaling in ClickHouse Cloud with SharedMergeTree architecture. Manual scaling for self-managed deployments with cluster rebalancing capabilities | | Elasticity – Scaling for higher concurrency | • Single warehouse supports many concurrent queries (MAX\_CONCURRENCY\_LEVEL=8 controls resource allocation per query, not query limit) • Multi-cluster warehouses enable thousands of concurrent queries with auto-scaling • Unlimited virtual warehouses can be created | Supports high concurrency with proper resource allocation and configuration. Vertical auto-scaling and horizontal manual scaling. Additional warehouses can idle to zero billing. Primary service always on in multi-warehouse configurations. | **Snowflake** scales very well both for data volumes and query concurrency. The decoupled storage/compute architecture supports resizing clusters without downtime, and in addition, supports auto-scaling horizontally for higher query concurrency during peak hours. **ClickHouse** doesn't offer any dedicated scaling features or mechanisms. While it can deliver linearly scalable performance for some types of queries, scaling itself has to be done manually. Hardware is self-managed in ClickHouse. This means that to scale you would have to provision a cluster and migrate. ## Performance [#performance] Performance is the biggest challenge with most data warehouses today. While decoupled storage and compute architectures improved scalability and simplified administration, for most data warehouses it introduced two bottlenecks; storage, and compute. Most modern cloud data warehouses fetch entire partitions over the network instead of just fetching the specific data needed for each query. While many invest in caching, most do not invest heavily in query optimization. Most vendors also have not improved continuous ingestion or semi-structured data analytics performance, both of which are needed for operational and customer-facing use cases. | Feature | Snowflake | ClickHouse | | ------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Indexes | • Search Optimization Service for point lookups and selective queries (additional cost) • Clustering keys for data organization and automatic clustering • Materialized views • Snowflake Optima automatic indexing on Generation 2 warehouses (no additional cost) • No traditional database indexes | • Primary indexes • Skipping indexes (minmax, set, bloom filters, ngrambf\_v1, tokenbf\_v1)
• MergeTree indexes
• Incremental Materialized views | | Compute tuning | • Warehouse T-shirt sizing (XS to 6XL) • Multi-cluster configuration and scaling policies • Generation 1 vs Generation 2 warehouse selection • MAX\_CONCURRENCY\_LEVEL parameter tuning • Query Acceleration Service for long-running queries | Configurable compute resources in cloud offering | | Storage format | Columnar micro-partitioned & compressed storage | Columnar, supports sorted, compressed, encoded & sparsely indexed files with native Apache Iceberg support. | | Table-level partition & pruning techniques | • Data automatically divided into micro-partitions • Automatic pruning at micro-partition level • Clustering keys for data organization with automatic clustering • Snowflake Optima provides additional automatic pruning optimization on Gen2 warehouses | Partitioning by date/time and custom partitions with MergeTree indexes. | | Result cache | Yes | Yes, results cache with TTL and query condition cache. | | Warm cache (SSD) | Yes, at micro-partition level granularity | Yes, at indexed data-range level granularity | | Support for semi-structured data & JSON functions within SQL | Yes | Yes, including Lambda expressions and native JSON data type (GA in v25.3) | | Vector Search and AI Capabilities | AI integration through Cortex AI and Snowpark ML | • Native vector search capabilities and embeddings
• MCP Server for AI driven analytics
• Natural Language to SQL
• SQL based Inference | | Query Optimizations | • Search Optimization Service for point lookups (additional cost)
• Query Acceleration Service (QAS) for long-running and unpredictable workloads • Snowflake Optima automatic optimization on Generation 2 warehouses (no additional cost) • Automatic clustering with background maintenance • Materialized views with automatic refresh • Result cache (24hrs) • Cost-based optimization with dynamic query rewriting | • Primary indexes (ORDER BY)
• Data skipping indexes (minmax, set, bloom filters, ngrambf\_v1, tokenbf\_v1)
• Materialized views
• Projections
• PREWHERE optimization
• Query analysis tools
• Automatic global join reordering (v25.9)
• Enhanced JSON query optimization
• Streaming secondary indices | **Snowflake** typically comes on top for most queries when it comes to performance in public TPC-based benchmarks when compared to BigQuery and Redshift, but only marginally. Its micro partition storage approach effectively scans less data compared to larger partitions. The ability to isolate workloads over the decoupled storage & compute architecture lets you avoid competition for resources compared to multi-tenant shared resource solutions, and the ability to increase warehouse sizes can often enhance performance (for a higher price), but not always linearly. Snowflake's recently released "Search optimization service" delivers index-like behavior for point queries, but comes at an additional cost. **ClickHouse** is famous for being one of the fastest local runtimes ever built for OLAP workloads. Its columnar storage, compression and indexing capabilities make it a consistent leader in benchmarks. Its lack of support for standard SQL and lack of query optimizer means that it's less suitable for traditional BI workloads, and more suitable for engineering managed workloads. While fast, it requires a lot of tuning and optimization. ## Use cases [#use-cases] There are a host of different analytics use cases that can be supported by a data warehouse. Look at your legacy technologies and their workloads, as well as the new possible use cases, and figure out which ones you will need to support in the next few years. | Feature | Snowflake | ClickHouse | | ---------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Low-latency dashboards | • Sub-second to seconds response times at TB+ scale with proper clustering and optimization • Enhanced by Query Acceleration Service and Search Optimization Service • Generation 2 warehouses provide \~2x performance improvement over Generation 1 • Snowflake Optima provides automatic optimization | • Sub-second load times at TB+ scale with proper indexing
• ClickHouse Cloud reduces engineering overhead with managed service • Proven low-latency performance (120ms at 2500 QPS in benchmarks)
• Purpose-built for low-latency OLAP and real-time analytics | | Enterprise BI | • Mature and comprehensive Enterprise DW feature set • Extensive integrations with Enterprise BI ecosystem • Multi-cloud deployment options with consistent experience • Strong SQL compliance and wide ecosystem support • Zero-copy data sharing capabilities | • Growing ecosystem with 50+ integrations including major BI tools
• Native MySQL protocol support enables broad BI tool compatibility • Strong SQL compliance with PostgreSQL compatibility
• Best suited for modern analytical workloads and engineering-managed use cases | | Data Apps and AI Applications (Customer-facing low-latency high concurrency) | • Multi-cluster warehouses support thousands of concurrent users with auto-scaling • Individual warehouses support many concurrent queries (not limited to 8 concurrent queries) • Sub-second to seconds response times with proper optimization • Generation 2 warehouses provide significant performance improvements for high-concurrency workloads • AI integration through Cortex AI | • Sub-second response times at TB+ scale
• Supports 1000 concurrent users per replica
• Strong price-performance on customer-facing applications
• Native vector search and embeddings | | Ad hoc | • Excellent for ad-hoc with decoupled storage/compute • Auto-scaling and instant compute provisioning • Minimal predefined optimization required • Query Acceleration Service handles unpredictable workloads automatically • Snowflake Optima provides automatic optimization for recurring patterns | • Good for ad-hoc queries with ClickHouse Cloud's separated storage/compute architecture
• Join optimizations enable more query complexity
• Strong sampling capabilities (TABLESAMPLE) for exploratory analysis
• Resource management through user quotas prevents query interference
• Materialized views offer performance improvements for common aggregation patterns, ad-hoc users specify directly in SQL | **Snowflake** is a well rounded general purpose cloud data warehouse, that can also span beyond traditional BI & Analytics use cases into Ad-Hoc and ML use cases. Thanks to the flexible decoupeld storage & compute architecture that allows you to isolate and control the amount of compute per workload, it's possible to tackle a broad spectrum of workloads. However, like its close siblings Redshift & BigQuery, it struggles to deliver low-latency query performance at scale, making it a lesser fit for operational use cases and customer-facing data apps. **ClickHouse** was not designed to be a data warehouse, but rather a low-latency query execution runtime. Managing it typically requires significant engineering overhead. Hence, it's a good fit for engineering managed operational use cases and customer-facing data apps, where low latency matters. It is not a good fit for a general purpose data warehouse, nor for Ad-Hoc analytics or ELT. # Snowflake vs Redshift vs Firebolt (2025) (/comparison/snowflake-vs-redshift) ## Architecture [#architecture] The biggest difference among cloud data warehouses are whether they separate storage and compute, how much they isolate data and compute, and what clouds they can run on. | Feature | Snowflake | Redshift | Firebolt | | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Separation of storage and compute | Yes | RA3 instances enable separation of compute and storage, but limited workload isolation compared to other platforms | Yes, separation of storage and metadata as well as compute from compute with full workload isolation. | | Supported cloud infrastructure | AWS, Azure, GCP with full feature parity across all three major clouds | AWS only | AWS (GCP coming soon) & anywhere (Firebolt Core) | | Isolated tenancy – option for dedicated resources | • Multi-tenant pooled resources
• Isolated tenancy available via VPS tier | • Isolated tenant & resources • Runs in your VPC | • Multi-tenant metadata layer
• Isolated tenancy for compute & storage per client | | Control vs abstraction of compute | • Configurable warehouse sizes (XS to 6XL)
• Multi-cluster warehouses with auto-scaling
• Choice between Generation 1 and Generation 2 standard warehouses
• MAX\_CONCURRENCY\_LEVEL parameter for resource allocation | • Configurable cluster size
• Configurable compute types | Uses engine abstraction:
• Each engine has configurable cluster size (1-128 nodes) for horizontal scaling.
• Configurable compute family (compute vs storage optimized) and type (XS, S, M, L, XL) for vertical scaling
• Number of clusters for concurrency (auto)scaling.
Provides full workload isolation across engines. | | Self-hosted and hybrid deployment options | Snowflake for Government Cloud and private cloud options available | Limited hybrid options with Redshift Serverless | • Firebolt Core: Forever free, self-hosted edition with full query engine capabilities
• Same performance and features as managed service
• Deploy anywhere: local laptop, cloud, datacenter, Kubernetes
• Production-grade distributed architecture
• No usage restrictions except building competing SaaS | | ACID Compliance and Transactions | Full ACID compliance with Time Travel and zero-copy cloning capabilities | ACID compliant at table level with some limitations on concurrent operations | • Full ACID compliance with snapshot isolation
• Multi-statement transactions supported
• Strong consistency across all operations
• Supports concurrent reads and writes
• Transactional integrity for data applications | **Snowflake** was one of the first decoupled storage and compute architectures, making it the first to have nearly unlimited compute scale and workload isolation, and horizontal user scalability. It runs on AWS, Azure and GCP. It is multi-tenant over shared resources in nature and requires you to move data out of your VPC and into the Snowflake cloud. “Virtual Private Snowflake” (VPS) is its highest-priced tier, and can run a dedicated isolated version of Snowflake. Its virtual warehouses can be T-shirt sized along an XS/S/M…/4XL axis, where each discrete T-shirt size is bundled with fixed HW properties that are abstracted from the users. Snowflake has recently added support for Snowflake managed Iceberg tables. **Redshift** has the oldest architecture, being the first Cloud DW in the group. Its architecture wasn’t designed to separate storage & compute. While it now has RA3 nodes which allow you to scale compute and only cache the data you need locally, all compute still operates together. You cannot separate and isolate different workloads over the same data, which puts it behind other decoupled storage/compute architectures. Redshift runs as an isolated tenant per customer, and unlike other cloud data warehouses, it is deployed in your VPC. Redshift offers a serverless option which is based on an abstracted unit called Redshift Processing Unit (RPU) ranging from 8 to 512 in increments of 8. Each RPU provides 2 vCPU and 16GB RAM. Thus, 8 RPU is equivalent to 16 vCPU / 128GB RAM. The minimum RPU is 8. **Firebolt** is built on a natively decoupled storage & compute architecture, on AWS only. Data has to be copied outside of your VPC into the Firebolt, where both your compute and data run in a dedicated and isolated tenant. A “Firebolt Engine” can be granularly configured across # of nodes and different CPU/RAM/SSD combinations. ## Scalability [#scalability] There are three big differences among data warehouses and query engines that limit scalability: decoupled storage and compute, dedicated resources, and continuous ingestion. | Feature | Snowflake | Redshift | Firebolt | | --------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Elasticity – Scaling for larger data volumes and faster queries | • Instant warehouse resize (XS to 6XL) with no downtime • Multi-cluster auto-scaling • Generation 2 warehouses provide \~2x performance improvement over Generation 1 | Available via Elastic Resize – slow and limited, downtime required | Granular cluster resize with node types, number of nodes and number of clusters. Zero downtime. | | Elasticity – Scaling for higher concurrency | • Single warehouse supports many concurrent queries (MAX\_CONCURRENCY\_LEVEL=8 controls resource allocation per query, not query limit) • Multi-cluster warehouses enable thousands of concurrent queries with auto-scaling • Unlimited virtual warehouses can be created | • 5 concurrent queries per WLM queue by default (up to 8 queues) • Concurrency Scaling enables thousands of concurrent queries | A single engine can handle hundreds of concurrent queries. Engines auto-scale the number of clusters up and down base on resource usage thresholds. Idle engines scale down to zero billing. | **Snowflake** scales very well both for data volumes and query concurrency. The decoupled storage/compute architecture supports resizing clusters without downtime, and in addition, supports auto-scaling horizontally for higher query concurrency during peak hours. **Redshift** is limited in scale because even with RA3, it cannot distribute different workloads across clusters. While it can scale to up to 10 clusters automatically to support query concurrency, it can only handle a maximum of 50 queued queries across all clusters by default. **Firebolt** can handle the largest data volumes and concurrency on a single comparable cluster size, thanks to its superior hardware efficiency. Thanks to its decoupled storage & compute architecture it scales very well to large data volumes. However, resizing an engine size isn’t instant and requires orchestration if avoiding downtime is necessary. A single Firebolt engine can support hundreds of concurrent queries, avoiding the need to scale out for most use cases. Scaling horizontally for even higher concurrency is manual. ## Performance [#performance] Performance is the biggest challenge with most data warehouses today. While decoupled storage and compute architectures improved scalability and simplified administration, for most data warehouses it introduced two bottlenecks; storage, and compute. Most modern cloud data warehouses fetch entire partitions over the network instead of just fetching the specific data needed for each query. While many invest in caching, most do not invest heavily in query optimization. Most vendors also have not improved continuous ingestion or semi-structured data analytics performance, both of which are needed for operational and customer-facing use cases. | Feature | Snowflake | Redshift | Firebolt | | ------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Indexes | • Search Optimization Service for point lookups and selective queries (additional cost) • Clustering keys for data organization and automatic clustering • Materialized views • Snowflake Optima automatic indexing on Generation 2 warehouses (no additional cost) • No traditional database indexes | None | • Sparse primary indexes
• Aggregating indexes
• Join indexes
• Optimizer driven index usage | | Compute tuning | • Warehouse T-shirt sizing (XS to 6XL) • Multi-cluster configuration and scaling policies • Generation 1 vs Generation 2 warehouse selection • MAX\_CONCURRENCY\_LEVEL parameter tuning • Query Acceleration Service for long-running queries | Choice over number of nodes and their type | SQL defined engines. Control number of nodes, node family and type per cluster, with one or more clusters per engine. Multiple engines isolate workloads. | | Storage format | Columnar micro-partitioned & compressed storage | Columnar & compressed storage (RA3 nodes) | Columnar, sorted & compressed & sparsely indexed storage (F3 – Firebolt File Format) with native Apache Iceberg support | | Table-level partition & pruning techniques | • Data automatically divided into micro-partitions • Automatic pruning at micro-partition level • Clustering keys for data organization with automatic clustering • Snowflake Optima provides additional automatic pruning optimization on Gen2 warehouses | • No table partitions • User-defined distribution & sort keys are used to optimize for speed | • User-defined table-level partitions are optional.
• Data is automatically sorted, compressed and indexed into F3 format.
• Pruning at indexed data-range level. | | Result cache | Yes | Yes | Yes, results and sub-results cache with transactional spoiling. | | Warm cache (SSD) | Yes, at micro-partition level granularity | Only with RA3 nodes at partition-level granularity | Yes, at indexed data-range level granularity | | Support for semi-structured data & JSON functions within SQL | Yes | Limited | Yes, including Lambda expressions and native nested array structures | | Vector Search and AI Capabilities | AI integration through Cortex AI and Snowpark ML | Limited AI capabilities – primarily through integrations | • Native vector search capabilities and embeddings
• MCP Server for AI driven analytics
• Natural Language to SQL
• SQL based Inference | | Query Optimizations | • Search Optimization Service for point lookups (additional cost)
• Query Acceleration Service (QAS) for long-running and unpredictable workloads • Snowflake Optima automatic optimization on Generation 2 warehouses (no additional cost) • Automatic clustering with background maintenance • Materialized views with automatic refresh • Result cache (24hrs) • Cost-based optimization with dynamic query rewriting | • Basic query optimizer • Materialized views • Result caching • ANALYZE for table statistics • Workload management (WLM) • Automated materialized views (AutoMV) • AI-driven scaling (Serverless preview) | • Primary indexes, aggregating indexes, join indexes, sparse indexes
• Sub-plan result caching
• F3 storage format optimization
• Automatic query optimizer with aggressive pruning
• Late column materialization
• Query analysis tools based on execution telemetry | **Snowflake** typically comes on top for most queries when it comes to performance in public TPC-based benchmarks when compared to BigQuery and Redshift, but only marginally. Its micro partition storage approach effectively scans less data compared to larger partitions. The ability to isolate workloads over the decoupled storage & compute architecture lets you avoid competition for resources compared to multi-tenant shared resource solutions, and the ability to increase warehouse sizes can often enhance performance (for a higher price), but not always linearly. Snowflake’s recently released “Search optimization service” delivers index-like behavior for point queries, but comes at an additional cost. **Redshift** does provide a result cache for accelerating repetitive query workloads and also has more tuning options than some others. But it does not deliver much faster compute performance than other cloud data warehouses in benchmarks. Sort keys can be used to optimize performance, but their contribution is limited. There is no support for indexes, and low-latency analytics at large data volumes is hard to achieve. Because Redshift decoupling of storage & compute is limited compared to other cloud data warehouses, it doesn’t support isolating workloads, which means performance can degrade under pressure and competition for resources. **Firebolt** is the fastest when it comes to query performance when compared to cloud data warehouses and services like Athena. Its unique approach to storage and indexing results in highly aggressive data pruning that scans dramatically less data compared to other technologies. While other technologies scan partitions or micro-partitions, Firebolt works with indexed data ranges that are significantly smaller. In addition, Firebolt lets users accelerate queries further with multiple index types (Aggregating index, Join index), and using its decoupled storage & compute architecture workloads can be easily isolated to guarantee consistent performance. ## Use cases [#use-cases] There are a host of different analytics use cases that can be supported by a data warehouse. Look at your legacy technologies and their workloads, as well as the new possible use cases, and figure out which ones you will need to support in the next few years. | Feature | Snowflake | Redshift | Firebolt | | ---------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Low-latency dashboards | • Sub-second to seconds response times at TB+ scale with proper clustering and optimization • Enhanced by Query Acceleration Service and Search Optimization Service • Generation 2 warehouses provide \~2x performance improvement over Generation 1 • Snowflake Optima provides automatic optimization | • Seconds to tens of seconds load times at 100s of GB scale • Can achieve faster performance with Concurrency Scaling and proper tuning | • 120ms query latency at 4000 QPS (FireScale benchmark 2025)
• Sub-second performance at TB+ scale with proper indexing
• Built for AI-driven analytics, dashboards, and real-time analytic applications | | Enterprise BI | • Mature and comprehensive Enterprise DW feature set • Extensive integrations with Enterprise BI ecosystem • Multi-cloud deployment options with consistent experience • Strong SQL compliance and wide ecosystem support • Zero-copy data sharing capabilities | • Mature and comprehensive Enterprise DW feature set • Extensive integrations with Enterprise BI ecosystem • Strong AWS ecosystem integration | • Growing ecosystem with focus on modern BI tools
• Strong SQL compliance with PostgreSQL
• Wire level compatibility drives expansion to PostgreSQL BI and ETL ecosystem | | Data Apps and AI Applications (Customer-facing low-latency high concurrency) | • Multi-cluster warehouses support thousands of concurrent users with auto-scaling • Individual warehouses support many concurrent queries (not limited to 8 concurrent queries) • Sub-second to seconds response times with proper optimization • Generation 2 warehouses provide significant performance improvements for high-concurrency workloads • AI integration through Cortex AI | • 5 concurrent queries per WLM queue by default (up to 8 queues) • Concurrency Scaling enables thousands of concurrent queries • Seconds-level response times typical • Automatic scaling for burst workloads • Limited AI application support | • 120ms latency at 4000+ QPS proven performance at TB+ scale
• Supports hundreds to thousands of concurrent queries on single engine
• Price-performance leader (8x better than Snowflake, 18x vs Redshift) • Purpose-built for AI agents and data-intensive applications
• Native vector search and embeddings | | Ad hoc | • Excellent for ad-hoc with decoupled storage/compute • Auto-scaling and instant compute provisioning • Minimal predefined optimization required • Query Acceleration Service handles unpredictable workloads automatically • Snowflake Optima provides automatic optimization for recurring patterns | • Performance dependent on predefined distribution & sort keys • Elastic Resize enables adding compute resources • Typically subset of data loaded for ad-hoc analysis | • Excellent performance out-of-the-box with engine optimized for star and snowflake joins and aggregations
• Self learning query plan optimizer
• Full workload isolation prevents ad-hoc complexity from affecting real-time workloads
• Aggregating indexes are automatically used by optimizer | **Snowflake** is a well rounded general purpose cloud data warehouse, that can also span beyond traditional BI & Analytics use cases into Ad-Hoc and ML use cases. Thanks to the flexible decoupeld storage & compute architecture that allows you to isolate and control the amount of compute per workload, it’s possible to tackle a broad spectrum of workloads. However, like its close siblings Redshift & BigQuery, it struggles to deliver low-latency query performance at scale, making it a lesser fit for operational use cases and customer-facing data apps. **Redshift** was originally designed to support traditional internal BI reporting and dashboard use cases for analysts. As such, it is typically used as a general-purpose Enterprise data warehouse. With deep integrations into the AWS ecosystem, it can also leverage AWS ML service, making it also useful for ML projects. However, given the coupling of storage & compute, and the difficulty in delivering low-latency analytics at scale, it is less suited for operational use cases and customer-facing use cases like Data Apps. The coupling of storage and compute, together with the need to predefine sort & dist keys for optimal performance, make it challenging to use for Ad-Hoc analytics. **Firebolt** stands out by being the fastest cloud data warehouse when compared to Snowflake, Redshift, BigQuery and Athena. It’s great for delivering sub-second analytics at scale, while remaining hardware efficient and high concurrency friendly. This makes it a great choice for operational use cases and customer-facing data apps. Given that it is not as feature-rich and integration rich as the more mature data warehouses makes it a lesser fit for a general-purpose Enterprise data warehouse. It is also not the best fit for ad-hoc use cases, because of the need to predefine indexing at the table level. # E-Commerce Analytics Primer (/free-sample-datasets/e-commerce) This primer introduces data engineers to sample use cases for data-driven decision-making in e-commerce. It walks through ingesting data and exploring performance optimizations using the Firebolt Cloud Data Warehouse, with data from the Open Customer Data Platform (CDP) Project. ## Understanding the e-commerce data model [#understanding-the-e-commerce-data-model] A well-thought-out schema is critical for efficiently managing the diverse data in an e-commerce warehouse — user interactions, product details, and transaction records. This dataset uses a single-table design: | Property | Data Type | Description | | --------------- | -------------- | --------------------------------------- | | `event_time` | TIMESTAMPTZ | Time in UTC | | `event_type` | TEXT | Customer event (view / cart / purchase) | | `product_id` | BIGINT | ID of a product | | `category_id` | TEXT | Product's category ID | | `category_code` | TEXT | Product's category code | | `brand` | TEXT | Product brand | | `price` | NUMERIC(38, 9) | Price of a product | | `user_id` | TEXT | Permanent user ID | | `user_session` | TEXT | Temporary user session ID | The schema supports user identification, event tracking, product trends, and session details for personalization and segmentation. ## Working with the e-commerce dataset [#working-with-the-e-commerce-dataset] The data comes from a Kaggle dataset loaded into a public Amazon S3 bucket (`firebolt-sample-datasets-public-us-east-1`). It spans seven months of activity (October 2019 – April 2020), roughly 32GB uncompressed as CSV, converted to Parquet for ingestion. Once ingested it is approximately **412 million records** — 52GB uncompressed, 21GB compressed. Start by creating a database and engine: ```sql CREATE DATABASE ecommercedb WITH DESCRIPTION = 'ECommerce Analytics Primer'; CREATE ENGINE ecommerceEngine; START ENGINE ecommerceEngine; USE ENGINE ecommerceEngine; USE ecommercedb; ``` Create the fact table: ```sql CREATE FACT TABLE IF NOT EXISTS "ecommerce" ( "event_time" TIMESTAMPTZ NOT NULL, "event_type" TEXT NOT NULL, "product_id" BIGINT NOT NULL, "category_id" TEXT NULL, "category_code" TEXT NULL, "brand" TEXT NULL, "price" NUMERIC(38, 9) NULL, "user_id" TEXT NULL, "user_session" TEXT NULL ); ``` Ingest the data: ```sql COPY INTO ecommerce FROM 's3://firebolt-sample-datasets-public-us-east-1/ecommerce_primer/parquet/' WITH PATTERN='*.gz.parquet' TYPE = PARQUET; SHOW TABLES; ``` ## Sample analytics on the e-commerce dataset [#sample-analytics-on-the-e-commerce-dataset] ### Customer lifetime value (LTV) [#customer-lifetime-value-ltv] Assess the total revenue generated by customers over their engagement with your brand. ```sql SELECT user_id, SUM(price) AS total_revenue, COUNT(DISTINCT user_session) AS total_sessions, (SUM(price) / COUNT(DISTINCT user_session)) AS average_revenue_per_session FROM ecommerce WHERE event_type = 'purchase' AND user_session IS NOT NULL GROUP BY user_id ORDER BY total_revenue DESC; ``` ### Funnel analysis for conversion optimization [#funnel-analysis-for-conversion-optimization] Track the customer journey from views to purchases and identify drop-off points. ```sql SELECT event_type, COUNT(DISTINCT user_id) AS unique_users FROM ecommerce WHERE event_type IN ('view', 'cart', 'purchase') GROUP BY event_type ORDER BY 2 DESC; ``` ### Product recommendations and cross-selling [#product-recommendations-and-cross-selling] Analyze co-purchase data to reveal products frequently bought together. ```sql WITH CoPurchaseCounts AS ( SELECT a.product_id AS product_A, b.product_id AS product_B, COUNT(*) AS purchase_count FROM ecommerce a JOIN ecommerce b ON a.user_session = b.user_session AND a.product_id < b.product_id WHERE a.event_type = 'purchase' AND b.event_type = 'purchase' GROUP BY a.product_id, b.product_id ) SELECT * FROM CoPurchaseCounts cp WHERE purchase_count > 2 ORDER BY purchase_count DESC LIMIT 10; ``` ### Seasonal sales trends and insights [#seasonal-sales-trends-and-insights] Analyze sales by month to understand seasonal variation. ```sql SELECT EXTRACT(MONTH FROM event_time) AS sales_month, SUM(price) AS total_revenue FROM ecommerce WHERE event_type = 'purchase' GROUP BY sales_month ORDER BY sales_month; ``` ### Customer segmentation for targeted marketing [#customer-segmentation-for-targeted-marketing] Segment customers by activity level and average revenue. ```sql WITH CustomerSegments AS ( SELECT user_id, COUNT(DISTINCT user_session) AS session_count, SUM(price) AS total_revenue FROM ecommerce WHERE event_type = 'purchase' GROUP BY user_id ) SELECT CASE WHEN session_count <= 5 THEN 'Low Activity' WHEN session_count <= 20 THEN 'Medium Activity' ELSE 'High Activity' END AS segment, COUNT(user_id) AS user_count, AVG(total_revenue) AS avg_revenue FROM CustomerSegments GROUP BY segment; ``` ### Cart abandonment rate [#cart-abandonment-rate] Identify where potential customers drop off in the purchase process. ```sql WITH CartAbandonment AS ( SELECT event_time::DATE as event_date, user_id, COUNT(CASE WHEN event_type = 'cart' THEN 1 ELSE NULL END) AS cart_count, COUNT(CASE WHEN event_type = 'purchase' THEN 1 ELSE NULL END) AS purchase_count FROM ecommerce WHERE event_type IN ('cart', 'purchase') GROUP BY event_date, user_id ) SELECT event_date, COUNT(*) AS total_users, SUM(CASE WHEN cart_count > 0 AND purchase_count = 0 THEN 1 ELSE 0 END) AS abandoned_carts, (SUM(CASE WHEN cart_count > 0 AND purchase_count = 0 THEN 1 ELSE 0 END) * 100.0 / COUNT(*)) AS abandonment_rate FROM CartAbandonment GROUP BY event_date ORDER BY event_date; ``` ### User purchase frequency distribution [#user-purchase-frequency-distribution] Analyze the distribution of user purchase frequencies for marketing planning. ```sql WITH UserPurchaseFrequency AS ( SELECT user_id, COUNT(DISTINCT event_time::DATE) AS purchase_frequency FROM ecommerce WHERE event_type = 'purchase' GROUP BY user_id ) SELECT purchase_frequency, COUNT(user_id) AS user_count FROM UserPurchaseFrequency GROUP BY purchase_frequency ORDER BY purchase_frequency; ``` ## Optimizing performance [#optimizing-performance] Use the `recommend_ddl` command to find an optimal primary index for the workload: ```sql CALL recommend_ddl ( ecommerce, ( SELECT query_text FROM information_schema.engine_query_history where query_text ilike 'select%' and end_time > NOW() - INTERVAL '30 minutes' ) ); ``` For this workload, Firebolt recommends a primary index on `event_type` and `user_session`, which achieves roughly 94% average data pruning. Recreate the table with that index and reload: ```sql CREATE FACT TABLE IF NOT EXISTS "ecommerce_pi" ( "event_time" TIMESTAMPTZ NOT NULL, "event_type" TEXT NOT NULL, "product_id" BIGINT NOT NULL, "category_id" TEXT NULL, "category_code" TEXT NULL, "brand" TEXT NULL, "price" double precision NULL, "user_id" TEXT NULL, "user_session" TEXT NULL ) PRIMARY INDEX event_type, user_session; INSERT INTO ecommerce_pi SELECT * FROM ecommerce; ``` Data analytics is a critical part of e-commerce operations today, and performance and efficiency are essential in cloud-based analytics to avoid slower performance and cost overruns. ## Appendix: querying the data lake with external tables [#appendix-querying-the-data-lake-with-external-tables] You can query data directly from S3 without loading it into the warehouse — useful for ad-hoc analysis. ```sql CREATE EXTERNAL TABLE IF NOT EXISTS ex_ecommerce ( event_time TIMESTAMPTZ NOT NULL, event_type TEXT NOT NULL, product_id BIGINT NOT NULL, category_id TEXT NULL, category_code TEXT NULL, brand TEXT NOT NULL, price NUMERIC(38, 9) NULL, user_id TEXT NULL, user_session TEXT NULL ) URL = 's3://firebolt-sample-datasets-public-us-east-1/ecommerce_primer/parquet/' OBJECT_PATTERN = '*.gz.parquet' TYPE= (PARQUET); ``` ```sql SELECT event_type, count(*) FROM ex_ecommerce GROUP BY ALL; ```