Market intelligence and analytics

A leading market research technology company provides digital market intelligence about millions of websites and mobile apps to thousands of clients worldwide by capturing all kinds of online information including ad, search and website traffic, and bringing it together within fast, interactive analytics.

The challenge

The company had a homegrown analytics solution that restricted its offering and market growth. While it was fast enough for specific queries and data sets, it was limited both in the types of analytics and the size of data it could support for each customer, and hard to change.

The evaluation process

They started evaluating different solutions to support segment analytics by its customers. They considered many alternatives as part of their evaluation.

The solution - Firebolt for segment analytics

Firebolt Appsflyer case study

They started using Firebolt for segment analytics. Their customers use segment analytics to help understand customer behavior, preferences, loyalty, and cross-selling opportunities.

They have over 200TB of customer data (compressed), spanning more than three years, in Firebolt. This has been increasing as they add data that they could not support in their previous solution. They now write SQL and run it on Firebolt to Extract, Load, and Transform (ELT) data from their data lake into Firebolt. All the segment analytics are accessed via REST APIs that invoke the SQL from their Web-based analytics UI.

Faster time to market for new features

Athena vs Firebolt performance table

With Firebolt, they are now much more agile. It used to take 6-9 months to roll out new analytics features to customers. The existing system required custom development to create the new schema and queries, and optimize for performance. When they first adopted Firebolt, the time it took to get started, become experts and deploy their first project into production was close to 3 months. But now that they are experts, new features take a few days.

Customer onboarding is now much faster as well because they were able to automate the personalization for each customer that used to take weeks of custom development. What made this possible was Firebolt’s native support for JSON. Part of the first project involved automating the personalization using two JSON columns for pages and timestamps and Lambda-style array functions in SQL. The SQL performs complex calculations that personalize queries for each customer.

The greater agility and flexibility comes from being able to do everything in SQL - from ELT to all the queries and personalization - and REST APIs. This has made it much easier to use 3rd party infrastructure and add all new kinds of analytics, including ad hoc.

Faster, more flexible ELT

Before Firebolt, it could take up to 3 months just to develop new ETL jobs. A data engineer would use AWS Lambda to process the data, perform complex calculations, and load it into the custom system.

With Firebolt, they now write SQL that uses lambda array functions, and deploy the SQL for ELT onto a Firebolt engine, which is a Firebolt compute cluster. You can deploy any number of engines to support different workloads like ELT or analytics, or use engines to scale out horizontally to support more users or queries. It now can take just hours to add any new data sets.

The same data engineer who used to write the AWS Lambda code was able to immediately start using lambda array functions in SQL, and loves it.

Runs all queries in 2 seconds or less

With Firebolt, every query they run is 2 seconds or less, with a lot of sub-second query times. A big part of what helped was indexing. Most of their query performance comes from a combination of Firebolt’s faster data access times, query optimization engine and sparse indexing. Sparse indexes are primary composite indexes on tables that include the most frequently used columns on each table. For one large with a lot of computed aggregates, they added an aggregating index, which you declare once in SQL and Firebolt automatically maintains. The index immediately doubled performance with minimal additional computing.

Scales for any data, queries, and users

The company’s previous custom system could not handle all of the data their customers wanted. With Firebolt, they can now load any data their customers need and easily analyze the entire 3 years of data if needed.

One of the biggest benefits came from Firebolt’s decoupled storage and compute architecture. Their customers could be assigned to different Firebolt engines. As customers perform queries, Firebolt only fetches and caches the data ranges they need, not entire partitions of data or the entire 200TB+ of data. Because this is a much smaller data set, it not only requires much less compute. The queries run much faster than they would against the entire data set. This architecture was ideal for the way they needed to scale and support their customers.

5-20x lower cost compared to the alternatives

Not only was Firebolt 5x lower in infrastructure costs compared to their existing system, which didn’t provide as much functionality, flexibility or hold as much data. It was also as much as 20x less expensive than the alternatives they evaluated. Not only were there even more savings from the lower costs of engineering and administration. There were also greater revenues through a much better service.

The future - more new customer analytics, faster

With segment analytics deployed, the company sees more opportunities to improve their analytics, such as keyword analytics. There are also opportunities upstream, The data prep team sees opportunities to move some Spark jobs to Firebolt for the greater performance and agility.

Talk to a Firebolt Expert