Data Teams' Outlook on Data Warehousing in 2023

December 2022 

Introduction and Key Findings 

Introduction & Methodology 

As companies continue to handle the impact of an economic slowdown, we want to shed light on what's really going on in the market. The data function of any organization is critical, so understanding the priorities of key stakeholders, namely the data team, is essential. Rather than a general state of the market, we used this survey as an opportunity to speak directly to the data teams across 300 organizations and ask for their outlook for the year ahead. 

With the rise of data apps, we’ve uncovered that data volumes are growing at a tremendous pace, and data practitioners continue to face price, performance and reliability challenges. The billion dollar question is: what will grow faster - the volume of data itself, the users’ expectations, or the technology that supports the usage of this data? 

One thing is clear: today’s data teams are tasked with a continual uphill battle to stay two steps ahead, continuously moving and improving performance. Which technology decisions will get them across the line? 

To get a true picture of the data function in today’s organizations, we commissioned a survey of 300 data stakeholders, who work in the Gaming, Internet, Software Development, and Marketing & Advertising industries in the USA. In all of these industries, there is a big shift towards data apps (e.g., data intensive, customer facing analytics). This report was administered online by Global Surveyz Research, a global research firm. The survey is based on responses from executives, analytics engineers, data engineers, and data architects, from company sizes between 150 to 3,000 employees. The respondents were recruited through a global B2B research panel, invited via email to complete the survey, with all responses collected during Q3 2022. The average amount of time spent on the survey was six minutes and 10 seconds. The answers to the non-numerical questions were randomized to prevent order bias in the answers.

Key Findings 

Technology isn't keeping up with the pace of data growth 

Nothing is going to stop data from growing exponentially. The total volume of data managed is projected to grow at a CAGR of 41% through 2023, with even more dramatic growth in certain industries, such as Marketing and Advertising. The data shows the complexity of big data is beyond what current solutions can handle. As usage explodes, 71% of companies recognize the expectation for query response time is between 10 seconds and one minute, and yet 70% admit the real-world response time is up to five minutes. 

Price and performance are the biggest data related challenges 

When respondents were asked for their main data-related challenges, price and performance came out on top. However, it’s interesting to compare these with practical considerations when investing in a data warehouse. In this case, cost didn’t make the top five considerations, and top criteria for buying a platform were security (4.12/5) performance (4.10/5) and the availability of a free trial (4.03/5). While cost is a major pain, it is considered as unavoidable when working with data warehouses. As a result, during the selection process of data warehouses, cost considerations rank lower than security, performance, ’try before you buy’, architecture, integrations and more. 

Big data investments aren’t slowing down, despite the recession 

As data volumes grow, and user expectations around response time and data usage become more demanding, what does 2023 look like? The atmosphere in the market is that things are slowing down, as seen by the 60% who expect their budgets to stay the same or even decrease. However, decision makers are still planning for growth. Executive team members such as Vice Presidents and Heads of Data are more than twice as likely to see the need for additional budget, compared to Architects and Engineers. In addition, 80% of respondents expect their data teams to grow in 2023 at an average growth rate of 11.8%. 

SQL is king

The most coveted analytic feature is ANSI SQL compliance. This could be tied to the hiring and upskilling challenge faced by companies who prefer to standardize their work and remove operational inefficiencies of using proprietary SQL dialects for different platforms. 

Data Volumes Managed, 2021-2023 

Within the segments surveyed, data volumes that companies manage are growing, and are expected to continue to grow from 107TB in 2021 on average, to 213TB in 2023 (a CAGR – Compound Annual Growth Rate of 41%). We then looked at the average data volumes managed by industry (figure 3). The largest growth is in Marketing & Advertising, with a projected CAGR of 54%, while the lowest is still a significant 35%, for the Internet segment.

Figure 2: Average Data Volumes Managed in TBs, 2021-2023 
Figure 2: Average Data Volumes Managed in TBs, 2021-2023 

Figure 3: Average Data Volumes Managed in TBs by Industry, 2021-2023
Figure 3: Average Data Volumes Managed in TBs by Industry, 2021-2023 


Query Response Times – Expected vs. Reality 

Survey respondents were asked to evaluate their most commonly used analytics workloads. Our questions were: what query response times are expected in their company for these workloads, and what are their real-world slowest query times in practice? 

We uncovered a disconnect between expectations and reality. 71% of respondents believe expectations are for a response time of between 10 seconds and one minute, and yet in reality – 70% say that the response time is five times longer, and that it can take up to five minutes to receive a response. It is clear that technology capabilities come up short against increasing performance expectations. 

*Percentages do not add up to 100% due to rounding up of numbers. 

Figure 4: Query Response Times – Expected vs. Reality 
Figure 4: Query Response Times – Expected vs. Reality


Top Considerations in Choosing Data Warehouse Technologies 

We asked survey respondents to rank their top considerations for choosing data warehouse technologies, on a scale of 1 (not important) to 5 (very important). 

Top considerations are security (4.12), performance (4.10), and having a free trial (4.03). This can be seen as a checklist of items that need to be met before a solution can be adopted for use, in order of importance. It is interesting to see that while price is the biggest challenge (as shown in figure 1), costs did not make the top five criteria when choosing data warehouse technologies. Companies are willing to pay the price, even if it’s painful, as long as the technology meets security and performance requirements. Simplicity of implementation and maintenance rank lower as most cloud data warehouses are managed services that inherently reduce the effort of operational tasks. 

*Question allowed more than one answer and as a result, percentages will add up to more than 100% 

Figure 5: Top Considerations for Choosing Data Warehouse Technologies (Weighted Average)
Figure 5: Top Considerations for Choosing Data Warehouse Technologies (Weighted Average) 


Key Data Analytics Features 

We also asked respondents to choose the most important features for data analytics. The most coveted analytic feature is ANSI SQL compliance (42%). This could be tied to the hiring and upskilling challenge faced by companies (as shown in Figure 1) who prefer to standardize their work and remove operational inefficiencies of using proprietary SQL dialects for different platforms. 

ANSI SQL is followed by auto scaling (39%), and sub-second response times (29%). These features rank higher than capabilities such as data observability, streaming and others. 

*Question allowed more than one answer and as a result, percentages will add up to more than 100%  

Figure 6: Most Important Analytic Features
Figure 6: Most Important Analytic Features 


Projected Growth of Data Teams in 2023 

80% of companies expect their data teams to grow in 2023, at an average growth rate of 11.8%. The data team is a mission critical part of the business, and despite the recession or the impact of the pandemic, only 1% are projecting that their data teams will shrink. 

Figure 7: Data Team Expected Growth, 2023
Figure 7: Data Team Expected Growth, 2023 


Projected Data Team Budget in 2023  

Only 40% of respondents are planning to increase their budgets in 2023 (Figure 8). 42% are planning to keep it the same. When diving into this cohort by role (Figure 9), we see the more senior executives are leading the charge for greater budget, (63% for Head/VP of Data). These executives recognize the planned growth in resources shown in the previous slide (Figure 7), while the practitioners don’t have this vantage point, and aren’t familiar with plans to grow technology-wise. Despite an atmosphere in the market that things are slowing down, executives plan to grow both budgets and teams.   

Figure 8: Data Team Budget in 2023
Figure 8: Data Team Budget in 2023 

Figure 9: Plan to Increase Data Team Budget in 2023 by Role
Figure 9: Plan to Increase Data Team Budget in 2023 by Role 


Demographics 

Figure 10: Industry
Figure 10: Industry 

Figure 11: Department
Figure 11: Department 

Figure 12: Role
Figure 12: Role 

Figure 13: Company Size
Figure 13: Company Size 

Figure 14: Cloud Provider
Figure 14: Cloud Provider 


Contact Firebolt

For more information about Firebolt

Contact us
Send me as pdf