Matillion, a provider of data transformation software for cloud data warehouses (CDWs), and IDG Research have released findings of an IDG Research MarketPulse survey, “Optimizing Business Analytics by Transforming Data in the Cloud.” The research exposes the challenges companies face and the strategies they use to prepare data for BI and analytics, with faster time-to-value for implementing analytics projects rising as the main driver for migrating to a cloud approach.
The survey polled more than 200 IT, data science, and data engineering professionals at North American organizations with at least 1,000 employees. The top takeaways include:
Enterprises struggle to use data volumes for actionable insights.
- On average, data volumes are growing at a rate of 63% a month
- 12% report that their data volumes are growing at 100% or more per month
- More than 20% of those surveyed report drawing from 1,000 or more data sources
Responsibility for exploiting data is divided between IT groups (55%) and business units (45%)
Enterprises are tapping the cloud to simplify and scale data management.
- While less than one-quarter (23%) of respondents have completely centralized their BI and analytics teams, virtually all plan to tap the cloud for data management, migrate or continue to migrate data to the cloud over the next 24 months
- 90% have already placed some data in CDWs
- 37% of organizational data is in CDWs, 35% is in on-premises data warehouses, and 25% is in offsite, non-CDWs
- Three CDWs dominate: Amazon Redshift (used by 54% of the respondents), Google BigQuery (50%), and Snowflake (26%)
- The top reason enterprises migrate their data to cloud platforms is faster time-to-value for implementing analytics projects
Despite the broad and growing use of CDWs, they aren’t a panacea; CDW adoption alone does not address all data analytics needs.
- More than 90% said it is challenging to make data available in a format usable for analytics
Respondents cited several obstacles slowing their data analytics projects, including a lack of necessary data granularity; manual coding of data pipelines; and difficulty connecting with multiple data sources
Part of the challenge is in how enterprises are using legacy ETL processes in a modern data environment.
- Many struggles with the sequence and process of data transformation; more than one-third (37%) manually code data into the necessary format before loading it into BI and analytics tools
- Only 28% load data into the cloud and then use the cloud platform to automate the transformation process
- Data portability (45%) and scalability (46%) top the list of perceived benefits of a modern approach to data transformation
