Data engineering is a critical process for any business that relies on data to make decisions. As more businesses depend on data, the need for effective and efficient data management solutions has never been greater.
dbt Labs recently announcement that its data management platform, dbt, now supports data transformation in Python. In addition to SQL capabilities, this new capability will enable data teams to tackle statistical analysis and predictive modeling. Before that, working with Python and SQL required separate workflows and infrastructure.
“We created dbt to harness the power of the modern cloud data platform and empower all data practitioners to participate in the data transformation process. Six years ago, that meant working exclusively in SQL, the native warehouse language,” said Tristan Handy, Founder and CEO of dbt Labs. “Today, with advancements in data platforms, we are excited to bring the power and accessibility of dbt to a new set of data workloads.”
dbt Labs says the release of this new feature will allow the 16,000 organizations using dbt today to take advantage of Python features available on major cloud data platforms. This includes Snowflake’s Python Connector for Snowpark, BigQuery’s Serverless Spark, and Databricks’ SQL from Databricks.
Whether dbt users prefer SQL or Python depends on the task. However, having the option to use either language will give data teams the flexibility they need to get the job done.
“Snowflake’s partnership with dbt Labs has been instrumental in modern analytics as we strive to enable data teams to securely and collaboratively deploy SQL code into production,” said Torsten Grabs, director of product management at Snowflake. “With dbt Labs’ introduction of Python Models and Snowpark for Python from Snowflake, joint customers can now effortlessly combine the power of SQL and Python for modern analytics and benefit from the wealth of innovation in data processing in the Python community. This will make it even easier for analytics, data engineering, and data science teams to be productive and collaborative in the Data Cloud.”
Sudhir Hasbe, senior director of product management at Google Cloud, said dbt has been a flexible workflow for BigQuery customers to manage and drive their data transformations. He believed that support for Python processing in BigQuery would give customers and the data community even more ways to solve business problems with data.
Photo credit: iStockphoto/Andrey Suslov