# Recent Google Cloud Updates to Delight Data Engineers
Written on
Chapter 1: Exciting Developments for Data Engineers
In October, Google introduced several intriguing updates aimed specifically at improving the workflow of Data Engineers. One of the most noteworthy advancements is the integration of BigQuery with Apache technologies. This collaboration promises to enhance data management and processing capabilities significantly.
This paragraph will result in an indented block of text, typically used for quoting other text.
Section 1.1: Unifying BigQuery and Apache
Google’s initiative to merge BigQuery with Apache Spark was previously discussed in detail in another article. This integration is set to foster a more seamless experience when working with large datasets.
Subsection 1.1.1: Analytics Hub Launch
In addition, Google has made its Analytics Hub broadly available. This innovative service within BigQuery facilitates secure data exchanges and the sharing of analytical resources both within and across various organizations. Through this platform, data providers can publish listings that point to shared datasets, enabling subscribers to access and engage with these resources.
Section 1.2: Enhanced Business Intelligence Features
Another significant enhancement relates to BigQuery and the Business Intelligence platform, Looker Studio (formerly Data Studio). Users can now monitor critical metrics, such as BI Engine Top Tables Cached Bytes, BI Engine Query Fallback Count, and Query Execution Count, directly from their dashboards. The BigQuery BI Engine provides rapid in-memory analytics, allowing for analysis of stored data with response times under one second.
Chapter 2: New Functionalities for Data Engineers
The first video titled "Google Cloud Professional Data Engineer Recertified in 2020, 2022 & 2024: My Observations on the Exam" explores insights and experiences related to the certification exam, further enriching the understanding of Google Cloud’s offerings for data professionals.
Additionally, Google has unveiled a significant feature for Data Engineers: Remote Functions. This allows users to call functions from Cloud Functions or Cloud Run directly within their Google Standard SQL queries, now widely available.
Furthermore, the Data Exploration Workbench in Dataplex is also now generally accessible. This tool provides a serverless data exploration experience powered by Spark, allowing access to Spark SQL scripts and Jupyter notebooks. This workbench enables Data Consumers to focus on deriving insights rather than juggling various tools and platforms.
The second video titled "Reviewing Google's Data Engineer Certificate - Is It Worth Your Time And What Will You Learn?" assesses the value and learning outcomes of Google's Data Engineer certification, offering additional context for professionals considering this path.
In summary, these updates from Google this October are particularly beneficial for Data Engineers, enhancing their capabilities and workflows.
Sources and Further Readings
[1] Google, BigQuery release notes (2022) [2] Google, Analytics Hub (2022) [3] Google, What’s new with Google Cloud (2022)