Data Engineering with GCP

Google Cloud Platform (GCP) has emerged as a premier leader in cloud analytics, making data engineering skills more critical than ever for modern business success. The current evolution of generative artificial intelligence (AI) and agentic AI has created a significant demand in the data engineering discipline since the accuracy and effectiveness of AI output primarily depend on the quality of data. Ensuring high-quality, curated data requires a robust and scalable data engineering platform that can cater to the velocity, veracity, and volume of data.

Google is a pioneer in data engineering solutions, which are provided through its GCP. Many of the Fortune 500 companies leverage GCP's services for transforming petabytes of data for analytics, AI, and machine learning (ML). This book begins with data engineering essentials like ETL, ELT, and big data roles before moving into GCP environment setup and security. You will learn BigQuery for data warehousing and SQL optimization, followed by real-time ingestion using Pub/Sub, Dataflow, and Datastream. You will learn to integrate machine learning via

Vertex AI pipelines. Finally, it will provide the skills to use the processed data for analytics, AI, and ML use cases.

After finishing this book, you will possess the technical competence to design, build, and monitor professional-grade data solutions on Google Cloud. You will be ready to tackle real-world challenges, from automating complex workflows to leveraging AI for predictive analytics in any enterprise environment.

What you will learn

● Develop highly scalable, modern data engineering solutions in GCP.

● Optimize BigQuery performance using advanced table partitioning and data clustering.

● Build streaming pipelines using Pub/Sub, Dataflow, and the Apache Beam framework.

● Deploy Spark and Hadoop clusters on Dataproc with GCS lakes.

● Apply data mesh, generative AI, and decentralized data strategies.

● Learn enterprise ETL and ELT architectures through managed Cloud Composer and Apache Airflow.

Who this book is for

This book is for data architects, data engineers, data analysts, and ML engineers working on transforming raw data to curated, quality data for enterprise consumption. It caters to beginners as well as experienced data professionals and students who want to become data professionals.

Table of Contents

1. Foundations of Data Engineering

2. Data Engineering Services in GCP

3. BigQuery Data Warehousing Service

4. Data Ingestion Using Pub/Sub and Dataflow

5. ETL and Orchestration Using Cloud Composer

6. Data Lakes Using Cloud Storage and Dataproc

7. Data Visualization Using BigQuery and Looker

8. Data Migration Using Database Migration Service

9. Data Integration and Machine Learning Pipelines in GCP

10. Cloud Monitoring, DevOps Automation and Best Practices

11. Data Exchange and Sharing Using BigQuery Sharing

12. Emerging Trends and Real-world Use Cases