Major updates to the course since the launchDecember 2022 - Sections 3, 4 & 5 updated to reflect recent UIchanges to Azure Databricks. Also included lessons on additional functionality included by Databricks recently to Databricks clusters. . Welcome! I am looking forward to helping you with learning one of the in-demand data engineering tools in the cloud, Azure Databricks! This course has been taught with implementing a data engineering solution using Azure Databricks and Spark core for a real world project of analysing and reporting on Formula1 motor racing data. This is like no other course in Udemy for Azure Databricks. Once you have completed the course including all the assignments, I strongly believe that you will be in a position to start a real world data engineering project on your own and also proficient on Azure Databricks. I have also included lessons on Azure Data Lake Storage Gen2, Azure Data Factory as well as PowerBI. The primary focus of the course is Azure Databricks and Spark core, but it also covers the relevant concepts and connectivity to the other technologies mentioned. Please note that the course doesn’t cover other aspects of Spark such as Spark streaming and Spark ML. Also the course has been taught using PySpark as well as Spark SQL; It doesn’t cover Scala or Java. The course follows a logical progression of a real world project implementation with technical concepts being explained and the Databricks notebooks being built at the same time. Even though this course is not specifically designed to teach you the skills required for passing the Azure Data Engineer Associate Certification Exam DP203, it can greatly help you get most of the necessary skills required for the exam. I value your time as much as I do mine. So, I have designed this course to be fast-paced and to the point. Also, the course has been taught with simple English and no jargons. I start the course from basics and by the end of the course you will be proficient in the technologies used. Currently the course teaches you the followingAzure DatabricksBuilding a solution architecture for a data engineering solution using Azure Databricks, Azure Data Lake Gen2, Azure Data Factory and Power BICreating and using Azure Databricks service and the architecture of Databricks within AzureWorking with Databricks notebooks as well as using Databricks utilities, magic commands etcPassing parameters between notebooks as well as creating notebook workflowsCreating, configuring and monitoring Databricks clusters, cluster pools and jobsMounting Azure Storage in Databricks using secrets stored in Azure Key VaultWorking with Databricks Tables, Databricks File System (DBFS) etcUsing Delta Lake to implement a solution using Lakehouse architectureCreating dashboards to visualise the outputsConnecting to the Azure Databricks tables from PowerBISpark (Only PySpark and SQL)Spark architecture, DataSources API and Dataframe APIPySpark - Ingestion of CSV, simple and complex JSON files into the data lake as parquet files/ tables. PySpark - Transformations such as Filter, Join, Simple Aggregations, GroupBy, Window functions etc. PySpark - Creating local and temporary viewsSpark SQL - Creating databases, tables and viewsSpark SQL- Transformations such as Filter, Join, Simple Aggregations, GroupBy, Window functions etc. Spark SQL - Creating local and temporary viewsImplementing full refresh and incremental load patterns using partitionsDelta LakeEmergence of Data Lakehouse architecture and the role of delta lake. Read, Write, Update, Delete and Merge to delta lake using both PySpark as well as SQLHistory, Time Travel and VacuumConverting Parquet files to Delta filesImplementing incremental load pattern using delta lakeAzure Data FactoryCreating pipelines to execute Databricks notebooksDesigning robust pipelines to deal with unexpected scenarios such as missing filesCreating dependencies between activities as well as pipelinesScheduling the pipelines using data factory triggers to execute at regular intervalsMonitor the triggers/ pipelines to check for errors/ outputs.
