Ultimate Azure Data Factory: Cloud Data Engineering

Focused View

Aravind Suri

7:52:22

12 View

1. Overview

1. Welcome.mp4

01:21

2. What you will learn.mp4

03:00

3. Goal of this course.mp4

01:33

4. Commitment.mp4

00:52

5.1 ADFTutorial.zip

5. Course Materials.mp4

00:33

6.1 01-Chapter1 - Course Overview.pdf

6.2 02-Chapter2 - Introduction to Azure Data Factory.pdf

6.3 03-Chapter3 - Project Overview.pdf

6.4 04-Chapter4 - Environment.pdf

6.5 05-Chapter5 - Building a Data Pipeline.pdf

6.6 06-Chapter6 - Pipeline Activities and Parameters.pdf

6.7 07-Chapter7 - Mapping Data Flows.pdf

6.8 08-Chapter8 - Implementing Flowlets.pdf

6.9 09-Chapter9 - Controlling Pipeline Flow.pdf

6.10 10-Chapter10 - Building the Data Warehouse - Part 1.pdf

6.11 11-Chapter11 - Building the Data Warehouse - Part2.pdf

6.12 12-Chapter12 - Building the Delta Lake.pdf

6.13 13-Chapter13 - Presentation Layer.pdf

6.14 14-Chapter14 - Overview of Triggers.pdf

6.15 15-Chapter15 - Monitoring.pdf

6.16 16-Chapter16 - Conclusion.pdf

6. Course Slides.html

2. Introduction

1.1 02-Chapter2 - Introduction to Azure Data Factory.pdf

1. Introduction to Azure Data Factory.mp4

00:18

2. Why Azure Data Factory.mp4

00:42

3. What is Azure Data Factory.mp4

01:03

4. Benefits of Azure Data Factory.mp4

01:57

5. Azure Account.mp4

02:05

6. User Interface Azure Portal.mp4

10:32

7. Module Summary.mp4

00:50

3. Project Overview

1.1 03-Chapter3 - Project Overview.pdf

1. Hands-On Project Overview.mp4

00:26

2. Business Case for the Project.mp4

01:33

3. Solution Requirements.mp4

01:25

4. Architectural Patterns.mp4

01:56

5. Modern Data Warehouse Architecture.mp4

01:36

6. Hands-On Project Architecture.mp4

02:13

7. Repositories.mp4

01:34

8. Module Summary.mp4

00:52

4. Environment

1.1 04-Chapter4 - Environment.pdf

1. Module Overview.mp4

00:19

2. Software Tools.mp4

01:32

3. Software Tools Setup.mp4

07:21

4. Azure Resources.mp4

00:45

5. Setup Azure Resources.mp4

01:54

6. Setup Azure Resources in Azure Portal.mp4

00:39

7. Setup Azure Resource Group.mp4

01:06

8. Setup Azure Data Lake Storage.mp4

01:41

9. Setup Azure Data Factory Resource.mp4

01:21

10. Setup Azure Sql DB Resource.mp4

03:03

11. Review Azure Resources.mp4

03:05

12. Setup Azure Data Studio.mp4

02:05

13. Setup Azure Storage Explorer.mp4

01:51

14. Module Summary.mp4

00:52

5. Building a Data Pipeline

1.1 05-Chapter5 - Building a Data Pipeline.pdf

1. Module Overview.mp4

00:31

2. Building Blocks of Azure Data Factory - Main Components.mp4

02:21

3. Building Blocks of Azure Data Factory - Pipelines and Activities.mp4

01:04

4. Building Blocks of Azure Data Factory - How they Tie Together.mp4

01:26

5. Azure Data Factory User Interface - Main Page.mp4

01:23

6. Azure Data Factory User Interface - Authoring Canvas.mp4

04:09

7. Data Sources.mp4

01:54

8. Data Sources - Data Ingestion.mp4

00:29

9. Data Sources - Data Organization.mp4

01:19

10. Building the Data Pipeline.mp4

00:26

11. Building the Data Pipeline - Creating the Containers.mp4

01:20

12. Building the Data Pipeline - Creating the Pipeline.mp4

03:01

13. Building the Data Pipeline - Review and Organize.mp4

02:53

14. Importing Semi-Structured Data.mp4

00:24

15. Importing Semi-Structured Data - Building the Pipeline.mp4

03:39

16. Importing Semi-Structured Data - Organizing the Pipeline.mp4

01:07

17. Importing Semi-Structured Data - Recap of the Lesson.mp4

01:58

18. Naming Conventions.mp4

04:50

19. Module Summary.mp4

00:38

6. Pipeline Activities and Parameters

1.1 06-Chapter6 - Pipeline Activities and Parameters.pdf

1. Module Overview.mp4

01:10

2. Activities.mp4

03:13

3. Activity Dependencies.mp4

01:15

4. Activity Dependencies - Examples.mp4

03:50

5. Copy Activity.mp4

02:26

6. Copy Activity Concepts - Examples.mp4

04:06

7. Expressions and Variables.mp4

02:52

8. Expressions and Variables - Examples.mp4

02:29

9. Parameters.mp4

02:40

10. Parameters - Examples.mp4

03:22

11. Azure Key Vault - Overview.mp4

01:52

12. Azure Key Vault - Setup.mp4

03:40

13. Azure Key Vault - Create Linked Service.mp4

02:19

14. Importing Semi-Structured Data.mp4

12:46

15. Module Summary.mp4

01:09

7. Mapping Data Flows

1.1 07-Chapter7 - Mapping Data Flows.pdf

1. Module Overview.mp4

00:42

2. Introduction to Mapping Data Flows.mp4

02:48

3. Scenarios for Mapping Data Flows.mp4

04:40

4. User Interface of Mapping Data Flows.mp4

03:01

5. User Interface of Mapping Data Flows - Debug Feature.mp4

02:13

6. Implementing a Mapping Data Flow - Overview.mp4

01:21

7. Implementing a Mapping Data Flow - Pipeline and Data Sources.mp4

03:01

8. Implementing a Mapping Data Flow - Adding Transformations.mp4

05:09

9. Implementing a Mapping Data Flow - Pipeline Execution.mp4

03:56

10. Mapping Data Flow - Concepts.mp4

01:05

11. Mapping Data Flow - Concepts Example.mp4

05:54

12. Performance of Mapping Data Flows - Integration Runtime.mp4

02:44

13. Performance of Mapping Data Flows.mp4

03:41

14. Module Summary.mp4

00:59

8. Implementing Flowlets

1.1 08-Chapter8 - Implementing Flowlets.pdf

1. Module Overview.mp4

00:27

2. Introduction to Flowlets.mp4

02:13

3. Scenarios for Flowlets.mp4

01:48

4. User Interface of Flowlets - Overview.mp4

01:08

5. User Interface of Flowlets - Create a Demo Flowlet.mp4

02:40

6. Implementing a Flowlet - Create Flowlet.mp4

03:40

7. Implementing a Flowlet - Use the Flowlet.mp4

00:59

8. Module Summary.mp4

00:25

9. Controlling Pipeline Flow

1.1 09-Chapter9 - Controlling Pipeline Flow.pdf

1. Module Overview.mp4

00:26

2. Asserts.mp4

03:25

3. Implementing Asserts - Assert Expect True.mp4

03:26

4. Implementing Asserts - Identifying Error Rows.mp4

02:00

5. Implementing Asserts - Processing Error Rows.mp4

04:47

6. Error Handling Overview.mp4

02:29

7. Implementing Error Handling - Fail Activity.mp4

02:38

8. Implementing Error Handling - Capturing Errors.mp4

02:23

9. Implementing Error Handling - Logging Errors.mp4

02:23

10. Implementing Error Handling - Review of Error Pipeline.mp4

02:38

11. Integrating Data Quality and Error Handling.mp4

04:43

12. Building Pipelines using Pre-Built Templates.mp4

05:31

13. Module Summary.mp4

00:41

10. Building the Data Warehouse - Part 1

1.1 10-Chapter10 - Building the Data Warehouse - Part 1.pdf

1. Module Overview.mp4

00:38

2. Data Warehouse Overview.mp4

02:25

3. Data Warehouse Models.mp4

01:55

4. Data Warehouse Vino World.mp4

01:43

5. Data Process.mp4

02:47

6. Building the Azure Sql Database - Create the Stage Tables.mp4

02:22

7. Building the Azure Sql Database - Create the DW Tables.mp4

01:21

8. Building the Staging Layer Master Data - Master data.mp4

02:56

9. Building the Staging Layer Master Data - Product data.mp4

02:22

10. Building the Staging Layer Master Data - Metadata approach.mp4

04:53

11. Building the Staging Layer Master Data - Create Parameter Datasets.mp4

03:56

12. Building the Staging Layer Master Data - Create Metadata Pipeline.mp4

05:12

13. Building the Staging Layer Master Data - Pipeline execution.mp4

03:04

14. Building the Staging Layer Transaction Data.mp4

05:19

15. Building the Staging Layer Product Data - Combine Product Data.mp4

02:22

16. Building the Staging Layer Transaction Data - Combine Sales Data.mp4

02:18

17. Module Summary.mp4

00:56

11. Building the Data Warehouse - Part 2

1.1 11-Chapter11 - Building the Data Warehouse - Part2.pdf

1. Module Overview.mp4

00:28

2. Dimensions - Overview of Dimensions.mp4

02:41

3. Dimensions - Slowly Changing Dimensions.mp4

03:06

4. Dimensions - Master Dimensions and SCD Type.mp4

01:47

5. Building Type1 Dimensions - Using Data Flows.mp4

04:17

6. Building Type 1 Dimensions - Pipeline Review.mp4

05:09

7. Building Type 1 Dimensions - Using Stored Procedures.mp4

04:16

8. Dimensions - Overview of Type2 Dimensions.mp4

02:16

9. Building Type 2 Dimensions - Product Dimension.mp4

01:30

10. Building Type 2 Dimensions - Using Data Flows - Step1.mp4

05:23

11. Building Type 2 Dimensions - Using Data Flows - Step2.mp4

06:35

12. Building Type 2 Dimensions - Pipeline Review.mp4

03:39

13. Building Type 2 Dimensions - Using Stored Procedures.mp4

06:05

14. Building Dimensions - Build remaining dimensions.mp4

01:24

15. Facts - Overview.mp4

01:41

16. Building Facts.mp4

04:40

17. Data Warehouse Review and Data Analysis.mp4

02:54

18. Module Summary.mp4

01:02

12. Building the Delta Lake

1. Module Overview.mp4

00:51

2. Recap of what we implemented.mp4

01:02

3. What we will implement.mp4

00:52

4. Azure Databricks.mp4

00:26

5. What is Azure Databricks.mp4

01:09

6. Core Artifacts of Azure Databricks.mp4

00:50

7. Setup Azure Databricks.mp4

00:21

8. Setup Databricks Resource.mp4

01:35

9. Databricks UI Overview.mp4

02:59

10. Databricks Cluster Overview.mp4

02:05

11. Create Databricks Cluster.mp4

03:57

12. Azure Service Principal and Access to Data Lake Storage.mp4

03:15

13. Mount Azure Data Lake Storage.mp4

02:44

14. Overview of Delta Lake Implementation.mp4

00:58

15. What is a Delta Lake.mp4

01:26

16. Create Data Source for the Delta Table.mp4

03:37

17. Create Delta Table.mp4

02:57

18. Load Delta Table.mp4

01:42

19. Update Delta Table.mp4

03:16

20. Delta Table Concepts.mp4

04:38

21. Create Linked Service to Databricks from Data Factory.mp4

02:27

22. Executing Databricks Notebook from Data Factory.mp4

04:19

23. Module Summary.mp4

00:52

13. Presentation Layer

1.1 13-Chapter13 - Presentation Layer.pdf

1. Module Overview.mp4

00:51

2. Overview - Modern Data Warehouse.mp4

00:47

3. Overview - What we implemented.mp4

01:02

4. Overview - What we will implement.mp4

00:28

5. PowerBI - Installation.mp4

02:24

6. PowerBI - Overview.mp4

02:13

7. PowerBI - Connecting to the Data Warehouse.mp4

02:45

8. PowerBI - Building the Tabular Model.mp4

01:51

9. PowerBI - Building the Report.mp4

05:03

10. PowerBI - Report Requirements.mp4

00:56

11. PowerBI - Report Review.mp4

02:02

12. Module Summary.mp4

00:55

14. Triggers

1.1 14-Chapter14 - Overview of Triggers.pdf

1. Module Overview.mp4

00:55

2. Overview of Triggers.mp4

02:50

3. Approach to Pipeline Execution.mp4

01:08

4. Implementing a Master Pipeline.mp4

01:53

5. Executing the Master Pipeline.mp4

01:31

6. Implementing Event-based triggers.mp4

03:23

7. Executing Event-based Triggers.mp4

03:07

8. Scheduling Pipelines.mp4

03:00

9. Creating a Tumbling Window Trigger.mp4

05:25

10. Module Summary.mp4

00:46

15. Monitoring

1.1 15-Chapter15 - Monitoring.pdf

1. Module Overview.mp4

00:37

2. Executing Event-based triggers.mp4

03:07

3. Overview of Data Factory Monitoring.mp4

02:40

4. What do we monitor in Azure Data Factory.mp4

02:17

5. Visual Monitoring in Azure Data Factory.mp4

06:59

6. Pipeline Recovery.mp4

05:12

7. Setup Alerts.mp4

04:29

8. Validate the Alert.mp4

03:21

9. Metrics.mp4

02:37

10. Module Summary.mp4

01:08

16. Section 16 Conclusion

1.1 16-Chapter16 - Conclusion.pdf

1. Summary.mp4

01:00

Description

Real world Modern Data Warehouse project for Data Engineers using Azure Data Factory, Sql, Data Lake, Databricks [DP203]

What You'll Learn?

You will learn how to build data pipelines in Azure Data Factory (ADF) through a step-by-step approach.
You will learn how to ingest data in different formats into Azure Data Lake Gen2 using Azure Data Factory (ADF)
You will learn how to use and build various types of transformations in Azure Data Factory (ADF)
You will learn hands-on implementations of building generic artifacts in Azure Data Factory (ADF) such as Flowlets and Templates
You will learn how to transform data into the Medallion layers in Azure Data Lake Gen2 using Data Flows in Azure Data Factory (ADF)
You will learn how to implement ETL/ELT using Azure Data Factory (ADF) in order to implement a Data Warehouse
You will learn how to create generic metadata driven pipelines in Azure Data Factory (ADF) to implement the ETL/ELT processes
You will learn the concepts of the Modern Data Warehouse Architecture and the Delta Lake
You will learn the concepts of Slowly Changing Dimensions and how to implement them in Azure Data Factory (ADF)
You will learn how to load transformed data from Azure Data Lake Storage Gen2 to Azure SQL Database using Azure Data Factory (ADF)
You will learn how to implement a Delta Lake using Databricks Notebook Activity in Azure Data Factory (ADF) and load into Azure Data Lake Storage Gen2
You will learn how to transform your raw data into a finished data warehouse using Azure Data Factory (ADF) and then visualize it in PowerBI
You will learn how to build pipelines using good practices and naming standards as in a typical real-world data engineering project
You will learn how to implement different types of Triggers in Azure Data Factory (ADF) and how to schedule your data pipelines
You will learn how to monitor pipelines using Azure Data Factory (ADF), Azure Monitor, and how to recover from pipeline failures
By the end of this course you will have learnt all the topics required on Azure Data Factory to pass the Azure Data Engineer Associate Certification Exam DP203

Who is this for?

Beginners or Students who want to break into the Data Engineering field

Developers who want to learn Data Engineering

Data Engineers who want to learn how to implement a Modern Data Warehouse through a step-by-step approach

Data Engineers/Data Warehouse developers who want to get the skills necessary in implementing cloud based data engineering solutions

Data Engineers who want to understand how to build and end-to-end solution using Azure Data Factory (ADF)

What You Need to Know?

Basic understanding of Sql will be beneficial

Basic understanding of cloud computing will be beneficial

Experience in Azure is not required, we will learn this step by step within this course as we build the project

Understanding of data warehouses will be beneficial, but not necessary, we will learn it as we build the project in the course

An Azure account is required for the course, we will learn how you can create it during the course

More details

Description
Welcome!
Data engineering is a thriving focus in the IT industry, with Microsoft's Azure Data Factory emerging as a sought-after tool in cloud-based data engineering.
Join this course for a step-by-step journey into mastering Azure Data Factory (ADF). Using a real-world scenario of an e-commerce company grappling with data integration and insights, we'll explore the data of an online wine retailer, showcasing how implementing a modern data warehouse with ADF can provide solutions.
Distinguishing itself from other Udemy offerings on Azure Data Factory and Data Engineering Technologies, this course guides you hands-on in transforming raw data into a Modern Data Warehouse using Azure Data Factory (ADF). Upon completion, you'll gain proficiency in ADF, ready to tackle real-world data engineering projects.
Given the course's focus on real-world business scenarios, it adopts a sequential approach mirroring how such requirements unfold in actual projects. This method ensures you not only implement business needs but also grasp the technical concepts explained at each stage of implementing data pipelines with Azure Data Factory (ADF).
This course covers more than just modern data warehouse concepts like architecture, medallion layers, and delta lake. You'll also gain expertise in utilizing diverse Azure ecosystem solutions, including Azure Data Lake Storage, Azure SQL Database, and Azure Databricks. Additionally, you'll learn to visually represent the completed data warehouse through Power BI reports.
This course enables you to grasp concepts and skills assessed in the Azure Data Engineer Associate Certification exam DP203. While it equips you with the necessary skills, it's important to note that the course is not designed solely for certification passing but for comprehensive learning.
I appreciate your time, and I've crafted this course to be practical and focused. I aim for simplicity and conciseness, starting from the basics and ensuring proficiency in the technologies covered.

Currently the course teaches you the following:

Azure Data Factory
Constructing a contemporary Data Warehouse architecture for a data engineering solution involves utilizing Azure Data Engineering technologies like Azure Data Factory (ADF), Azure Data Lake Gen2, Azure SQL Database, Azure Databricks, Azure KeyVault, and Microsoft PowerBI.
Incorporating data from varied sources with diverse formats into Azure Data Lake Gen2 is achieved through the use of Azure Data Factory.
Comprehending Azure concepts, including resources and their provisioning methods.
Learning to incorporate and use tools such as Azure Storage Explorer, Azure Data Studio, and Visual Studio Code in the development workflow.
Implementing Azure Data Factory (ADF) pipelines using different control flow activities such as Get Metadata, ForEach, If Conditions, etc.
Using Parameters and Variables in Pipelines, Datasets and LinkedServices to create generic parameter driven pipelines in Azure Data Factory (ADF).
Using parameters in conjunction with Azure KeyVault to create generic parameter driven piplines in Azure Data Factory (ADF).
Implementing Mapping Data Flows to create transformation logic to handle a variety of transformation scenarios such as Filter, Conditional Split, Derived Column, Aggregate, Join,Â Select, and Sink transformation.
Developing universal components in data pipelines, such as Flowlets, and mastering the swift development of data processing needs through pre-built pipeline templates.
Learning how to implement error handling in data pipelines and controlling pipeline flow.
Implementig data quality rules using the Assert transformation within a data pipeline.
Implementing data pipelines to handle common slowly changing dimension scenarios such as SCD Type 1 and SCD Type 2.
Implementing data pipleines to implement a Fact table.
Learning how to debug data pipelines and resolving issues.
Implementing pipeline scheduling using different types of triggers such as Event Trigger, Schedule Trigger and Tumbling Window Trigger in Azure Data Factory (ADF)
Implementing Azure Data Factory pipelines to invoke Mapping Data Flows and executing them.
Creating ADF pipelines to execute Databricks Notebook activities to carry out transformations and implement a Delta Lake table.
Creating pipeline dependencies and using the Pipeline activity to orchestrate the ETL/ELT process.
Implementing trigger dependencies to understand how to chain pipelines and orchestrate the data flow.
Monitoring data pipelines, creating alert notifications, and reporting data factory metrics using Azure Data Factory Monitor.
Understanding how to monitor Azure Data Factory pipelines using Azure Monitor using specific Data FactoryÂ metrics.
Modern Data Warehouse
Understand the different types of Data Warehouse Architectures.
Understand the concepts of a Delta Lake.
Understand the Dimensional Model and a Star Schema based Data Warehouse.
Understand the concept of Medallion Layers and how to implement it within the Azure Data Lake Storage.
Azure Databricks
Understand the creation of an Azure Databricks Workspace, Databricks clusters, Mounting storage accounts, Creating Databricks notebooks, performing transformations using Databricks notebooks, and Invoking Databricks notebooks from Azure Data Factory.
Understand the implementation of a Delta Lake table using Azure Databricks Notebook activity from an Azure Data Factory pipeline.
Understand the concepts of Optimizing a Delta Lake Table, Time Travel, Vacuuming, and Delta Logs.
Azure Resources and Azure Storage Solutions
Learn the different approaches to creating Azure Resources.
Learn how to create an Azure Storage Account resource, creating containers, and how to upload data through the Azure Portal or through Azure Storage Explorer into the Azure storage resource.
Learn how to create an Azure SQL Database resource, understand the Pricing Tiers, Creating an Admin User, Creating Tables, Loading Data, Querying the database and interacting with Azure Sql Database through Azure Data Studio.
Who this course is for:
Beginners or Students who want to break into the Data Engineering field
Developers who want to learn Data Engineering
Data Engineers who want to learn how to implement a Modern Data Warehouse through a step-by-step approach
Data Engineers/Data Warehouse developers who want to get the skills necessary in implementing cloud based data engineering solutions
Data Engineers who want to understand how to build and end-to-end solution using Azure Data Factory (ADF)

User Reviews

Rating

average 0

Total votes0

Focused display

Hello! As a Big Data and Analytics Manager, I bring over two decades of expertise in designing and developing Data and Analytics Solutions across diverse industries, including Finance, Airlines, Pharmaceuticals, and AutomotiveBringing substantial expertise, I have successfully designed and implemented BI solutions for prominent global corporations like Prudential Securities, Air Canada, Goldman Sachs, Merck, and Daimler. My professional journey spans countries, including the USA, Canada, and Germany. I excel in guiding teams to deliver solutions across Classical Data Warehouse, Big Data, Streaming, Cloud BI, and Data Science domains.With over two decades of experience, Data Analytics is a passion of mine. Holding an MBA in Finance from the Richard Ivey School of Business in Canada and being certified in Risk Management by the Global Association of Risk Professionals (GARP), I bring substantial expertise. My background includes significant experience in Quantitative Finance, coupled with adeptness in utilizing various development tools.I've delivered projects on both on-premise and cloud platforms, including Azure. My expertise extends beyond technical aspects to encompass managerial experience in establishing Agile Data Analytic teams in Cloud and Big Data domains. As an avid reader and lifelong learner, I find joy in crafting technical solutions.Passionate about teaching, I find joy in sharing knowledge and witnessing student development. My teaching approach is grounded in real-world business requirements, guiding you through practical implementations. This method ensures a step-by-step understanding and ingrains concepts through real-world project experience. Completing my course will equip you with the knowledge and foundation to confidently engage in real projects in the field.

Udemy

View courses Udemy

Students take courses primarily to improve job-related skills.Some courses generate credit toward technical certification. Udemy has made a special effort to attract corporate trainers seeking to create coursework for employees of their company.