Master Data Engineering using Azure Data Analytics

Focused View

Durga Viswanatha Raju Gadiraju

13:32:26

121 View

1. Introduction

1. Introduction.html

2. Setup Environment for Data Engineering using Azure

1. Setup VS Code on Windows.mp4

03:22

2. Setup Python 3.9 on Windows.mp4

05:42

3. Configure Environment Variable PATH for Python on Windows.mp4

04:17

4. Integrate VSCode with Python on Windows.mp4

05:01

3. Getting Started with Azure for Data Engineering

1. Sign up for Azure Portal.mp4

01:41

2. Sign up for Azure Subscription.mp4

04:13

3. Overview of Azure CLI and Azure Cloud Shell.mp4

03:24

4. Setup Azure CLI on Windows or Mac or Linux.mp4

03:06

5. Configure Azure CLI against Azure Portal Account.mp4

05:45

6. Overview of Cost Management and Billing in Azure Portal.mp4

03:16

7. Review Resources used by Azure Cloud Shell.mp4

02:18

4. Getting Started with Azure Resource Groups

1. Create Azure Resource Group using Azure Portal.mp4

03:57

2. Add Storage Account as Resource to Azure Resource Group.mp4

05:04

3. Overview of Azure Resource Groups and Resources.mp4

03:23

5. Setup Data Sets for Data Engineering

1. Download Data Sets for Data Engineering from Git Repository.mp4

02:10

2. Create Container with in Azure Storage Account.mp4

01:47

3. Review Upload Feature of Azure Storage Account using Azure Portal.mp4

01:21

4. Setup Azure Storage Explorer on Windows or Mac.mp4

02:57

5. Upload Local Folder into Azure Storage Container using Storage Explorer.mp4

04:08

6. Validate Data Sets using Azure Portal.mp4

02:15

7. Create ADLS Storage Account in Azure.mp4

02:03

8. Upgrade Azure Blob Storage to ADLS Gen 2.mp4

03:32

6. Getting Started with Azure Data Factory

1. Introduction to Getting Started with Azure Data Factory.mp4

02:07

2. Setup Azure Data Factory and Launch ADF Studio.mp4

02:25

3. Overview of Azure Data Factory Studio.mp4

02:21

4. Create ADF Linked Service to Storage Account.mp4

02:20

5. Create ADF Dataset using ADF Studio.mp4

04:06

6. Review ADF Dataset CSV Properties.mp4

05:30

7. Create Azure Dataset for Sink using Parquet.mp4

02:14

8. Understand the Schema of Data Set.mp4

01:50

9. Create Data Flow Source using Azure Dataset.mp4

04:29

10. Define Cache Sink to ADF Data Flow.mp4

01:52

11. Create ADF Pipeline for File Format Converter.mp4

03:05

12. Run and Review ADF Data Pipelines.mp4

03:54

13. Update ADF Data Flow with ADLS Dataset as Sink.mp4

04:41

14. Conclusion to Getting Started with Azure Data Factory.mp4

02:16

15. Exercise - Simple ADF Data Flow and Pipeline for Order Items.mp4

02:09

7. ADF Data Flow for ETL Logic to Compute Daily Product Revenue

1. Introduction to ADF Data Flow for ETL Logic to Compute Daily Product Revenue.mp4

02:25

2. Create Data Flow to Compute Daily Product Revenue.mp4

02:25

3. Filter Transformation in ADF Data Flow.mp4

04:21

4. Create ADF Pipeline to Validate Data Flow.mp4

03:44

5. Create ADF Integration Runtime to run ADF Pipelines.mp4

03:54

6. Validate Custom ADF Integration Runtime using ADF Pipeline.mp4

03:12

7. ADF Data Flow Filter Transformation using in.mp4

04:38

8. ADF Data Flow Join Trasformation between 2 Data Sets.mp4

02:38

9. Validate ADF Data Flow Join Transformation using ADF Pipeline.mp4

02:30

10. ADF Data Flow Aggregate Transformation to Compute Daily Product Revenue.mp4

05:31

11. ADF Data Flow Sink to Save Results to Azure Storage using Parquet.mp4

03:17

12. Run and Review ADF Pipeline with ETL Data Flow.mp4

05:40

13. Access JSON Code of ADF Data Flow and Pipeline.mp4

03:50

8. Run ADF Pipelines Dynamically using Parameters

1. Introduction to Running ADF Pipelines Dynamically using Parameters.mp4

01:54

2. Create ADF Data Set using Parameter for Dynamic Path.mp4

04:21

3. Define Parameter and Use in Filter Transformation of ADF Data Flow.mp4

04:08

4. Create ADF Pipeline with Parameter.mp4

04:16

5. Run ADF Pipeline with Parameters.mp4

03:31

9. Run Baseline ETL Loads using ADF Pipeline

1. Overview of Common ADF Pipeline Activities.mp4

02:46

2. Overview of ADF Pipeline ForEach.mp4

06:03

3. Create ADF Pipeline for Baseline load using ForEach and Execute Pipeline.mp4

04:48

4. Run ADF Pipeline for Baseline Load.mp4

03:26

10. Performance Tuning of ADF Data Flows and Pipelines

1. Introduction to Prerformance Tuning of ADF Data Flows and Pipelines.mp4

01:09

2. Create Integration Runtime with right Compute Size.mp4

03:18

3. Troubleshoot Performance Bottleneck of Baseline ADF Pipeline.mp4

03:28

4. Reduce Cluster Startup Time using Custom Integration Runtime.mp4

06:03

5. Using Paralllel in ADF Pipeline ForEach Activity.mp4

03:41

6. Troubleshoot Shuffling and Too Many Small Files Issue.mp4

06:28

7. Reduce Shuffle Partitions in ADF Data Flow Aggregate Transformation.mp4

05:04

8. Conclusion of Performance Tuning of ADF Pipelines and Data Flows.mp4

01:51

11. Getting Started with Azure SQL Database

1. Setup Azure SQL Database Server.mp4

03:30

2. Setup Database in Azure SQL Database Server.mp4

03:44

3. Overview of SQL Server Databases in Azure.mp4

02:07

4. Setup Azure Data Studio on Windows or Mac or Linux.mp4

02:19

5. Connect to Azure SQL Database using Azure Data Studio.mp4

02:15

12. ADF Data Copy to Copy Data From Files to SQL Server Tables

1. Create table in Azure SQL Database.mp4

02:53

2. Create Linked Service and Dataset for Azure SQL Database Table.mp4

04:03

3. Copy ADF Dataset into a folder.mp4

01:11

4. Create ADF Pipeline with Data Copy to Copy CSV Data to SQL Table.mp4

04:27

5. Define Mapping in ADF Data Copy.mp4

03:33

6. Merge from CSV to SQL Table using ADF Pipeline Data Copy.mp4

02:58

7. Exercise to Copy Data to SQL Table using ADF Data Copy.mp4

02:28

13. Getting Started with Azure Synapse Analytics

1. Create Azure Synapse Analytics Workspace.mp4

04:05

2. Getting Started with Azure Synapse Studio.mp4

02:35

3. Overview of Azure Synapse Serverless SQL Pool.mp4

01:18

4. Link Azure Storage Account with Azure Synapse Workspace.mp4

02:57

5. Generate Azure Synapse Query using ADLS Files.mp4

02:25

6. Run Queries against ADLS files using Azure Synapse Serverless Workspace.mp4

04:07

7. Integrating Azure Synapse Workspace and Data in ADLS Files.mp4

03:40

8. Specifying Schema for Azure Synapse Queries on ADLS Files.mp4

03:09

9. Creating External Tables on ADLS Files using Azure Synapse Workspace.mp4

04:02

10. Managing SQL Scripts in Azure Synapse Studio.mp4

02:29

11. Create Dedicated SQL Pool in Azure Synapse Workspace.mp4

02:38

12. Create Table and Copy Data into Azure Synapse Dedicated SQL Pool Database.mp4

04:20

13. Overview of Development Tools in Azure Synapse Studio.mp4

02:20

14. Copy Data into Azure Synapse Tables using Copy Data Tool.mp4

06:50

15. Exercise to get started with Azure Synapse Analytics.mp4

03:15

14. Build ADF Data Flow using Azure SQL and Synapse Analytics

1. Introduction to ETL Logic and Application Architecture.mp4

05:25

2. Overview of ETL using ADF Data Flow.mp4

03:34

3. Getting Started with Azure Data Factory for the ETL Logic.mp4

01:58

4. Review Linked Service and Datasets for Azure SQL Database and Tables.mp4

02:43

5. Create ADF Data Flow with Azure SQL Database Table as Source.mp4

04:18

6. Review ADF Data Flow Source Options for Database Table.mp4

03:11

7. Create ADF Data Flow Source using Azure SQL Query.mp4

04:28

8. Run ADF Pipeline with Source using Azure SQL Query.mp4

02:35

9. Define and Use Parameters in ADF Data Flow Source with Azure SQL Query.mp4

05:24

10. Run ADF Pipeline with SQL Query Source using Parameters.mp4

03:32

11. ADF Data Flow Join Orders and Order Items.mp4

05:02

12. Run ADF Data Flow with Join between Orders and Order Items.mp4

05:17

13. Troubleshoot Issue in ADF Data Flow Source Query.mp4

03:35

14. Run and Validate ADF Data Flow with fix.mp4

03:49

15. Add Aggregate to ADF Data Flow.mp4

03:16

16. Run ADF Pipeline to Compute Daily Product Revenue.mp4

05:33

17. Create Target Table in Azure Synapse Dedicated SQL Pool.mp4

03:04

18. Create Linked Service and Dataset for Azure Synapse Table.mp4

04:04

19. Update and Run ADF Data Flow Sink with Synapse Dataset.mp4

05:26

20. Pause or Delete Azure Synapse Dedicated SQL Pool.mp4

02:26

15. Getting Started with Azure Databricks

1. Create Azure Databricks Workspace using Premium Trial.mp4

01:42

2. Launch Azure Databricks Workspace or Environment.mp4

02:12

3. Getting Started with Databricks Clusters on Azure.mp4

04:43

4. Increase Azure Quota for Databricks Clusters.mp4

03:49

5. Getting Started with Databricks Notebook on Azure.mp4

04:04

6. Overview of Azure Databricks Workspace Infrastructure.mp4

06:28

7. Overview of Azure Databricks and other Azure Services.mp4

03:54

8. Delete Azure Databricks Workspace.mp4

03:41

9. Setup Databricks CLI on Windows.mp4

04:22

10. Configure Databricks CLI and Validate.mp4

06:17

11. Troubleshoot and Reconfigure Databricks CLI using Token.mp4

05:38

16. Integration of Azure Storage and Databricks

1. Introduction to Integration of Azure Storage and Databricks.mp4

02:00

2. Setup Databricks Personal Compute Cluster.mp4

01:44

3. Setup Data Set in DBFS using Databricks CLI Commands.mp4

05:09

4. Overview of fs magic in Databricks Notebooks.mp4

03:45

5. Access Files in Azure Storage Account using Credentials Passthrough.mp4

04:53

6. Access Files in Azure Storage Account using Access Key.mp4

03:31

7. Create Spark SQL View using Data in Azure Storage Accounts.mp4

06:00

8. Validate using Spark SQL and Exercise to Create Spark SQL View.mp4

02:25

9. Integration of Azure Storage and Pyspark Demo.mp4

06:17

17. Overview of Databricks Secrets

1. Introduction to Overview of Databricks Secrets.mp4

01:33

2. Managing Databricks Secrets using Databricks CLI Commands.mp4

03:38

3. Create Databricks Secret for Azure Storage Account Key.mp4

03:15

4. Access Secret Details in Databricks Applications using dbutils secrets APIs.mp4

06:05

5. Authenticate Application with Azure Storage Account using Databricks Secrets.mp4

05:13

6. Steps involved in using Databricks Secrets.mp4

02:56

18. Basic Transformations using Spark SQL

1. Process Data in DBFS using Databricks Spark SQL.mp4

05:03

2. Getting Started with Spark SQL Example using Databricks.mp4

04:44

3. Create Temporary Views using Spark SQL.mp4

06:34

4. Exercise to create temporary views using Spark SQL.mp4

01:27

5. Spark SQL Query to compute Daily Product Revenue.mp4

06:10

6. Save Query Result to DBFS using Spark SQL.mp4

04:25

19. Ranking using Spark SQL Windowing Functions

1. Ranking using Spark SQL Windowing Functions.mp4

01:31

2. Create Temporary View for ranking using Spark SQL Windowing Functions.mp4

01:36

3. Compute Global Rank using Spark SQL Windowing Functions.mp4

05:37

4. Compute Ranks Per Key using Spark SQL Windowing Functions.mp4

03:55

5. Difference Between rank and dense_rank.mp4

04:10

6. Filter on Ranks using Spark SQL Windowing Functions.mp4

06:59

20. Getting Started with PySpark Data Frame APIs

1. Overview of Pyspark Examples on Databricks.mp4

01:04

2. Process Schema Details in JSON using Pyspark.mp4

07:32

3. Create Dataframe with Schema from JSON File using Pyspark.mp4

06:03

4. Transform Data using Spark APIs.mp4

04:13

5. Get Schema Details for all Data Sets using Pyspark.mp4

04:08

6. Convert CSV to Parquet with Schema using Pyspark.mp4

05:01

21. Databricks Jobs and Workflows

1. Overview of Databricks Workflows.mp4

03:10

2. Pass Arguments to Databricks Python Notebooks.mp4

07:15

3. Pass Arguments to Databricks SQL Notebooks.mp4

03:16

4. Create and Run First Databricks Job.mp4

07:31

5. Run Databricks Jobs and Tasks with Parameters.mp4

05:40

6. Create and Run Orchestrated Pipeline using Databricks Job.mp4

06:53

7. Import ELT Data Pipeline Applications into Databricks Environment.mp4

02:56

8. Spark SQL Application to Cleanup Database and Datasets.mp4

03:52

9. Review File Format Converter Pyspark Code.mp4

05:11

10. Review Databricks SQL Notebooks for Tables and Final Results.mp4

03:57

11. Validate Applications for ELT Pipeline using Databricks.mp4

07:36

12. Build ELT Pipeline using Databricks Job in Workflows.mp4

09:22

13. Run and Review Execution details of ELT Data Pipeline using Databricks Job.mp4

05:00

22. Build ELT Pipelines using Databricks Jobs and Workflows

1. Overview of Databricks Workflows.mp4

03:10

2. Pass Arguments to Databricks Python Notebooks.mp4

07:15

3. Pass Arguments to Databricks SQL Notebooks.mp4

03:16

4. Create and Run First Databricks Job.mp4

07:31

5. Run Databricks Jobs and Tasks with Parameters.mp4

05:40

6. Create and Run Orchestrated Pipeline using Databricks Job.mp4

06:53

7. Import ELT Data Pipeline Applications into Databricks Environment.mp4

02:56

8. Spark SQL Application to Cleanup Database and Datasets.mp4

03:52

9. Review File Format Converter Pyspark Code.mp4

05:11

10. Review Databricks SQL Notebooks for Tables and Final Results.mp4

03:57

11. Validate Applications for ELT Pipeline using Databricks.mp4

07:36

12. Build ELT Pipeline using Databricks Job in Workflows.mp4

09:22

13. Run and Review Execution details of ELT Data Pipeline using Databricks Job.mp4

05:00

14. Cleanup Databricks Environment on GCP.mp4

03:17

23. Orchestrate Azure Databricks Applications using ADF Pipelines

1. Getting Started with ADF to integrate with Databricks.mp4

01:57

2. Create ADF Linked Service for Azure Databricks.mp4

02:58

3. Trigger Azure Databricks Application from ADF Pipeline.mp4

04:19

4. Develop Core Logic using Databricks Notebook for ADF Pipeline Integration.mp4

04:00

5. Overview of Parameters using Databricks Notebooks.mp4

03:32

6. Add Parameters to ADF Pipeline and Databricks Activity.mp4

03:11

7. Run ADF Pipeline with Databricks Activity using Parameters.mp4

04:12

24. Build Data Pipelines using ADF Pipelines and Databricks

1. Introduction to Data Pipelines using ADF Pipelines and Databricks.mp4

01:17

2. Code Review to Compute Daily Product Revenue.mp4

06:42

3. Overview of ADF Pipeline and Databricks Integration Features.mp4

04:17

4. Different Options for ADF Pipelines and Databricks Notebook Integration.mp4

06:37

5. Create Driver Databricks Notebook for ADF Pipeline.mp4

03:08

6. Create ADF Linked Service for Databricks Job Cluster.mp4

06:25

7. Create ADF Pipeline with Databricks Driver Notebook.mp4

04:38

8. Run ADF Pipeline with Databricks Driver Notebook.mp4

04:12

9. ADF Pipeline to Orchestrate Databricks Notebooks.mp4

07:37

10. ADF Pipeline to Orchestrate Databricks Notebooks.mp4

04:50

11. Orchestrate Databricks Applications using ADF Pipeline.mp4

04:58

12. Import ELT Data Pipeline Applications into Databricks Environment.mp4

02:56

Description

Learn Azure Storage for Data Lake, ADF for ETL, BigQuery for Data Warehouse, Databricks for Big Data Pipeline, etcs

What You'll Learn?

Data Engineering leveraging Services under Azure Data Analytics such as Azure Storage, Data Factory, Azure SQL, Synapse, Databricks, etc.
Setup Development Environment using Visual Studio Code on Windows
Building Data Lake using Azure Storage (Blob and ADLS)
Build Data Warehouse using Azure Synapse
Implement ETL Logic using ADF Data Flow with Azure Storage as Source and Target
In Depth Coverage of Orchestration using ADF Pipeline
Overview of Azure SQL and Azure Synapse Serverless and Dedicated Pool Features
Implement ETL Logic using ADF Data Flow with Azure SQL as Source and Azure Synapse as Target
Using Data Copy to copy data between different sources and targets
Performance Tuning Scenarios of ADF Data Flow and Pipelines
Build Big Data Solutions using Azure Databricks
Overview of Spark SQL and Pyspark Data Frame APIs
Build ELT Pipelines using Databricks Jobs and Workflows
Orchestrate Databricks Notebooks using ADF Pipelines

Who is this for?

Beginner or Intermediate Data Engineers who want to learn Key Azure Analytics Services for Data Engineering such as Azure Storage, ADF, Synapse, Databricks, etc

Intermediate Application Engineers who want to explore Data Engineering using Azure Analytics Services for Data Engineering such as Azure Storage, ADF, Synapse, Databricks, etc

Data and Analytics Engineers who want to learn Data Engineering Azure Analytics Services for Data Engineering such as Azure Storage, ADF, Synapse, Databricks, etc

Testers who want to learn key skills to test Data Engineering applications built using Azure Analytics Services for Data Engineering such as Azure Storage, ADF, Synapse, Databricks, etc

More details

Description
Data Engineering is all about building Data Pipelines to get data from multiple sources into Data Lakes or Data Warehouses and then from Data Lakes or Data Warehouses to downstream systems. As part of this course, I will walk you through how to build Data Engineering Pipelines using Azure Data Analytics Stack. It includes services such as Azure Storage (both Blob and ADLS), ADF Data Flow, ADFÂ Pipeline, Azure SQL, Azure Synapse, Azure Databricks, and many more.
As part of this course, first, you will go ahead and set up the environment to learn using VS Code on Windows and Mac.
Once the environment is ready, you need to sign up for Azure Portal. We will provide all the instructions to sign up for Azure Portal Account including reviewing billing as well as getting USD 200 Credit valid for up to a month.
We typically use Azure Storage as Data Lake. As part of this course, you will learn how to use Azure Storage as Data Lake along with how to manage the files in Azure Storage using tools such as Azure Storage Explorer.
ADF is used for both ETL as well as Orchestration. First, you will understand how to perform ETLÂ using ADF Data Flow. The source and target will be Files in Azure Storage Account. As part of this process, you will also learn how to set up Linked Services and Data Sets in ADF.
Once ADF Data Flow is ready, you will go ahead and build Pipeline for Orchestration using ADF Pipeline. You will also learn how to parameterize and also how to take care of baseline load.
You will also understand key performance tuning techniques using ADF Pipeline such as controlling the number of partitions, custom integration runtimes (IR), etc.
AzureÂ provides RDBMS as different services for Postgres, SQLÂ Server, etc. You will learn how to set up Azure SQL Once the Azure SQLÂ is set up, you will also understand how to create required tables and run queries against them.
ADF provides ADFÂ Data Copy to copy data from different sources and different targets. Once the Database tables are ready you will use ADF Data Copy to copy data into the tables.
Azure provides Synapse Analytics for Data Warehouse. You will get an overview of both serverless as well as dedicated pools. You will end up setting up Dedicated Pool for ETL using ADF.
Once Azure SQL and Azure Synapse are ready, you will build ETLÂ Pipeline using ADF Data Flow and Orchestrate using ADF Pipeline.
Azure Databricks is the service for Big Data Processing using Spark Engine. You will learn how to set up Azure Databricks, integrate with ADLS, and also managing secrets.
You will also get an overview of Spark SQLÂ and Pyspark Data Frame APIs using Azure Databricks.
You will also build ELT Pipeline using Databricks Jobs and Workflows where tasks are defined based on Pyspark as well as Spark SQL.
You will also understand how to build ADF Pipelines to orchestrate Databricks Notebooks.
Who this course is for:
Beginner or Intermediate Data Engineers who want to learn Key Azure Analytics Services for Data Engineering such as Azure Storage, ADF, Synapse, Databricks, etc
Intermediate Application Engineers who want to explore Data Engineering using Azure Analytics Services for Data Engineering such as Azure Storage, ADF, Synapse, Databricks, etc
Data and Analytics Engineers who want to learn Data Engineering Azure Analytics Services for Data Engineering such as Azure Storage, ADF, Synapse, Databricks, etc
Testers who want to learn key skills to test Data Engineering applications built using Azure Analytics Services for Data Engineering such as Azure Storage, ADF, Synapse, Databricks, etc

User Reviews

Rating

average 0

Total votes0

Focused display

20+ years of experience in executing complex projects using a vast array of technologies including Big Data and the Cloud.ITVersity, Inc. - is a US-based organization that provides quality training for IT professionals and we have a track record of training hundreds of thousands of professionals globally.Building an IT career for people with required tools such as high-quality material, labs, live support, etc to upskill and cross-skill is paramount for our organization.At this time our training offerings are focused on the following areas:* Application Development using Python and SQL* Big Data and Business Intelligence* Cloud* Datawarehousing, Databases

Udemy

View courses Udemy

Students take courses primarily to improve job-related skills.Some courses generate credit toward technical certification. Udemy has made a special effort to attract corporate trainers seeking to create coursework for employees of their company.