Masterclass on Azure Data Factory - Data Engineer for 2024

Focused View

Atchyut Kumar

16:45:11

0 View

01 - Initial Concepts

001 Data Factory creation.mp4

07:38

002 Resource Group.mp4

05:25

003 Top Level Concepts in ADF.mp4

08:22

004 Linked Service and Datasets.mp4

11:41

005 Blob Storage.mp4

14:04

006 Variables in ADF.mp4

06:24

007 Parameters.mp4

12:07

008 JSON Structure.mp4

10:21

009 Integration Runtime.mp4

09:31

02 - Pipeline Activities

001 Copy Activity.mp4

17:38

002 Append Variable Activity.mp4

10:28

003 Delete Activity.mp4

10:21

004 Execute Pipeline Activity.mp4

07:10

005 Fail Activity.mp4

04:21

006 Get Metadata Activity.mp4

10:06

007 Lookup Activity.mp4

12:21

008 Set Variable Activity.mp4

12:23

009 Wait Activity.mp4

04:38

03 - Conditional and Loop Activities

001 Switch Activity.mp4

14:00

002 Filter Activity.mp4

09:02

003 ForEach Activity.mp4

22:59

004 Until Activity.mp4

15:04

005 If Condition.mp4

10:17

04 - Data Flow Transformations

001 Conditional Split Transformation.mp4

19:20

002 Exists Transformation.mp4

16:01

003 Union Transformation.mp4

22:16

004 Join Transformation.mp4

24:28

005 Aggregate Transformation.mp4

13:12

006 Surrogate Key Transformation.mp4

12:01

007 Select Transformation.mp4

10:41

008 Lookup Transformation.mp4

15:11

009 Derived Column Transformation.mp4

18:03

010 Pivot Transformation.mp4

17:32

011 Unpivot Transformation.mp4

15:27

012 Rank Transformation.mp4

14:54

013 Window Transformation.mp4

22:13

014 Flatten Transformation.mp4

12:58

015 Filter Transformation.mp4

13:16

05 - Real Time Scenarios

001 Part 1 - Copy files dynamically from one storage location to other.mp4

10:36

002 Part 2 - Copy files dynamically from one storage location to other.mp4

10:01

003 Set Variable self assignment handling.mp4

15:16

004 Fetch file names from folder and assign to variable.mp4

11:19

005 Copying specific file pattern files from Source to Sink.mp4

16:39

006 ADF pipeline logging.mp4

08:41

007 Process CSV file and split single column into multiple columns.mp4

14:23

008 Copy Missed files from Source to Destination.mp4

28:38

009 Run ADF pipeline manaully-Call pipeline from other ADF pipeline (with REST API).mp4

12:32

06 - Email Notification with ADF and Logic App

001 Email notification theory.mp4

14:01

002 Logic App Creation in Azure.mp4

02:41

003 Logic App and ADF Configuration for Email notification.mp4

30:56

004 Email notification configuration before and after pipeline with Stored procedure.mp4

48:25

07 - Data Load for Multiple Tables and multiple approaches

001 Full Load and Incemental load Theory.mp4

08:07

002 Copying multiple tables from database to database using ADF array.mp4

55:45

08 - Slowly Changing Dimensions

001 Fact and Dimension tables introduction.mp4

06:37

002 Slowly Chaning Dimension - Type 1.mp4

39:39

003 Slowly Changing Dimension - Type 2.mp4

25:58

004 Slowly Changing Dimension - Type 6.mp4

06:39

09 - Logs Maintenece in ADF

001 Logs Table Creation.mp4

08:07

002 Starting Logs Stored Procedure Creation.mp4

10:38

10 - Azure Key Vault

001 Azure Key Vault - Creation.mp4

10:10

11 - Azure Databricks - PySpark

001 Azure Databricks Service -Creation.mp4

03:45

002 JDBC Connection -How to connect to SQL server & read table from Azure Databricks.mp4

08:31

003 Mount to ADLS or Blob from Azure Databricks using PySpark.mp4

10:15

004 Read CSV file from DBFS with schema and header.mp4

11:32

005 withColumn() Transformation Function.mp4

38:25

12 - Azure Databricks - Delta Lake & Delta Table

001 Delta lake introduction.mp4

02:17

002 Datawarehouse Vs Data Lake Vs Delta Lake.mp4

12:28

003 Delta Table Architecture.mp4

05:47

004 Delta Table creation and observing its structure.mp4

22:29

Description

Learn ETL & Big Data Processing - Azure Data Factory(ADF), SQL Server, Blob, Datalake, REST API, PySpark, Databricks

What You'll Learn?

You will learn how to build a real time data pipeline in Azure Data Factory (ADF).
You will learn how to transform data using Data Flows in Azure Data Factory (ADF) and load into ADLS2, Blob storage.
You will learn how to ingest JSON data from SQL Server to API end point.
You will learn how to build production ready pipelines and good practices and naming standards.
You will learn how to monitor pipelines using Azure Data Factory (ADF), Azure Monitor and Log Analytics with a real-world project.
You will learn about manual/Triggers in Azure Data Factory (ADF) and how to use them to schedule the data pipelines.
Create Data driven, fully integrated, dynamic and automated production grade pipeline creation and orchestration.
Data Warehousing Concepts like Fact & dimension tables, SCD type1, type2, Incremental loading and implement using ADF.
Connect & copy data from On Premise data stores, On premise SQL server via Self Hosted Integration Runtime to cloud.
How to ingest data from sources such as REST API, Azure Blob Storage, SQL DB into Azure Data Lake Gen2 using Azure Data Factory (ADF)
Learn Azure Data Factory(ADF) with real time projects - ADF, SQL Server, Blob Storage, Datalake G1,G2, REST API.

Who is this for?

For all levels.

Students, Working Professionals

What You Need to Know?

Basic concepts about cloud computing will be useful, but not mandatory.

An Azure Account is required, If not, we will create a free account in the course.

More details

Description
Welcome to the Course "Azure Data Factory - Data Engineer : Real Time Projects".
Learn ETL and Big Data Processing on Azure with Azure Data Factory V2

Azure Data Factory (ADF) is one of the in-demand data engineering tool in the Cloud. This course has been taught with covering a data engineering solutions using ADF for real world scenarios.
Once you complete this course including assignments, quizzes, you will be in a position to take up or handle confidently any kind of projects one equal to 10+ years of experience.
Entire course built based on real time scenarios which are going to use very frequently and common to any type of projects on various domains.
This course will gives you necessary skills and help you greatly to pass for Certification exams like DP200 and DP203.

What this course will teach you?
You will learn how to build a real time data pipeline in Azure Data Factory (ADF).
You will learn how to transform data using Data Flows in Azure Data Factory (ADF) and load into ADLS2, Blob storage.
You will learn how to ingest JSON data from SQL Server to API end point.
You will learn how to build production ready pipelines and good practices and naming standards.
You will learn how to monitor pipelines using Azure Data Factory (ADF), Azure Monitor and Log Analytics with a real-world project.
You will learn about manual/Triggers in Azure Data Factory (ADF) and how to use them to schedule the data pipelines.
Create Data driven, fully integrated, dynamic and automated production grade pipeline creation and orchestration.
Data Warehousing Concepts like Fact & dimension tables, SCD type1, type2, Incremental loading and implement using ADF.
Connect & copy data from On Premise data stores, On premise SQL server via Self Hosted Integration Runtime to cloud.
How to ingest data from sources such as REST API, Azure Blob Storage, SQL DB into Azure Data Lake Gen2 using Azure Data Factory (ADF)
Who this course is for:
For all levels.
Students, Working Professionals

User Reviews

Rating

average 0

Total votes0

Focused display

Microsoft Azure

Atchyut Kumar

Instructor's Courses

Atchyut Kumar is a renowned programming and GATE CS/IT expert!With a Masters from the prestigious National Institute of Technology-Calicut and a remarkable GATE CS/IT top ranker with an exceptional 99.97 percentile, Atchyut brings a wealth of knowledge and expertise to our Azure Data Engineering training.Having spent 6+ years in database development and as a skilled Data Engineer, Atchyut possesses a remarkable 7+ years of experience as a GATE faculty. In total, he boasts an impressive 13+ years of teaching, research, and industry experience, having successfully mentored over 50,000 students through both classroom and online courses.Atchyut is not only an expert in problem-solving techniques, algorithms, and competitive programming, but he also holds a specialization in data integration, transformation, and consolidation from various structured, unstructured, and streaming data systems. This unique skill set enables him to construct suitable schemas for building robust analytics solutions.His unparalleled guidance has paved the way for many of his former students, who are now thriving in esteemed organizations such as Amazon, Samsung, Oracle, Google, Adobe, and many others. Some have even pursued higher studies, such as MS, M Tech, or Ph.D., further emphasizing the quality and impact of his teaching.

Udemy

View courses Udemy

Students take courses primarily to improve job-related skills.Some courses generate credit toward technical certification. Udemy has made a special effort to attract corporate trainers seeking to create coursework for employees of their company.