Azure Data Factory +Synapse Analytics End to End ETL project

Focused View

Shanmukh Sattiraju

4:21:16

70 View

1 - Introduction

1 - Welcome to the course.mp4

00:28

2 - Main Focus and Prerequisites.mp4

01:42

3 - Services used in this project.mp4

01:16

4 - Project Overview.mp4

02:00

5 - Project Architecture.mp4

02:04

6 - Additional section on CICD setup for Azure Data Factory.mp4

00:36

7 - Course Structure.mp4

01:23

8 - Understanding OTT dataset.mp4

02:12

9 - Project-Files.zip

9 - Resources.html

2 - Environment Setup

10 - Environment setup Intro.mp4

00:35

11 - Creating a budget for our Project.mp4

06:42

12 - Creating a resource group.mp4

01:58

13 - Creating an Azure Data Factory.mp4

02:47

14 - Creating an Azure Datalake Storage Gen2.mp4

03:05

15 - Creating an Azure Synapse Analytics Workspace.mp4

05:38

16 - Suggestion on Saving costs for Azure SQL Database.mp4

00:39

17 - Creating an Azure SQL Database.mp4

05:03

18 - Installing Power BI Desktop.mp4

02:08

3 - Data Ingestion

19 - Data Ingestion Intro.mp4

01:00

20 - Data Ingestion Integration Runtimes.mp4

02:48

21 - Data Ingestion What is Self Hosted Integration Runtime.mp4

03:08

22 - Overview of Onpremise data source and Datalake.mp4

03:55

23 - Downloading and installing Self Hosted IR in Onpremise Environment.mp4

07:00

24 - Creating and adding Secrets to Azure Key vault.mp4

04:40

25 - Creating Linked Service for Azure Key vault Demo.mp4

02:41

26 - Creating Linked Service and Dataset for Onpremise File Storage.mp4

05:15

27 - Creating Linked Service and Dataset for Azure Datalake.mp4

04:58

28 - Creating Copy Activity to copy all files from Onpremise to Azure Datalake.mp4

05:16

29 - Incremental data loading using Last Modified Date of File.mp4

07:23

30 - Incremental Load based on File Name Demo.mp4

04:07

31 - Incremental Data loading based on Filename Practical.mp4

15:39

4 - Transformation

32 - Transformation Intro.mp4

01:28

33 - Azure Synapse Analytics Introduction.mp4

02:29

34 - Assigning Role for Synapse in Azure Datalake Demo.mp4

01:50

35 - Assigning role and creating linked service in Azure Synapse Analytics Practical.mp4

03:37

36 - Reading CSV files from ADLS from Synapse Notebook Practical.mp4

07:42

36 - microsoft official documentation link for code.zip

37 - Stop spark session manually to save cost.mp4

00:55

38 - Identify and delete duplicate rows Demo.mp4

01:27

39 - Identify and remove duplicate rows Practical.mp4

03:44

40 - Identify and Remove or Replace NULL values Demo.mp4

01:20

41 - Identify and Remove or Replace NULL values Practical.mp4

07:11

42 - New column based on IMDB Rating Demo.mp4

01:23

43 - New column based on IMDB Rating Practical.mp4

04:32

44 - New column based on Runtime in Hours Demo.mp4

00:41

45 - New column based on Runtime in Hours Practical.mp4

02:19

46 - Practise Activity for creating a new column.html

47 - Solution PySpark code for Practise Activity.html

48 - Changing data types from String to Date Type Demo.mp4

00:56

49 - Changing String to Date Data Type Practical.mp4

04:05

50 - Writing transformed data to ADLS Demo.mp4

01:04

51 - Writing transformed data to Datalake Practical.mp4

04:00

52 - End of Writing Transformation Code.mp4

01:15

53 - Calling Synapse Notebook Activity from Azure Data Factory Demo.mp4

01:23

54 - Calling Synapse notebook from Azure Data Factory Practical.mp4

06:12

55 - Transformation Conclusion.mp4

00:26

5 - Data Loading

56 - Data Loading Section Intro.mp4

00:53

57 - Installing and Accessing Azure SQL Database from SSMS.mp4

04:35

58 - Loading Data to SQL Database Demo.mp4

01:07

59 - Copying data to SQL Database Practical.mp4

06:51

60 - Fix error in Linked service while creating for SQL Database.mp4

01:35

61 - Conclusion.mp4

00:27

6 - Enchancements

62 - Enhancements Section Intro.mp4

00:33

63 - Enhancement for Copy data from Onpremise to ADLS Demo.mp4

02:12

64 - Enhancement For Copy data from Onpremise Practical.mp4

05:07

65 - Enhancement for synapse notebook Demo.mp4

01:59

66 - Enhancement for synapse notebook to transform only todays file.mp4

05:27

7 - Orchestration

67 - Orchestration Section Intro.mp4

00:48

68 - Orchestrating the pipelines Demo.mp4

01:34

69 - Orchestrating all the pipelines and make it an automated pipeline Practical.mp4

05:17

70 - Send an automatic alert email notification when pipeline failed in ADF Demo.mp4

01:26

71 - Send an alert email notification when pipeline is failed Practical.mp4

07:50

72 - Executing all pipelines for loading data into SQL.mp4

02:54

8 - Reporting

73 - Reporting Section Intro.mp4

00:54

74 - Reporting Data in Power BI Practical.mp4

05:11

9 - Additional Section Configuring Continuous Integration Continuous deployment

75 - CICD Introduction.mp4

00:53

76 - What is Continuous Integration.mp4

05:42

77 - What is Continuous Deployement.mp4

03:14

78 - CICD Part 1.mp4

09:34

79 - CICD Part 2.mp4

04:47

80 - CICD Part 3.mp4

06:24

81 - CICD Part 4.mp4

05:35

10 - Conclusion

82 - Conclusion for the course.mp4

00:22

Description

Complete end to end Project on OTT platform using Azure Data Factory and Azure Synapse Analytics from Scratch with Labs

What You'll Learn?

Create incremental pipeline based on last modified date
Calling Azure Synapse notebook from Azure Data Factory
Configure CICD from Scratch in Azure Data Factory
Build end to end project using Azure Data Engineering Services
Create a Self Hosted Integration Runtime and ingest data from On-premise Environment
How to transform data with Azure Synapse Analytics
Create and configure an Azure Data Lake
Project Oriented Learning of Azure Data Engineering services
Create an Azure Key vault service
Using Secrets of Key vault in Linked services
Create and configure an Azure SQL Database
Writing PySpark transformation logic in Synapse Notebook
Use of Triggers in Azure Data Factory
Create an incremental data load pipeline to ingest file in daily basis
Build an Automated pipeline in Azure Data Factory
Setup CICD environment from Scratch
Create incremental pipeline based on file name and pick today's file
Perform Data Analysis Using PySpark Code using Synapse Notebook
Create and configure an Azure Data Factory
Create and configure an Azure Synapse Analytics
Orchestrate all pipelines in Azure Data Factory
Loading Data into Azure SQL Database
Ingest Data from On-premise to Azure Datalake in Azure Cloud
Send an Automatic Alert email when a pipeline is failed
Transform only Today's file in Azure Synapse Analytics
Report transformed data in Power BI
Complete hands-on project using Azure Data Engineering Services
Provides insights on services that needed to clear DP-203
Understand how GetMetaData, ForEach, If Condition Activities works

Who is this for?

Azure Data Engineers curious to do Hands-on practical Labs

Data Engineers who want real time project experience using Azure Data Engineering Services

Students aspiring to learn real time project on Azure Data Engineering

Folks preparing for DP-203 to get practical understanding

More details

Description
Are you looking to build an end-to-end ETL project using Azure data engineering services? Look no further than this comprehensive course on Azure Data Factory and Azure Synapse Analytics. This course is designed to guide you through the process of creating all the necessary services from scratch, before building an ETL project from start to finish.
100+ Enrolments within 24 hours of Course Launch!!!!
Throughout this course, you'll gain practical, hands-on experience with Azure Data Factory and Azure Synapse Analytics, learning how to use these powerful data engineering tools to create a highly effective ETL solution. You'll explore the many features and capabilities of these platforms, as well as their integration with other Azure services like
1. Azure SQL Database
2. Azure Synapse Analytics
3. Azure Key Vault
4. Azure Data Factory for Orchestration,
5. Azure Storage solutions (Azure Datalake Gen2)
6. Microsoft Power BI
7. Azure Logic Apps

In addition to the core ETL project, this course also includes an additional section on CICD (Continuous Integration and Continuous Deployment) on Azure Data Factory, helping you to fully automate your data engineering workflow.
Whether you're a beginner or an experienced data engineer, this course is designed to help you gain a comprehensive understanding of Azure Data Factory and Azure Synapse Analytics. By the end of the course, you'll be able to confidently create and manage your own ETL projects on the Azure cloud.
Enrol now and take your data engineering skills to the next level!
Who this course is for:
Azure Data Engineers curious to do Hands-on practical Labs
Data Engineers who want real time project experience using Azure Data Engineering Services
Students aspiring to learn real time project on Azure Data Engineering
Folks preparing for DP-203 to get practical understanding

User Reviews

Rating

average 0

Total votes0

Focused display

Microsoft Azure

Shanmukh Sattiraju

Instructor's Courses

An Azure Data Engineer and having vast experience on Azure Data Engineering Services and building ETL Pipelines. I have developed expertise in managing large-scale data solutions on the Microsoft Azure cloud platform. My knowledge and experience in Azure services, such as Azure Data Factory, Azure Synapse , and other data engineering services in Azure, enable me to design and implement robust data pipelines and optimize data processing workflows.In addition to my work as a data engineer, I am also a passionate blogger and instructor at Udemy. Through my blog and online courses, I share my insights and knowledge in data engineering and related topics with 200+ students on Udemy, helping them to build their skills and knowledge in the field.As a data professional, I am committed to continuous learning and staying up-to-date with the latest industry trends and technologies/With passion on learning Cloud Technologies with hands-on learning and Certified with- Microsoft Azure Data Engineer (DP-203)- Microsoft Certified Power BI Data Analyst (PL-300)- Microsoft Certified Azure Administrator (AZ-104)- Data bricks Certified Lakehouse Fundamentals- AWS Certified Solutions Architect - Associate- AWS Certified Cloud Practitioner- Microsoft Certified Azure Fundamentals (AZ-900)- Microsoft Certified Azure Data Fundamentals (DP-900)- Microsoft Certified Azure Security, Compliance, and Identity Fundamentals (SC-900)"Evolve ourselves along with the trending technology by learning and enhance the skill set to master it"

Udemy

View courses Udemy

Students take courses primarily to improve job-related skills.Some courses generate credit toward technical certification. Udemy has made a special effort to attract corporate trainers seeking to create coursework for employees of their company.