Azure Databricks end to end project with Unity Catalog CICD

Focused View

Shanmukh Sattiraju

15:01:03

13 View

1. Introduction

1. Course Introduction.mp4

01:13

2. Project Architecture and Concepts.mp4

06:17

3. Course prerequisites and benefits.mp4

02:53

4.1 cicd.zip

4.2 CICD.zip

4.3 notebooks.zip

4.4 notebooks.zip

4. Project Complete Code.html

2. Environment Setup

1. Section Introduction.mp4

00:35

2. Creating a budget for project.mp4

04:22

3. Creating an Azure Databricks Workspace.mp4

03:24

4. Creating an Azure Datalake Storage Gen2.mp4

01:49

5. Walkthough on databricks Workspace UI.mp4

04:03

3. Azure Databricks - An Introduction

1. Section Introduction.mp4

00:56

2. Introduction to Distributed Data Processing.mp4

13:39

3. What is Azure Databricks.mp4

05:36

4. Azure Databricks Architecture.mp4

08:14

5. Cluster types and configuration.mp4

24:21

6. Behind the scenes when creating cluster.mp4

06:39

7.1 Login page.html

7.2 Signup page.html

7. Sign up for Databricks Community Edition.mp4

05:18

8.1 01. understanding notebook and markdowns.zip

8. Understanding notebook and Markdown basics.mp4

11:21

9.1 02. magic commands.zip

9. Notebook - Magic Commands.mp4

08:09

10.1 03. dbutis - file system utilities.zip

10. DBUitls -File System Utilities.mp4

18:56

11.1 04. dbuitls - widget utilities.zip

11. DBUitls -Widget Utilities.mp4

13:49

12.1 dbutils - notebooks utils - child - para.zip

12.2 dbutils - notebooks utils - child.zip

12.3 dbutils notebook utils - parent - para.zip

12.4 dbutils notebook utils - parent.zip

12. DBUtils - Notebook Utils.mp4

13:00

13. Basic knowledge on Azure databricks.html

4. Delta lake

1. Section Intro.mp4

00:50

2. Drawbacks of Azure Datalake.mp4

04:06

3. What is delta lake.mp4

01:33

4. Understanding Lakehouse Architecture.mp4

06:19

5. Creating databricks workspace and ADLS for delta lake.mp4

03:19

6.1 Countries1.csv

6.2 Grant the service principal access to Azure Data Lake Storage Gen2.html

6.3 test access.zip

6. Accessing Datalake storage using service principal.mp4

11:00

7.1 01. drawbacks of adls.zip

7. Drawbacks of ADLS - practical.mp4

10:52

8.1 01. drawbacks of adls - delta.zip

8. Creating Delta lake.mp4

04:28

9. Understanding the delta format.mp4

05:15

10.1 02. understanding the transaction log.zip

10. Understanding Transaction Log.mp4

12:40

11.1 03. creating delta tables.zip

11. Creating delta tables using SQL Command.mp4

19:39

12.1 04. creating delta tables using pyspark.zip

12. Creating Delta table using PySpark Code.mp4

06:02

13.1 SchemaLessCols.csv

13.2 SchemaMoreCols.csv

13. Uploading files for next lectures.mp4

00:32

14.1 05. schema enforce.zip

14.2 SchemaLessCols.csv

14.3 SchemaManagementDelta.csv

14.4 SchemaMoreCols.csv

14. Schema Enforcement.mp4

11:16

15.1 05. schema evolution.zip

15. Schema Evolution.mp4

05:18

16.1 06. versioning and time travel.zip

16.2 SchemaManagementDelta.csv

16. Time Travel and Versioning.mp4

19:21

17.1 07. vacuum command.zip

17. Vacuum Command.mp4

10:05

18.1 08. convert to delta.zip

18. Convert to Delta.mp4

04:36

19. Understanding Optimize Command - Demo.mp4

05:24

20.1 09. optimize command.zip

20. Optimize Command - Practical.mp4

14:27

21.1 10. upsert using merge.zip

21. UPSERT using MERGE.mp4

08:49

22. Test your Delta lake knowledge.html

5. Unity Catalog

1. Section Introduction.mp4

00:42

2. What is Unity Catalog.mp4

06:13

3. Creating Access Connector for Databricks.mp4

04:57

4. Creating Metastore in Unity Catalog.mp4

11:10

5. Unity Catalog Object Model.mp4

06:41

6. Roles in Unity Catalog.mp4

02:30

7. Creating users in Azure Entra ID.mp4

03:44

8. User and groups management Practical.mp4

12:08

9.1 Cluster Policy Project Defaults.txt

9.2 Compute Policy Definition.html

9. Cluster Policies.mp4

12:18

10. What are cluster pools.mp4

03:39

11.1 Cluter policy for pool.txt

11. Creating Cluster Pool.mp4

09:41

12. Creating a Dev Catalog.mp4

06:12

13.1 unity catalog privileges.zip

13. Unity Catalog Privileges.mp4

14:42

14. Understanding Unity Catalog.mp4

06:04

15. Creating and accessing External location and storage credentials.mp4

14:05

16.1 managed vs external tables.zip

16. Managed and External Tables in Unity Catalog.mp4

10:42

17. Check your knowledge on Unity Catalog.html

6. Spark Structured Streaming

1. Section Introduction.mp4

01:26

2. Spark Structured Streaming - basics.mp4

02:29

3. Understanding micro batches and background query.mp4

15:39

4. Supported Sources and Sinks.mp4

02:33

5.1 01 - streaming basics.zip

5. WriteStream and checkpoints.mp4

11:17

6. Community Edition Drop databases.mp4

06:44

7.1 02. outputmodes.zip

7. Understanding outputModes.mp4

19:08

8.1 03. triggers.zip

8. Understanding Triggers.mp4

20:58

9. Autoloader - Intro.mp4

04:16

10.1 04. autoloader basics.zip

10. Autoloader - Schema inference.mp4

11:59

11. Schema Evolution - Demo.mp4

05:08

12.1 05. autoloader - schema evolution.zip

12. Schema Evolution - Practical.mp4

13:14

13. Check your knowledge on Streaming.html

7. Project Overview

1. Section Introduction.mp4

00:47

2. Typical Medallion Architecture.mp4

05:47

3. Project Architecture.mp4

03:13

4. Understanding the dataset.mp4

04:51

8. Project Setup

1. Section Introduction.mp4

00:51

2. Expected Setup.mp4

03:05

3. Creating containers and External Locations.mp4

07:08

4.1 01. project setup.zip

4. Creating all schemas dynamically.mp4

19:36

5.1 01. project setup.zip

5. Creating bronze Tables Dynamically.mp4

07:28

9. Ingestion to Bronze

1. Section Introduction.mp4

00:49

2. Ingesting data to bronze layer - Demo.mp4

04:29

3.1 Sample files.zip

3. Ingesting raw traffic data to bronze table.mp4

14:42

4. Assignment to get the raw roads data to bronze table.html

5.1 02. load to bronze.zip

5. Ingesting raw roads data to bronze Table.mp4

08:32

6. To prove autoloader handles incremental loading.mp4

05:36

10. Silver Layer Transformations

1. Section Introduction.mp4

00:50

2.1 03. silver - traffic transformations.zip

2. Transforming Silver Traffic data.mp4

29:46

3. To prove only incremented records were being transformed.mp4

05:28

4.1 04. common.zip

4. Creating a common Notebook.mp4

02:32

5. Run one notebook from another notebook.mp4

04:36

6.1 05. silver - roads transformation.zip

6. Transforming Silver Roads data.mp4

13:18

11. Loading to Gold Layer

1. Section Introduction.mp4

00:34

2. Getting data to Gold Layer.mp4

01:22

3.1 06. gold - final transformations and loading.zip

3. Gold Layer Transformations and loading.mp4

09:33

12. Orchestrating with Workflows

1. Section Introduction.mp4

00:44

2.1 notebooks.zip

2. Adding run for common notebook in all notebooks.mp4

00:59

3. Creating Jobs and executing end to end flow.mp4

10:42

4. Attaching trigger to workflows.mp4

08:37

13. Reporting with Power BI

1. Installing Power BI Desktop.mp4

01:55

2.1 final report.zip

2. Reporting data to Power BI.mp4

06:50

14. Continuous Integration and Continuous Deployment (CICD)

1. Section Introduction.mp4

01:21

2. Expected Setup.mp4

01:30

3. Understanding Continuous Integration.mp4

06:28

4. Understanding Continuous Deployment.mp4

02:50

5. Creating Required resources for UAT.mp4

08:00

6. Configuring storage containers and external locations for UAT.mp4

05:36

7. Login and create repository in Azure DevOps.mp4

05:46

8. Integrating Azure Devops with Databricks.mp4

04:40

9. Creating feature branch and pull request to main branch.mp4

10:17

10. Creating pull request as new user.mp4

10:37

11.1 Code.zip

11. Uploading and understanding YAML Files for CICD.mp4

07:30

12. Creating CI pipeline to have live folder.mp4

19:01

13. Permissions to see Live Folder.mp4

02:39

14.1 For CD Pipeline.zip

14. Creating Deployment pipeline and deploying.mp4

13:48

15. End to end test CICD pipeline.mp4

04:15

16. Running notebooks in UAT.mp4

05:47

15. Delta Live Tables (DLT)

1. Section Intro.mp4

00:51

2. Origin of Delta live tables.mp4

02:44

3. Considerations in Lakehouse Architecture.mp4

02:10

4. Understanding Declarative ETL.mp4

06:28

5. Limitations of Delta Live Tables.mp4

02:09

6. Defining Tables from datasets.mp4

07:05

7. Creating DLT Pipeline.mp4

08:54

8.1 dlt.zip

8. End to end DLT Pipeline.mp4

13:08

9. Deleting cluster by DLT pipeline.mp4

01:16

16. Conclusion

1.1 My LinkedIn profile.html

1. Course completion.mp4

00:49

2. My other Data Engineering Courses.html

Description

Azure Databricks Mastery: Hands-on project with Unity Catalog , Delta lake, CI/CD implementing Medallion Architecture

What You'll Learn?

Understand and implement Unity Catalog
Implement project with incremental loading
Understanding the Spark Structured Streaming
Implement Continuous Integration and Continuous Deployment in project
Real time hands-on project experience
Implement and work with Delta Lake
Understand the features of Delta Lake
Implement Medallion Architecture in your project
Evolution of Delta lake from Datalake
Understand workflows in Azure Databricks
Simulate real time environment with Unity Catalog
Implement and understand the governance with Unity Catalog
Master the compute cluster creation and Management
How spark structured streaming works
Implement structured streaming in Azure databricks
Understand incremental loading with Autoloader
Code that can run in any environment
Understand and implement the Unity Catalog Object Model
Build an end to end CICD pipeline
Understand the implementation of Delta Live tables
Practise tests to check your knowledge

Who is this for?

Data Engineers who want to get real time experience using Azure Databricks

Data professionals who want to build an end to end project in Azure Databricks

Engineers who want to learn Azure Databricks and its implementation

What You Need to Know?

Basic knowledge on Python and SQL

Basic knowledge on Azure Cloud

An Azure account to implement the end to end project

More details

Description
Embark on a transformative journey to master Azure Databricks with our comprehensive hands-on Udemy course. Tailored not just for learning but also to equip you with practical concepts essential for passing the Databricks Certified Data Engineer Associate certification, this course is your key to success.
Immerse yourself in real-world projects where you'll leverage the capabilities of Unity Catalog, Delta Lake, and CI/CD methodologies, all while implementing the cutting-edge Medallion Architecture. This training program serves as your gateway to seamlessly integrate and process data in the cloud, offering invaluable insights into the latest practices in data engineering.
Throughout the course, delve into the intricacies of Delta Lake, refine your skills with Unity Catalog, and become proficient in the art of Continuous Integration and Continuous Deployment. Whether you're a seasoned data professional aiming to enhance your skill set or a budding enthusiast eager to explore the world of data engineering, this course provides the tools and knowledge to elevate your expertise in Azure Databricks.
Join us on this educational journey to unlock the full potential of cloud-based data engineering, propelling yourself towards success in contemporary data projects. Enrich your career and knowledge with this comprehensive Udemy course, ensuring you don't miss the opportunity to become a proficient Azure Databricks Engineer. Your transformation begins here!
Who this course is for:
Data Engineers who want to get real time experience using Azure Databricks
Data professionals who want to build an end to end project in Azure Databricks
Engineers who want to learn Azure Databricks and its implementation

User Reviews

Rating

average 0

Total votes0

Focused display

An Azure Data Engineer and having vast experience on Azure Data Engineering Services and building ETL Pipelines. I have developed expertise in managing large-scale data solutions on the Microsoft Azure cloud platform. My knowledge and experience in Azure services, such as Azure Data Factory, Azure Synapse , and other data engineering services in Azure, enable me to design and implement robust data pipelines and optimize data processing workflows.In addition to my work as a data engineer, I am also a passionate blogger and instructor at Udemy. Through my blog and online courses, I share my insights and knowledge in data engineering and related topics with 200+ students on Udemy, helping them to build their skills and knowledge in the field.As a data professional, I am committed to continuous learning and staying up-to-date with the latest industry trends and technologies/With passion on learning Cloud Technologies with hands-on learning and Certified with- Microsoft Azure Data Engineer (DP-203)- Microsoft Certified Power BI Data Analyst (PL-300)- Microsoft Certified Azure Administrator (AZ-104)- Data bricks Certified Lakehouse Fundamentals- AWS Certified Solutions Architect - Associate- AWS Certified Cloud Practitioner- Microsoft Certified Azure Fundamentals (AZ-900)- Microsoft Certified Azure Data Fundamentals (DP-900)- Microsoft Certified Azure Security, Compliance, and Identity Fundamentals (SC-900)"Evolve ourselves along with the trending technology by learning and enhance the skill set to master it"

Udemy

View courses Udemy

Students take courses primarily to improve job-related skills.Some courses generate credit toward technical certification. Udemy has made a special effort to attract corporate trainers seeking to create coursework for employees of their company.