Data Engineering Pipeline Management with Apache Airflow

Focused View

Janani Ravi

2:08:30

126 View

01 - Introduction

01 - Features for data engineering pipeline management.mp4

06:11

02 - 1. Working with Role-Based Access Control

01 - Prerequisites.mp4

00:41

02 - Quick install overview.mp4

03:29

03 - Creating an admin user and exploring roles.mp4

05:05

04 - Creating users with different roles.mp4

03:06

05 - Executing a simple branching DAG.mp4

02:39

06 - Executing a simple SQL DAG.mp4

03:46

07 - The public and viewer roles.mp4

03:39

08 - The user role.mp4

02:34

09 - The op role.mp4

01:41

10 - Actions, resources, and permissions.mp4

02:35

11 - Adding permissions to the public role.mp4

03:31

12 - Creating and configuring a custom role.mp4

03:17

03 - 2. Managing SLAs

01 - Configuring emails for SLA management.mp4

03:37

02 - Configuring task-level SLAs.mp4

03:02

03 - Triggering and viewing SLA misses.mp4

03:01

04 - Configuring DAG-level SLAs.mp4

03:33

05 - Configuring DAG failed action.mp4

05:46

04 - 3. Scheduling DAGs with Datasets

01 - Dataset producer pipeline.mp4

04:13

02 - Dataset consumer pipeline.mp4

02:00

03 - Data-aware scheduling.mp4

03:12

04 - Purchases producer pipeline and join pipeline.mp4

05:03

05 - Data-aware scheduling with multiple datasets.mp4

05:52

05 - 4. Working with Airflow Plugins

01 - Introducing plugins.mp4

03:40

02 - Adding menu items using plugins.mp4

02:40

03 - Exploring the CSV reader plugin.mp4

02:14

04 - Implementing the CSV reader plugin.mp4

06:42

06 - 5. Scaling Airflow

01 - Scaling Apache Airflow.mp4

04:44

02 - Basic setup for the transformation pipeline.mp4

03:27

03 - DAG for the transformation pipeline.mp4

06:00

04 - Install RabbitMQ on macOS and Linux.mp4

04:05

05 - Set up an admin user for RabbitMQ.mp4

01:55

06 - Configuring the CeleryExecutor for Airflow.mp4

01:46

07 - Executing tasks on a single Celery worker.mp4

03:42

08 - Executing tasks on multiple Celery workers.mp4

01:22

09 - Assigning tasks to queues.mp4

02:58

07 - Conclusion

01 - Summary and next steps.mp4

01:42

Description

Take a deeper dive into data engineering pipeline management using Apache Airflow. In this course, certified Google cloud architect and data engineer Janani Ravi guides you through using Apache Airflow to complete your data engineering pipeline management workflows. Learn how to work with role-based access control, including creating users with different roles, executing a branching DAG and a SQL DAG, recalling actions that users with different roles can perform, and more. Go over how to manage SLAs and schedule DAGs with datasets. Find out how to work with AirFlow plugins and explore the CSV reader plugin. Plus, discover how to scale Apache Airflow, set up a data transformation pipeline, execute tasks, and more.

This course was created by Janani Ravi. We are pleased to host this training in our library.

More details

User Reviews

Rating

average 0

Total votes0

Focused display

Data Science

Data Analysis

Software Engineering

Janani Ravi

Instructor's Courses

Linkedin Learning

View courses Linkedin Learning

LinkedIn Learning is an American online learning provider. It provides video courses taught by industry experts in software, creative, and business skills. It is a subsidiary of LinkedIn. All the courses on LinkedIn fall into four categories: Business, Creative, Technology and Certifications. It was founded in 1995 by Lynda Weinman as Lynda.com before being acquired by LinkedIn in 2015. Microsoft acquired LinkedIn in December 2016.