Data Engineering - Apache Airflow

Focused View

Eman Ali Mughal

4:06:44

769 View

1. Introduction

1. Introduction.mp4

07:23

2. Getting Started

1. Workflows.mp4

05:12

2. Medallion Architecture and ETL.mp4

06:01

3. Problem Statement.mp4

10:55

4. Introduction To Apache Airflow.mp4

07:51

5. Summary.mp4

01:29

6. From This Section.html

3. Airflow Installation and Configuration

1. Overview.mp4

01:38

2. GCP - Google Cloud Platform - Creating Virtual Machine.mp4

04:21

3.1 db-scripts.zip

3.2 install-airflow.zip

3.3 install-airflow2.zip

3. Apache Airflow Installation on Ubuntu.mp4

09:06

4.1 install-docker-and-docker-compsoe.zip

4. Docker Installation and Using AIrflow with Docker Image.mp4

10:30

5. Airflow Configurations - Part 1.mp4

05:32

6. Airflow Configuration - Part 2.mp4

10:17

7. Summary.mp4

01:31

8. 4 Questions In This Quiz, All related to videos in this section..html

4. Apache Airflow Architecture and Concepts

1. Airflow Architecture.mp4

05:53

2. Airflow Core Concepts Part 1.mp4

15:25

3. Airflow Core Concepts Part 2.mp4

11:57

4. Airflow Core Concepts Part 3.mp4

12:26

5. Summary.mp4

01:33

6. 6 Questions from this section.html

5. Apache AIrflow UI Demo

1. Dag Page.mp4

03:01

2. Top Navigation Bar.mp4

13:19

3. Dag Detail Page.mp4

06:09

4. Branching Operator Dag.mp4

07:13

5. Summary.mp4

01:32

6. XCom Example Dag.mp4

06:19

6. Developing Big Data Pipeline in Apache Airflow

1. Use Cases.mp4

03:20

2.1 bronz.zip

2.2 gold.zip

2.3 medallion-airflow.zip

2.4 silver.zip

2. Medallion Airflow Dag.mp4

07:45

3.1 medicaldevicesdata.zip

3.2 medicaldevicesdata.zip

3. Medical Devices Airflow Dag.mp4

13:08

4.1 jdbcoperator.zip

4.2 mysqloperator.zip

4. JDBC And MySQL Operator Airflow Dag.mp4

12:45

5. Custom Operators Airflow Dag.mp4

09:59

6.1 readandclean.zip

6.2 savetoparquet.zip

6.3 transformdata.zip

6.4 weathersourcedata.zip

6.5 weathersourceetldag.zip

6. Weather Source Airflow Dag.mp4

10:31

7.1 dimcustomers.zip

7.2 dimorders.zip

7.3 dimproducts.zip

7.4 ecommercedag.zip

7.5 factorderdetails.zip

7.6 shoppingcartdata.zip

7. ECommerce Airflow Dag.mp4

10:27

8.1 snowflakedag.zip

8. Snowflake Dag.mp4

03:52

9.1 elasticsearchdag.zip

9. Elasticsearch Dag.mp4

05:25

10. Summary.mp4

02:59

Description

Apache Airflow is a workflow orchestration tool, and primarily used in Big Data related project.

What You'll Learn?

To get the basic understanding of Workflows, Apache Airflow basic features, Core Concepts, Familiarization with Airflow UI, XComs, Variables, Connections Hooks
You will also learn how to install Apache Airflow on Linux / Ubunto, in 2 different flavours, Install in Docker Container and Vanilla Installation
This course will help you enhance knowledge of ETL pipelines and how to orchestrate those ETL pipelines in Airflow Dags.
Dags in this course contains working examples of Apache Spark, Elasticsearch, Snowflake, Ecommerce Dag as well.

Who is this for?

This Course is intended for Data Engineers, Big Data Architects, or for students those does not have prior knowledge of Apache Airflow.

What You Need to Know?

You should be familiar with Python, Scala Languages, Little bit of knowledge about concepts of Big Data, Some understanding about Apache Spark will be help full in understanding of this course.

More details

Description
Airflow is an extensive tool and has many different features. I will explain in details how we can take benefit from Airflow in our Big Data-related projects. In this Apache Airflow I will start from basic concepts, then move to Installation, Later will deep dive into advanced concepts. In the end, I will create a few big data-related projects and create Airflow Dag to give you a flavor of how all components of Airflow work together.

Please note that Apache Airflow is not a Data Flow tool rather its a Workflow Orchestration tool, With such a nice and beautiful User Interface and a bundle of rich components available. When data is flowing through different processes it is difficult to monitor each and every process, if we are working in CLIÂ based environment. Rather using Apache Airflow we can get notified by color scheming of tasks, and we can start the task right from where it has failed.

I will show you different use cases of Big Data universe, in which we can take leverage from Apache Airflow.

After completing this course Part 1, you will be find your self equipped with a new tool just to solve your Big Data use cases.
Who this course is for:
This Course is intended for Data Engineers, Big Data Architects, or for students those does not have prior knowledge of Apache Airflow.

User Reviews

Rating

average 0

Total votes0

Focused display

Data Science

Eman Ali Mughal

Instructor's Courses

I have shifted towards Big Data for the last 5 years, Architected and Designed Big Data pipelines, Ingesting terabytes of data. Technologies use Apache Spark, Apache Airflow, Snowflake, Dataricks, AWS, GCP, Scala, Python, Apache NiFi, Linux, Ubuntu, Bash scripting, Hive, Hue, Zepplin, and Jupiter.For the last 21 years, I have been in the field of software development, I have worked in various different technologies like .NET Technologies, SQL Server, Desktop application development, ASP.NET, ASP.NET MVC, SPA, WCF, WPF, MVVM, Design Patterns, Web Api, Angular JS, Knockout JS, Jquery, Azure Development, Developing Class Libraries, Android and IOs development.For the last 8 years I have been running my own software development company name Vibrant Illusions.We are a group of elite professionals, programmers, and graphics designers providing software solutions to different organizations.

Udemy

View courses Udemy

Students take courses primarily to improve job-related skills.Some courses generate credit toward technical certification. Udemy has made a special effort to attract corporate trainers seeking to create coursework for employees of their company.