Build Real World Big Data Projects 2023

Focused View

MD Imran

5:37:00

102 View

1. Build ETL Data Pipeline on AWS EMR Cluster

1. Exploration of the dataset.mp4

04:11

2. Creating EMR Cluster.mp4

06:52

3. Login into EMR part 1.mp4

08:20

4. Login into EMR part 2.mp4

02:36

5. Upload Data into Amazon S3.mp4

02:51

6. using HIve as ETL Tool.mp4

14:11

7. Hive Data Insertion.mp4

02:05

8. Install Tableau Desktop.mp4

06:27

9. Install Driver.mp4

02:14

10. Connect Tableau to Amazon EMR Hive.mp4

02:46

11. Add data schema and Table.mp4

00:52

12. plot charts in Tableau part 1.mp4

06:28

13. plot charts in Tableau part 2.mp4

09:27

14. plot charts in Tableau part 3.mp4

02:51

15. plot charts in Tableau part 4.mp4

04:38

16. plot charts in Tableau part 5.mp4

04:58

17. Building Dashboard and story.mp4

14:58

2. Build Modern ETL Data Pipeline using IICS

1. Tour to Architecture diagram.mp4

01:33

2. Exploration of the dataset.mp4

02:52

3. Upload data to AWS S3.mp4

02:05

4. Create Postgresql in aws.mp4

06:00

5. Download Pgadmin.mp4

03:34

6. set up postgres sql and create schemas.mp4

01:44

7. Create schemas and order table in your postgres instance.mp4

10:38

8. set up infromatica cloud account.mp4

04:57

9. Add S3 Connection part 1.mp4

10:31

10. Add S3 Connection part 2.mp4

00:45

11. Add postgres Connection.mp4

02:48

12. Create customer Destination in Datawarehouse.mp4

05:33

13. EL for aws s3 to data warehouse.mp4

10:34

14. Create order Destination in Datawarehouse.mp4

04:41

15. EL for app database to data warehouse.mp4

04:39

16. Create DBT account.mp4

01:47

17. dbt part 1.mp4

07:33

18. dbt part 2.mp4

07:01

19. dbt part 3.mp4

06:05

20. dbt part 4.mp4

02:00

21. dbt part 5.mp4

01:43

22. dbt part 6.mp4

06:32

23. dbt part 7.mp4

01:47

24. dbt part 8.mp4

08:32

25. Tableau and postgres set up.mp4

02:33

26. How to Build charts in Tableau.mp4

02:03

3. Create A Data Pipeline based on Messaging Using PySpark and Airflow

1. Tour to Architecture diagram.mp4

03:15

2. Create EC2 Instance.mp4

05:21

3. SSH into EC2 Instance.mp4

02:14

4. Envirnoment setup with docker.mp4

04:26

5. Copy Important folder from local to ec2 and give required permissions.mp4

17:07

6. To connect to different services locally after port forwarding.mp4

05:28

7. To get into bash shell of different containers.mp4

07:46

8. Insert Nifi Template.mp4

02:45

9. Data Extraction with Nifi.mp4

11:10

10. Data encryption parsing.mp4

09:43

11. Data sources hdfs kafka part 1.mp4

06:17

12. Data sources hdfs kafka part 2.mp4

06:03

13. Data sources hdfs kafka part 3.mp4

00:47

14. streaming data from kafka to pyspark.mp4

05:40

15. pyspark streaming output kafka nifi hdfs part 1.mp4

04:52

16. pyspark streaming output kafka nifi hdfs part 2.mp4

07:36

17. Move Data HDFS to hive Table part 1.mp4

03:47

18. Move Data HDFS to hive Table part 2.mp4

03:07

19. Dataflow Orchestration with Airflow part 1.mp4

08:39

20. Dataflow Orchestration with Airflow part 2.mp4

01:01

21. Connecting with Data Visualization Tool.mp4

01:41

22. Plot charts.mp4

01:00

Description

Work with Big Data Tools, SQL Databases, AWS, ETL, Data Integration Tools & more to master real-world Big Data Projects

What You'll Learn?

How to Build a Scalable Data Pipeline using various Components
Data Warehouse Design
Data Preparation,Cleaning, Data Transformation and Manipulation
Industry Project Ready projects

Who is this for?

People with some software background who want to learn the New technology in big data analysis will want to check this out. This course focuses on Various Big data Tools; we introduce some Data Engineering and data Science concepts along the way, but that's not the focus. If you want to learn how to Build Data Engineering Projects , then this course is for you.

Data analysts and Data Engineer who are curious about Big Data Tools and how it relates to their work.

More details

Description
A real data engineering project usually involves multiple components. Setting up a data engineering project, while conforming to best practices can be extremely time-consuming. If you are

A data analyst, student, scientist, or engineer looking to gain data engineering experience, but are unable to find a good starter project.
1. Wanting to work on a data engineering project that simulates a real-life project.
2. Looking for an end-to-end data engineering project.
3. Looking for a good project to get data engineering experience for job interviews.

Then this Course is for you. In this Course, you will
Learn How to Set up data infrastructure such as Airflow, Redshift, Snowflake, etc
Learn data pipeline best practices.
Learn how to spot failure points in data pipelines and build systems resistant to failures.
Learn how to design and build a data pipeline from business requirements.
Learn How to Build End to End ETL Pipeline
Set up Apache Airflow, AWS EMR, AWS Redshift, AWS Spectrum, and AWS S3.

Tech stack:Â Â
âž”Language: Python
âž”Package: PySpark
âž”Services: Docker, Kafka, Amazon Redshift,S3, IICS, DBT Many More

Requirements
This courseÂ presume that students have prior knowledge of AWS or its Big Data services.
Having a fair understanding of Python and SQL would help but it is not mandatory.

Every Month New Projects will be added

Who this course is for:
People with some software background who want to learn the New technology in big data analysis will want to check this out. This course focuses on Various Big data Tools; we introduce some Data Engineering and data Science concepts along the way, but that's not the focus. If you want to learn how to Build Data Engineering Projects , then this course is for you.
Data analysts and Data Engineer who are curious about Big Data Tools and how it relates to their work.

User Reviews

Rating

average 0

Total votes0

Focused display

Big Data

MD Imran

Instructor's Courses

Experienced Data Engineer with a demonstrated history of working in service and product companies. Solved data mysteries for different domains like Aviation, Pharmaceutical, FinTech, Telecom and Employee Services. Have designed scalable & optimized data pipelines to handle PetaBytes of data, with Batch & Real Time frequency. Got good exposure on different BigData frameworks (Hadoop, Spark, Hive, Sqoop, Flume, Flink, Kafka, Docker), Databases (MySQL, HBase, Cassandra, Redshift, Elastic Search), AWS Services (S3, Lambda, EMR, Glue, Cloudwatch, Redshift, SNS, SQS, Athena, Appflow), Dashboarding Tools (Grafana, Kibana, QuickSight, DataDog, Data Studio), Monitoring Tools (Airflow, Azkaban), Web Development (HTML, CSS, Scala Play, Django, Rest API, JavaScript, Ajax, JQuery), Good command over programming languages (Python, Java, Scala, Shell Scripting) and strong Data Structures & Algorithm fundamentals.

Udemy

View courses Udemy

Students take courses primarily to improve job-related skills.Some courses generate credit toward technical certification. Udemy has made a special effort to attract corporate trainers seeking to create coursework for employees of their company.