Transform Data into Insights with Dagster and Deepnote

Focused View

Simon Szalai

8:41:51

92 View

1 - Introduction

1 - Welcome to the World of Data Engineering.mp4

01:22

2 - The Power of Clean Organized Data.mp4

01:57

3 - The Skills and Tools Needed to be a Successful Data Engineer.mp4

02:56

4 - An ETL pipeline for Small and MediumSized Businesses.mp4

01:32

2 - Exploring the Tools of the ETL pipeline Deepnote Dagster and Metabase

5 - DeepNote.mp4

08:53

6 - Dagster.mp4

12:41

7 - Metabase.mp4

05:21

8 - Other tools.mp4

02:54

3 - Designing the ETL Pipeline From Data Sources to Dashboards

9 - Building the Solution Architecture.mp4

08:51

4 - Setting Up Your Development Environment and Generating Dummy Data

10 - Creating a PostgreSQL Database on Google Cloud.mp4

06:59

11 - Generating Synthetic Data of a Hypothetical Client.mp4

08:20

11 - github repo to start with.zip

12 - Explanation of the data generation process optional.mp4

14:47

13 - Verifying the Generated Data.mp4

06:00

5 - Getting Started with Deepnote An Introduction to Python and SQL for Data Explor

14 - Extracting and Viewing Data in Deepnote.mp4

14:39

15 - Digging Deeper Identifying Data Issues.mp4

09:20

16 - Digging Deeper Coming Up with a Strategy.mp4

12:47

17 - Creating a Database Table for Storing Normalized Data.mp4

08:35

6 - Data Preprocessing in Deepnote Cleaning and Normalizing Data

18 - Preprocessing Relational Data POS Transactions.mp4

40:15

19 - Preprocessing Relation Data Crypto Transactions.mp4

26:00

20 - Preprocessing JSON Data.mp4

15:17

21 - Preprocessing Excel Sheets Loading Files from Google Drive.mp4

38:37

22 - Preprocessing Excel Sheets Market Transactions.mp4

57:26

23 - Refactoring Business Logic Challenge.mp4

11:24

24 - Refactoring Business Logic Solution.mp4

07:42

25 - Unit Testing.mp4

08:15

7 - Setting up the ETL pipeline with Dagster

26 - Overview of Dagster Concepts.mp4

17:51

27 - Set up Local Dagster Development.mp4

20:15

28 - Extracting Data.mp4

23:49

29 - Transforming and Loading Data.mp4

15:55

30 - Partitioned Processing.mp4

17:15

31 - Job Configuration.mp4

16:38

32 - Streamed Data Processing.mp4

18:23

33 - Processing Files.mp4

06:32

34 - Creating Dagster Schedules.mp4

04:32

35 - Creating Dagster Sensors.mp4

19:02

36 - Deploying to Dagster Cloud.mp4

17:54

8 - Visualizing Data in Metabase

37 - Creating Visualizations from the Processed Data.mp4

10:55

9 - Bonus Content

38 - Bonus Lecture.html

Description

Data Engineering for Empowered Business Decisions: ETL, Exploration & Visualization

What You'll Learn?

Turn messy, real-world data into actionable insights.
Gain familiarity with tools such as Deepnote, Dagster, and Metabase.
Use Deepnote as a data engineering development environment.
Generate realistic development data for analysis and visualization.
Learn data exploration and preprocessing techniques using Python and SQL.
Clean and normalize data from various sources, such as relational databases, JSON, .xls files and more.
Set up Dagster to orchestrate your data pipeline.
Integrate the processing logic into a scalable ETL pipeline with Dagster.
Deploy your pipeline to Dagster Cloud (serverless)
Optimize processing through techniques such as parallelization or streamed processing.
Create powerful data visualizations using Metabase.

Who is this for?

Developers seeking to build scalable and efficient ETL pipelines.

Entrepreneurs looking to leverage data for business growth.

Data analysts and scientists who want to streamline their data processing workflow.

Business professionals looking to improve their data-driven decision-making abilities.

Students and recent graduates interested in a career in data engineering.

Data managers tasked with organizing and making data accessible for analysis.

Project managers looking to implement data-driven solutions for clients or company.

Individuals interested in learning cutting-edge tools and techniques in data engineering.

More details

Description
Do you struggle with making data-driven decisions for your business due to scattered, inconsistent, and inaccessible data? This course is the solution! Learn to build a streamlined and efficient ETL pipeline that will allow you to turn data into actionable insights.
This course teaches you how to build a system that collects data from multiple sources, normalizes it, and stores it in a consistent and accessible format. You will learn how to extract data, explore and preprocess it, and ultimately visualize it to support better decision-making and optimize business processes.
Forget about big data and cluster management headaches, this course is designed to get you up and running quickly with a real-time ETL pipeline. With infrastructure costs under $50 a month, you can start seeing immediate results and return on investment for your clients or company.
In the first part of the course, I will walk you through the architecture and introduce you to the tools we will be using:
Deepnote, as a setup-free development environment
Dagster, as the pipeline orchestrator
Metabase, as a low-code data visualization platform
While the course will introduce you to the relevant features of Deepnote and Metabase, it is mostly focused on Dagster.
In the next part, we will get started by generating dummy sales data of a hypothetical company using Deepnote. The code will be provided for this. Once we have the data, the course will dive into data exploration and preprocessing techniques using Python and SQL in Deepnote, including cleaning and normalizing data from various sources such as relational and JSON data, Excel sheets, and more. We will implement the processing logic in Deepnote, then commit it to a Git repository that will be shared with Dagster.
In the following section, we will wrap the business logic with Dagster operations and jobs, then deploy them to Dagster Cloud (self-hosted option also available), which will allow you to manage everything from a single, unified view. In this section, you will also learn a few tricks to speed up and optimize processing, such as parallelization or streamed processing.
In the final section of this course, you'll bring your preprocessed data to life with Metabase. With a few simple clicks, even non-technical individuals will be able to create stunning, powerful visualizations that unlock the full potential of your data.
By the end of this course, you'll have a comprehensive understanding of the tools used and how they work together, empowering you to provide tangible benefits to your clients or company from day one, measured in thousands or tens of thousands of dollars.
The choice is yours - will you seize this opportunity to deliver massive benefits to your company or clients, and claim your fair share of the rewards?
Who this course is for:
Developers seeking to build scalable and efficient ETL pipelines.
Entrepreneurs looking to leverage data for business growth.
Data analysts and scientists who want to streamline their data processing workflow.
Business professionals looking to improve their data-driven decision-making abilities.
Students and recent graduates interested in a career in data engineering.
Data managers tasked with organizing and making data accessible for analysis.
Project managers looking to implement data-driven solutions for clients or company.
Individuals interested in learning cutting-edge tools and techniques in data engineering.

User Reviews

Rating

average 0

Total votes0

Focused display

Self-taught software engineer with 5+ years of experience using data science and engineering to solve real-world problems.Worked at several startups in Europe and Canada in Health Tech, Food Tech, Enterprise Software, and Consulting.Most code written in Python and JavaScript. Interested in Robotics, AI, IoT, Sensor Networks, Quantum Computing, and Biotechnology.

Udemy

View courses Udemy

Students take courses primarily to improve job-related skills.Some courses generate credit toward technical certification. Udemy has made a special effort to attract corporate trainers seeking to create coursework for employees of their company.