Data Pipelines with Snowflake and Streamlit

Focused View

Marcos Vinicius Oliveira

5:17:27

0 View

1 - Introduction

1 -Introduction.mp4

00:43

2 - Setup - Part 1

1 -Snowflake Setup - 1.mp4

01:29

2 -Snowflake Setup - 2.mp4

06:39

3 -Kaggle Setup.mp4

02:09

3 -screenshot 1.zip

3 -screenshot 2.zip

3 - External Access Integration Request.html

4 -SerpAPI Setup.mp4

02:32

5 -VS Code Setup.mp4

11:55

6 -AWS Account Setup.mp4

01:54

3 - Sample download code

1 -Kaggle download script - 1.mp4

07:24

2 -Kaggle download script - 2.mp4

06:12

3 -SerpAPI download script - 1.mp4

07:10

4 -SerpAPI download script - 2.mp4

07:59

4 - Setup - Part 2

1 -Snowflake EAI request completion.mp4

03:20

5 - Database preparation

1 -Database preparation - 1.mp4

09:56

2 -Database preparation - 2.mp4

09:50

6 - Kaggle Python procedure

1 -Kaggle Python procedure - 1.mp4

12:39

2 -Kaggle Python procedure - 2.mp4

07:14

3 -Kaggle Python procedure - 3.mp4

14:42

7 - SerpAPI Python procedure

1 -SerpAPI Python procedure - 1.mp4

16:38

2 -SerpAPI Python procedure - 2.mp4

02:12

3 -SerpAPI Python procedure - 3.mp4

04:21

4 -SerpAPI Python procedure - 4.mp4

17:29

8 - Task design and DWH layer

1 -Task design - 1.mp4

09:32

2 -Task design - 2.mp4

12:08

3 -DWH design - 1.mp4

08:13

4 -DWH design - 2.mp4

13:39

9 - Streamlit app

1 -Streamlit app - 1.mp4

06:23

2 -Streamlit app - 2.mp4

04:00

10 - Pipeline enhancements

1 -Improvements summary.mp4

04:27

2 -Kaggle procedure update.mp4

16:33

3 -SerpAPI procedure update - 1.mp4

06:13

4 -SerpAPI procedure update - 2.mp4

06:11

5 -SerpAPI procedure update - 3.mp4

11:46

6 -SerpAPI procedure update - 4.mp4

10:18

7 -SerpAPI procedure update - 5.mp4

08:55

8 -SerpAPI procedure update - 6.mp4

17:11

9 -SerpAPI procedure update - 7.mp4

11:56

10 -SerpAPI procedure update - 8.mp4

11:36

11 - Conclusion

1 -Conclusion.mp4

03:59

2 -ddl.zip

2 - Course content.html

Description

Using Snowflake to data engineer Kaggle and Google Trends data with Python procedures and tasks

What You'll Learn?

Setup Snowflake and AWS Accounts
Work with Kaggle and SerpAPI
Download and manipulate data with Jupyter Notebooks on VS Code
Work with External Access Integration and Storage Integration on Snowflake
Create Snowflake Python based procedures
Create Snowflake tasks
Create Streamlit apps inside of Snowflake

Who is this for?

Data Engineers looking to get proficient on Snowflake and Streamlit for building data pipelines

What You Need to Know?

Proficient knowledge on SQL and basic knowledge on Snowflake database

Basic knowledge on data modeling and engineering

Proficient Python knowledge

More details

Description
This course focuses on building a data engineering pipeline that integrates multiple data sources, including Kaggle datasets and Google Trends data (fetched via SerpAPI), to analyze the relationship between Netflix show releases and the popularity of actors. You'll learn to gather and combine data on Netflix actors and their trends on Google, particularly in the weeks following a show's release.
You will use Kaggle as a source for the Netflix shows and actors dataset and Google Trends (accessed via SerpAPI) to fetch real-time search data for the actors. This data will be stored and processed within the Snowflake database, leveraging its cloud-native architecture for optimal scalability and performance.
Technical Stack Overview:
Snowflake Database: The central repository for storing and querying data.
Streamlit in Snowflake: A web app framework to visualize the data directly inside Snowflake.
AWS S3: For data storage and retrieval, particularly for intermediate datasets.
Snowflake Python Procedures: Automating data manipulation and pipeline processes.
Snowflake External Access & Storage Integrations: Managing secure access to external services and storage.
By the end of the course, you'll have a fully functional data pipeline that processes and combines streaming data, cloud storage, and APIs for trend analysis, visualized through an interactive Streamlit app within Snowflake.
Who this course is for:
Data Engineers looking to get proficient on Snowflake and Streamlit for building data pipelines

User Reviews

Rating

average 0

Total votes0

Focused display

Data Science

Marcos Vinicius Oliveira

Instructor's Courses

Senior Data Engineer with more than 12 years of experience with Data and Analytics, plus 4 years of experience with Snowflake and Snowflake - SnowPro Core Certified. Informatica Cloud Expert mainly on Data Integration and Mass Ingestion. Python Expert with focus on data analysis with pandas and data engineering scripts for production environments. Streamlit Expert on the Snowflake platform.

Udemy

View courses Udemy

Students take courses primarily to improve job-related skills.Some courses generate credit toward technical certification. Udemy has made a special effort to attract corporate trainers seeking to create coursework for employees of their company.