Snowpark Python: Ingest JSON Data Automatically In Snowflake

Focused View

DE Simplified

1:35:03

18 View

1. How JSON Data Files Are Processed In Snowflake

1. Challenges With JSON Data Onboarding & Processing Using DDLDML.mp4

05:03

2. JSON Data To DDLDML Automation Scope In Snowflake.mp4

01:28

2. JSON Data Processing Automation Requirement

1. Technical Pre-Requisite For This Course.mp4

01:16

2. What Would Not Be Covered In This Course.mp4

01:14

3. How Much DDLDML To Be Produced To Ingest Data

1.1 approach-1.zip

1.2 in-scope-out-of-scope.zip

1.3 order-item-data-flow.zip

1.4 order.zip

1. Amount of SQL Code (DDLDML) Required - Manual Effort Requirement.mp4

03:39

2.1 approach-2.zip

2. Approach-2 Where SnowPipe Is Used To Copy Data.mp4

01:05

4. Tree Map

1.1 course-tree-map.zip

1. Complete Course Structure Discussion.mp4

03:17

5. JSON Data File & Role Of Infer-Schema

1.1 emp json with array.zip

1.2 emp json with comma.zip

1.3 emp json with dic.zip

1.4 emp json with newline.zip

1.5 infer-schema-json.zip

1.6 simple emp single entity.zip

1. The Power & Limitation Of Infer Schema When Using With JSON Files.mp4

08:40

6. Creating Objects In Snowflake

1.1 user-role-etc.zip

1. Creating User & Role.mp4

02:22

2. Creating Database & Schema.mp4

01:45

3. Creating File Format & Stages.mp4

02:37

4.1 employee data.zip

4. Uploading JSON Data Using Snowsight.mp4

01:51

5.1 employee data cli.zip

5. Uploading JSON Data Using SnowSQL.mp4

01:36

7. Setting Up Python Version Environment

1. Setup Python Environment.mp4

01:47

8. Authentication

1.1 simple-auth.zip

1. SnowparkSnowflake Connectivity Using Basic Authentication.mp4

03:06

2.1 rsa-key-based-auth-snowpark.zip

2. SnowparkSnowflake Key Pair (RSA) Based Authentication.mp4

05:14

3.1 sso-based-authentication-snowpark.zip

3. SnowparkSnowflake Single Sign On (SSO) Based Authentication.mp4

03:24

9. SnowparkSnowflake JSON Ingestion Automation

1.1 python-src.zip

1. Challenges With Infer Schema.mp4

01:29

2. Design Approach.mp4

02:46

3. High Level Code Explanation.mp4

02:27

4. Jijja Template Discussion.mp4

04:04

5. List JSON.mp4

06:36

6. Infer Schema Logic.mp4

02:21

7. Jinja Execution.mp4

10:19

8. SQL Execution.mp4

05:02

10. Section-05

1.1 employee-domain-data-10k.zip

1.2 python-src-code.zip

1.3 sql-output.zip

1. Multiple JSON Entities & Run Automation Script To Generate DDLDML.mp4

09:28

2. Final Comment.mp4

01:07

Description

Automate DDL & DML SQL Generation For JSON Data Files & Save 95% Manual Effort

What You'll Learn?

How to automate JSON Data ingestion from named stages to snowflake table using Snowpark Python API
How to use infer-schema to automate JSON data ingestion activities.
How to build automation utilities using Snowpark Python API
How to build end to end data onboarding tool using Snowpark Python API

Who is this for?

Snowflake Cloud Data Developer

Snowflake Cloud Data Engineer

Snowflake Cloud Data Architect

Python Data Developer

Snowflake Data Engineer

What You Need to Know?

Basic working knowledge with Python Programming Language

Working knowledge with Snowflake Cloud Data Warehouse

Basic working knowledge with Snowpark Python API

More details

Description
In many situations, Snowflake gets lots of JSON data files and JSON entities, and data development teams write complex stored procedures to make DDL and DML SQL scripts to process and flatten the JSON data entities. Creating & building DDL &Â DML statement manually is a time consuming and error prone process. This also hampers overall development process.
This tutorial helps you create a simple and sophisticated utility using Snowflake's Snowpark Python API. This python utility uses Infer-schema table function along with Python JSON library to figure out JSON structures and helps to create all your landing/bronze/silver layer snowflake object requirements. With Python-Jinja2 templates, it doesn't just create DDL commands; it also makes copy commands, streams, tasks, and stored procedures. This makes it easy to automate moving data from your external storage to your bronze/silver layers.

This tutorial will explain the current challenges and how to solve this problem
We'll look at the Infer Schema Table Function and its limits in detecting JSON structures.
We'll discuss the usual patterns and repetition when making DDL & DML statements for JSON entities.
You'll learn how to create a simple data ingestion solution using Snowpark and Python without writing any SQL code
We'll show you how to put this design into practice with the Snowpark Python API.
Finally, we'll demonstrate the whole process using multiple JSON data examples, and you'll see how it can quickly load a large number of records in under 2 minutes.
Who this course is for:
Snowflake Cloud Data Developer
Snowflake Cloud Data Engineer
Snowflake Cloud Data Architect
Python Data Developer
Snowflake Data Engineer

User Reviews

Rating

average 0

Total votes0

Focused display

At Data Engineering Simplified, our mission is to offer the data engineering developer community compact yet powerful technical tips and digests. We focus on providing concise and impactful content that can benefit data engineers in their work. Our aim is to simplify complex concepts and deliver valuable insights, empowering developers to enhance their skills and efficiency in the field of data engineering.

Udemy

View courses Udemy

Students take courses primarily to improve job-related skills.Some courses generate credit toward technical certification. Udemy has made a special effort to attract corporate trainers seeking to create coursework for employees of their company.