Companies Home Search Profile
TrainingHub TrainingHub

Snowflake cloud database with ELT(Airflow+Python+Talend)

Focused View

8:49:50

0 View
  • 1. Module 1 Intro.mp4
    02:27
  • 1.1 course ddl stmts.zip
  • 1. Setup snowflake environment.mp4
    01:09
  • 2.1 Setting up virtual machine.docx
  • 2.2 Virtual machine download alternate link 2.html
  • 2.3 Virtual box download.html
  • 2.4 Virtual machine download.html
  • 2.5 Virtual machine download alternate link.html
  • 2. Virtual machine download.mp4
    04:40
  • 3.1 airflow.zip
  • 3. Mount local folder to vm and start vm.mp4
    02:44
  • 4.1 ssh pwd.zip
  • 4.2 Visual studio code download.html
  • 4. Setup Vs code for VM.mp4
    04:21
  • 5. Configure connection details.mp4
    02:26
  • 6. Github link.html
  • 1. Airflow introduction.mp4
    02:31
  • 2. Airflow architecture.mp4
    03:33
  • 3. First Dag.mp4
    08:42
  • 4.1 Postgress airflow metadata db connection details.docx
  • 4.2 postgress airflow metadata db connection details.zip
  • 4. Introduction to metadata DB.mp4
    03:32
  • 5. Airflow scheduler.mp4
    03:26
  • 6. Scheduling in airflow.mp4
    05:12
  • 7. Executors in airflow.mp4
    03:19
  • 8. Airflow webserver.mp4
    08:55
  • 9. Handle failures.mp4
    02:57
  • 1. Introduction.mp4
    00:25
  • 2. Load to snowflake example 1.mp4
    03:37
  • 3. [Demo] Load to snowflake example 1.mp4
    05:21
  • 4. Load to snowflake example 2.mp4
    05:12
  • 5. Load to snowflake example 3.mp4
    04:41
  • 1.1 Python API.html
  • 1.2 python api.zip
  • 1.3 Python API.xlsx
  • 1. Introduction.mp4
    02:57
  • 2. Connection test 1.mp4
    03:01
  • 3. Connection test 2.mp4
    03:15
  • 4. execute method.mp4
    06:24
  • 5. cursor attributes.mp4
    06:52
  • 6. execute many.mp4
    04:00
  • 7. execute async part 1.mp4
    08:47
  • 8. execute async part 2.mp4
    03:45
  • 9. execute async part 3.mp4
    03:13
  • 10. Formatting pyformat.mp4
    03:10
  • 11. Formatting qmark.mp4
    04:37
  • 12. Result metadata.mp4
    02:16
  • 13. Type code.mp4
    01:48
  • 1. Fetch pandas arrow.mp4
    03:59
  • 2. Context manager.mp4
    02:45
  • 3. Result batch.mp4
    08:11
  • 4. Get result batch.mp4
    08:59
  • 5. Fetch arrow batches.mp4
    05:16
  • 6. Fetch pandas batches.mp4
    01:54
  • 7. To pandas arrow.mp4
    02:42
  • 8. Execute stream.mp4
    04:31
  • 9. Execute string.mp4
    02:51
  • 10. Cursor dict.mp4
    03:47
  • 1. Copy data s3 to snowflake.mp4
    09:50
  • 2. Copy data s3 to snowflake ( parallel ).mp4
    05:14
  • 3. Execute ELT query Python way.mp4
    06:47
  • 4. Schedule airflow job For python code.mp4
    06:31
  • 1. Talend as open source ETL tool..mp4
    02:23
  • 2.1 Installing Talend.docx
  • 2. Download and install talend..mp4
    03:39
  • 3. Talend workspace introduction..mp4
    06:49
  • 4.1 Installing Talend.docx
  • 4.2 Section 2 lec 4 emp data.zip
  • 4.3 section 2 lec 4 emp ddl.zip
  • 4.4 Snowflake project.zip
  • 4.5 TOS DI-20181026 1147-V7.1.1.zip
  • 4. Build first job and configure talend.mp4
    12:38
  • 5. Build sample job Level 1.mp4
    08:17
  • 6. Build sample job Level 2.mp4
    01:01
  • 7. Build sample job Level 3.mp4
    04:58
  • 8. Building sample job Level4.mp4
    04:41
  • 9. Build sample job Level5.mp4
    05:11
  • 10. Build sample job Schedule using airflow.mp4
    03:26
  • 11. Summary.mp4
    00:55
  • 1. Module 2 intro.mp4
    02:51
  • 1. Analyst requirement..mp4
    01:29
  • 2.1 S3snowcp.zip
  • 2. S3 to snowflake copy app demo.mp4
    10:11
  • 3. Requirement to process data..mp4
    01:48
  • 4.1 Eltjobdemo.zip
  • 4. Elt app demo.mp4
    06:35
  • 5. Scheduling job..mp4
    00:29
  • 6. Writing generic dag code part1.mp4
    04:31
  • 7. Writing generic dag code part2.mp4
    04:22
  • 8. Writing generic dag code part3.mp4
    02:45
  • 1. Introduction.mp4
    04:59
  • 2.1 Data for this section.html
  • 2. High level design.mp4
    02:02
  • 3.1 Data for this section.html
  • 3. Copy data from s3.mp4
    04:53
  • 4. Parsing data.mp4
    07:42
  • 5. Audit src records.mp4
    09:40
  • 6. Archive and create view part 1.mp4
    05:16
  • 7. Archive and create view part 2.mp4
    07:03
  • 8. Score records part 1.mp4
    06:56
  • 9. Score records part 2.mp4
    07:00
  • 10. prepare dag code.mp4
    03:43
  • 11. Execute airflow job.mp4
    05:22
  • 12. Improve the approach.mp4
    04:51
  • 13. Parameter registry.mp4
    06:32
  • 14. Summary.mp4
    02:48
  • 1. Introduction..mp4
    00:41
  • 2.1 Nyc website.html
  • 2.2 Nyc data volume.xlsx
  • 2.3 Nyc data.7z
  • 2. Nyc traffic data overview.mp4
    09:55
  • 3. Load sample data.mp4
    06:38
  • 4. Nyc full data load.mp4
    07:22
  • 5.1 Queries.zip
  • 5. Snow sight Dashboard.mp4
    05:11
  • 1. Bonus section.html
  • 1.1 create stage obj audit tbls.zip
  • 1.2 Creating integration object.docx
  • 1. Building talend generic copy job..mp4
    16:05
  • 2. Evaluating audit tables..mp4
    10:51
  • 3. ELT transform job demo..mp4
    20:30
  • 4.1 elt audit tbls ddl.zip
  • 4. ELT Generic job code demo..mp4
    17:50
  • 1. Introduction..mp4
    00:35
  • 2. Managers requirement..mp4
    01:04
  • 3. Solution step 1 preparing query..mp4
    04:32
  • 4. Solution step 2 create talend job..mp4
    06:48
  • 5. Solution step 3 parameterise talend job..mp4
    03:05
  • 6.1 manager requirement1.zip
  • 6. Solution step 4 create jar application..mp4
    13:07
  • 7. Managers requirement 2..mp4
    01:51
  • 8. Solution step 1 creating talend job..mp4
    09:13
  • 9.1 managers requirement2.zip
  • 9. Solution step 2 using context groups..mp4
    07:01
  • 10. Developer problem.mp4
    00:49
  • 11.1 cost warning.zip
  • 11. Cost warning app.mp4
    12:24

    • Description

    Learn to integrate ETL tools with Snowflake and leverage Airflow for ELT with snowflake.Process 250+ GB data.ELT flow.

    What You'll Learn?

    • Leveraging snowflake cloud data warehouse using Talend.
    • Talend basics.
    • Airflow basics.
    • Building airflow Dags.
    • Connect snowflake with python.
    • Auditing snowflake commands.
    • Capturing cost and performance metrics.
    • Virtual machine with preconfigured airflow.
    • End to End project to load Nyc traffic data(250 + GB ).

    Who is this for?

  • Developers interested to know how to build workflows for snowflake.

    • What You Need to Know?

  • This course will use virtual box software.
  • Disk space of ~30 Gb is required to run the virtual machine.

    • More details

    Description

    In our previous course Snowflake Masterclass [Real time demos+Best practices+Labs] we deep-dived and understood the fundamentals of snowflake, solved lot of assignments, and understood best practices to load data and unload data.

    Also, we closely evaluated most of the snowflake features understanding how they work under the hood. Through these discussions, you realized how to use Snowflake efficiently.

    There was one missing piece, How to build and orchestrate ETL workflows on Snowflake. This course is just about that.

    In this course, we are going to learn,

    1. Build workflows in Airflow.

    2. We will leverage Talend capabilities to build generic code to ingest data and process data in snowflake.       

    3. We will build audit tables and record every command we fire on the snowflake. We will record the time consumed for each task and capture snowflake credits.

    4. Once we build the framework we will build a workflow to process and transform 250 + GB volume of NYC traffic data.

    5. At last, we will connect the Snowflake with python and write code to capture stats of data we loaded to the snowflake.

    6. you will also get access to preconfigured Jupyter notebook to run your python code on the Snowflake.


    If you have previously not worked with Talend, Airflow and Python don't worry they are very simple tools I will provide the necessary introduction.

    I am sure you will learn a lot from this journey. See you in the course!!


    Who this course is for:

    • Developers interested to know how to build workflows for snowflake.

    User Reviews
    Rating
    0
    0
    0
    0
    0
    average 0
    Total votes0
    Focused display
    Category

    Data Science

    Udemy
    Udemy
    View courses Udemy
    Students take courses primarily to improve job-related skills.Some courses generate credit toward technical certification. Udemy has made a special effort to attract corporate trainers seeking to create coursework for employees of their company.
    • language english
    • Training sessions 101
    • duration 8:49:50
    • English subtitles has
    • Release Date 2024/10/31

    Courses related to Data Science

    Apache Solr 7: Search Playbook
    Pluralsight Omar Perez
    Omar Perez
    Apache Solr 7: Search Playbook
    1:27:01
    10/04/2023
    Machine Learning and Deep Learning Projects in Python
    Udemy S. Emadedin Hashemi
    S. Emadedin Hashemi
    Machine Learning and Deep Learning Projects in Python
    5:32:43
    10/02/2023
    Subtitle
    Top 10 Skills for Healthcare Data Analysts
    Linkedin Learning Ben Sullins and Free the Data Academy
    Ben Sullins and Free the Data Academy
    Top 10 Skills for Healthcare Data Analysts
    30:52
    English subtitles
    07/16/2023