Optimizing Apache Spark on Databricks

Focused View

Janani Ravi

2:00:05

15 View

01. Course Overview

01. Course Overview.mp4

02:05

02. Exploring and Mitigating Data Ingestion Problems

02. Prerequisites and Course Outline.mp4

01:59

03. Delta Lake.mp4

07:01

04. Data Ingestion-Definition Challenges and Best Practices.mp4

06:08

05. Auto Loader for Data Ingestion.mp4

02:52

06. Demo-Creating an External Cloud Storage Source for Ingestion of Files.mp4

04:33

07. Demo-Ingesting Streaming Data into Delta Lake.mp4

06:29

08. Demo-Tracking Processed Files using Auto Loader.mp4

03:21

09. Demo-Ingesting Batch Data into Delta Lake.mp4

03:03

10. Demo-Ingesting Data into Delta Lake Using SQL.mp4

03:09

11. Databricks Data Ingestion Network.mp4

03:13

03. Diagnosing and Mitigating Performance Problems

12. Performance Issues in Spark.mp4

04:05

13. Performance Bottlenecks in Spark-Serialization and Skew.mp4

04:39

14. Performance Bottlenecks in Spark-Spill Shuffle and Memory.mp4

04:59

15. Memory Partitions and Disk Partitions.mp4

01:16

16. Demo-Disk Partitioning.mp4

07:11

17. Data Skipping and Z-order Clustering.mp4

03:12

18. Demo-Z-ordering on a Small Delta Table.mp4

04:30

19. Demo-Z-ordering on a Large Delta Table.mp4

03:45

20. Bucketing to Optimize Joins.mp4

02:15

21. Demo-Bucketed and Unbucketed Tables.mp4

04:30

22. Demo-Joining Bucketed and Unbucketed Tables.mp4

05:39

04. Optimizing Spark for Performance

23. FIFO and Fair Schedulers.mp4

05:34

24. Demo-Default Pool FIFO Scheduling.mp4

05:12

25. Demo-Configuring Different Pools to Share Resources.mp4

03:57

26. Delta Cache.mp4

04:27

27. Demo-Configuring the Delta Cache on a Cluster.mp4

01:51

28. Demo-Running Queries on Cached Data.mp4

04:56

29. New Features in Apache Spark 3.0.mp4

02:47

30. Summary and Further Study.mp4

01:27

Description

This course will teach you how to optimize the performance of Spark clusters on Azure Databricks by identifying and mitigating various issues such as data ingestion problems and performance bottlenecks

What You'll Learn?

The Apache Spark unified analytics engine is an extremely fast and performant framework for big data processing. However, you might find that your Apache Spark code running on Azure Databricks still suffers from a number of issues. These could be due to the difficulty in ingesting data in a reliable manner from a variety of sources or due to performance issues that you encounter because of disk I/O, network performance, or computation bottlenecks.

In this course, Optimizing Apache Spark on Databricks, you will first explore and understand the issues that you might encounter ingesting data into a centralized repository for data processing and insight extraction. Then, you will learn how Delta Lake on Azure Databricks allows you to store data for processing, insights, as well as machine learning on Delta tables and you will see how you can mitigate your data ingestion problems using Auto Loader on Databricks to ingest streaming data.

Next, you will explore common performance bottlenecks that you are likely to encounter while processing data in Apache Spark, issues dealing with serialization, skew, spill, and shuffle. You will learn techniques to mitigate these issues and see how you can improve the performance of your processing code using disk partitioning, z-order clustering, and bucketing.

Finally, you will learn how you can share resources on the cluster using scheduler pools and fair scheduling and how you can reduce disk read and write operations using caching on Delta tables.

When you are finished with this course, you will have the skills and knowledge of optimizing performance in Spark needed to get the best out of your Spark cluster.

More details

User Reviews

Rating

average 0

Total votes0

Focused display

Apache Spark

Microsoft Azure

Janani Ravi

Instructor's Courses

Janani has a Masters degree from Stanford and worked for 7+ years at Google. She was one of the original engineers on Google Docs and holds 4 patents for its real-time collaborative editing framework. After spending years working in tech in the Bay Area, New York, and Singapore at companies such as Microsoft, Google, and Flipkart, Janani finally decided to combine her love for technology with her passion for teaching. She is now the co-founder of Loonycorn, a content studio focused on providing high-quality content for technical skill development. Loonycorn is working on developing an engine (patent filed) to automate animations for presentations and educational content.

Pluralsight

View courses Pluralsight

Pluralsight, LLC is an American privately held online education company that offers a variety of video training courses for software developers, IT administrators, and creative professionals through its website. Founded in 2004 by Aaron Skonnard, Keith Brown, Fritz Onion, and Bill Williams, the company has its headquarters in Farmington, Utah. As of July 2018, it uses more than 1,400 subject-matter experts as authors, and offers more than 7,000 courses in its catalog. Since first moving its courses online in 2007, the company has expanded, developing a full enterprise platform, and adding skills assessment modules.