Windowing and Join Operations on Streaming Data with Apache Spark on Databricks

Focused View

Janani Ravi

2:02:08

64 View

01. Course Overview

01. Course Overview.mp4

02:06

02. Performing Windowing Operations on Data

02. Prerequisites and Course Outline.mp4

02:12

03. Stateless and Stateful Transformations.mp4

05:26

04. Tumbling Sliding and Global Windows.mp4

05:05

05. Event Time Ingestion Time and Processing Time.mp4

06:19

06. Demo-Reading Streaming Data from a File Source.mp4

03:36

07. Demo-Operations Using Global Windows.mp4

03:48

08. Demo-Operations Using Tumbling Windows.mp4

03:08

09. Demo-More Operations Using Tumbling Windows.mp4

05:29

10. Demo-Operations Using Sliding Windows.mp4

03:39

03. Exploring Aggregations Using Watermarks

11. Demo-Provisioning an HDInsight Kafka Cluster.mp4

06:05

12. Demo-Configuring Kafka to Avertise IP Addresses.mp4

02:55

13. Demo-Accessing the Kafka Broker Zookeeper Hostname and IP Addresses.mp4

02:08

14. Demo-Creating a Kafka Topic and Setting up a Producer.mp4

03:04

15. Demo-Peering the Kafka Cluster with the Databricks Cluster.mp4

03:26

16. Demo-Tumbling Windows Using Event Time.mp4

05:46

17. Demo-Sliding Windows Using Event Time.mp4

01:08

18. Watermarks and Late Data.mp4

03:02

19. Configuring Watermarks in Spark.mp4

03:52

20. Watermarking to Limit State.mp4

05:21

21. Demo-Azure Event Hubs as a Streaming Source.mp4

02:36

22. Demo-Publishing Events to Azure Event Hubs.mp4

05:03

23. Demo-Configuring Watermarks on Streams.mp4

07:48

04. Performing Join Operations on Data

24. Streaming Joins.mp4

04:49

25. Demo-Streaming-static Joins-Full Outer Join.mp4

04:06

26. Demo-Streaming-static Joins-Other Join Operations.mp4

04:57

27. Demo-Setting up Multiple Streaming Sources.mp4

03:38

28. Demo-Streaming-streaming Joins.mp4

03:32

29. Demo-Inner Joins with Watermarks.mp4

04:00

30. Demo-Left Outer and Left Semi Joins with Watermarks.mp4

02:46

31. Summary and Further Study.mp4

01:18

Description

This course will teach you how to leverage windowing, watermarking, and join operations on streaming data in Spark for your specific use cases.

What You'll Learn?

Structured Streaming in Apache Spark treats real-time data as a table that is being constantly appended. In such a stream processing model the burden of stream processing shifts from the user to the system, making it very easy and intuitive to process streaming data with Spark. Apache Spark supports a range of windowing and join operations on streaming data using processing time and event time.

In this course, Windowing and Join Operations on Streaming Data with Apache Spark on Databricks, you will learn the difference between stateless operations that operate on a single streaming entity and stateful operations that operate on multiple entities accumulated in a stream. Then, you will explore the different kinds of windows supported by Apache Spark which includes tumbling windows, sliding windows, and global windows.

Next, you will understand the differences between event time, ingestion time, and processing time and see how you can perform windowing operations using both processing time as well as event time. Along the way, you will connect to an HDInsight Kafka cluster to read records for your input stream. You will then use watermarking to deal with late-arriving data and see how you can use watermarks to limit the state that Apache Spark stores.

Finally, you will perform join operations using streams and explore the types of joins that Spark supports for static-stream joins and stream-stream joins. You will also see how you can connect to Azure Event Hubs to read records.

When you are finished with this course, you will have the skills and knowledge of windowing and join operations needed to identify when these powerful transformations should be performed and how they are performed.

More details

User Reviews

Rating

average 0

Total votes0

Focused display

Apache Spark

Microsoft Azure

Janani Ravi

Instructor's Courses

Janani has a Masters degree from Stanford and worked for 7+ years at Google. She was one of the original engineers on Google Docs and holds 4 patents for its real-time collaborative editing framework. After spending years working in tech in the Bay Area, New York, and Singapore at companies such as Microsoft, Google, and Flipkart, Janani finally decided to combine her love for technology with her passion for teaching. She is now the co-founder of Loonycorn, a content studio focused on providing high-quality content for technical skill development. Loonycorn is working on developing an engine (patent filed) to automate animations for presentations and educational content.

Pluralsight

View courses Pluralsight

Pluralsight, LLC is an American privately held online education company that offers a variety of video training courses for software developers, IT administrators, and creative professionals through its website. Founded in 2004 by Aaron Skonnard, Keith Brown, Fritz Onion, and Bill Williams, the company has its headquarters in Farmington, Utah. As of July 2018, it uses more than 1,400 subject-matter experts as authors, and offers more than 7,000 courses in its catalog. Since first moving its courses online in 2007, the company has expanded, developing a full enterprise platform, and adding skills assessment modules.