Azure Databricks and Spark SQL (Python)

Focused View

Malvik Vaghadia

6:58:55

147 View

1. Course Overview Introduction to Spark and Databricks

1. Course Introduction.mp4

03:44

2. Big Data.mp4

03:12

3. Hadoop, Spark and Databricks.mp4

04:53

4. Apache Spark Architecture.mp4

02:41

5. Spark vs Databricks Comparison.mp4

01:33

6. Resource Comparing Apache Spark vs Databricks.html

2. Azure and Databricks Set Up

1. Azure Account Set Up.mp4

02:48

2. Azure UI Overview.mp4

09:02

3. Resource Azure Resources.html

4. Creating your Databricks Service.mp4

04:02

5. Databricks UI Overview.mp4

04:37

6. Clusters.mp4

07:39

7. Resource Pricing, Cluster Pools and Runtime Versions.html

8. How to use Databricks Notebooks.mp4

08:44

9. Mix Languages and add Markdown text in your Notebook.mp4

03:25

10. Databricks Utilities Module and FileStore Utilities.mp4

04:26

11. Resource How to use Notebooks.html

12.1 databricks course notebooks and resources.zip

12. IMPORTANT - Download Course Resource Notebooks.mp4

04:09

13. Cost Management and Cancelling your Subscription.mp4

01:24

14. Resource Cancelling your Azure Subscription.html

3. Reading and Writing Data

1.1 Countries.zip

1. Dataset Download.mp4

00:13

2. Databricks FileStore.mp4

07:06

3. Resource File Types.html

4. Reading Data.mp4

20:37

5. Writing Data.mp4

12:05

6. Parquet Files.mp4

02:29

7. Deleting Files and Folders.mp4

04:06

4. Data Analysis and Transformation with SparkSQL

1. Selecting and Renaming Columns.mp4

14:43

2. Adding New Columns.mp4

04:22

3. Changing Data Types.mp4

03:47

4. Math Functions and Simple Arithmetic.mp4

05:40

5. Sort Functions.mp4

02:15

6. String Functions.mp4

03:33

7. Datetime Functions.mp4

05:41

8. Filtering DataFrames.mp4

08:32

9. Conditional Statements.mp4

03:20

10. Using SQL Expressions with expr().mp4

05:24

11. Dropping Columns.mp4

03:16

12. Grouping your DataFrame.mp4

09:45

13. Pivot your DataFrame.mp4

01:45

14. Joining DataFrames.mp4

09:44

15. Union.mp4

01:51

16. Unpivot your DataFrame.mp4

01:45

17. Pandas.mp4

01:45

5. Utilising the Medallion Architecture in Databricks

1. Medallion Architecture.mp4

02:49

2. Resource Medallion Architecture.html

6. Challenge Section Customer Orders

1.1 Customer Orders.zip

1. Dataset Download and DBFS Upload - to animate.mp4

03:45

2. Assignment 1 Bronze to Silver.mp4

03:24

3. Assignment 1 Solutions Walkthrough.mp4

07:52

4. Assignment 2 Silver to Gold.mp4

02:18

5. Assignment 2 Solutions Walkthrough.mp4

09:03

7. Visualizations and Dashboards

1. Visualizations and Dashboards.mp4

13:23

8. Accessing Data from Azure Data Lake Storage (ADLS) with Databricks

1. Creating an ADLS Gen2 Account.mp4

07:23

2. (Optional) Storage Explorer.mp4

03:01

3. Accessing via Access Keys.mp4

07:01

4. Accessing via SAS Token.mp4

04:16

5. Mounting ADLS to DBFS Overview.mp4

02:35

6. Mounting ADLS to DBFS Demo.mp4

09:49

7. Secret Scopes.mp4

12:23

8. End to End Walkthrough Example.mp4

10:12

9. Hive Metastore, Databases, Tables and Views

1. Running SQL on DataFrames.mp4

10:12

2. Hive Metastore and Creating Databases.mp4

04:07

3. Managed Tables.mp4

14:27

4. Specifying a Location for your Underlying Managed Table Data.mp4

05:49

5. Unmanaged (External) Tables.mp4

06:15

6. Permanent Views.mp4

04:55

10. Challenge Section Employees

1.1 Employees.zip

1. Dataset Download and ADLS Upload.mp4

03:14

2. Assignment Employees.mp4

01:47

3. Assignment Solutions Walkthrough.mp4

08:14

11. Databricks Data Lakehouse Delta Lake

1. Databricks Data Lakehouse Delta Lake Overview.mp4

05:39

2. Delta Lake Data Files.mp4

12:31

3. Deleting and Updating Records.mp4

15:32

4. Merge Into.mp4

09:34

5. Table Utility Commands.mp4

05:53

12. Modularize Code and Link Notebooks

1. Running a Notebook from another Notebook.mp4

07:08

2. Text Widgets.mp4

04:21

Description

Hands-on course focusing on data engineering and analysis on Azure Databricks using Spark SQL

What You'll Learn?

Azure Databricks
Data Lakehouse
Delta Lakes
Spark SQL
PySpark
Big Data
Real World Scenarios

Who is this for?

Anyone interested in working with Big Data and Spark

Anyone interested in working with Databricks

Anyone interested in working with cloud platforms

More details

Description
Databricks is one of the most in demand big data tools around. It is a fast, easy, and collaborative Spark based big data analytics service designed for data science, ML and data engineering workflows.
The course is packed with lectures, code-along videos and dedicated challenge sections. This should be more than enough to keep you engaged and learning! As an added bonus you will also have lifetime access to all the lecturesâ€¦ and I have provided detailed notebooks as a downloadable asset, the notebooks will contain step by step documentation with additional resources and links.
I have ensured that the delivery of the course is engaging and concise, the curriculum is extensive yet delivered in an efficient way. The course will provide you with hands-on training utilising a variety of different data sets.
The course is aimed at teaching you PySpark, Spark SQL in Python and the Databricks Lakehouse Architecture.
You will primarily be using Databricks on Microsoft Azure in addition to other services such as Azure Data Lake Storage Gen 2.
The course will cover a variety of areas including:
Set Up and Overview
Azure Databricks Notebooks
Spark SQL
Reading and Writing Data
Data Analysis and Transformation with Spark SQL in Python
Charts and Dashboards in Databricks Notebooks
Databricks Medallion Architecture
Accessing Data in Cloud Object Storage
Hive Metastore
Databases, Tables and Views in Databricks
Delta Lake / Databricks Lakehouse Architecture
Who this course is for:
Anyone interested in working with Big Data and Spark
Anyone interested in working with Databricks
Anyone interested in working with cloud platforms

User Reviews

Rating

average 0

Total votes0

Focused display

Malvik is an Instructor specialising in Data Analytics and Visualization.He has spent 10+ years working for and consulting a number of publicly listed / unlisted companies as a Data and BI specialist. Over the course of his career Malvik has developed a skillset in data engineering, data analytics and visualisation, with expertise across a number of technologies and programming languages including SQL, Python and R. He has worked extensively with a number of leading software platforms including Azure, Hadoop, Spark, Databricks, MongoDB, Oracle, MySQL, MS SQL Server, Qlik and Microsoft Power Platforms.

Udemy

View courses Udemy

Students take courses primarily to improve job-related skills.Some courses generate credit toward technical certification. Udemy has made a special effort to attract corporate trainers seeking to create coursework for employees of their company.