Developing Spark Applications with Python

If you are going to work with Big Data or Machine Learning, you need to learn Apache Spark. If you need to learn Spark, you should get this book.

About the Book: Ever since the dawn of civilization, humans have had a need for organizing data. Accounting has existed for thousands of years. It was initially used to account for crops and herds, but later on was adopted for many other uses. Simple analog methods were used at first, which at some point evolved into mechanical devices.

Fast-forward a few years, and we get to the digital era, where things like databases and spreadsheets started to be used to manage ever-growing amounts of data. How much data? A lot. More than what a human could manage in their mind or using analog methods, and its still growing.

Paraphrasing a smart man, developing applications that worked with data went something like this: You took a group of developers, put them into a room, fed them a lot of pizza, and wrote a big check for the largest database that you could buy, and another one for the largest metal box on the market. Eventually, you got an application capable of handling large amounts of data for your enterprise. But as expected, things changethey always do, dont they?

We reached an era of information explosion, in large part thanks to the internet. Data started to be created at an unprecedented rate; so much so that some of these data sets cannot be managed and processed using traditional methods.
In fact, we can say that the internet is partly responsible for taking us into the Big Data era. Hadoop was created at Yahoo to help crawl the internet, something that could not be done with traditional methods. The Yahoo engineers that created Hadoop were inspired by two papers released by Google that explained how they solved the problem of working with large amounts of data in parallel.

But Big Data was more than just Hadoop. Soon enough, Hadoop, which initially was meant to refer to the framework used for distributed processing of large amounts of data (MapReduce), started to become more of an umbrella term to describe an ecosystem of tools and platforms capable of massive parallel processing of data. This included Pig, Hive, Impala, and many more.

But sometime around 2009, a research project in UC Berkeley AMPLab was started by Matei Zaharia. At first, according to legend, the original project was building a cluster management framework, known as mesos. Once mesos was born, they wanted to see how easy it was to build a framework from scratch in mesos, and thats how Spark was born.

Spark can help you process large amounts of data, both in the Data Engineering world, as well as in the Machine Learning one.

Welcome to the Spark era!

Table of Contents
1 The Spark Era
2 Understanding Apache Spark
3 Getting Technical with Spark
4 Sparks RDDs
5 Going Deeper into Spark Core
6 Data Frames and Spark SQL
7 Spark SQL
8 Understanding Typed API: DataSet
9 Spark Streaming
10 Exploring NOOAs Datasets
11 Final words
12 About the Authors

Download pdf

ISBN-10

ISBN-13

9781676414155

Publisher

Packt Publishing

Price

7.22

File Type

PDF

Page No.

Developing Spark Applications with Python

Python

Apache Spark

Xavier Morera

Packt Publishing

Xavier Morera

Similar Books

Other Authors' Books

Other Publishing Books

User Reviews

Rating