Instructor-led training

Introduction to Spark with Python Training in New York City(nyc), New York

Python Programming training from Hartmann Software Group, delivered onsite in New York City(nyc), New York or live online.

Request a Training Proposal View Course Outline

What you'll learn

This course introduces the Apache Spark distributed computing engine, and is suitable for developers, data analysts, architects, technical managers, and anyone who needs to use Spark in a hands-on manner. It is based on the Spark 2.x release. All examples and labs use Python for programming.

The course provides a solid technical introduction to the Spark architecture and how Spark works. It covers the basic building blocks of Spark (e.g. RDDs and the distributed compute engine), as well as higher-level constructs that provide a simpler and more capable interface (e.g. DataFrames and Spark SQL). It includes in-depth coverage of Spark SQL and DataFrames, which are now the preferred programming API. This includes exploring possible performance issues and strategies for optimization.

The course also covers more advanced capabilities such as the use of Spark Streaming to process streaming data, and integrating with the Kafka server.

The course is very hands-on, with many labs. Participants will interact with Spark through the pyspark shell (for interactive, ad-hoc processing) as well as through programs using the Spark API. After taking this course, you will be ready to work with Spark in an informed and productive manner.

Prerequisites

Working knowledge of some programming language - no Java experience needed

Course Outline

Session 1: Introduction to Spark

Overview, Motivations, Spark Systems

Spark Ecosystem

Spark vs. Hadoop

Acquiring and Installing Spark

The Spark Shell, SparkContext

Session 2: RDDs and Spark Architecture

RDD Concepts, Lifecycle, Lazy Evaluation

RDD Partitioning and Transformations

Working with RDDs - Creating and Transforming (map, filter, etc.)

Session 3: Spark SQL, DataFrames, and DataSets

Overview

SparkSession, Loading/Saving Data, Data Formats (JSON, CSV, Parquet, text ...)

Introducing DataFrames (Creation and Schema Inference)

Supported Data Formats (JSON, Text, CSV, Parquet)

Working with the DataFrame (untyped) Query DSL (Column, Filtering, Grouping, Aggregation)

SQL-based Queries

Mapping and Splitting (flatMap(), explode(), and split())

DataFrames vs. RDDs

Session 4: Shuffling Transformations and Performance

Grouping, Reducing, Joining

Shuffling, Narrow vs. Wide Dependencies, and Performance Implications

Exploring the Catalyst Query Optimizer (explain(), Query Plans, Issues with lambdas)

The Tungsten Optimizer (Binary Format, Cache Awareness, Whole-Stage Code Gen)

Session 5: Performance Tuning

Caching - Concepts, Storage Type, Guidelines

Minimizing Shuffling for Increased Performance

Using Broadcast Variables and Accumulators

General Performance Guidelines

Session 6: Creating Standalone Applications

Core API, SparkSession.Builder

Configuring and Creating a SparkSession

Building and Running Applications - sbt/build.sbt and spark-submit

Application Lifecycle (Driver, Executors, and Tasks)

Cluster Managers (Standalone, YARN, Mesos)

Logging and Debugging

Session 7: Spark Streaming

Introduction and Streaming Basics

Streaming Introduction

Structured Streaming (Spark 2+)

Continuous Applications

Table Paradigm, Result Table

Steps for Structured Streaming

Sources and Sinks

Consuming Kafka Data

Kafka Overview

Structured Streaming - "kafka" format

Processing the Stream

Training locations

Introduction to Spark with Python training across the U.S. and beyond.

We provide Introduction to Spark with Python training onsite and live online for organizations throughout the United States and internationally.

View all Introduction to Spark with Python training locations

Ready to train your team?

Let's customize this training for your organization.

Tell us about your employees, your goals, and your preferred schedule. We'll prepare a training proposal covering scope, delivery format, and pricing.

Request a Training Proposal

or call 303.377.6176