HARTMANN software group

Introduction to Spark Programming - 4 Day Training in Schenectady

Enroll in or hire us to teach our Introduction to Spark Programming - 4 Day class in Schenectady, New York by calling us @303.377.6176. Like all HSG classes, Introduction to Spark Programming - 4 Day may be offered either onsite or via instructor led virtual training. Consider looking at our public training schedule to see if it is scheduled: Public Training Classes
Provided there are enough attendees, Introduction to Spark Programming - 4 Day may be taught at one of our local training facilities.
We offer private customized training for groups of 3 or more attendees.
get pricing information

Course Description

  
This course introduces the Apache Spark distributed computing engine, and is suitable for developers, data analysts, architects, technical managers, and anyone who needs to use Spark in a hands-on manner. It is based on the Spark 2.x release. The course provides a solid technical introduction to the Spark architecture and how Spark works. It covers the basic building blocks of Spark (e.g. RDDs and the distributed compute engine), as well as higher-level constructs that provide a simpler and more capable interface (e.g. DataSets/DataFrames and Spark SQL). It includes in-depth coverage of Spark SQL, DataFrames, and DataSets, which are now the preferred programming API. This includes exploring possible performance issues and strategies for optimization. The course also covers more advanced capabilities such as the use of Spark Streaming to process streaming data, and integrating with the Kafka server. The course is very hands-on, with many labs. Participants will interact with Spark through the Spark shell (for interactive, ad-hoc processing) as well as through programs using the Spark API. After taking this course, you will be ready to work with Spark in an informed and productive manner. Labs currently support Scala - contact us for Python/Java support.
Course Length: 4 Days
Course Tuition: $1690 (US)

Prerequisites
Reasonable programming experience. An overview of Scala is provided for those who don't know it.

Course Outline

  

Session 1 (Optional): Scala Ramp Up

  • Scala Introduction, Variables, Data Types, Control Flow
  • The Scala Interpreter
  • Collections and their Standard Methods (e.g. map())
  • Functions, Methods, Function Literals
  • Class, Object, Trait, case Class

Session 2: Introduction to Spark

  • Overview, Motivations, Spark Systems
  • Spark Ecosystem
  • Spark vs. Hadoop
  • Acquiring and Installing Spark
  • The Spark Shell, SparkContext

Session 3: RDDs and Spark Architecture

  • RDD Concepts, Lifecycle, Lazy Evaluation
  • RDD Partitioning and Transformations
  • Working with RDDs - Creating and Transforming (map, filter, etc.)

Session 4: Spark SQL, DataFrames, and DataSets

  • Overview
  • SparkSession, Loading/Saving Data, Data Formats (JSON, CSV, Parquet, text ...)
  • Introducing DataFrames and DataSets (Creation and Schema Inference)
  • Supported Data Formats (JSON, Text, CSV, Parquet)
  • Working with the DataFrame (untyped) Query DSL (Column, Filtering, Grouping, Aggregation)
  • SQL-based Queries
  • Working with the DataSet (typed) API
  • Mapping and Splitting (flatMap(), explode(), and split())
  • DataSets vs. DataFrames vs. RDDs

Session 5: Shuffling Transformations and Performance

  • Grouping, Reducing, Joining
  • Shuffling, Narrow vs. Wide Dependencies, and Performance Implications
  • Exploring the Catalyst Query Optimizer (explain(), Query Plans, Issues with lambdas)
  • The Tungsten Optimizer (Binary Format, Cache Awareness, Whole-Stage Code Gen)

Session 6: Performance Tuning

  • Caching - Concepts, Storage Type, Guidelines
  • Minimizing Shuffling for Increased Performance
  • Using Broadcast Variables and Accumulators
  • General Performance Guidelines

Session 7: Creating Standalone Applications

  • Core API, SparkSession.Builder
  • Configuring and Creating a SparkSession
  • Building and Running Applications - sbt/build.sbt and spark-submit
  • Application Lifecycle (Driver, Executors, and Tasks)
  • Cluster Managers (Standalone, YARN, Mesos)
  • Logging and Debugging

Session 7: Spark Streaming

  • Introduction and Streaming Basics
  • Spark Streaming (Spark 1.0+)
    • DStreams, Receivers, Batching
    • Stateless Transformation
    • Windowed Transformation
    • Stateful Transformation
  • Structured Streaming (Spark 2+)
    • Continuous Applications
    • Table Paradigm, Result Table
    • Steps for Structured Streaming
    • Sources and Sinks
  • Consuming Kafka Data
    • Kafka Overview
    • Structured Streaming - "kafka" format
    • Processing the Stream

Course Directory [training on all levels]

Technical Training Courses
Software engineer/architect, System Admin ... Welcome!
Business Training Courses
Project Managers, Business Analysts, Paralegals ... Welcome!
Upcoming Classes
Gain insight and ideas from students with different perspectives and experiences.

Interesting Reads Take a class with us and receive a book of your choosing for 50% off MSRP.