Introduction to Spark 3 with Scala Training in South Bend
We offer private customized training for groups of 3 or more attendees.
|
||
Course Description |
||
This course introduces the Apache Spark distributed computing engine, and is suitable for developers, data analysts, architects, technical managers, and anyone who needs to use Spark in a hands-on manner. It is based on the Spark 3.x release. The course provides a solid technical introduction to the Spark architecture and how Spark works. It covers the basic building blocks of Spark (e.g. RDDs and the distributed compute engine), as well as higher-level constructs that provide a simpler and more capable interface (e.g. DataSets/DataFrames and Spark SQL). It includes in-depth coverage of Spark SQL, DataFrames, and DataSets, which are now the preferred programming API. This includes exploring possible performance issues and strategies for optimization. The course also covers more advanced capabilities such as the use of Spark Streaming to process streaming data, and integrating with the Kafka server. The course is very hands-on, with many labs. Participants will interact with Spark through the Spark shell (for interactive, ad-hoc processing) as well as through programs using the Spark API. After taking this course, you will be ready to work with Spark in an informed and productive manner. Labs are supported in Scala. There is a separate course for Python users.
Course Length: 4 Days
Course Tuition: $1890 (US) |
Prerequisites |
|
Working knowledge of some programming language - no Java experience needed |
Course Outline |
Session 1 (Optional): Scala Ramp Up
Scala Introduction, Variables, Data Types, Control Flow
The Scala Interpreter
Collections and their Standard Methods (e.g. map())
Functions, Methods, Function Literals
Class, Object, Trait, case Class
Session 2: Introduction to Spark
Overview, Motivations, Spark Systems
Spark Ecosystem
Spark vs. Hadoop
Acquiring and Installing Spark
The Spark Shell, SparkContext
Session 3: RDDs and Spark Architecture
RDD Concepts, Lifecycle, Lazy Evaluation
RDD Partitioning and Transformations
Working with RDDs - Creating and Transforming (map, filter, etc.)
Session 4: Spark SQL, DataFrames, and DataSets
Overview
SparkSession, Loading/Saving Data, Data Formats (JSON, CSV, Parquet, text ...)
Introducing DataFrames and DataSets (Creation and Schema Inference)
Supported Data Formats (JSON, Text, CSV, Parquet)
Working with the DataFrame (untyped) Query DSL (Column, Filtering, Grouping, Aggregation)
SQL-based Queries
Working with the DataSet (typed) API
Mapping and Splitting (flatMap(), explode(), and split())
DataSets vs. DataFrames vs. RDDs
Session 5: Shuffling Transformations and Performance
Grouping, Reducing, Joining
Shuffling, Narrow vs. Wide Dependencies, and Performance Implications
Exploring the Catalyst Query Optimizer (explain(), Query Plans, Issues with lambdas)
The Tungsten Optimizer (Binary Format, Cache Awareness, Whole-Stage Code Gen)
Session 6: Performance Tuning
Caching - Concepts, Storage Type, Guidelines
Minimizing Shuffling for Increased Performance
Using Broadcast Variables and Accumulators
General Performance Guidelines
Session 7: Creating Standalone Applications
Core API, SparkSession.Builder
Configuring and Creating a SparkSession
Building and Running Applications - sbt/build.sbt and spark-submit
Application Lifecycle (Driver, Executors, and Tasks)
Cluster Managers (Standalone, YARN, Mesos)
Logging and Debugging
Session 8: Spark Streaming
Introduction and Streaming Basics
Structured Streaming (Spark 2+)
Continuous Applications
Table Paradigm, Result Table
Steps for Structured Streaming
Sources and Sinks
Consuming Kafka Data
Kafka Overview
Structured Streaming - "kafka" format
Processing the Stream
|
Course Directory [training on all levels]
- .NET Classes
- Agile/Scrum Classes
- Ajax Classes
- Android and iPhone Programming Classes
- Blaze Advisor Classes
- C Programming Classes
- C# Programming Classes
- C++ Programming Classes
- Cisco Classes
- Cloud Classes
- CompTIA Classes
- Crystal Reports Classes
- Design Patterns Classes
- DevOps Classes
- Foundations of Web Design & Web Authoring Classes
- Git, Jira, Wicket, Gradle, Tableau Classes
- IBM Classes
- Java Programming Classes
- JBoss Administration Classes
- JUnit, TDD, CPTC, Web Penetration Classes
- Linux Unix Classes
- Machine Learning Classes
- Microsoft Classes
- Microsoft Development Classes
- Microsoft SQL Server Classes
- Microsoft Team Foundation Server Classes
- Microsoft Windows Server Classes
- Oracle, MySQL, Cassandra, Hadoop Database Classes
- Perl Programming Classes
- Python Programming Classes
- Ruby Programming Classes
- Security Classes
- SharePoint Classes
- SOA Classes
- Tcl, Awk, Bash, Shell Classes
- UML Classes
- VMWare Classes
- Web Development Classes
- Web Services Classes
- Weblogic Administration Classes
- XML Classes
- Introduction to Spring 5 (2022)
16 December, 2024 - 18 December, 2024 - Introduction to C++ for Absolute Beginners
16 December, 2024 - 17 December, 2024 - See our complete public course listing
Java Programming Uses & Stats
Difficulty
|
Popularity
|
Year Created 1995 |
Pros
Most Commonly Used:
Great Career Choice:
Android Apps Development:
It Can Run On Any Platform:
Great Supporting IDE's: |
Cons
Uses a Lot of Memory:
Difficulty in Learning:
Slow Start Up Times:
Verbose and Complex Code:
Commercial License Cost: |
Java Programming Job Market |
Average Salary
|
Job Count
|
Top Job Locations
New York City |
Complimentary Skills to have along with Java Programming
- If you are an experienced Java developer, learning a complimentary language to Java should come much more naturally. As an example JetBrains recently created the Kotlin programming language which is officially supported by Google for mobile development. Kotlin compiles to Java bytecode and runs on the JVM; it's purported to address many of Java's shortcomings... |