Introduction to Spark 3 with Python Training in Layton
|
We offer private customized training for groups of 3 or more attendees.
|
||
Course Description |
||
| This course introduces the Apache Spark distributed computing engine, and is suitable for developers, data analysts, architects, technical managers, and anyone who needs to use Spark in a hands-on manner. It is based on the Spark 3.x release. All examples and labs use Python for programming. The course provides a solid technical introduction to the Spark architecture and how Spark works. It covers the basic building blocks of Spark (e.g. RDDs and the distributed compute engine), as well as higher-level constructs that provide a simpler and more capable interface (e.g. DataFrames and Spark SQL). It includes in-depth coverage of Spark SQL and DataFrames, which are now the preferred programming API. This includes exploring possible performance issues and strategies for optimization. The course also covers more advanced capabilities such as the use of Spark Streaming to process streaming data, and integrating with the Kafka server.
The course is very hands-on, with many labs. Participants will interact with Spark through the pyspark shell (for interactive, ad-hoc processing) as well as through programs using the Spark API. After taking this course, you will be ready to work with Spark in an informed and productive manner.
Course Length: 4 Days
Course Tuition: $1890 (US) |
||
Prerequisites |
|
| Working knowledge of some programming language - no Java experience needed | |
Course Outline |
|
Session 1: Introduction to Spark
Overview, Motivations, Spark Systems
Spark Ecosystem
Spark vs. Hadoop
Acquiring and Installing Spark
The Spark Shell, SparkContext
Session 2: RDDs and Spark Architecture
RDD Concepts, Lifecycle, Lazy Evaluation
RDD Partitioning and Transformations
Working with RDDs - Creating and Transforming (map, filter, etc.)
Session 3: Spark SQL, DataFrames, and DataSets
Overview
SparkSession, Loading/Saving Data, Data Formats (JSON, CSV, Parquet, text ...)
Introducing DataFrames (Creation and Schema Inference)
Supported Data Formats (JSON, Text, CSV, Parquet)
Working with the DataFrame (untyped) Query DSL (Column, Filtering, Grouping, Aggregation)
SQL-based Queries
Mapping and Splitting (flatMap(), explode(), and split())
DataFrames vs. RDDs
Session 4: Shuffling Transformations and Performance
Grouping, Reducing, Joining
Shuffling, Narrow vs. Wide Dependencies, and Performance Implications
Exploring the Catalyst Query Optimizer (explain(), Query Plans, Issues with lambdas)
The Tungsten Optimizer (Binary Format, Cache Awareness, Whole-Stage Code Gen)
Session 5: Performance Tuning
Caching - Concepts, Storage Type, Guidelines
Minimizing Shuffling for Increased Performance
Using Broadcast Variables and Accumulators
General Performance Guidelines
Session 6: Creating Standalone Applications
Core API, SparkSession.Builder
Configuring and Creating a SparkSession
Building and Running Applications - sbt/build.sbt and spark-submit
Application Lifecycle (Driver, Executors, and Tasks)
Cluster Managers (Standalone, YARN, Mesos)
Logging and Debugging
Session 7: Spark Streaming
Introduction and Streaming Basics
Streaming Introduction
Structured Streaming (Spark 2+)
Continuous Applications
Table Paradigm, Result Table
Steps for Structured Streaming
Sources and Sinks
Consuming Kafka Data
Kafka Overview
Structured Streaming - "kafka" format
Processing the Stream |
Course Directory [training on all levels]
- .NET Classes
- Agile/Scrum Classes
- Ajax Classes
- Android and iPhone Programming Classes
- Blaze Advisor Classes
- C Programming Classes
- C# Programming Classes
- C++ Programming Classes
- Cisco Classes
- Cloud Classes
- CompTIA Classes
- Crystal Reports Classes
- Design Patterns Classes
- DevOps Classes
- Foundations of Web Design & Web Authoring Classes
- Git, Jira, Wicket, Gradle, Tableau Classes
- IBM Classes
- Java Programming Classes
- JBoss Administration Classes
- JUnit, TDD, CPTC, Web Penetration Classes
- Linux Unix Classes
- Machine Learning Classes
- Microsoft Classes
- Microsoft Development Classes
- Microsoft SQL Server Classes
- Microsoft Team Foundation Server Classes
- Microsoft Windows Server Classes
- Oracle, MySQL, Cassandra, Hadoop Database Classes
- Perl Programming Classes
- Python Programming Classes
- Ruby Programming Classes
- Security Classes
- SharePoint Classes
- SOA Classes
- Tcl, Awk, Bash, Shell Classes
- UML Classes
- VMWare Classes
- Web Development Classes
- Web Services Classes
- Weblogic Administration Classes
- XML Classes
- Microsoft Azure AI Fundamentals (AI-900T00)
25 November, 2024 - 25 November, 2024 - Ruby Programming
2 December, 2024 - 4 December, 2024 - Ruby on Rails
5 December, 2024 - 6 December, 2024 - Fast Track to Java 17 and OO Development
9 December, 2024 - 13 December, 2024 - Introduction to Spring 5 (2022)
16 December, 2024 - 18 December, 2024 - See our complete public course listing
Python Programming Uses & Stats
|
Difficulty
|
Popularity
|
Year Created 1991 |
|
Pros
Easy to Learn:
The learning curve is very mild and the language is versatile and fast to develop.
Massive Libraries:
You can find a library for basically anything: from web development, through game development, to machine learning.
Do More with Less Code:
You can build prototypes and test out ideas much quicker in Python than in other language
|
Cons
Speed Limitations: It is an interpretive language and therefore much slower than compiled languages. Problems with Threading: Multi-threaded CPU-bound programs may be slower than single-threaded ones do to the Global Interpreter Lock (GIL) that allows only one thread to execute at a time. Weak on Mobile: Although, there are a number or libraries that provide a way to develop for both Android and iOS using Python currently Android and iOS don’t support Python as an official programming language. |
| Python Programming Job Market |
Average Salary
|
Job Count
|
Top Job Locations
New York City Mountain View San Francisco |
|
Complimentary Skills to have along with Python Programming
The potential for career growth, whether you are new to the industry or plan to expand your current skills, depends upon your interests:
- For knowledge in building in PC or windows, phone apps or you are looking your future in Microsoft learn C#
- For android apps and also cross platform apps then learn Java
- If you are an Apple-holic and want to build iOS and MAC apps and then choose Objective C or Swift
- Interested in game development? C++
- Data mining or statistics then go with R programming or MATLAB
- Building an operating systems? C
|
Interesting Reads Take a class with us and receive a book of your choosing for 50% off MSRP.
