Apache Spark – building systems for real-time data processing

Course objectives

Introduction to Apache Spark as an effective data processing tool.

Benefits

Participants will learn and understand:

• Apache Spark platform architecture
• Development environment
• Principles of building scalable applications
• Building data flows using Apache Spark
• Modern concepts addressing many known problems in the world of Big Data (Delta Lake)

Participants will be able to:

• Create applications using Apache Spark
• Create applications that process large amounts of data

Training scope

• Apache Spark – architecture
• Historical overview
  • Solution architecture
  • Running the application
  • Monitoring
  • Troubleshooting / debugging
• Data processing with Apache Spark
  • RDDs, DataFrames, and DataSets
  • Spark SQL
  • Joins
  • File formats
  • Data aggregation
• Preparing the development environment to work with Apache Spark (part conducted using Scala)
  • Working with IntelliJ
  • Introduction to SBT
  • Passing parameters / configuration using external libraries
  • Code testing
• Delta Lake – a format that facilitates data processing
  • Introduction to the concept
  • Writing / reading data using the Delta format
  • The most important functions and differences compared to classic files (Parquet / ORC)

Audience

All people involved in the software development process, members of project teams. Developers familiar with the basics of programming, who want to learn the basics of the Scala language necessary to effectively write applications using Apache Spark.