Apache Spark – building systems for real-time data processing

Why take this course

If you want to learn a distributed computing platform and build applications that process large amounts of data, then this course is for you. You’ll gain knowledge and an understanding of Apache Spark’s architecture and learn how to build scalable applications. You will be able to create data pipelines using Apache Spark and discover modern concepts that solve problems in the world of Big Data, such as Delta Lake.

What you'll learn

• Create applications using Apache Spark.
• Build applications that process large volumes of data.
• The architecture of the Apache Spark platform.
• The development environment.
• The principles of building scalable applications.
• How to build data pipelines using Apache Spark.
• Modern concepts that address many common problems in the world of Big Data (Delta Lake).

Certification & Exam

Upon completion of the training, you will receive a certificate of participation, confirming your skills in creating data processing applications with Apache Spark. There is no final exam—all you need is active participation in the classes.

Who is this course for

This course is designed for anyone involved in the software development process and members of project teams. The course is also for developers who know the basics of programming and want to learn the fundamentals of the Scala language, which are necessary for writing effective applications using Apache Spark.

Topics covered

• Apache Spark – architecture
• Historical overview
  • Solution architecture
  • Running the application
  • Monitoring
  • Troubleshooting / debugging
• Data processing with Apache Spark
  • RDDs, DataFrames, and DataSets
  • Spark SQL
  • Joins
  • File formats
  • Data aggregation
• Preparing the development environment to work with Apache Spark (part conducted using Scala)
  • Working with IntelliJ
  • Introduction to SBT
  • Passing parameters / configuration using external libraries
  • Code testing
• Delta Lake – a format that facilitates data processing
  • Introduction to the concept
  • Writing / reading data using the Delta format
  • The most important functions and differences compared to classic files (Parquet / ORC)