Apache Airflow – a modern way to orchestrate tasks

Course objectives

Introducing Apache Airflow as a central tool for convenient orchestrating tasks between different platforms.

Benefits

Participants will learn and understand:

• Apache Airflow from the user’s point of view
• ways of testing DAGs, building complex dependencies between tasks and good practices of working with the tool

Participants will be able to:

• be fluent in the environment,
• orchestrate data flows knowing all the possibilities and limitations of the tool.

Training scope

• Apache Airflow – a new approach to task orchestration
• Theoretical introduction
  • Historical overview
  • Architecture overview
  • The most common use cases
• Construction of data flows
  • The concept of DAGs and the way of defining them
  • DAG testing
  • Operators – using existing ones and writing your own
  • Airflow REST API
  • Transferring information between tasks – Airflow XComs
  • Advanced ways of defining dependencies between tasks – Trigger Rules
  • Organization of tasks – SubDAGs and groups of tasks
  • Defining conditional relations between tasks – Branching

Audience

Data engineers looking to explore one of the most popular Open Source task orchestration tools, Apache Airfow. Basic knowledge of Python is required.