Introducing Apache Airflow as a central tool for convenient orchestrating tasks between different platforms.
Participants will learn and understand:
- Apache Airflow from the user’s point of view
- ways of testing DAGs, building complex dependencies between tasks and good practices of working with the tool
Participants will be able to:
- be fluent in the environment,
- orchestrate data flows knowing all the possibilities and limitations of the tool.
- Apache Airflow – a new approach to task orchestration
- Theoretical introduction
- Historical overview
- Architecture overview
- The most common use cases
- Construction of data flows
- The concept of DAGs and the way of defining them
- DAG testing
- Operators – using existing ones and writing your own
- Airflow REST API
- Transferring information between tasks – Airflow XComs
- Advanced ways of defining dependencies between tasks – Trigger Rules
- Organization of tasks – SubDAGs and groups of tasks
- Defining conditional relations between tasks – Branching
Data engineers looking to explore one of the most popular Open Source task orchestration tools, Apache Airfow. Basic knowledge of Python is required.