Flashcards
Airflow
Cards in this deck (15)
- 1DAGDirected Acyclic Graph
- Collection of tasks, their dependencies and settings.
- Defined in .py script as code. - 2XComFeature for cross communication between tasks.
- 3dags_folder- The folder where airflow pipelines live.
- This path must be absolute.
- Airflow looks in your DAGS_FOLDER for modules that contain DAG objects in their GLOBAL NAMESPACE and adds the objects it finds in the DagBag. - 4DAG Run- An instance of a DAG, containing task instances that run for a specific execution_date.
- Created by the Airflow scheduler or an external trigger. - 5Task- A Task defines a unit of work within a DAG; it is represented as a node in the DAG graph, and it is written in Python.
- Each task is an implementation of an Operator. - 6OperatorAn operator describes a single task in a workflow.
- 7SensorAn Operator that waits (polls) for a certain time, file, database row, S3 key, etc.
- 8chain(op1, [op2, op3], [op4, op5], op6)op1 >> [op2, op3]
op2 >> op4
op3 >> op5
[op4, op5] >> op6 - 9Task InstanceAn instance of a task - that has been assigned to a DAG and has a state associated with a specific DAG run (i.e for a specific execution_date).
- 10execution_dateThe logical date and time for a DAG Run and its Task Instances.
- 11JinjaJinja is a modern and designer-friendly templating language for Python, modelled after Django’s templates.
- 12Hooks- Hooks are interfaces to external platforms and databases like Hive, S3, MySQL, Postgres, HDFS, and Pig.
- Hooks implement a common interface when possible, and act as a building block for operators. - 13PoolsAirflow pools can be used to limit the execution parallelism on arbitrary sets of tasks.
- 14ConnectionsThe information needed to connect to external systems is stored in the Airflow metastore database. A conn_id is defined there, and hostname / login / password / schema information attached to it. Airflow pipelines retrieve centrally-managed connections information by specifying the relevant conn_id.
- 15—