YAML File - Dag Definition

The YAML file for our DAG lays out how the DAG should be setup. Let’s break it down below:

Defaults Section

This section provides a set of default_args that we can re-use. This provides access to the same default interface that is used in a normal python-DAG definition:

Defaults Defintion

Example DAG Defintions

In these next few lines, we see a DAG definition. A few things to note:

  1. We can overwrite the default args just like a normal DAG definition in python
  2. We include a set of tasks
  3. Each task includes an operator and a command.
  4. Some tasks contain dependencies and these taken together create our graph

Example DAG

The example DAG above overrides the owner from our default args. In addition it runs a custom Python function from our print_hello.py file. This demonstrates how executable python code (written by engineering team members) can be included with the no-code template.

The remaining DAGs just spawn simple bash operators with commands. Note however that multiple DAGs can be templated in one file. This can enable teams to submit DAG definitions or to otherwise group DAGs relating to a particular use case.

Example DAG2

Let’s go over to the Admin UI to see these DAGs in action.