Managed Airflow

Setting up the Managed Airflow Instance

  1. Go to Managed Apache Airflow Console

  2. Enter a name for the Airflow instance - airflow-yourname-instance

  3. Select the Airflow version 1.10.12 from the drop down Airflow Instance

This workshop is currently designed for Airflow version 1.10.12. Please make sure you have selected this specific version to create the MWAA cluster.

  1. Enter the name of the S3 Bucket - s3://airflow-yourname-bucket

  2. For DAGs folder, enter - s3://airflow-yourname-bucket/dags

  3. For the Plugins, enter - s3://airflow-yourname-bucket/plugins/

  4. For the Requirements file, enter - s3://airflow-yourname-bucket/requirements/requirements.txt Airflow S3 Bucket

    Modifying or adding the Plugins or the requirements file at a later point in time will require an Environment update for the changes to take into effect on your Airflow instances.

  5. Click on Next

  6. Click on Create MWAA VPC Airflow S3 Bucket

  7. It will open up a new Tab with CloudFormation. Scroll to the bottom and click on Create stack

  8. Wait for the status to change to CREATE_COMPLETE for the MWAA-VPC Stack

  9. Once complete, go back to the Managed Apache Airflow console, and select the newly created VPC from the drop down.

    You may need to refresh the drop down to be able to see the new VPC created by CFN.

  10. Under Web server access, change the selection to Public network.

    Selection of Public network for web server access in Production workloads is not recommended for security best practices.

  11. Under Security group(s), leave the default selecting where Create new security group is checked. Airflow S3 Bucket

  12. Under Environment class, proceed with mw1.small and change the Maximum worker count to 4

  13. Under Monitoring, enable the Airflow DAG processing logs, and change the Log level to INFO for the processing logs

  14. Under Airflow configuration options, click on Add custom configuration value

  15. Configuration option => core.enable_xcom_pickling , Custom value => True
    *(This is required for the Sagemaker Module)* Airflow S3 Bucket

  16. Click on Next

  17. Review the configurations and Click on Create environment at the bottom of the page.

The creation of the Airflow instance will take a few minutes to complete.

Meanwhile, note down the name of the Airflow Execution role created and attached to the instance. Airflow S3 Bucket

Let’s proceed to assign the necessary permissions to the Airflow IAM role, we will come back to check on the status here later.