Enter a name for the Airflow instance -
Select the Airflow version 2.0.2 from the drop down
This workshop is currently designed for Airflow version 2.0.2. Please make sure you have selected this specific version to create the MWAA cluster.
Enter the name of the S3 Bucket -
For DAGs folder, enter -
For the Plugins, enter -
For the Requirements file, enter -
Modifying or adding the Plugins or the requirements file at a later point in time will require an Environment update for the changes to take into effect on your Airflow instances.
Click on Next
Click on Create MWAA VPC
It will open up a new Tab with CloudFormation. Scroll to the bottom and click on Create stack
Wait for the status to change to CREATE_COMPLETE for the MWAA-VPC Stack
Once complete, go back to the Managed Apache Airflow console, and select the newly created VPC from the drop down.
You may need to refresh the drop down to be able to see the new VPC created by CFN.
Under Web server access, change the selection to Public network.
Selection of Public network for web server access in Production workloads is not recommended for security best practices.
Under Security group(s), leave the default selecting where Create new security group is checked.
Under Environment class, proceed with mw1.small and change the Maximum worker count to
Under Monitoring, enable the Airflow DAG processing logs, and change the Log level to INFO for the processing logs
Under Airflow configuration options, click on Add custom configuration value
Configuration option =>
core.enable_xcom_pickling , Custom value =>
*(This is required for the Sagemaker Module)*
Click on Next
Review the configurations and Click on Create environment at the bottom of the page.
Meanwhile, note down the name of the Airflow Execution role created and attached to the instance.
Let’s proceed to assign the necessary permissions to the Airflow IAM role, we will come back to check on the status here later.