Orchestration using Amazon Managed Workflows for Apache Airflow on provisioned clusters
Amazon Managed Workflows for Apache Airflow (MWAA) brings automation to life by managing complex data pipelines from beginning to end. At its core, it handles Apache Airflow, which creates, schedules, and monitors your workflows with precision. Each data pipeline breaks down into smaller, interconnected tasks that work together seamlessly in a coordinated flow.
Using Python, developers craft these workflows as Directed Acyclic Graphs (DAGs), defining the exact path and sequence for data processing. The system grows and adapts through powerful plugins, while a user interface provides clear visibility into every workflow’s status and progress. Working hand in hand with Amazon Redshift serverless, it creates a complete ecosystem for data processing, all while Amazon manages the underlying infrastructure.
In this recipe, we will build the underlying infrastructure used for Apache Airflow...