Orchestration using AWS Step Functions on provisioned clusters
AWS Step Functions allow you to author a workflow where each step is decoupled, but the application state can be maintained. AWS Step Functions is integrated with multiple AWS services that allow flexibility to call the specific service in each of the tasks.
AWS Step Functions supports the Amazon States Language, which allows the workflow to be authored and maintained like a JSON file. You can harness AWS Step Functions to execute any complex ETL workflow in Amazon Redshift. AWS Step Functions also integrates with Amazon Redshift serverless.
In this recipe, we will use AWS Step Functions to orchestrate a simple ETL workflow that will submit queries to Amazon Redshift asynchronously using the Amazon Redshift Data API. We will start with creating an AWS Lambda function that will be used to submit and status poll for the queries.
Getting ready
To complete this recipe, you will need the following:
-
...