Event-driven applications using AWS Lambda on Amazon Redshift provisioned clusters
AWS Lambda helps you build an event-driven microservice. This serverless process can be invoked using a variety of events such as when a file arrives, when a notification is received, and so on. This helps build a decoupled data workflow that can be invoked as soon as the upstream dependencies are met, instead of a schedule-based workflow. For example, let’s say we have a website that is continuously sending clickstream logs every 15 minutes into Amazon S3.
Instead of accumulating all the log files and processing them at midnight in a typical ETL process, Amazon S3 can send an event to a Lambda function when an object is created and processed immediately. This provides several advantages, such as processing in smaller batch sizes to meet an SLA and to have the data current within the provisioned cluster.
In this recipe, you will learn how to use Python-based AWS Lambda to copy data into...