Building Serverless Python Web Services with Zappa

There are some technical prerequisites before moving ahead. We are going to demonstrate AWS through a web console and the AWS CLI. The following prerequisites should be considered:

Web hosting has changed drastically since it started. Physical server machines were shared among many web applications, and it was a really big challenge when it came to scale. It proved to be very expensive for any individual or company to afford an entire server machine to host their web application.

But, thanks to virtualization, the need for a physical server for any web application has been eliminated. Virtualization provides the ability to create many virtual servers as opposed to a single physical server.

Now, the new era of serverless is making the developer's life easier, as we can focus our hard work on development instead of investing time and money on deployment.

Amazon introduced Amazon Elastic Compute Cloud (Amazon EC2) as a cloud computing solution. Amazon EC2 makes it possible to create an array of virtual servers or instances the Amazon Cloud without investing in hardware. You can scale it as per your requirements in terms of networking, computing, and storage.

The serverless approach is nothing but the process of eliminating the manual workload of setting up the hosting environment. Cloud service providers provide serverless services, and so you never actually own any server. Instead, the cloud service provider executes your code in a high-availability infrastructure.

Many cloud service providers introduced different services for the serverless infrastructure. Amazon introduced AWS Lambda as a compute service, where you just provide your code and AWS Lambda executes the code in a highly scalable infrastructure by itself. You don't need to worry about managing the services manually. You need to pay for the compute time of your code execution, and there are no charges when your code is not running.

AWS Lambda executes the code as needed in response to events such as data storage events on an S3 bucket, Amazon DynamoDB events, and HTTP request events via the API Gateway. AWS Lambda is able to execute the code based on scheduled time events via AWS CloudWatch Events. AWS Lambda supports Python, Node.js, C#, and Java.

You need to write a function, which will be executed by AWS Lambda on your behalf.

AWS Lambda is implemented on a container-based model that supports a runtime environment and executes the code as per the Lambda function configuration. When the Lambda function is invoked, it launches the container (an execution environment) based on the AWS Lambda configuration and enables the basic runtime environment, which is required to execute the code.

Let's start with some practical work:

There are three options available for creating a role:

HelloWorld Lambda function

The Lambda function includes three sections:

Let's look at detailed information about the configuration and monitoring. We will have a separate section for triggers. 

def lambda_handler(event, context): 
    print("value1 = " + event['key1']) 
    print("value2 = " + event['key2']) 

Monitoring

AWS CloudWatch is a monitoring service for AWS resources and manages all activity logs. It creates metric data to generate statistical data. CloudWatch enables real-time monitoring of AWS resources. It also monitors hardware information related to AWS EC2 or RDS database instances and other resources.

Lambda monitoring sections display the last 24 hours' analytics data related to the Lambda function's activity and performance. The following screenshot shows the monitored analytics information about our hello world Lambda function:

Let's move on to the next section, where we are going to look at the Lambda function execution.

AWS Lambda supports several methods of execution. Let's start with the basic execution from its own web console interface. AWS Lambda provides the capability to test the function manually, where you can define the test event context. If you want to test against some other Amazon services, then there are built-in event templates available.

The following screenshot demonstrates the test event creation:

As shown in the preceding screenshot, a single Lambda function can have a maximum of 10 test events and the test events are persisted, so you can reuse them whenever you want to test your Lambda function.

I created the test event with the event name as HelloWorld and now I am going to execute the HelloWorld function, when converting the Lambda function as a Python microservice, as shown in the following code:

from __future__ import print_function 
import json 
 
print('Loading function') 
 
def lambda_handler(event, context): 
    print("Received event: " + json.dumps(event, indent=2)) 
    print("value1 = " + event['key1']) 
    print("value2 = " + event['key2']) 
    print("value3 = " + event['key3']) 
    return "Hello World" 

Here, we are printing the event data and then returning back to the Hello World string:

Lambda manages some information on every request execution, such as a request ID and billing information. The Lambda price model is based on the time consumption on request processing, whereas the request ID is the unique identification of every request.

In the Log output, you can see all the print statements output. Now, let's raise an error and see how Lambda responds and returns the logs.

We are going to replace the current code with the following snippet:

from __future__ import print_function 
import json 
 
print('Loading function') 
 
 
def lambda_handler(event, context): 
    print("Received event: " + json.dumps(event, indent=2)) 
    raise Exception('Exception raised manually.') 

The following screenshot is the log snippet of the execution result:

Here, Lambda responded with the complete stack trace information and logged it as well. You can check the CloudWatch logs, as CloudWatch is preconfigured with the AWS Lambda execution.

We learned about the Lambda function execution from the Lambda console, and now it's time to execute the Lambda function from a schedule trigger. In our project, we often need to have a cron job schedule to execute some functionality at a particular time period.

Lambda triggers will help us to set up the triggers based on events. Let's move ahead to introduce the trigger to our hello world function.

The Lambda function can be configured in response to events. AWS provides a list of triggers that support lots of events. These triggers belong to their associated AWS services.

You can add a trigger to your Lambda function from the triggers section.

I am going to slightly modify the hello world Lambda function. Here, we are printing the request ID, which is received in the context object as an aws_request_id attribute. It also prints the timestamp:

Now, we are going to add a trigger to our Lambda function that will execute our Lambda function every minute.

The following screenshot shows the Add trigger flow, where you can easily configure any trigger from the left-hand panel with your Lambda function:

We are going to configure the CloudWatch Events trigger. CloudWatch Events deliver near real-time system events that describe the changes in AWS resources.

You can set up simple event rules with operational events in AWS resources as they occur, and you can also schedule automated events that self-trigger based on cron or the rate expression.

Here, we are going to schedule a rate expression to execute our hello world Lambda function every minute. We need to select CloudWatch Events from the triggers dropdown.

To create the CloudWatch event rule, we are going to create a new rule. We need to set up the rule with some required information, such as the rule name, which is a unique identifier. So, we are going to name the rule as hello-world-every-minute and the rule type as either the event pattern or schedule expression. In our case, it would be a schedule expression as the rate (1 minute), as shown in the preceding screenshot.

Once we set the trigger and enable it, the scheduled event would get triggered as per the schedule expression. Let's see our hello world Lambda logs after five minutes.

To view the logs related to any services, you need to do the following:

The following screenshot describes the CloudWatch log access:

By selecting the HelloWorld Lambda function log groups, you see the logging activity related to our HelloWorld Lambda function. The following screenshot shows the logs of the HelloWorld function:

Here, you can see that our hello world Lambda function is executed exactly every minute since the time we have enabled the trigger.  

Now, let's move ahead to create a serverless RESTful API.

Let's understand the microservice scenario where we are going to deploy a serverless hello world function that will respond to HTTP events through the API Gateway.

The Amazon API Gateway service enables you to create, manage, and publish a RESTful API to interact with AWS resources at any scale. The API Gateway provides an interface where you expose the backend through the REST application programming interface.

To enable the AWS serverless infrastructure, the API Gateway plays an important role, as it can be configured to execute the Lambda functions. 

Now, we are going to configure an API Gateway service to executes the Lambda function

Here is the hello world function:

When we integrate the AWS Lambda function with the API Gateway, the Lambda function must return a dictionary object with the required keys as statusCode, headers, and body.  The value of the body attribute must be in a JSON string. Hence, we converted the Python dictionary into a JSON string.

It's time to integrate the API Gateway with the Lambda function. As we have seen in our previous discussion about triggers, we are going to add a trigger with the API Gateway:

We are going to create an API Gateway service with the name as LambdaMicroservice. The API Gateway enables you to create and maintain a deployment stage as per your requirement.

If you want to secure your API then you have two options—using AWS IAM and opening it with the access key, or keeping it as open, making it publicly available.

AWS IAM (Identity Access Management) is an AWS cloud service that is helpful in creating a secure access credential in order to access AWS cloud services.

Opening with the access key feature allows you to generate the key from the API Gateway console. In our case, we are going to keep the security open only, as we need to access our API publicly:

Once you add and save the changes, the REST API is ready within a few seconds. The invoke URL is our REST API endpoint.

Let's hit the invoke URL using the curl command-line tool and see what happens:

$ curl https://cfi6872cxa.execute-api.us-east-2.amazonaws.com/prod/HelloWorld
{"message": "Hello World returned in JSON"}

That's it. We are done with creating a serverless RESTful API using AWS Lambda and the API Gateway. Now, we are going to see how we can interact with the AWS services using the AWS CLI.

The AWS CLI is an open source tool developed on top of AWS SDK for Python using the Boto library, which provides commands to interact with AWS services. With the very minimum configuration, you can manage any AWS services from the CLI. It provides direct access to AWS services and you can develop shell scripts to manage your resources.

For example, if you want to upload the file to the S3 bucket, then you can do so by just a single command from the CLI:

$ aws s3 cp index.html s3://bucket-name/

aws s3 cp is a shell-like command that performs the multi-part file upload operation in order to complete the operation.

It also supports customization for some of the AWS services. You can see the list of AWS services supported by aws-cli by using the --help command.

$ pip install awscli --upgrade 

Configuring the AWS CLI

awscli directly accesses AWS services but we need to configure and authenticate it in order to access AWS services.

Run the aws configure command to configure the AWS CLI with your Amazon account:

You can get the AWS access key ID and AWS secret access key from the My Security Credentials option, as shown in the following screenshot:

Let's configure the AWS Lambda function using AWS CLI.

Let's configure our hello world Lambda function and triggers using the awscli utility command.

The AWS CLI supports all available AWS services. You can see a detailed description of the aws command using aws help and it will also list all the available services.

We are interested in Lambda, as we are going to create a Lambda function with a simple hello world context:

$ aws lambda help 

This will list a complete description of the Lambda service and all the available commands that are required to manage the AWS Lambda service.

Creating a Lambda function

Here, we are going to create a new Lambda function with the aws lambda create-function command. To run this command, we need to pass the required and optional arguments.

Make sure you have a role with permission for the lambda:CreateFunction action.

Previously, in the AWS Lambda console, we chose the code entry point as inline editing. Now, we will be using a ZIP file as a deployment package.

Before creating the Lambda function, we should create a Lambda function deployment package.

This deployment package will be a ZIP file consisting of your code and any dependencies.

If your project has some dependencies, then you must install the dependencies in the root directive of the project. For example:

   $ pip install requests -t <project-dir> OR  
   $ pip install -r requirements.txt  -t <project-dir>

Here, the -t option indicates the target directory.

Create a simple lambda_handler function in a file named as handler.py, as shown in the following screenshot:

Now, let's make a deployment package as a ZIP file consisting of the preceding code:

Now, we are ready to create the Lambda function. The following screenshot describes the command execution:

You can see that, in the AWS Lambda console, the Lambda function immediately got created:

Let's discuss the required and optional parameters that we used with the aws lambda create-function command:

• --function-name (required): The name is self-explanatory. We need to pass the Lambda function name that we are intending to create.
• --role (required): This is a required parameter where we need to use the AWS role ARN as a value. Make sure this role has permissions to create the Lambda function.
• --runtime (required): We need to mention the runtime environment for the Lambda function execution. As we mentioned earlier, AWS Lambda supports Python, Node.js, C#, and Java. So these are the possible values:
  • python2.7
  • python3.6
  • nodejs
  • nodejs4.3
  • nodejs6.10
  • nodejs4.3-edge
  • dotnetcore1.0
  • java8
• --handler (required): Here, we mention the function path that will be an execution entry point by the AWS Lambda. In our case, we used handler.lambda_function, where the handler is the file that contains the lambda_function.
• --description: This option lets you add some text description about your Lambda function.
• --zip-file: This option is used to upload the deployment package file of your code from your local environment/machine. Here, you need to add fileb:// as a prefix to your ZIP file path.
• --code: This option helps you upload the deployment package file from the AWS S3 bucket.

You should pass a string value with a pattern, such as the one shown here:

 "S3Bucket=<bucket-name>,S3Key=<file-name>,S3ObjectVersion=<file-version-id>". 

There are many other optional parameters available that you can see with the help command, such as aws lambda create-function help. You can use them as per your requirement.

Now we will see the Lambda function invocation using the command $ aws lambda invoke.

 

 

Invoking the function

The Lambda CLI provides a command to directly invoke the Lambda function:

$ aws lambda invoke --function-name <value> <outfile>

Let's look at the parameters:

• --function-name (required): This parameter asks for a Lambda function name
• outfile (required): Here, you need to mention a filename where the returned output or response by the Lambda function will be stored

Here are other optional parameters available that you can list by the help command.

Let's invoke our recently created HelloWorldCLI function:

When we invoked the Lambda function, it immediately responded with a status code and the Lambda function returned the output data stored in the newly created lambda_output.txt file by the lambda invoke command.

In this chapter, we learned about the manual process of creating a simple AWS Lambda and configuring some triggers against it. Also, we looked at the AWS Lambda configuration using the AWS CLI. It's really amazing to implement a serverless application. These AWS services play an essential part in creating a serverless infrastructure, where you can develop your application and deploy it as serverless.