Amazon EC2 Cookbook

You need to ask yourself several questions in order to choose the right AWS EC2 instance for meeting your requirements. These include: What is the primary purpose of the EC2 instance being provisioned? What is the duration of your need for a particular machine? Do you need high performance storage? Should you go for dedicated or shared tenancy? Will the machine be used for compute-intensive or memory-intensive processing? What are the scalability, availability, and security requirements? What are your networking requirements? There are several options available for each of these parameters, and we will describe them in our recipes for making the right choices. For low latency, you can host your application in the AWS region nearest to the end user. Each AWS region is a separate geographic area, and has multiple isolated locations called availability zones. These availability zones are individual data centers in each region. They are used to deploy fault-tolerant and highly available applications. The latency between these availability zones is very low. If something goes wrong in an availability zone, then it does not affect the systems in another availability zone.

An EC2 instance is a virtual machine hosted on the AWS Cloud. As an instance creator, you have root privileges on any instances you started. An EC2 instance can be used to host one or more of web servers, application servers, database servers, or backend processes/services requiring heavy compute or graphics processing. Depending on your application architecture, you can choose to host various components distributed across multiple EC2 instances.

AWS offers different types of storage attachments viz. SSD and magnetic. If you require higher storage performance, then ensure that the EC2 instance type you choose supports SSD.

There are three distinct purchasing options available for provisioning the AWS EC2 instances:

There are several AWS EC2 instance families available for different types of application workloads. These include general purpose, memory optimized, compute optimized, storage optimized, and GPU instances. Choosing the right instance type is a key decision in provisioning EC2 instances.

We recommend that you start with a minimum required instance type that meets your requirements. In many cases, choosing a general-purpose EC2 instance is a good starting point. You can then load test your application on this instance for overall performance and stability. If your applications are not meeting your performance objectives on the current instance type, you can easily upgrade the size or choose a more specialized instance type, though this process does require a reboot of your instance. This approach can help you optimize your instance sizes and types.

To achieve high performance or meet compliance requirements or to just avoid noisy neighbors, the type of tenancy chosen is a critical decision. On AWS, there are two types of tenancy, dedicated and shared. In the case of dedicated tenancy, AWS provisions your instance on dedicated hardware. These instances are isolated from instances created using the shared tenancy option and instances created by other tenants. Tenancy can be configured at the instance level or at the VPC level. Once the option is selected, changing the tenancy type (instance or VPC level) is not allowed. There are cost implications of using dedicated tenancy versus shared tenancy.

In addition, if we want to set the Provisioned IOPS parameter, then we have to use the EBS-optimized instance types. Amazon EBS-optimized instances deliver dedicated throughput to Amazon EBS, with options ranging between 500 Mbps and 2,000 Mbps (depending on the instance type selected). EBS-optimized flag provides dedicated and more consistent link between EC2 and EBS. EBS optimized EC2 instances also allocate dedicated bandwidth to its attached volumes.

AWS CLI is a set of unified command-line tools to work with multiple AWS services. Using AWS CLI tools you can manage EC2 resources (such as instances, security groups, and volumes) and your VPC resources (such as VPCs, subnets, route tables, and Internet gateways).

Your EC2 instance receives a private IP address from the EC2-Classic range each time it's started, whereas your instance receives a static private IP address from the address range in EC2-VPC. You can only have one private IP address in EC2-Classic, but in EC2-VPC, we have multiple private IP addresses. If you attach an EIP (Elastic IP) to EC2-Classic instance, it will get dissociated when you stop the instance. But for VPC EC2 instance, it remains associated even after you stop it. We can create subnets, routing tables, and Internet gateways in VPC. For on-premise connectivity, we need VPC.

If you created your AWS account after 2013-12-04, then only EC2-VPC is supported. In this case, a default VPC is created in each AWS region. Therefore, unless you create your own VPC and specify it when you launch an instance, your instances are launched in your default VPC.

If you created your AWS account before 2013-03-18, then both EC2-Classic and EC2-VPC are supported in the regions you used before, and only EC2-VPC in regions that you didn't use. In this case, a default VPC is created in each region in which you haven't created any AWS resources. Therefore, unless you create your own VPC and specify it when you launch an instance in a region (that you haven't used before), the instance is launched in your default VPC for that region. However, if you launch an instance in a region that you've used before, the instance is launched in EC2-Classic.

In this recipe, we will launch EC2 instances using EC2-Classic and EC2-VPC.

$ aws ec2 describe-images
--filter [Filter] 

$ aws ec2 describe-images
--filter
"Name=virtualization-type,Values=paravirtual"
"Name=root-device-type,Values=ebs" "Name=architecture,Values=x86_64"
"Name=name,Values=ubuntu/images/ebs/ubuntu-precise-12.04-amd64-server-20130204"

$ aws ec2 run-instances 
--image-id [ImageId] 
--count [InstanceCount] 
--instance-type [InstanceType] 
--key-name [KeyPairName] 
--security-group-ids [SecurityGroupIds]

$ aws ec2 run-instances 
--image-id ami-7e2c612c 
--count 1 
--instance-type t1.micro 
--key-name WebServerKeyPair 
--security-group-ids sg-ad70b8c9

$ aws ec2 run-instances 
--image-id [ImageId] 
--count [InstanceCount] 
--instance-type [InstanceType] 
--key-name [KeyPairName] 
--security-group-ids [SecurityGroupIds]
--subnet-id [SubnetId]

$ aws ec2 run-instances 
--image-id ami-7e2c612c 
--count 1 
--instance-type t1.micro 
--key-name WebServerKeyPair
--security-group-ids sg-ad70b8c8 
--subnet-id subnet-aed11acb

Elastic IP (EIP) address is the static public IP address. You can attach and detach the EIP from EC2 instance at any time. Instances in EC2-Classic support only one private IP address and corresponding EIP. Instances in EC2-VPC support multiple private IP addresses, and each one can have a corresponding EIP. If you stop the instance in EC2-Classic the EIP is disassociated from instance, and you have to associate it again when you start the instance. But if you stop the instance in EC2-VPC, the EIP remains associated with the EC2 instance.

In this recipe, we list the commands for allocating an Elastic IP address in a VPC and associating it with the network interface.

With multiple NICs, you can better manage your network traffic. Multiple NICs is one of the prerequisite for high availability. The number of NICs attached to the EC2 instance will depend on the type of EC2 instance. ENI's and multiple private IP addresses are only available for instances running in a VPC. In cases of instance failure, we can detach and then re-attach the ENI to a standby instance, where DNS changes are not required for achieving business continuity. We can attach multiple ENIs from different subnets to an instance, but they both should be in the same availability zone. This enables us to separate the public-facing traffic from the management traffic.

We can have one primary address and one or more secondary addresses for an NIC. We can detach and then attach NIC from one instance to another. We can attach one Elastic IP to each private address. When you launch an instance, a public IP address can be autoassigned to the network interface for eth0. This is possible only when you create a network interface for eth0 instead of using an existing network interface. You can detach secondary NIC (ethN) when an instance is running or stopped. However, you can't detach the primary (eth0) interface. In addition, you can attach security groups to NIC. If you set the instance termination policy to delete on termination, then the NIC will automatically be deleted, if you delete the EC2 instance.

$ aws ec2 attach-network-interface 
--network-interface-id [NetworkInterfaceId]
--instance-id [InstanceId]
--device-index [DeviceIndex]

$ aws ec2 attach-network-interface 
--network-interface-id eni-5c88f739 
--instance-id i-2e7dace3 
--device-index 1

$ aws ec2 associate-address
--network-interface-id [NetworkInterfaceId]
--allocation-id [AllocationId]
--private-ip-address [PrivateIpAddress]

$ aws ec2 associate-address
--network-interface-id eni-5c88f739
--allocation-id eipalloc-d59f80b7
--private-ip-address 10.0.0.26

Instance storage consists of disks that are physically attached to the host computer. Data on these disks is lost once the instance restarts. For persistence across restarts, we need to use EBS volumes.

EBS volumes are automatically replicated within its availability zone to protect against component failures.

AWS EBS volumes are persisted independently from your EC2 instances. These are connected through Network Attached Storage (NAS). If you lose the EC2 instance, then the data stored on EBS will still be available to a newly provisioned EC2 instance. You can attach as many EBS volumes as you want. However, an EBS volume can only be attached to one EC2 instance at a time. You can detach EBS volume from one EC2 instance, and then attach to a different EC2 instance. An I/O request of up to 256 Kilobytes is counted as a single I/O operation (IOP).

If we use standard EBS volumes as the boot device volume, then the boot process of a Windows or Linux machine is fast. We can have storage up to 16 TB and 10,000 IOPS per volume. General purpose SSD is best for boot device volumes, and small and medium sized databases. These SSD volumes can deliver a maximum throughput of 160 Mbps when attached to EBS-optimized instances.

Provisioned IOPS (SSD) volumes deliver within 10% of the IOPS performance 99.9% of the time over a given year. If we have a 200 GB volume with 1,000 IOPS, then 99.9% of the time, actual I/O on this volume will be at 900 IOPS or higher. Many database workloads need provisioned IOPS for consistent performance. We can configure storage up to 16 TB and 20,000 IOPS per volume. Provisioned IOPS volumes can deliver 320 Mbps when attached to EBS-optimized instances.

Magnetic disks are a lower cost option for EBS volumes. If data read frequency is low then this type of EBS volume is a good option.

Encryption is also possible while using the EBS volumes. Encryption is done for data at rest, data in transit, and disk I/O. Using encrypted EBS volumes have a minor effect on I/O latency, but the performance remains the same. To encrypt EBS volume, you just need to select the Encrypt this volume checkbox when creating EBS volume from AWS console. In this recipe, we list the commands for creating an EBS volume, and then attaching it to an EC2 instance.

$ aws ec2 describe-availability-zones

$ aws ec2 create-volume 
--availability-zone [AvailabilityZone] 
--volume-type [VolumeType]
--iops [IOPS]
--size [Size]

$ aws ec2 create-volume 
--availability-zone ap-southeast-1b 
--volume-type io1 
--iops 1000 
--size 90

$ aws ec2 attach-volume 
--volume-id [VolumeId]
--instance-id [InstanceId]
--device [Device]

$ aws ec2 attach-volume 
--volume-id vol-64e54f6a 
--instance-id i-2e7dace3 
--device /dev/sdf

Tags represent metadata for your AWS resources. Tags are used to separate your AWS resources from one another. These are key/value pairs. If we use good tags, then it's easy to filter resources by tag names. It is also helpful for analyzing your bill; we can get the billing information of all tags by filtering on tags associated with the AWS resources. For example, you can tag several resources with a specific application name, and then organize your billing information to see the total cost for that application across several AWS services. If we add a tag that has the same key as an existing tag, then the new value will override the old value. You can edit tag keys and values at any time, and you can also remove them at any time.

In this recipe, we describe the command for creating tags for our AWS resources.

$ aws ec2 create-tags 
--resources [Resources] 
--tags [Tags]

$ aws ec2 create-tags 
--resources i-2e7dace3 
--tags 
Key=Name,Value=Tomcat Key=Group,Value='FronEnd Server Group'

Security groups are like firewalls for your EC2 instances. If you don't specify the security group while creating instance in EC2-VPC, then AWS automatically assigns the default security group of the EC2-VPC to the instance. We can configure the inbound and outbound rules for security groups. We can also change these inbound and outbound rules while the instance is running. These changes are automatically applied.

For every VPC, we get a default security group, which we can't delete. You can't use a security group that you created for EC2-VPC when you launch an instance in EC2-Classic. You also can't use security group that you created for EC2-Classic, when you launch an instance in EC2-VPC. After you launch an instance in EC2-Classic, you can't change its security group but you can add and delete rules, which are then applied, automatically. But after you launch an instance in EC2-VPC, you can change its security groups, and add and remove rules, which are then applied, automatically.

When you specify a security group as the source or destination for a rule, the rule affects all instances associated with the security group The security groups created for EC2-Classic can only have inbound rules, but security groups created for EC2-VPC can have both inbound and outbound rules.

The limit to create security groups for each region is 500. You can create up to 100 security groups per VPC. You can also assign an unlimited number of security groups to the instance launched in EC2-Classic, whereas only 5 security groups can be assigned to an instance launched in VPC. The number of rules that can be added to each security group on EC2-Classic is 100 and for VPC it is 50.

$ aws ec2 create-security-group 
--group-name [SecurityGroupName]
--description [Description]

$ aws ec2 create-security-group 
--group-name WebServerSecurityGroup 
--description "Web Server Security Group"

$ aws ec2 create-security-group 
--group-name [SecurityGroupName]
--description [Description] 
--vpc-id [VPCId]

$ aws ec2 create-security-group 
--group-name WebServerSecurityGroup 
--description "Web Server Security Group" 
--vpc-id vpc-1f33c27a

$ aws ec2 authorize-security-group-ingress 
--group-id [SecurityGroupId] 
--protocol [Protocol]
--port [Port]
--cidr [CIDR]

$ aws ec2 authorize-security-group-ingress 
--group-id sg-c6b873a3 
--protocol tcp 
--port 22 
--cidr 123.252.223.114/32

$ aws ec2 authorize-security-group-egress 
--group-id [SecurityGroupId] 
--protocol [Protocol] 
--port [Port]
--cidr [CIDR]

$ aws ec2 authorize-security-group-egress 
--group-id sg-c6b873a3 
--protocol tcp 
--port 3866 
--cidr 123.252.223.114/24

$ aws ec2 modify-instance-attribute 
--instance-id [InstanceId] 
--groups [SecurityGroupIds]

$ aws ec2 modify-instance-attribute 
--instance-id i-2e7dace3 
--groups sg-c6b873a3 sg-ccb873a9

AWS can authenticate using the public-private key mechanism. The recommended authentication mechanism is public-private key authentication instead of passwords to remotely log in to your instances with SSH. We upload the public key to AWS, and store the private key on our local machine. If anyone has your private key, then they can easily log in to your EC2 instances. It's a best practice to store these private keys in a secure place. We can create the public and private key from our machine using tools like PuTTY Key Generator.

You should include a passphrase with the private key to prevent unauthorized persons from logging in to your EC2 instance. When you include a passphrase, you have to enter the passphrase whenever you log in to the EC2 instance. A passphrase on a private key is an extra layer of protection. If you lost your private key for an EBS-backed instance, you can regain access to your instance by executing the following steps:

EC2 instances can be grouped using placement groups. For example, instances requiring low latency and high bandwidth communication can be placed in the same placement group. When instances are placed in this placement group, they have access to low latency, non-blocking 10 Gbps networking when communicating with other instances in the placement group (within a single availability zone). AWS recommends launching all the instances within the cluster placement group at the same time.

$ aws ec2 create-placement-group 
--group-name [GroupName] 
--strategy [Strategy]

$ aws ec2 create-placement-group 
--group-name WebServerGroup 
--strategy cluster

$ aws ec2 run-instances 
--image-id [ImageId] 
--count [Count]
--instance-type [InstanceType]
--key-name [KeyPairName]
--security-group-ids [SecurityGroupIds]
--subnet-id [SubnetId]
--placement [Placement]

$ aws ec2 run-instances 
--image-id ami-7e2c612c 
--count 1 
--instance-type c3.large 
--key-name WebServerKeyPair
--security-group-ids sg-ad70b8c8 
--subnet-id subnet-aed11acb 
--placement GroupName= WebServerGroup

The Elastic Load Balancer (ELB) works within a single AWS region. You can scale both horizontally (adding more EC2 instances) and vertically (increasing EC2 instance size) within AWS, but it's best practice to scale horizontally. It can, however, load balance across several instances in multiple availability zones. If you don't want to load balance instances across multiple availability zones, then you can also disable it. If we want to load balance the instances across multiple regions, then we have to use Route 53 (instead of an ELB). ELB continuously checks the health of the instances, and only routes traffic to healthy instances. The health check frequency and the URL parameters are configurable.

If a healthy instance comes online, then the ELB recognizes the instance and routes traffic to it. ELB can be used to implement high-availability application architectures. If we use Route 53 with ELB, we can enable failover to a different region. ELB can also be configured with autoscaling, thereby enabling load balancing across new instances created by auto-scaling groups.

ELB can work with instances in EC2-Classic and VPC. There are two types of load balancers we can create internal or internet facing. We can't create internal load balancer without VPC. We can create both internal and internet facing load balancers within VPC. You can also enable sticky sessions on ELB using either application generated cookies or ELB generated cookies. In addition, you can assign security groups to ELBs. If you don't assign any security group while creating the ELB in VPC, it uses the default security group of the VPC. SSL termination is also supported in ELB, using this obviates the need to install SSL certificate on each and every EC2 instance.

$ aws elb create-load-balancer 
--load-balancer-name [LoanBalancerName] 
--listeners [Listeners]
--subnets [SubnetIds] 
--security-groups [SecurityGroups]

$ aws elb create-load-balancer
--load-balancer-name WebLoadBalancer
--listeners Protocol=HTTP,LoadBalancerPort=80,InstanceProtocol=HTTP,InstancePort=8080 
--subnets subnet-aed11acb 
--security-groups sg-c6b873a3

$ aws elb configure-health-check 
--load-balancer-name [LoanBalancerName] 
--health-check [HealthCheckup]

$ aws elb configure-health-check 
--load-balancer-name WebLoadBalancer
--health-check Target=HTTP:8080/index.html,Interval=30,UnhealthyThreshold=2,HealthyThreshold=2,Timeout=3 

$ aws elb register-instances-with-load-balancer
--load-balancer-name [LoanBalancerName] 
--instances [Instances]

$ aws elb register-instances-with-load-balancer 
--load-balancer-name WebLoadBalancer 
--instances i-d3ff2c1e i-2e7dace3

Application and network errors can render the system unavailable to the user. Multi-availability zone deployments are used for building high-availability applications at the AWS region level. For implementing fault tolerance for region level failures, we have to deploy our application in availability zones spanning across different regions. If we use multiple regions, we have to use Route 53 for failover. If the primary region goes down, Route 53 fails over to the secondary region.

Increasing load on system can also cause system availability issues, but the autoscaling feature can help us solve the problem by autoscaling the number of servers during a spike in load. The number of servers is automatically reduced when the load comes back to normal levels. Detailed explanation on autoscaling is in Chapter 3, Managing AWS Resources Using AWS CloudFormation.

Building loosely coupled applications can also help avoid single points of failure. We can use Simple Queue Service (SQS) to build loosely coupled applications. Using the SQS queue size as a parameter, we can auto-scale our EC2 instances. For RDS high availability, we can configure a multi availability zone-deployment option. This will deploy the primary and secondary database instances in two different availability zones.

The AWS Marketplace helps customers find software from a set of third-party vendors. There is no need to set up a new billing account for another company; those bills can be paid via the AWS monthly bills. We can read reviews from other customers to help us make the most appropriate selection. We can also share or sell our AMIs with the public so that the wider community can use them.

In this recipe, we list the commands for creating AMIs for offering them to other users on AWS Marketplace.

$ aws ec2 create-image 
--instance-id [InstanceId]
--name [Name] 
--description [Description]

$ aws ec2 create-image 
--instance-id i-2e7dace3 
--name "WebServerImage" 
--description "Image of web server"

$ aws ec2 modify-image-attribute 
--image-id [ImageId] 
--launch-permission [LaunchPermission]

$ aws ec2 modify-image-attribute 
--image-id ami-97e6cbc5 
--launch-permission "{\"Add\": [{\"Group\":\"all\"}]}"