Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Newsletter Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials

7019 Articles
article-image-rdo-installation
Packt
09 Aug 2016
26 min read
Save for later

RDO Installation

Packt
09 Aug 2016
26 min read
In this article by Dan Radez, author of OpenStack Essentials - Second Edition, we will see how OpenStack has a very modular design, and because of this design, there are lots of moving parts. It is overwhelming to start walking through installing and using OpenStack without understanding the internal architecture of the components that make up OpenStack. In this article, we'll look at these components. Each component in OpenStack manages a different resource that can be virtualized for the end user. Separating the management of each of the types of resources that can be virtualized into separate components makes the OpenStack architecture very modular. If a particular service or resource provided by a component is not required, then the component is optional to an OpenStack deployment. Once the components that make up OpenStack have been covered, we will discuss the configuration of a community-supported distribution of OpenStack called RDO. (For more resources related to this topic, see here.) OpenStack architecture Let's start by outlining some simple categories to group these services into. Logically, the components of OpenStack are divided into three groups: Control Network Compute The control tier runs the Application Programming Interface (API) services, web interface, database, and message bus. The network tier runs network service agents for networking, and the compute tier is the virtualization hypervisor. It has services and agents to handle virtual machines. All of the components use a database and/or a message bus. The database can be MySQL, MariaDB, or PostgreSQL. The most popular message buses are RabbitMQ, Qpid, and ActiveMQ. For smaller deployments, the database and messaging services usually run on the control node, but they could have their own nodes if required. In a simple multi-node deployment, the control and networking services are installed on one server and the compute services are installed onto another server. OpenStack could be installed on one node or more than two nodes, but a good baseline for being able to scale out later is to put control and network together and compute by itself Now that a base logical architecture of OpenStack has been defined, let's look at what components make up this basic architecture. To do that, we'll first touch on the web interface and then work toward collecting the resources necessary to launch an instance. Finally, we will look at what components are available to add resources to a launched instance. Dashboard The OpenStack dashboard is the web interface component provided with OpenStack. You'll sometimes hear the terms dashboard and Horizon used interchangeably. Technically, they are not the same thing. This article will refer to the web interface as the dashboard. The team that develops the web interface maintains both the dashboard interface and the Horizon framework that the dashboard uses. More important than getting these terms right is understanding the commitment that the team that maintains this code base has made to the OpenStack project. They have pledged to include support for all the officially accepted components that are included in OpenStack. Visit the OpenStack website (http://www.openstack.org/) to get an official list of OpenStack components. The dashboard cannot do anything that the API cannot do. All the actions that are taken through the dashboard result in calls to the API to complete the task requested by the end user. Throughout this article, we will examine how to use the web interface and the API clients to execute tasks in an OpenStack cluster. Next, we will discuss both the dashboard and the underlying components that the dashboard makes calls to when creating OpenStack resources. Keystone Keystone is the identity management component. The first thing that needs to happen while connecting to an OpenStack deployment is authentication. In its most basic installation, Keystone will manage tenants, users, and roles and be a catalog of services and endpoints for all the components in the running cluster. Everything in OpenStack must exist in a tenant. A tenant is simply a grouping of objects. Users, instances, and networks are examples of objects. They cannot exist outside of a tenant. Another name for a tenant is a project. On the command line, the term tenant is used. In the web interface, the term project is used. Users must be granted a role in a tenant. It's important to understand this relationship between the user and a tenant via a role. For now, understand that a user cannot log in to the cluster unless they are a member of a tenant. Even the administrator has a tenant. Even the users the OpenStack components use to communicate with each other have to be members of a tenant to be able to authenticate. Keystone also keeps a catalog of services and endpoints of each of the OpenStack components in the cluster. This is advantageous because all of the components have different API endpoints. By registering them all with Keystone, an end user only needs to know the address of the Keystone server to interact with the cluster. When a call is made to connect to a component other than Keystone, the call will first have to be authenticated, so Keystone will be contacted regardless. Within the communication to Keystone, the client also asks Keystone for the address of the component the user intended to connect to. This makes managing the endpoints easier. If all the endpoints were distributed to the end users, then it would be a complex process to distribute a change in one of the endpoints to all of the end users. By keeping the catalog of services and endpoints in Keystone, a change is easily distributed to end users as new requests are made to connect to the components. By default, Keystone uses username/password authentication to request a token and the acquired tokens for subsequent requests. All the components in the cluster can use the token to verify the user and the user's access. Keystone can also be integrated into other common authentication systems instead of relying on the username and password authentication provided by Keystone Glance Glance is the image management component. Once we're authenticated, there are a few resources that need to be available for an instance to launch. The first resource we'll look at is the disk image to launch from. Before a server is useful, it needs to have an operating system installed on it. This is a boilerplate task that cloud computing has streamlined by creating a registry of pre-installed disk images to boot from. Glance serves as this registry within an OpenStack deployment. In preparation for an instance to launch, a copy of a selected Glance image is first cached to the compute node where the instance is being launched. Then, a copy is made to the ephemeral disk location of the new instance. Subsequent instances launched on the same compute node using the same disk image will use the cached copy of the Glance image. The images stored in Glance are sometimes called sealed-disk images. These images are disk images that have had the operating system installed but have had things such as the Secure Shell (SSH) host key and network device MAC addresses removed. This makes the disk images generic, so they can be reused and launched repeatedly without the running copies conflicting with each other. To do this, the host-specific information is provided or generated at boot. The provided information is passed in through a post-boot configuration facility called cloud-init. Usually, these images are downloaded from distribution's download pages. If you search the internet for your favorite distribution's name and cloud image, you will probably get a link to where to download a generic pre-built copy of a Glance image, also known as a cloud image. The images can also be customized for special purposes beyond a base operating system installation. If there was a specific purpose for which an instance would be launched many times, then some of the repetitive configuration tasks could be performed ahead of time and built into the disk image. For example, if a disk image was intended to be used to build a cluster of web servers, it would make sense to install a web server package on the disk image before it was used to launch an instance. It would save time and bandwidth to do it once before it is registered with Glance instead of doing this package installation and configuration over and over each time a web server instance is booted. There are quite a few ways to build these disk images. The simplest way is to do a virtual machine installation manually, make sure that the host-specific information is removed, and include cloud-init in the built image. Cloud-init is packaged in most major distributions; you should be able to simply add it to a package list. There are also tools to make this happen in a more autonomous fashion. Some of the more popular tools are virt-install, Oz, and appliance-creator. The most important thing about building a cloud image for OpenStack is to make sure that cloud-init is installed. Cloud-init is a script that should run post boot to connect back to the metadata service. Neutron Neutron is the network management component. With Keystone, we're authenticated, and from Glance, a disk image will be provided. The next resource required for launch is a virtual network. Neutron is an API frontend (and a set of agents) that manages the Software Defined Networking (SDN) infrastructure for you. When an OpenStack deployment is using Neutron, it means that each of your tenants can create virtual isolated networks. Each of these isolated networks can be connected to virtual routers to create routes between the virtual networks. A virtual router can have an external gateway connected to it, and external access can be given to each instance by associating a floating IP on an external network with an instance. Neutron then puts all the configuration in place to route the traffic sent to the floating IP address through these virtual network resources into a launched instance. This is also called Networking as a Service (NaaS). NaaS is the capability to provide networks and network resources on demand via software. By default, the OpenStack distribution we will install uses Open vSwitch to orchestrate the underlying virtualized networking infrastructure. Open vSwitch is a virtual managed switch. As long as the nodes in your cluster have simple connectivity to each other, Open vSwitch can be the infrastructure configured to isolate the virtual networks for the tenants in OpenStack. There are also many vendor plugins that would allow you to replace Open vSwitch with a physical managed switch to handle the virtual networks. Neutron even has the capability to use multiple plugins to manage multiple network appliances. As an example, Open vSwitch and a vendor's appliance could be used in parallel to manage virtual networks in an OpenStack deployment. This is a great example of how OpenStack is built to provide flexibility and choice to its users. Networking is the most complex component of OpenStack to configure and maintain. This is because Neutron is built around core networking concepts. To successfully deploy Neutron, you need to understand these core concepts and how they interact with one another. Nova Nova is the instance management component. An authenticated user who has access to a Glance image and has created a network for an instance to live on is almost ready to tie all of this together and launch an instance. The last resources that are required are a key pair and a security group. A key pair is simply an SSH key pair. OpenStack will allow you to import your own key pair or generate one to use. When the instance is launched, the public key is placed in the authorized_keys file so that a password-less SSH connection can be made to the running instance. Before that SSH connection can be made, the security groups have to be opened to allow the connection to be made. A security group is a firewall at the cloud infrastructure layer. The OpenStack distribution we'll use will have a default security group with rules to allow instances to communicate with each other within the same security group, but rules will have to be added for Internet Control Message Protocol (ICMP), SSH, and other connections to be made from outside the security group. Once there's an image, network, key pair, and security group available, an instance can be launched. The resource's identifiers are provided to Nova, and Nova looks at what resources are being used on which hypervisors, and schedules the instance to spawn on a compute node. The compute node gets the Glance image, creates the virtual network devices, and boots the instance. During the boot, cloud-init should run and connect to the metadata service. The metadata service provides the SSH public key needed for SSH login to the instance and, if provided, any post-boot configuration that needs to happen. This could be anything from a simple shell script to an invocation of a configuration management engine. Cinder Cinder is the block storage management component. Volumes can be created and attached to instances. Then they are used on the instances as any other block device would be used. On the instance, the block device can be partitioned and a filesystem can be created and mounted. Cinder also handles snapshots. Snapshots can be taken of the block volumes or of instances. Instances can also use these snapshots as a boot source. There is an extensive collection of storage backends that can be configured as the backing store for Cinder volumes and snapshots. By default, Logical Volume Manager (LVM) is configured. GlusterFS and Ceph are two popular software-based storage solutions. There are also many plugins for hardware appliances. Swift Swift is the object storage management component. Object storage is a simple content-only storage system. Files are stored without the metadata that a block filesystem has. These are simply containers and files. The files are simply content. Swift has two layers as part of its deployment: the proxy and the storage engine. The proxy is the API layer. It's the service that the end user communicates with. The proxy is configured to talk to the storage engine on the user's behalf. By default, the storage engine is the Swift storage engine. It's able to do software-based storage distribution and replication. GlusterFS and Ceph are also popular storage backends for Swift. They have similar distribution and replication capabilities to those of Swift storage. Ceilometer Ceilometer is the telemetry component. It collects resource measurements and is able to monitor the cluster. Ceilometer was originally designed as a metering system for billing users. As it was being built, there was a realization that it would be useful for more than just billing and turned into a general-purpose telemetry system. Ceilometer meters measure the resources being used in an OpenStack deployment. When Ceilometer reads a meter, it's called a sample. These samples get recorded on a regular basis. A collection of samples is called a statistic. Telemetry statistics will give insights into how the resources of an OpenStack deployment are being used. The samples can also be used for alarms. Alarms are nothing but monitors that watch for a certain criterion to be met. Heat Heat is the orchestration component. Orchestration is the process of launching multiple instances that are intended to work together. In orchestration, there is a file, known as a template, used to define what will be launched. In this template, there can also be ordering or dependencies set up between the instances. Data that needs to be passed between the instances for configuration can also be defined in these templates. Heat is also compatible with AWS CloudFormation templates and implements additional features in addition to the AWS CloudFormation template language. To use Heat, one of these templates is written to define a set of instances that needs to be launched. When a template launches, it creates a collection of virtual resources (instances, networks, storage devices, and so on); this collection of resources is called a stack. When a stack is spawned, the ordering and dependencies, shared configuration data, and post-boot configuration are coordinated via Heat. Heat is not configuration management. It is orchestration. It is intended to coordinate launching the instances, passing configuration data, and executing simple post-boot configuration. A very common post-boot configuration task is invoking an actual configuration management engine to execute more complex post-boot configuration. OpenStack installation The list of components that have been covered is not the full list. This is just a small subset to get you started with using and understanding OpenStack. Further components that are defaults in an OpenStack installation provide many advanced capabilities that we will not be able to cover. Now that we have introduced the OpenStack components, we will illustrate how they work together as a running OpenStack installation. To illustrate an OpenStack installation, we first need to install one. Let's use the RDO Project's OpenStack distribution to do that. RDO has two installation methods; we will discuss both of them and focus on one of them throughout this article. Manual installation and configuration of OpenStack involves installing, configuring, and registering each of the components we covered in the previous part, and also multiple databases and a messaging system. It's a very involved, repetitive, error-prone, and sometimes confusing process. Fortunately, there are a few distributions that include tools to automate this installation and configuration process. One such distribution is the RDO Project distribution. RDO, as a name, doesn't officially mean anything. It is just the name of a community-supported distribution of OpenStack. The RDO Project takes the upstream OpenStack code, packages it in RPMs and provides documentation, forums, IRC channels, and other resources for the RDO community to use and support each other in running OpenStack on RPM-based systems. There are no modifications to the upstream OpenStack code in the RDO distribution. The RDO project packages the code that is in each of the upstream releases of OpenStack. This means that we'll use an open source, community-supported distribution of vanilla OpenStack for our example installation. RDO should be able to be run on any RPM-based system. We will now look at the two installation tools that are part of the RDO Project, Packstack and RDO Triple-O. We will focus on using RDO Triple-O in this article. The RDO Project recommends RDO Triple-O for installations that intend to deploy a more feature-rich environment. One example is High Availability. RDO Triple-O is able to do HA deployments and Packstack is not. There is still great value in doing an installation with Packstack. Packstack is intended to give you a very lightweight, quick way to stand up a basic OpenStack installation. Let's start by taking a quick look at Packstack so you are familiar with how quick and lightweight is it. Installing RDO using Packstack Packstack is an installation tool for OpenStack intended for demonstration and proof-of-concept deployments. Packstack uses SSH to connect to each of the nodes and invokes a puppet run (specifically, a puppet apply) on each of the nodes to install and configure OpenStack. RDO website: http://openstack.redhat.com Packstack installation: http://openstack.redhat.com/install/quickstart The RDO Project quick start gives instructions to install RDO using Packstack in three simple steps: Update the system and install the RDO release rpm as follows: sudo yum update -y sudo yum install -y http://rdo.fedorapeople.org/rdo-release.rpm Install Packstack as shown in the following command: sudo yum install -y openstack-packstack Run Packstack as shown in the following command: sudo packstack --allinone The all-in-one installation method works well to run on a virtual machine as your all-in-one OpenStack node. In reality, however, a cluster will usually use more than one node beyond a simple learning environment. Packstack is capable of doing multinode installations, though you will have to read the RDO Project documentation for Packstack on the RDO Project wiki. We will not go any deeper with Packstack than the all-in-one installation we have just walked through. Don't avoid doing an all-in-one installation; it really is as simple as the steps make it out to be, and there is value in getting an OpenStack installation up and running quickly. Installing RDO using Triple-O The Triple-O project is an OpenStack installation tool developed by the OpenStack community. A Triple-O deployment consists of two OpenStack deployments. One of the deployments is an all-in-one OpenStack installation that is used as a provisioning tool to deploy a multi-node target OpenStack deployment. This target deployment is the deployment intended for end users. Triple-O stands for OpenStack on OpenStack. OpenStack on OpenStack would be OOO, which lovingly became referred to as Triple-O. It may sound like madness to use OpenStack to deploy OpenStack, but consider that OpenStack is really good at provisioning virtual instances. Triple-O applies this strength to bare-metal deployments to deploy a target OpenStack environment. In Triple-O, the two OpenStacks are called the undercloud and the overcloud. The undercloud is a baremetal management enabled all-in-one OpenStack installation that will build for you in a very prescriptive way. Baremetal management enabled means it is intended to manage physical machines instead of virtual machines. The overcloud is the target deployment of OpenStack that is intended be exposed to end users. The undercloud will take a cluster of nodes provided to it and deploy the overcloud to them, a fully featured OpenStack deployment. In real deployments, this is done with a collection of baremetal nodes. Fortunately, for learning purposes, we can mock having a bunch of baremetal nodes by using virtual machines. Mind blown yet? Let's get started with this RDO Manager based OpenStack installation to start unraveling what all this means. There is an RDO Manager quickstart project that we will use to get going. The RDO Triple-O wiki page will be the most up-to-date place to get started with RDO Triple-O. If you have trouble with the directions in this article, please refer to the wiki. OpenSource changes rapidly and RDO Triple-O is no exception. In particular, note that the directions refer to the Mitaka release of OpenStack. The name of the release will most likely be the first thing that changes on the wiki page that will impact your future deployments with RDO Triple-O. Start by downloading the pre-built undercloud image from the RDO Project's repositories. This is something you could build yourself but it would take much more time and effort to build than it would take to download the pre-built one. As mentioned earlier, the undercloud is a pretty prescriptive all-in-one deployment which lends itself well to starting with a pre-built image. These instructions come from the readme of the triple-o quickstart github repository (https://github.com/redhat-openstack/tripleo-quickstart/): myhost# mkdir -p /usr/share/quickstart_images/ myhost# cd /usr/share/quickstart_images/ myhost# wget https://ci.centos.org/artifacts/rdo/images/mitaka/delorean/stable/undercloud.qcow2.md5 https://ci.centos.org/artifacts/rdo/images/mitaka/delorean/stable/undercloud.qcow2 Make sure that your ssh key exists: Myhost# ls ~/.ssh If you don't see the id_rsa and id_rsa.pub files in that directory list, run the command ssh-keygen. Then make sure that your public key is in the authorized keys file: myhost# cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys Once you have the undercloud image and you ssh keys pull a copy of the quickstart.sh file, install the dependencies and execute the quickstart script: myhost# cd ~ myhost# wget https://raw.githubusercontent.com/redhat-openstack/tripleo-quickstart/master/quickstart.sh myhost#sh quickstart.sh -u file:///usr/share/quickstart_images/undercloud.qcow2 localhost quickstart.sh will use Ansible to set up the undercloud virtual machine and will define a few extra virtual machines that will be used to mock a collection of baremetal nodes for an overcloud deployment. To see the list of virtual machines that quickstack.sh created, use virsh to list them: myhost# virsh list --all Id Name State ---------------------------------------------------- 17 undercloud running - ceph_0 shut off - compute_0 shut off - control_0 shut off - control_1 shut off - control_2 shut off Along with the undercloud virtual machine, there are ceph, compute, and control virtual machine definitions. These are the nodes that will be used to deploy the OpenStack overcloud. Using virtual machines like this to deploy OpenStack is not suitable for anything but your own personal OpenStack enrichment. These virtual machines represent physical machines that would be used in a real deployment that would be exposed to end users. To continue the undercloud installation, connect to the undercloud virtual machine and run the undercloud configuration: myhost# ssh -F /root/.quickstart/ssh.config.ansible undercloud undercloud# openstack undercloud install The undercloud install command will set up the undercloud machine as an all-in-one OpenStack installation ready be told how to deploy the overcloud. Once the undercloud installation is completed, the final steps are to seed the undercloud with configuration about the overcloud deployment and execute the overcloud deployment: undercloud# source stackrc undercloud# openstack overcloud image upload undercloud# openstack baremetal import --json instackenv.json undercloud# openstack baremetal configure boot undercloud# neutron subnet-list undercloud# neutron subnet-update <subnet-uuid> --dns-nameserver 8.8.8.8 There are also some scripts and other automated ways to make these steps happen: look at the output of the quickstart script or Triple-O quickstart docs in the GitHub repository to get more information about how to automate some of these steps. The source command puts information into the shell environment to tell the subsequent commands how to communicate with the undercloud. The image upload command uploads disk images into Glance that will be used to provision the overcloud nodes.. The first baremetal command imports information about the overcloud environment that will be deployed. This information was written to the instackenv.json file when the undercloud virtual machine was created by quickstart.sh. The second configures the images that were just uploaded in preparation for provisioning the overcloud nodes. The two neutron commands configure a DNS server for the network that the overclouds will use, in this case Google's. Finally, execute the overcloud deploy: undercloud# openstack overcloud deploy --control-scale 1 --compute-scale 1 --templates --libvirt-type qemu --ceph-storage-scale 1 -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml Let's talk about what this command is doing. In OpenStack, there are two basic node types, control and compute. A control node runs the OpenStack API services, OpenStack scheduling service, database services, and messaging services. Pretty much everything except the hypervisors are part of the control tier and are segregated onto control nodes in a basic deployment. In an HA deployment, there are at least three control nodes. This is why you see three control nodes in the list of virtual machines quickstart.sh created. RDO Triple-O can do HA deployments, though we will focus on non-HA deployments in this article. Note that in the command you have just executed, control scale and compute scale are both set to one. This means that you are deploying one control and one compute node. The other virtual machines will not be used. Take note of the libvirt-type parameter. It is only required if the compute node itself it virtualized, which is what we are doing with RDO Triple-O, to set the configuration properly for the instances to nested. Nested virtualization is when virtual machines are running inside of a virtual machine. In this case, the instances will be virtual machines running inside of the compute node, which is a virtual machine. Finally, the ceph storage scale and storage environment file will deploy Ceph at the storage backend for Glance and Cinder. If you leave off the Ceph and storage environment file parameters, one less virtual machine will be used for deployment. The indication the overcloud deploy has succeeded will give you a Keystone endpoint and a success message: Overcloud Endpoint: http://192.0.2.6:5000/v2.0 Overcloud Deployed Connecting to your Overcloud Finally, before we dig into looking at the OpenStack components that have been installed and configured, let's identify three ways that you can connect to the freshly installed overcloud deployment: From the undercloud: This is the quickest way to access the overcloud. When the overcloud deployment completed, a file named overcloudrc was created. Install the client libraries: Both RDO Triple-O and Packstack were installed from the RDO release repository. By installing this release repository, in the same way that was demonstrated earlier for Packstack on another computer, the OpenStack client libraries can be installed on that computer. If these libraries are installed on a computer that can route the network the overcloud was installed on then the overcloud can be accessed from that computer the same as it can from the undercloud. This is helpful if you do not want to be tied to jumping through the undercloud node to access the overcloud: laptop# sudo yum install -y http://rdo.fedorapeople.org/rdo-release.rpm laptop# sudo yum install python-openstackclient In addition to the client package, you will also need the overcloudrc file from the undercloud. As an example, you can install the packages on the host machine you have just run quickstart.sh and make the overcloud routable by adding an IP address to the OVS bridge the virtual machines were attached to: myhost# sudo ip addr add 192.0.2.222/24 dev bridget myhost# sudo ip link set up dev bridget Once this is done, the commands in the subsequent parts could be run from the host machine instead of the undercloud virtual machine. The OpenStack dashboard: OpenStack's included web interface is called the dashboard. In the installation you have just completed, you can access the overcloud's dashboard by first running the two ip commands used in the second option above then connecting to the IP address indicated as the overcloud endpoint but on port 80 instead of 5000: http://192.0.2.6/. Summary After looking at the components that make up an OpenStack installation, we used RDO Triple-O as a provisioning tool. We now have OpenStack installed and running. Now that OpenStack is installed and running, let's walk through each of the components discussed to learn how to use each of them. Resources for Article: Further resources on this subject: Keystone – OpenStack Identity Service [article] Concepts for OpenStack [article] Setting up VPNaaS in OpenStack [article]
Read more
  • 0
  • 0
  • 10278

article-image-conference-app
Packt
09 Aug 2016
4 min read
Save for later

Conference App

Packt
09 Aug 2016
4 min read
In this article, Indermohan Singh, the author of Ionic 2 Blueprints we will create a conference app. We will create an app which will provide list of speakers, schedule, directions to the venue, ticket booking, and lots of other features. We will learn the following things: Using the device's native features Leveraging localStorage Ionic menu and tabs Using RxJS to build a perfect search filter (For more resources related to this topic, see here.) Conference app is a companion application for conference attendees. In this application, we are using Lanyrd JSON Exportand hardcoded JSON file as our backend. We will have a tabs and side menu interface, just like our e-commerce application. When a user opens our app, the app will show a tab interface with SpeakersPageopen. It will have SchedulePage for conference schedule and AboutPage for information about conference. We will also make this app work offline, without any Internet connection. So, your user will still be able to view speakers, see the schedule, and do other stuff without using the Internet at all. JSON data In the application, we have used a hardcoded JSON file as our Database. But in the truest sense, we are actually using a JSON export of a Lanyrd event. I was trying to make this article using Lanyrd as the backend, but unfortunately, Lanyrd is mostly in maintenance mode. So I was not able to use it. In this article, I am still using a JSON export from Lanyrd, from a previous event. So, if you are able to get a JSON export for your event, you can just swap the URL and you are good to go. Those who don't want to use Lanyrd and instead want to use their own backend, must have a look at the next section. I have described the structure of JSON, which I have used to make this app. You can create your REST API accordingly. Understanding JSON Let's understand the structure of the JSON export. The whole JSON database is an object with two keys, timezone and sessions, like the following: { timezone: "Australia/Brisbane", sessions: [..] } Timezone is just a string, but sessions key is an array of lists of all the sessions of our conference. Items in the sessions array are divided according to days of the conference. Each item represents a day of the conference and has the following structure: { day: "Saturday 21st November", sessions: [..] } So, the sessions array of each day has actual sessions as items. Each item has the following structure: { start_time: "2015-11-21 09:30:00", topics: [], web_url: "url of event times: "9:30am - 10:00am", id: "sdtpgq", types: [ ], end_time_epoch: 1448064000, speakers: [], title: "Talk Title", event_id: "event_id", space: "Space", day: "Saturday 21st November", end_time: "2015-11-21 10:00:00", other_url: null, start_time_epoch: 1448062200, abstract: "<p>Abstract of Talk</p>" }, Here, the speakers array has a list of all speakers. We will use that speakers array to create a list of all speakers in an array. You can see the whole structure here: That's all we need to understand for JSON. Defining the app In this section, we will define various functionalities of our application. We will also show the architecture of our app using an app flow diagram. Functionalities We will be including the following functionalities in our application: List of speakers Schedule detail Search functionality using session title, abstract, and speaker's names Hide/Show any day of the schedule Favorite list for sessions Adding favorite sessions to the device calendar Ability to share sessions to other applications Directions to venue Offline working App flow This is how the control will flow inside our application: Let's understand the flow: RootComponent: RootComponent is the root Ionic component. It is defined inside the /app/app.ts file. TabsPage: TabsPage acts as a container for our SpeakersPage, SchedulePage, and AboutPage. SpeakersPage: SpeakersPage shows a list of all the speakers of our conference. SchedulePage: SchedulePage shows us the schedule of our conference and allows us various filter features. AboutPage: AboutPage provides us information about the conference. SpeakersDetail: SpeakerDetail page shows the details of the speaker and a list of his/her presentations in this conference. SessionDetail: SessionDetail page shows the details of a session with the title and abstract of the session. FavoritePage: FavoritePage shows a list of the user's favorite sessions. Summary In this article, we discussed about the JSON files that will used as database in our app. We also defined the the functionalities of our app and understood the flow of our app. Resources for Article:  Further resources on this subject: First Look at Ionic [article] Ionic JS Components [article] Creating Our First App with Ionic [article]
Read more
  • 0
  • 0
  • 21222

article-image-expanding-your-data-mining-toolbox
Packt
09 Aug 2016
15 min read
Save for later

Expanding Your Data Mining Toolbox

Packt
09 Aug 2016
15 min read
In this article by Megan Squire, author of Mastering Data Mining with Python, when faced with sensory information, human beings naturally want to find patterns to explain, differentiate, categorize, and predict. This process of looking for patterns all around us is a fundamental human activity, and the human brain is quite good at it. With this skill, our ancient ancestors became better at hunting, gathering, cooking, and organizing. It is no wonder that pattern recognition and pattern prediction were some of the first tasks humans set out to computerize, and this desire continues in earnest today. Depending on the goals of a given project, finding patterns in data using computers nowadays involve database systems, artificial intelligence, statistics, information retrieval, computer vision, and any number of other various subfields of computer science, information systems, mathematics, or business, just to name a few. No matter what we call this activity – knowledge discovery in databases, data mining, data science – its primary mission is always to find interesting patterns. (For more resources related to this topic, see here.) Despite this humble-sounding mission, data mining has existed for long enough and has built up enough variation in how it is implemented that it has now become a large and complicated field to master. We can think of a cooking school, where every beginner chef is first taught how to boil water and how to use a knife before moving to more advanced skills, such as making puff pastry or deboning a raw chicken. In data mining, we also have common techniques that even the newest data miners will learn: how to build a classifier and how to find clusters in data. The aim is to teach you some of the techniques you may not have seen yet in earlier data mining projects. In this article, we will cover the following topics: What is data mining? We will situate data mining in the growing field of other similar concepts, and we will learn a bit about the history of how this discipline has grown and changed. How do we do data mining? Here, we compare several processes or methodologies commonly used in data mining projects. What are the techniques used in data mining? In this article, we will summarize each of the data analysis techniques that are typically included in a definition of data mining. How do we set up a data mining work environment? Finally, we will walk through setting up a Python-based development environment. What is data mining? We explained earlier that the goal of data mining is to find patterns in data, but this oversimplification falls apart quickly under scrutiny. After all, could we not also say that finding patterns is the goal of classical statistics, or business analytics, or machine learning, or even the newer practices of data science or big data? What is the difference between data mining and all of these other fields, anyway? And while we are at it, why is it called data mining if what we are really doing is mining for patterns? Don't we already have the data? It was apparent from the beginning that the term data mining is indeed fraught with many problems. The term was originally used as something of a pejorative by statisticians who cautioned against going on fishing expeditions, where a data analyst is casting about for patterns in data without forming proper hypotheses first. Nonetheless, the term rose to prominence in the 1990s, as the popular press caught wind of exciting research that was marrying the mature field of database management systems with the best algorithms from machine learning and artificial intelligence. The inclusion of the word mining inspires visions of a modern-day Gold Rush, in which the persistent and intrepid miner will discover (and perhaps profit from) previously hidden gems. The idea that data itself could be a rare and precious commodity was immediately appealing to the business and technology press, despite efforts by early pioneers to promote more the holistic term knowledge discovery in databases (KDD). The term data mining persisted, however, and ultimately some definitions of the field attempted to re-imagine the term data mining to refer to just one of the steps in a longer, more comprehensive knowledge discovery process. Today, data mining and KDD are considered very similar, closely related terms. What about other related terms, such as machine learning, predictive analytics, big data, and data science? Are these the same as data mining or KDD? Let's draw some comparisons between each of these terms: Machine learning is a very specific subfield of computer science that focuses on developing algorithms that can learn from data in order to make predictions. Many data mining solutions will use techniques from machine learning, but not all data mining is trying to make predictions or learn from data. Sometimes we just want to find a pattern in the data. In fact, in this article we will be exploring a few data mining solutions that do use machine learning techniques, and many more that do not. Predictive analytics, sometimes just called analytics, is a general term for computational solutions that attempt to make predictions from data in a variety of domains. We can think of the terms business analytics, media analytics, and so on. Some, but not all, predictive analytics solutions will use machine learning techniques to perform their predictions. But again, in data mining, we are not always interested in prediction. Big data is a term that refers to the problems and solutions of dealing with very large sets of data, irrespective of whether we are searching for patterns in that data, or simply storing it. In terms of comparing big data to data mining, many data mining problems are made more interesting when the data sets are large, so solutions discovered for dealing with big data might come in handy to solve a data mining problem. Nonetheless, these two terms are merely complementary, not interchangeable. Data science is the closest of these terms to being interchangeable with the KDD process, of which data mining is one step. Because data science is an extremely popular buzzword at this time, its meaning will continue to evolve and change as the field continues to mature. To show the relative search interest for these various terms over time, we can look at Google Trends. This tool shows how frequently people are searching for various keywords over time. In the following figure, the newcomer term data science is currently the hot buzzword, with data mining pulling into second place, followed by machine learning, data science, and predictive analytics. (I tried to include the search term knowledge discovery in databases as well, but the results were so close to zero that the line was invisible.) The y-axis shows the popularity of that particular search term as a 0-100 indexed value. In addition, I combined the weekly index values that Google Trends gives into a monthly average for each month in the period 2004-2015. Google Trends search results for four common data-related terms How do we do data mining? Since data mining is traditionally seen as one of the steps in the overall KDD process, and increasingly in the data science process, in this article we get acquainted with the steps involved. There are several popular methodologies for doing the work of data mining. Here we highlight four methodologies: two that are taken from textbook introductions to the theory of data mining, one taken from a very practical process used in industry, and one designed for teaching beginners. The Fayyad et al. KDD process One early version of the knowledge discovery and data mining process was defined by Usama Fayyad, Gregory Piatetsky-Shapiro, and Padhraic Smyth in a 1996 article (The KDD Process for Extracting Useful Knowledge from Volumes of Data). This article was important at the time for refining the rapidly-changing KDD methodology into a concrete set of steps. The following steps lead from raw data at the beginning to knowledge at the end: Data selection: The input to this step is raw data, and the output of this selection step is a smaller subset of the data, called the target data. Data pre-processing: The target data is cleaned, oddities and outliers are removed, and missing data is accounted for. The output of this step is pre-processed data, or cleaned data. Data transformation: The cleaned data is organized into a format appropriate for the mining step, and the number of features or variables is reduced if need be. The output of this step is transformed data. Data Mining: The transformed data is mined for patterns using one or more data mining algorithms appropriate to the problem at hand. The output of this step is the discovered patterns. Data Interpretation/Evaluation: The discovered patterns are evaluated for their ability to solve the problem at hand. The output of this step is knowledge. Since this process leads from raw data to knowledge, it is appropriate that these authors were the ones who were really committed to the term knowledge discovery in databases rather than simply data mining. The Han et al. KDD process Another version of the knowledge discovery process is described in the popular data mining textbook Data Mining: Concepts and Techniques by Jiawei Han, Micheline Kamber, and Jian Pei as the following steps, which also lead from raw data to knowledge at the end: Data cleaning: The input to this step is raw data, and the output is cleaned data Data integration: In this step, the cleaned data is integrated (if it came from multiple sources). The output of this step is integrated data. Data selection: The data set is reduced to only the data needed for the problem at hand. The output of this step is a smaller data set. Data transformation: The smaller data set is consolidated into a form that will work with the upcoming data mining step. This is called transformed data. Data Mining: The transformed data is processed by intelligent algorithms that are designed to discover patterns in that data. The output of this step is one or more patterns. Pattern evaluation: The discovered patterns are evaluated for their interestingness and their ability to solve the problem at hand. The output of this step is an interestingness measure applied to each pattern, representing knowledge. Knowledge representation: In this step, the knowledge is communicated to users through various means, including visualization. In both the Fayyad and Han methodologies, it is expected that the process will iterate multiple times over steps, if such iteration is needed. For example, if during the transformation step the person doing the analysis realized that another data cleaning or pre-processing step is needed, both of these methodologies specify that the analyst should double back and complete a second iteration of the incomplete earlier step. The CRISP-DM process A third popular version of the KDD process that is used in many business and applied domains is called CRISP-DM, which stands for CRoss-Industry Standard Process for Data Mining. It consists of the following steps: Business Understanding: In this step, the analyst spends time understanding the reasons for the data mining project from a business perspective. Data Understanding: In this step, the analyst becomes familiar with the data and its potential promises and shortcomings, and begins to generate hypotheses. The analyst is tasked to reassess the business understanding (step 1) if needed. Data Preparation: This step includes all the data selection, integration, transformation, and pre-processing steps that are enumerated as separate steps in the other models. The CRISP-DM model has no expectation of what order these tasks will be done in. Modeling: This is the step in which the algorithms are applied to the data to discover the patterns. This step is closest to the actual data mining steps in the other KDD models. The analyst is tasked to reassess the data preparation step (step 3) if the modeling and mining step requires it. Evaluation: The model and discovered patterns are evaluated for their value in answering the business problem at hand. The analyst is tasked with revisiting the business understanding (step 1) if necessary. Deployment: The discovered knowledge and models are presented and put into production to solve the original problem at hand. One of the strengths of this methodology is that iteration is built in. Between specific steps, it is expected that the analyst will check that the current step is still in agreement with certain previous steps. Another strength of this method is that the analyst is explicitly reminded to keep the business problem front and center in the project, even down in the evaluation steps. The Six Steps process When I teach the introductory data science course at my university, I use a hybrid methodology of my own creation. This methodology is called the Six Steps, and I designed it to be especially friendly for teaching. My Six Steps methodology removes some of the ambiguity that inexperienced students may have with open-ended tasks from CRISP-DM, such as Business Understanding, or a corporate-focused task such as Deployment. In addition, the Six Steps method keeps the focus on developing students' critical thinking skills by requiring them to answer Why are we doing this? and What does it mean? at the beginning and end of the process. My Six Steps method looks like this: Problem statement: In this step, the students identify what the problem is that they are trying to solve. Ideally, they motivate the case for why they are doing all this work. Data collection and storage: In this step, students locate data and plan their storage for the data needed for this problem. They also provide information about where the data that is helping them answer their motivating question came from, as well as what format it is in and what all the fields mean. Data cleaning: In this phase, students carefully select only the data they really need, and pre-process the data into the format required for the mining step. Data mining: In this step, students formalize their chosen data mining methodology. They describe what algorithms they used and why. The output of this step is a model and discovered patterns. Representation and visualization: In this step, the students show the results of their work visually. The outputs of this step can be tables, drawings, graphs, charts, network diagrams, maps, and so on. Problem resolution: This is an important step for beginner data miners. This step explicitly encourages the student to evaluate whether the patterns they showed in step 5 are really an answer to the question or problem they posed in step 1. Students are asked to state the limitations of their model or results, and to identify parts of the motivating question that they could not answer with this method. Which data mining methodology is the best? A 2014 survey of the subscribers of Gregory Piatetsky-Shapiro's very popular data mining email newsletter KDNuggets included the question What main methodology are you using for your analytics, data mining, or data science projects? 43% of the poll respondents indicated that they were using the CRISP-DM methodology 27% of the respondents were using their own methodology or a hybrid 7% were using the traditional KDD methodology These results are generally similar to the 2007 results from the same newsletter asking the same question. My best advice is that it does not matter too much which methodology you use for a data mining project, as long as you just pick one. If you do not have any methodology at all, then you run the risk of forgetting important steps. Choose one of the methods that seems like it might work for your project and your needs, and then just do your best to follow the steps. We will vary our data mining methodology depending on which technique we are looking at in a given article. For example, even though the focus of the article as a whole is on the data mining step, we still need to motivate of project with a healthy dose of Business Understanding (CRISP-DM) or Problem Statement (Six Steps) so that we understand why we are doing the tasks and what the results mean. In addition, in order to learn a particular data mining method, we may also have to do some pre-processing, whether we call that data cleaning, integration, or transformation. But in general, we will try to keep these tasks to a minimum so that our focus on data mining remains clear. Finally, even though data visualization is typically very important for representing the results of your data mining process to your audience, we will also keep these tasks to a minimum so that we can remain focused on the primary job at hand: data mining. Summary In this article, we learned what it would take to expand our data mining toolbox to the master level. First we took a long view of the field as a whole, starting with the history of data mining as a piece of the knowledge discovery in databases (KDD) process. We also compared the field of data mining to other similar terms such as data science, machine learning, and big data. Next, we outlined the common tools and techniques that most experts consider to be most important to the KDD process, paying special attention to the techniques that are used most frequently in the mining and analysis steps. To really master data mining, it is important that we work on problems that are different than simple textbook examples. For this reason we will be working on more exotic data mining techniques such as generating summaries and finding outliers, and focusing on more unusual data types, such as text and networks.  Resources for Article: Further resources on this subject: Python Data Structures [article] Mining Twitter with Python – Influence and Engagement [article] Data mining [article]
Read more
  • 0
  • 0
  • 6491

article-image-building-grid-system-susy
Packt
09 Aug 2016
14 min read
Save for later

Building a Grid System with Susy

Packt
09 Aug 2016
14 min read
In this article by Luke Watts, author of the book Mastering Sass, we will build a responsive grid system using the Susy library and a few custom mixins and functions. We will set a configuration map with our breakpoints which we will then loop over to automatically create our entire grid, using interpolation to create our class names. (For more resources related to this topic, see here.) Detailing the project requirements For this example, we will need bower to download Susy. After Susy has been downloaded we will only need two files. We'll place them all in the same directory for simplicity. These files will be style.scss and _helpers.scss. We'll place the majority of our SCSS code in style.scss. First, we'll import susy and our _helpers.scss at the beginning of this file. After that we will place our variables and finally our code which will create our grid system. Bower and Susy To check if you have bower installed open your command line (Terminal on Unix or CMD on Windows) and run: bower -v If you see a number like "1.7.9" you have bower. If not you will need to install bower using npm, a package manager for NodeJS. If you don't already have NodeJS installed, you can download it from: https://nodejs.org/en/. To install bower from your command line using npm you will need to run: npm install -g bower Once bower is installed cd into the root of your project and run: bower install susy This will create a directory called bower_components. Inside that you will find a folder called susy. The full path to file we will be importing in style.scss is bower_components/susy/sass/_susy.scss. However we can leave off the underscore (_) and also the extension (.scss). Sass will still load import the file just fine. In style.scss add the following at the beginning of our file: // style.scss @import 'bower_components/susy/sass/susy'; Helpers (mixins and functions) Next, we'll need to import our _helpers.scss file in style.scss. Our _helpers.scss file will contain any custom mixins or functions we'll create to help us in building our grid. In style.scss import _helpers.scss just below where we imported Susy: // style.scss @import 'bower_components/susy/sass/susy'; @import 'helpers'; Mixin: bp (breakpoint) I don't know about you, but writing media queries always seems like bit of a chore to me. I just don't like to write (min-width: 768px) all the time. So for that reason I'm going to include the bp mixin, which means instead of writing: @media(min-width: 768px) { // ... } We can simply use: @include bp(md) { // ... } First we are going to create a map of our breakpoints. Add the $breakpoints map to style.scss just below our imports: // style.scss @import 'bower_components/susy/sass/susy'; @import 'helpers'; $breakpoints: ( sm: 480px, md: 768px, lg: 980px ); Then, inside _helpers.scss we're going to create our bp mixin which will handle creating our media queries from the $breakpoints map. Here's the breakpoint (bp) mixin: @mixin bp($size: md) { @media (min-width: map-get($breakpoints, $size)) { @content; } } Here we are setting the default breakpoint to be md (768px). We then use the built in Sass function map-get to get the relevant value using the key ($size). Inside our @media rule we use the @content directive which will allows us pass any Sass or CSS directly into our bp mixin to our @media rule. The container mixin The container mixin sets the max-width of the containing element, which will be the .container element for now. However, it is best to use the container mixin to semantically restrict certain parts of the design to your max width instead of using presentational classes like container or row. The container mixin takes a width argument, which will be the max-width. It also automatically applies the micro-clearfix hack. This prevents the containers height from collapsing when the elements inside it are floated. I prefer the overflow: hidden method myself, but they do the same thing essentially. By default, the container will be set to max-width: 100%. However, you can set it to be any valid unit of dimension, such as 60em, 1160px, 50%, 90vw, or whatever. As long as it's a valid CSS unit it will work. In style.scss let's create our .container element using the container mixin: // style.scss .container { @include container(1160px); } The preceding code will give the following CSS output: .container { max-width: 1160px; margin-left: auto; margin-right: auto; } .container:after { content: " "; display: block; clear: both; } Due to the fact the container uses a max-width we don't need to specify different dimensions for various screen sizes. It will be 100% until the screen is above 1160px and then the max-width value will kick in. The .container:after rule is the micro-clearfix hack. The span mixin To create columns in Susy we use the span mixin. The span mixin sets the width of that element and applies a padding or margin depending on how Susy is set up. By default, Susy will apply a margin to the right of each column, but you can set it to be on the left, or to be padding on the left or right or padding or margin on both sides. Susy will do the necessary work to make everything work behind the scenes. To create a half width column in a 12 column grid you would use: .col-6 { @include span(6 of 12); } The of 12 let's Susy know this is a 12 column grid. When we define our $susy map later we can tell Susy how many columns we are using via the columns property. This means we can drop the of 12 part and simply use span(6) instead. Susy will then know we are using 12 columns unless we explicitly pass another value. The preceding SCSS will output: .col-6 { width: 49.15254%; float: left; margin-right: 1.69492%; } Notice the width and margin together would actually be 50.84746%, not 50% as you might expect. Therefor two of these column would actually be 101.69492%. That will cause the last column to wrap to the next row. To prevent this, you would need to remove the margin from the last column. The last keyword To address this, Susy uses the last keyword. When you pass this to the span mixin it lets Susy know this is the last column in a row. This removes the margin right and also floats the element in question to the right to ensure it's at the very end of the row. Let's take the previous example where we would have two col-6 elements. We could create a class of col-6-last and apply the last keyword to that span mixin: .col-6 { @include span(6 of 12); &-last { @include span(last 6 of 12) } } The preceding SCSS will output: .col-6 { width: 49.15254%; float: left; margin-right: 1.69492%; } .col-6-last { width: 49.15254%; float: right; margin-right: 0; } You can also place the last keyword at the end. This will also work: .col-6 { @include span(6 of 12); &-last { @include span(6 of 12 last) } } The $susy configuration map Susy allows for a lot of configuration through its configuration map which is defined as $susy. The settings in the $susy map allow us to set how wide the container should be, how many columns our grid should have, how wide the gutters are, whether those gutters should be margins or padding, and whether the gutters should be on the left, right or both sides of each column. Actually, there are even more settings available depending what type of grid you'd like to build. Let's, define our $susy map with the container set to 1160px just after our $breakpoints map: // style.scss $susy: ( container: 1160px, columns: 12, gutters: 1/3 ); Here we've set our containers max-width to be 1160px. This is used when we use the container mixin without entering a value. We've also set our grid to be 12 columns with the gutters, (padding or margin) to be 1/3 the width of a column. That's about all we need to set for our purposes, however, Susy has a lot more to offer. In fact, to cover everything in Susy would need an entirely book of its own. If you want to explore more of what Susy can do you should read the documentation at http://susydocs.oddbird.net/en/latest/. Setting up a grid system We've all used a 12 column grid which has various sizes (small, medium, large) or a set breakpoint (or breakpoints). These are the most popular methods for two reasons...it works, and it's easy to understand. Furthermore, with the help of Susy we can achieve this with less than 30 lines of Sass! Don't believe me? Let's begin. The concept of our grid system Our grid system will be similar to that of Foundation and Bootstrap. It will have 3 breakpoints and will be mobile-first. It will have a container, which will act as both .container and .row, therefore removing the need for a .row class. The breakpoints Earlier we defined three sizes in our $breakpoints map. These were: $breakpoints: ( sm: 480px, md: 768px, lg: 980px ); So our grid will have small, medium and large breakpoints. The columns naming convention Our columns will use a similar naming convention to that of Bootstrap. There will be four available sets of columns. The first will start from 0px up to the 399px (example: .col-12) The next will start from 480px up to 767px (example: .col-12-sm) The medium will start from 768px up to 979px (example: .col-12-md) The large will start from 980px (example: .col-12-lg) Having four options will give us the most flexibility. Building the grid From here we can use an @for loop and our bp mixin to create our four sets of classes. Each will go from 1 through 12 (or whatever our Susy columns property is set to) and will use the breakpoints we defined for small (sm), medium (md) and large (lg). In style.scss add the following: // style.scss @for $i from 1 through map-get($susy, columns) { .col-#{$i} { @include span($i); &-last { @include span($i last); } } } These 9 lines of code are responsible for our mobile-first set of column classes. This loops from one through 12 (which is currently the value of the $susy columns property) and creates a class for each. It also adds a class which handles removing the final columns right margin so our last column doesn't wrap onto a new line. Having control of when this happens will give us the most control. The preceding code would create: .col-1 { width: 6.38298%; float: left; margin-right: 2.12766%; } .col-1-last { width: 6.38298%; float: right; margin-right: 0; } /* 2, 3, 4, and so on up to col-12 */ That means our loop which is only 9 lines of Sass will generate 144 lines of CSS! Now let's create our 3 breakpoints. We'll use an @each loop to get the sizes from our $breakpoints map. This will mean if we add another breakpoint, such as extra-large (xl) it will automatically create the correct set of classes for that size. @each $size, $value in $breakpoints { // Breakpoint will go here and will use $size } Here we're looping over the $breakpoints map and setting a $size variable and a $value variable. The $value variable will not be used, however the $size variable will be set to small, medium and large for each respective loop. We can then use that to set our bp mixin accordingly: @each $size, $value in $breakpoints { @include bp($size) { // The @for loop will go here similar to the above @for loop... } } Now, each loop will set a breakpoint for small, medium and large, and any additional sizes we might add in the future will be generated automatically. Now we can use the same @for loop inside the bp mixin with one small change, we'll add a size to the class name: @each $size, $value in $breakpoints { @include bp($size) { @for $i from 1 through map-get($susy, columns) { .col-#{$i}-#{$size} { @include span($i); &-last { @include span($i last); } } } } } That's everything we need for our grid system. Here's the full stye.scss file: / /style.scss @import 'bower_components/susy/sass/susy'; @import 'helpers'; $breakpoints: ( sm: 480px, md: 768px, lg: 980px ); $susy: ( container: 1160px, columns: 12, gutters: 1/3 ); .container { @include container; } @for $i from 1 through map-get($susy, columns) { .col-#{$i} { @include span($i); &-last { @include span($i last); } } } @each $size, $value in $breakpoints { @include bp($size) { @for $i from 1 through map-get($susy, columns) { .col-#{$i}-#{$size} { @include span($i); &-last { @include span($i last); } } } } } With our bp mixin that's 45 lines of SCSS. And how many lines of CSS does that generate? Nearly 600 lines of CSS! Also, like I've said, if we wanted to create another breakpoint it would only require a change to the $breakpoint map. Then, if we wanted to have 16 columns instead we would only need to the $susy columns property. The above code would then automatically loop over each and create the correct amount of columns for each breakpoint. Testing our grid Next we need to check our grid works. We mainly want to check a few column sizes for each breakpoint and we want to be sure our last keyword is doing what we expect. I've created a simple piece of HTML to do this. I've also add a small bit of CSS to the file to correct box-sizing issues which will happen because of the additional 1px border. I've also restricted the height so text which wraps to a second line won't affect the heights. This is simply so everything remains in line so it's easy to see our widths are working. I don't recommend setting heights on elements. EVER. Instead using padding or line-height if you can to give an element more height and let the content dictate the size of the element. Create a file called index.html in the root of the project and inside add the following: <!doctype html> <html lang="en-GB"> <head> <meta charset="UTF-8"> <title>Susy Grid Test</title> <link rel="stylesheet" type="text/css" href="style.css" /> <style type="text/css"> *, *::before, *::after { box-sizing: border-box; } [class^="col"] { height: 1.5em; background-color: grey; border: 1px solid black; } </style> </head> <body> <div class="container"> <h1>Grid</h1> <div class="col-12 col-10-sm col-2-md col-10-lg">.col-sm-10.col-2-md.col-10-lg</div> <div class="col-12 col-2-sm-last col-10-md-last col-2-lg-last">.col-sm-2-last.col-10-md-last.col-2-lg-last</div> <div class="col-12 col-9-sm col-3-md col-9-lg">.col-sm-9.col-3-md.col-9-lg</div> <div class="col-12 col-3-sm-last col-9-md-last col-3-lg-last">.col-sm-3-last.col-9-md-last.col-3-lg-last</div> <div class="col-12 col-8-sm col-4-md col-8-lg">.col-sm-8.col-4-md.col-8-lg</div> <div class="col-12 col-4-sm-last col-8-md-last col-4-lg-last">.col-sm-4-last.col-8-md-last.col-4-lg-last</div> <div class="col-12 col-7-sm col-md-5 col-7-lg">.col-sm-7.col-md-5.col-7-lg</div> <div class="col-12 col-5-sm-last col-7-md-last col-5-lg-last">.col-sm-5-last.col-7-md-last.col-5-lg-last</div> <div class="col-12 col-6-sm col-6-md col-6-lg">.col-sm-6.col-6-md.col-6-lg</div> <div class="col-12 col-6-sm-last col-6-md-last col-6-lg-last">.col-sm-6-last.col-6-md-last.col-6-lg-last</div> </div> </body> </html> Use your dev tools responsive tools or simply resize the browser from full size down to around 320px and you'll see our grid works as expected. Summary In this article we used Susy grids as well as a simple breakpoint mixin (bp) to create a solid, flexible grid system. With just under 50 lines of Sass we generated our grid system which consists of almost 600 lines of CSS.  Resources for Article: Further resources on this subject: Implementation of SASS [article] Use of Stylesheets for Report Designing using BIRT [article] CSS Grids for RWD [article]
Read more
  • 0
  • 0
  • 14613

article-image-developing-middleware
Packt
08 Aug 2016
16 min read
Save for later

Developing Middleware

Packt
08 Aug 2016
16 min read
In this article by Doug Bierer, author of the book PHP 7 Programming Cookbook, we will cover the following topics: (For more resources related to this topic, see here.) Authenticating with middleware Making inter-framework system calls Using middleware to cross languages Introduction As often happens in the IT industry, terms get invented, and then used and abused. The term middleware is no exception. Arguably the first use of the term came out of the Internet Engineering Task Force (IETF) in the year 2000. Originally, the term was applied to any software which operates between the transport (that is, TCP/IP) and the application layer. More recently, especially with the acceptance of PHP Standard Recommendation number 7 (PSR-7), middleware, specifically in the PHP world, has been applied to the web client-server environment. Authenticating with middleware One very important usage of middleware is to provide authentication. Most web-based applications need the ability to verify a visitor via username and password. By incorporating PSR-7 standards into an authentication class, you will make it generically useful across the board, so to speak, being secure that it can be used in any framework that provides PSR-7-compliant request and response objects. How to do it… We begin by defining a ApplicationAclAuthenticateInterface class. We use this interface to support the Adapter software design pattern, making our Authenticate class more generically useful by allowing a variety of adapters, each of which can draw authentication from a different source (for example, from a file, using OAuth2, and so on). Note the use of the PHP 7 ability to define the return value data type: namespace ApplicationAcl; use PsrHttpMessage { RequestInterface, ResponseInterface }; interface AuthenticateInterface { public function login(RequestInterface $request) : ResponseInterface; } Note that by defining a method that requires a PSR-7-compliant request, and produces a PSR-7-compliant response, we have made this interface universally applicable. Next, we define the adapter that implements the login() method required by the interface. We make sure to use the appropriate classes, and define fitting constants and properties. The constructor makes use of ApplicationDatabaseConnection: namespace ApplicationAcl; use PDO; use ApplicationDatabaseConnection; use PsrHttpMessage { RequestInterface, ResponseInterface }; use ApplicationMiddleWare { Response, TextStream }; class DbTable implements AuthenticateInterface { const ERROR_AUTH = 'ERROR: authentication error'; protected $conn; protected $table; public function __construct(Connection $conn, $tableName) { $this->conn = $conn; $this->table = $tableName; } The core login() method extracts the username and password from the request object. We then do a straightforward database lookup. If there is a match, we store user information in the response body, JSON-encoded: public function login(RequestInterface $request) : ResponseInterface { $code = 401; $info = FALSE; $body = new TextStream(self::ERROR_AUTH); $params = json_decode($request->getBody()->getContents()); $response = new Response(); $username = $params->username ?? FALSE; if ($username) { $sql = 'SELECT * FROM ' . $this->table . ' WHERE email = ?'; $stmt = $this->conn->pdo->prepare($sql); $stmt->execute([$username]); $row = $stmt->fetch(PDO::FETCH_ASSOC); if ($row) { if (password_verify($params->password, $row['password'])) { unset($row['password']); $body = new TextStream(json_encode($row)); $response->withBody($body); $code = 202; $info = $row; } } } return $response->withBody($body)->withStatus($code); } } Best practice Never store passwords in clear text. When you need to do a password match, use password_verify(), which negates the need to reproduce the password hash. The Authenticate class is a wrapper for an adapter class that implements AuthenticationInterface. Accordingly, the constructor takes an adapter class as an argument, as well as a string that serves as the key in which authentication information is stored in $_SESSION: namespace ApplicationAcl; use ApplicationMiddleWare { Response, TextStream }; use PsrHttpMessage { RequestInterface, ResponseInterface }; class Authenticate { const ERROR_AUTH = 'ERROR: invalid token'; const DEFAULT_KEY = 'auth'; protected $adapter; protected $token; public function __construct( AuthenticateInterface $adapter, $key) { $this->key = $key; $this->adapter = $adapter; } In addition, we provide a login form with a security token, which helps prevent Cross Site Request Forgery (CSRF) attacks: public function getToken() { $this->token = bin2hex(random_bytes(16)); $_SESSION['token'] = $this->token; return $this->token; } public function matchToken($token) { $sessToken = $_SESSION['token'] ?? date('Ymd'); return ($token == $sessToken); } public function getLoginForm($action = NULL) { $action = ($action) ? 'action="' . $action . '" ' : ''; $output = '<form method="post" ' . $action . '>'; $output .= '<table><tr><th>Username</th><td>'; $output .= '<input type="text" name="username" /></td>'; $output .= '</tr><tr><th>Password</th><td>'; $output .= '<input type="password" name="password" />'; $output .= '</td></tr><tr><th>&nbsp;</th>'; $output .= '<td><input type="submit" /></td>'; $output .= '</tr></table>'; $output .= '<input type="hidden" name="token" value="'; $output .= $this->getToken() . '" />'; $output .= '</form>'; return $output; } Finally, the login() method in this class checks whether the token is valid. If not, a 400 response is returned. Otherwise, the login() method of the adapter is called: public function login( RequestInterface $request) : ResponseInterface { $params = json_decode($request->getBody()->getContents()); $token = $params->token ?? FALSE; if (!($token && $this->matchToken($token))) { $code = 400; $body = new TextStream(self::ERROR_AUTH); $response = new Response($code, $body); } else { $response = $this->adapter->login($request); } if ($response->getStatusCode() >= 200 && $response->getStatusCode() < 300) { $_SESSION[$this->key] = json_decode($response->getBody()->getContents()); } else { $_SESSION[$this->key] = NULL; } return $response; } } How it works… Go ahead and define the classes presented in this recipe, summarized in the following table: Class Discussed in these steps ApplicationAclAuthenticateInterface 1 ApplicationAclDbTable 2 - 3 ApplicationAclAuthenticate 4 - 6 You can then define a chap_09_middleware_authenticate.php calling program that sets up autoloading and uses the appropriate classes: <?php session_start(); define('DB_CONFIG_FILE', __DIR__ . '/../config/db.config.php'); define('DB_TABLE', 'customer_09'); define('SESSION_KEY', 'auth'); require __DIR__ . '/../Application/Autoload/Loader.php'; ApplicationAutoloadLoader::init(__DIR__ . '/..'); use ApplicationDatabaseConnection; use ApplicationAcl { DbTable, Authenticate }; use ApplicationMiddleWare { ServerRequest, Request, Constants, TextStream }; You are now in a position to set up the authentication adapter and core class: $conn = new Connection(include DB_CONFIG_FILE); $dbAuth = new DbTable($conn, DB_TABLE); $auth = new Authenticate($dbAuth, SESSION_KEY); Be sure to initialize the incoming request, and set up the request to be made to the authentication class: $incoming = new ServerRequest(); $incoming->initialize(); $outbound = new Request(); Check the incoming class method to see if it is POST. If so, pass a request to the authentication class: if ($incoming->getMethod() == Constants::METHOD_POST) { $body = new TextStream(json_encode( $incoming->getParsedBody())); $response = $auth->login($outbound->withBody($body)); } $action = $incoming->getServerParams()['PHP_SELF']; ?> The display logic looks like this: <?= $auth->getLoginForm($action) ?> Here is the output from an invalid authentication attempt. Notice the 401 status code on the right. In this illustration, you could add a var_dump() of the response object. Here is a successful authentication: Making inter-framework system calls One of the primary reasons for the development of PSR-7 (and middleware) was a growing need to make calls between frameworks. It is of interest to note that the main documentation for PSR-7 is hosted by PHP Framework Interop Group (PHP-FIG). How to do it… The primary mechanism used in middleware inter-framework calls is to create a driver program that executes framework calls in succession, maintaining a common request and response object. The request and response objects are expected to represent PsrHttpMessageServerRequestInterface and PsrHttpMessageResponseInterface respectively. For the purposes of this illustration, we define a middleware session validator. The constants and properties reflect the session thumbprint, which is a term we use to incorporate factors such as the website visitor's IP address, browser, and language settings: namespace ApplicationMiddleWareSession; use InvalidArgumentException; use PsrHttpMessage { ServerRequestInterface, ResponseInterface }; use ApplicationMiddleWare { Constants, Response, TextStream }; class Validator { const KEY_TEXT = 'text'; const KEY_SESSION = 'thumbprint'; const KEY_STATUS_CODE = 'code'; const KEY_STATUS_REASON = 'reason'; const KEY_STOP_TIME = 'stop_time'; const ERROR_TIME = 'ERROR: session has exceeded stop time'; const ERROR_SESSION = 'ERROR: thumbprint does not match'; const SUCCESS_SESSION = 'SUCCESS: session validates OK'; protected $sessionKey; protected $currentPrint; protected $storedPrint; protected $currentTime; protected $storedTime; The constructor takes a ServerRequestInterface instance and the session as arguments. If the session is an array (such as $_SESSION), we wrap it in a class. The reason why we do this is in case we are passed a session object, such as JSession used in Joomla. We then create the thumbprint using the factors previously mentioned factors. If the stored thumbprint is not available, we assume this is the first time, and store the current print as well as stop time, if this parameter is set. We used md5() because it's a fast hash, and is not exposed externally and is therefore useful to this application: public function __construct( ServerRequestInterface $request, $stopTime = NULL) { $this->currentTime = time(); $this->storedTime = $_SESSION[self::KEY_STOP_TIME] ?? 0; $this->currentPrint = md5($request->getServerParams()['REMOTE_ADDR'] . $request->getServerParams()['HTTP_USER_AGENT'] . $request->getServerParams()['HTTP_ACCEPT_LANGUAGE']); $this->storedPrint = $_SESSION[self::KEY_SESSION] ?? NULL; if (empty($this->storedPrint)) { $this->storedPrint = $this->currentPrint; $_SESSION[self::KEY_SESSION] = $this->storedPrint; if ($stopTime) { $this->storedTime = $stopTime; $_SESSION[self::KEY_STOP_TIME] = $stopTime; } } } It's not required to define __invoke(), but this magic method is quite convenient for standalone middleware classes. As is the convention, we accept ServerRequestInterface and ResponseInterface instances as arguments. In this method we simply check to see if the current thumbprint matches the one stored. The first time, of course, they will match. But on subsequent requests, the chances are an attacker intent on session hijacking will be caught out. In addition, if the session time exceeds the stop time (if set), likewise, a 401 code will be sent: public function __invoke( ServerRequestInterface $request, Response $response) { $code = 401; // unauthorized if ($this->currentPrint != $this->storedPrint) { $text[self::KEY_TEXT] = self::ERROR_SESSION; $text[self::KEY_STATUS_REASON] = Constants::STATUS_CODES[401]; } elseif ($this->storedTime) { if ($this->currentTime > $this->storedTime) { $text[self::KEY_TEXT] = self::ERROR_TIME; $text[self::KEY_STATUS_REASON] = Constants::STATUS_CODES[401]; } else { $code = 200; // success } } if ($code == 200) { $text[self::KEY_TEXT] = self::SUCCESS_SESSION; $text[self::KEY_STATUS_REASON] = Constants::STATUS_CODES[200]; } $text[self::KEY_STATUS_CODE] = $code; $body = new TextStream(json_encode($text)); return $response->withStatus($code)->withBody($body); } We can now put our new middleware class to use. The main problems with inter-framework calls, at least at this point, are summarized here. Accordingly, how we implement middleware depends heavily on the last point: Not all PHP frameworks are PSR-7-compliant Existing PSR-7 implementations are not complete All frameworks want to be the "boss" As an example, have a look at the configuration files for Zend Expressive, which is a self-proclaimed PSR7 Middleware Microframework. Here is the file, middleware-pipeline.global.php, which is located in the config/autoload folder in a standard Expressive application. The dependencies key is used to identify middleware wrapper classes that will be activated in the pipeline: <?php use ZendExpressiveContainerApplicationFactory; use ZendExpressiveHelper; return [ 'dependencies' => [ 'factories' => [ HelperServerUrlMiddleware::class => HelperServerUrlMiddlewareFactory::class, HelperUrlHelperMiddleware::class => HelperUrlHelperMiddlewareFactory::class, // insert your own class here ], ], Under the middleware_pipline key you can identify classes that will be executed before or after the routing process occurs. Optional parameters include path, error, and priority: 'middleware_pipeline' => [ 'always' => [ 'middleware' => [ HelperServerUrlMiddleware::class, ], 'priority' => 10000, ], 'routing' => [ 'middleware' => [ ApplicationFactory::ROUTING_MIDDLEWARE, HelperUrlHelperMiddleware::class, // insert reference to middleware here ApplicationFactory::DISPATCH_MIDDLEWARE, ], 'priority' => 1, ], 'error' => [ 'middleware' => [ // Add error middleware here. ], 'error' => true, 'priority' => -10000, ], ], ]; Another technique is to modify the source code of an existing framework module, and make a request to a PSR-7-compliant middleware application. Here is an example modifying a Joomla! installation to include a middleware session validator. Next, add this code the end of the index.php file in the /path/to/joomla folder. Since Joomla! uses Composer, we can leverage the Composer autoloader: session_start(); // to support use of $_SESSION $loader = include __DIR__ . '/libraries/vendor/autoload.php'; $loader->add('Application', __DIR__ . '/libraries/vendor'); $loader->add('Psr', __DIR__ . '/libraries/vendor'); We can then create an instance of our middleware session validator, and make a validation request just before $app = JFactory::getApplication('site');: $session = JFactory::getSession(); $request = (new ApplicationMiddleWareServerRequest())->initialize(); $response = new ApplicationMiddleWareResponse(); $validator = new ApplicationSecuritySessionValidator( $request, $session); $response = $validator($request, $response); if ($response->getStatusCode() != 200) { // take some action } How it works… First, create the ApplicationMiddleWareSessionValidator test middleware class described in steps 2 - 5. Then you will need to go to getcomposer.org and follow the directions to obtain Composer. Next, build a basic Zend Expressive application, as shown next. Be sure to select No when prompted for minimal skeleton: cd /path/to/source/for/this/chapter php composer.phar create-project zendframework/zend-expressive-skeleton expressive This will create a folder /path/to/source/for/this/chapter/expressive. Change to this directory. Modify public/index.php as follows: <?php if (php_sapi_name() === 'cli-server' && is_file(__DIR__ . parse_url( $_SERVER['REQUEST_URI'], PHP_URL_PATH)) ) { return false; } chdir(dirname(__DIR__)); session_start(); $_SESSION['time'] = time(); $appDir = realpath(__DIR__ . '/../../..'); $loader = require 'vendor/autoload.php'; $loader->add('Application', $appDir); $container = require 'config/container.php'; $app = $container->get(ZendExpressiveApplication::class); $app->run(); You will then need to create a wrapper class that invokes our session validator middleware. Create a SessionValidateAction.php file that needs to go in the /path/to/source/for/this/chapter/expressive/src/App/Action folder. For the purposes of this illustration, set the stop time parameter to a short duration. In this case, time() + 10 gives you 10 seconds: namespace AppAction; use ApplicationMiddleWareSessionValidator; use ZendDiactoros { Request, Response }; use PsrHttpMessageResponseInterface; use PsrHttpMessageServerRequestInterface; class SessionValidateAction { public function __invoke(ServerRequestInterface $request, ResponseInterface $response, callable $next = null) { $inbound = new Response(); $validator = new Validator($request, time()+10); $inbound = $validator($request, $response); if ($inbound->getStatusCode() != 200) { session_destroy(); setcookie('PHPSESSID', 0, time()-300); $params = json_decode( $inbound->getBody()->getContents(), TRUE); echo '<h1>',$params[Validator::KEY_TEXT],'</h1>'; echo '<pre>',var_dump($inbound),'</pre>'; exit; } return $next($request,$response); } } You will now need to add the new class to the middleware pipeline. Modify config/autoload/middleware-pipeline.global.php as follows. Modifications are shown in bold: <?php use ZendExpressiveContainerApplicationFactory; use ZendExpressiveHelper; return [ 'dependencies' => [ 'invokables' => [ AppActionSessionValidateAction::class => AppActionSessionValidateAction::class, ], 'factories' => [ HelperServerUrlMiddleware::class => HelperServerUrlMiddlewareFactory::class, HelperUrlHelperMiddleware::class => HelperUrlHelperMiddlewareFactory::class, ], ], 'middleware_pipeline' => [ 'always' => [ 'middleware' => [ HelperServerUrlMiddleware::class, ], 'priority' => 10000, ], 'routing' => [ 'middleware' => [ ApplicationFactory::ROUTING_MIDDLEWARE, HelperUrlHelperMiddleware::class, AppActionSessionValidateAction::class, ApplicationFactory::DISPATCH_MIDDLEWARE, ], 'priority' => 1, ], 'error' => [ 'middleware' => [ // Add error middleware here. ], 'error' => true, 'priority' => -10000, ], ], ]; You might also consider modifying the home page template to show the status of $_SESSION. The file in question is /path/to/source/for/this/chapter/expressive/templates/app/home-page.phtml. Simply adding var_dump($_SESSION) should suffice. Initially, you should see something like this: After 10 seconds, refresh the browser. You should now see this: Using middleware to cross languages Except in cases where you are trying to communicate between different versions of PHP, PSR-7 middleware will be of minimal use. Recall what the acronym stands for: PHP Standards Recommendations. Accordingly, if you need to make a request to an application written in another language, treat it as you would any other web service HTTP request. How to do it… In the case of PHP 4, you actually have a chance in that there was limited support for object-oriented programming. There is not enough space to cover all changes, but we present a potential PHP 4 version of ApplicationMiddleWareServerRequest. The first thing to note is that there are no namespaces! Accordingly, we use a classname with underscores, _, in place of namespace separators: class Application_MiddleWare_ServerRequest extends Application_MiddleWare_Request implements Psr_Http_Message_ServerRequestInterface { All properties are identified in PHP 4 using the key word var: var $serverParams; var $cookies; var $queryParams; // not all properties are shown The initialize() method is almost the same, except that syntax such as $this->getServerParams()['REQUEST_URI'] was not allowed in PHP 4. Accordingly, we need to split this out into a separate variable: function initialize() { $params = $this->getServerParams(); $this->getCookieParams(); $this->getQueryParams(); $this->getUploadedFiles; $this->getRequestMethod(); $this->getContentType(); $this->getParsedBody(); return $this->withRequestTarget($params['REQUEST_URI']); } All of the $_XXX super-globals were present in later versions of PHP 4: function getServerParams() { if (!$this->serverParams) { $this->serverParams = $_SERVER; } return $this->serverParams; } // not all getXXX() methods are shown to conserve space The null coalesce" operator was only introduced in PHP 7. We need to use isset(XXX) ? XXX : ''; instead: function getRequestMethod() { $params = $this->getServerParams(); $method = isset($params['REQUEST_METHOD']) ? $params['REQUEST_METHOD'] : ''; $this->method = strtolower($method); return $this->method; } The JSON extension was not introduced until PHP 5. Accordingly, we need to be satisfied with raw input. We could also possibly use serialize() or unserialize() in place of json_encode() and json_decode(): function getParsedBody() { if (!$this->parsedBody) { if (($this->getContentType() == Constants::CONTENT_TYPE_FORM_ENCODED || $this->getContentType() == Constants::CONTENT_TYPE_MULTI_FORM) && $this->getRequestMethod() == Constants::METHOD_POST) { $this->parsedBody = $_POST; } elseif ($this->getContentType() == Constants::CONTENT_TYPE_JSON || $this->getContentType() == Constants::CONTENT_TYPE_HAL_JSON) { ini_set("allow_url_fopen", true); $this->parsedBody = file_get_contents('php://stdin'); } elseif (!empty($_REQUEST)) { $this->parsedBody = $_REQUEST; } else { ini_set("allow_url_fopen", true); $this->parsedBody = file_get_contents('php://stdin'); } } return $this->parsedBody; } The withXXX() methods work pretty much the same in PHP 4: function withParsedBody($data) { $this->parsedBody = $data; return $this; } Likewise, the withoutXXX() methods work the same as well: function withoutAttribute($name) { if (isset($this->attributes[$name])) { unset($this->attributes[$name]); } return $this; } } For websites using other languages, we could use the PSR-7 classes to formulate requests and responses, but would then need to use an HTTP client to communicate with the other website. Here is the example: $request = new Request( TARGET_WEBSITE_URL, Constants::METHOD_POST, new TextStream($contents), [Constants::HEADER_CONTENT_TYPE => Constants::CONTENT_TYPE_FORM_ENCODED, Constants::HEADER_CONTENT_LENGTH => $body->getSize()] ); $data = http_build_query(['data' => $request->getBody()->getContents()]); $defaults = array( CURLOPT_URL => $request->getUri()->getUriString(), CURLOPT_POST => true, CURLOPT_POSTFIELDS => $data, ); $ch = curl_init(); curl_setopt_array($ch, $defaults); $response = curl_exec($ch); curl_close($ch); Summary In this article, we learned about providing authentication to a system, to make calls between frameworks, and to make a request to an application written in another language. Resources for Article: Further resources on this subject: Middleware [Article] Building a Web Application with PHP and MariaDB – Introduction to caching [Article] Data Tables and DataTables Plugin in jQuery 1.3 with PHP [Article]
Read more
  • 0
  • 0
  • 5757

article-image-tiered-application-architecture-docker-compose-part-3
Darwin Corn
08 Aug 2016
6 min read
Save for later

Tiered Application Architecture with Docker Compose, Part 3

Darwin Corn
08 Aug 2016
6 min read
This is the third part in a series that introduces you to basic web application containerization and deployment principles. If you're new to the topic, I suggest reading Part 1 and Part 2 . In this post, I attempt to take the training wheels off, and focus on using Docker Compose. Speaking of training wheels, I rode my bike with training wheels until I was six or seven. So in the interest of full disclosure, I have to admit that to a certain degree I'm still riding the containerization wave with my training wheels on. That's not to say I’m not fully using container technology. Before transitioning to the cloud, I had a private registry running on a Git server that my build scripts pushed to and pulled from to automate deployments. Now, we deploy and maintain containers in much the same way as I've detailed in the first two Parts in this series, and I take advantage of the built-in registry covered in Part 2 of this series. Either way, our use case multi-tiered application architecture was just overkill. Adding to that, when we were still doing contract work, Docker was just getting 1.6 off the ground. Now that I'm working on a couple of projects where this will be a necessity, I'm thankful that Docker has expanded their offerings to include tools like Compose, Machine and Swarm. This post will provide a brief overview of a multi-tiered application setup with Docker Compose, so look for future posts to deal with the latter two. Of course, you can just hold out for a mature Kitematic and do it all in a GUI, but you probably won't be reading this post if that applies to you. All three of these Docker extensions are relatively new, and so the entirety of this post is subject to a huge disclaimer that even Docker hasn't fully developed these extensions to be production-ready for large or intricate deployments. If you're looking to do that, you're best off holding out for my post on alternative deployment options like CoreOS and Kubernetes. But that's beyond the scope of what we're looking at here, so let's get started. First, you need to install the binary. Since this is part 3, I'm going to assume that you have the Docker Engine already installed somewhere. If you're on Mac or Windows, the Docker Toolbox you used to install it also contained an option to install Compose. I'm going to assume your daily driver is a Linux box, so these instructions are for Linux. Fortunately, the installation should just be a couple of commands--curling it from the web and making it executable: # curl -L https://github.com/docker/compose/releases/download/1.6.2/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose # chmod +x /usr/local/bin/docker-compose # docker-compose -v That last command should output version info if you've installed it correctly. For some reason, the linked installation doc thinks you can run that chmod as a regular user. I'm not sure of any distro that lets regular users write to /usr/local/bin, so I ran them both as root. Docker has its own security issues that are beyond the scope of this series, but I suggest reading about them if you're using this in production. My lazy way around it is to run every Docker-related command elevated, and I'm sure someone will let me have it for that in the comments. Seems like a better policy than making /usr/local/bin writeable by anyone other than root. Now that you have Compose installed, let's look at how to use it to coordinate and deploy a layered application. I'm abandoning my sample music player of the previous two posts in favor of something that's already separated its functionality, namely the Taiga project. If you're not familiar, it's a slick flat JIRA-killer, and the best part is that it's open source with a thorough installation guide. I've done the heavy lifting, so all you have to do is clone the docker-taiga repo into wherever you keep your source code and get to Composin'. $ git clone https://github.com/ndarwincorn/docker-taiga.git $ cd docker-taiga You'll notice a few things. In the root of the app, there's an .envfile where you can set all the environmental variables in one place. Next, there are two folders with taiga- prefixes. They correspond to the layers of the application, from the Angular frontend to the websocket and backend Django server. Each contains a Dockerfile for building the container, as well as relevant configuration files. There's also a docker-entrypoint-initdb.d folder that contains a shell script that creates the Taiga database when the postgres container is built. Having covered container creation in part 1, I'm more concerned with the YAML file in the root of the application, docker-compose.yml. This file coordinates the container/image creation for the application, and full reference can be found on Docker's website. Long story short, the compose YAML file gives the containers a creation order (databases, backend/websocket, frontend) and links them together, so that ports exposed in each container don't need to be published to the host machine. So, from the root of the application, let's run a # docker-compose up and see what happens. Provided there are no errors, you should be able to navigate to localhost:8080 and see your new Taiga deployment! You should be able to log in with the admin user and password 123123. Of course, there's much more to do--configure automated e-mails, link it to your Github organization, configure TLS. I'll leave that as an exercise for you. For now, enjoy your brand-new layered project management application. Of course, if you're deploying such an application for an organization, you don't want all your eggs in one basket. The next two parts in the series will deal with leveraging Docker tools and alternatives to deploy the application in a clustered, high-availability setup. About the Author Darwin Corn is a systems analyst for the Consumer Direct Care Network. He is a mid-level professional with diverse experience in the information technology world.
Read more
  • 0
  • 0
  • 7655
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at $19.99/month. Cancel anytime
article-image-preparing-simple-migration
Packt
08 Aug 2016
9 min read
Save for later

Preparing for a Simple Migration

Packt
08 Aug 2016
9 min read
In this article by, Ian Waters, David Greve, and Loryan Strant, the authors of the book, Microsoft Office 365: Exchange Online Implementation and Migration - Second Edition, so far, we have prepared you for purchasing and beginning to use Office 365. However, a key factor yet to be completed is to migrate mailbox content from your existing mail system. We will discuss the various options available for performing a migration to Exchange Online and what needs to be done to prepare. This will cover the following topics: Overview of a simple migration Available migration options Differences between source platforms The migration process Preparing your environment for migration (For more resources related to this topic, see here.) What is a simple migration? When an organization looks to migrate mail in a single direction to Exchange Online, it is called a simple migration due to the fact that it is one way. Organizations looking to retain on-premises Exchange Server functionality as well as utilize Exchange Online would choose a hybrid environment over a simple migration. A hybrid environment is a mixture of both the environments, which is more complex to implement and maintain than a simple migration. Migration options There are really only five main mail platform types used by organizations that wish to migrate to Exchange Online: POP e-mail systems IMAP e-mail systems Hosted Exchange Server Gmail or Google apps Exchange Server 2003 or 2007 Based on these platforms we have created several scenarios which are reflective of common customer environments and situations. POP e-mail Many small businesses currently utilize POP e-mail provided by their domain name or website host. These are the simplest of migrations as the mailbox content is migrated by relying on the PST file import functionality in Outlook. Users simply need to import their existing PST file into the new Exchange Online mailbox. IMAP Some organizations may be utilizing IMAP either from a hosted service provider or on an on-premises server (such as Novell GroupWise). These mail systems can be imported natively by using the migration wizard within the Exchange Online Control Panel. It is important to note that IMAP only contains mail objects and folders (not calendars, contacts, tasks, or any other Outlook-based items as these are stored in PST files). Hosted Exchange or Gmail For organizations that have already made a move to cloud services and are using email from either a hosted Exchange Server provider or Gmail (aka Google Apps), they are better off utilizing third party cloud-based migration systems. An example of this is www.migrationwiz.com which allows for mailbox content to be moved from the original mailbox directly into the new Exchange Online mailbox. Exchange Server 2003 and 2007 It is quite common to find organizations still running Exchange Server 2003 or 2007 either as a standalone solution or as part of Small Business Server 2003 or 2008. Effectively, all that is required is the enabling of RPC over HTTP for any of the versions. However, this can prove to be challenging as the functionality was not native to Exchange Server/SBS 2003. In Exchange Server 2007 / SBS 2008, however, there are more complexities around the provisioning of the actual certificates and utilization of the Autodiscover functionality. The actual migration is performed by running the migration wizard within the Exchange Online Control Panel. Staged migration A staged migration allows you to choose which mailboxes get migrated to Exchange Online and move at your own pace. During this process, however, users of both Exchange Server on-premises and Exchange Online are in separate environments. As a result, these users will not have access to a shared Global Address List or Free/Busy information. During the migration, all emails that are routed via the on-premises Exchange Server are also retained locally, which provides for rollback if required. Cutover migration A cutover migration simply begins the process of migrating all mailboxes from your on-premises Exchange Server to Exchange Online in a single step. Migration option comparison For simple digestion of the preceding scenarios, we have broken down the differences between the options into the following table and listed requirements for each: Migration type / Requirements PST Migration IMAP 3rd Party Tool Exchange Staged Exchange Cutover Server access No No Possibly Yes Yes Directory Synchronization No No No Yes No Define your own migration schedule Yes Yes Yes Yes No SSL certificate purchase No No Possibly Yes Yes PC access to migrate content Yes Yes No No No The migration process – in a nutshell The steps involved in the actual migration process will differ depending on the option chosen. Effectively, they will follow the same basic pattern: Creating user accounts Activating Exchange Online mailboxes Migrating mailbox content Generally, the Mail eXchange (MX) record is cut over after the completion of a successful mailbox migration. However, some administrators prefer to do this beforehand to ensure that no new mail items are left behind. However, the key to a successful migration is not the actual mailbox content migration itself, but the planning and preparation. Planning for migration Several key factors must be addressed when planning the migration to Exchange Online to ensure it is done successfully. These are both technical factors as well as human factors. Technical considerations There are numerous technical considerations which must be taken into account when planning for a migration. Some of the more common questions are as follows: Which of the previous example scenarios mirrors yours? Have users been informed of the change? How much data can you send through your Internet link to Exchange Online (meaning how many gigabytes of mail can be uploaded)? Does your monthly Internet download allowance cater for the mailboxes being downloaded back into the Outlook .OST file for each user? Do you plan to start users on Exchange Online with their full mailbox or just a portion of recent content? Do you have access to all desktops so you can configure the new account in Outlook? How many computers will you have to re-configure, and do you have the resources to do them all in the timeframe? Asking questions such as these will help determine your migration plan. For example, if you have 80 users, each with mailboxes of 5 GB, then it is not likely that you will be able to transfer 400 GB of data in a single weekend. This is especially important as Exchange Online only supports mailbox transfer speeds of 500 MB per hour (higher speeds can be achieved by raising a support request before commencing the migration). Therefore, you would most likely go for a staged migration approach (if using Exchange Server) or alternatively a third party solution such as www.migrationwiz.com which allows multiple mailbox copy passes. People considerations Another key element of any migration plan is change management and a communication plan with a view to ensure that the end user experience is not a negative one. It is important to notify the users of any changes to their day-to-day operations that may impact their productivity as any disruption could further delay the migration or leave a sour taste in their mouth. Part of the change management is to inform the users about the migration procedure at a high level so that they are made to feel part of the process. There will also be an element of having to re-configure the desktops and mobile devices. So the more comprehensive your change management and communications, the more you will be able to empower the users to do the work themselves. However, this can also be simplified by utilizing solutions such as https://www.bittitan.com/products/deploymentpro/about (from the same company that makes MigrationWiz) to automate the desktop configuration process. Preparing your environment for migration Before beginning the migration process it is important to ensure that your environment has been prepared and that all requirements have been met. Due to the varying requirements and processes, we have broken these down between types of migrations. PST-based migrations are not documented as they are merely a file export/import procedure which requires little preparation. You may want to document a procedure which states where the PST files are stored and how they are named, and even list the PST file passwords if password protection is required. IMAP In order for Exchange Online to access your IMAP server/service, it must be accessible via the IMAP protocol from the Internet – which is generally the case. It is important that the users be created via the Office 365 administrative interface, either individually or in bulk. You also need to prepare a listing of the existing IMAP mailboxes in CSV format which is imported into the Exchange Online Control Panel. This CSV maps the IMAP credentials (e-mail address, usernames, and passwords) to the Exchange Online mailbox. Hosted Exchange/Gmail When migrating to Exchange Online from a hosted Exchange or Gmail environment, you will generally have less access to the environment and as such less preparation work can be done. The process will be similar to that of IMAP whereby users are created in Office 365, followed by you working through the process of the third party provider to get access to the existing mail system and have the migration provider (for example, MigrationWiz) move the content directly. On-Premises Exchange 2003 or later A requirement of preparing to migrate to Exchange Online from Exchange Server 2003 or later is to ensure that the server can be reached using the RPC over HTTPS (Outlook Anywhere) method on port 443. You must have a valid SSL certificate installed from a trusted root Certificate Authority. Self-signed SSL certificates issued by your server will not work as they are not recognized by external sources as "trusted". To ensure that your Exchange Server is accessible by Office 365 and ready for migration, it is recommended that you test Outlook Connectivity by using the Microsoft Remote Connectivity Analyzer (https://testconnectivity.microsoft.com). Summary We have walked you through some of the common scenarios seen when performing a simple migration to Exchange Online. By now your environment should be ready, and both you and your users ready to take the final step and begin migrating your emails to the cloud. Resources for Article: Further resources on this subject: Microsoft Office Outlook Programming Using VSTO 3.0 and C#: Part 2 [article] Microsoft Office Live Small Business: Extending the Personal Website [article] Microsoft Office Excel Programming Using VSTO 3.0 [article]
Read more
  • 0
  • 0
  • 1405

article-image-key-elements-time-series-analysis
Packt
08 Aug 2016
7 min read
Save for later

Key Elements of Time Series Analysis

Packt
08 Aug 2016
7 min read
In this article by Jay Gendron, author of the book, Introduction to R for Business Intelligence, we will see that the time series analysis is the most difficult analysis technique. It is true that this is a challenging topic. However, one may also argue that an introductory awareness of a difficult topic is better than perfect ignorance about it. Time series analysis is a technique designed to look at chronologically ordered data that may form cycles over time. Key topics covered in this article include the following: (For more resources related to this topic, see here.) Introducing key elements of time series analysis Time series analysis is an upper-level college statistics course. It is also a demanding topic taught in econometrics. This article provides you an understanding of a useful but difficult analysis technique. It provides a combination of theoretical learning and hands-on practice. The goal is to provide you a basic understanding of working with time series data and give you a foundation to learn more. Use Case: forecasting future ridership The finance group approached the BI team and asked for help with forecasting future trends. They heard about your great work for the marketing team and wanted to get your perspective on their problem. Once a year they prepare an annual report that includes ridership details. They are hoping to include not only last year's ridership levels, but also a forecast of ridership levels in the coming year. These types of time-based predictions are forecasts. The Ch6_ridership_data_2011-2012.csv data file is available at the website—http://jgendron.github.io/com.packtpub.intro.r.bi/. This data is a subset of the bike sharing data. It contains two years of observations, including the date and a count of users by hour. Introducing key elements of time series analysis You just applied a linear regression model to time series data and saw it did not work. The biggest problem was not a failure in fitting a linear model to the trend. For this well-behaved time series, the average formed a linear plot over time. Where was the problem? The problem was in seasonal fluctuations. The seasonal fluctuations were one year in length and then repeated. Most of the data points existed above and below the fitted line, instead of on it or near it. As we saw, the ability to make a point estimate prediction was poor. There is an old adage that says even a broken clock is correct twice a day. This is a good analogy for analyzing seasonal time series data with linear regression. The fitted linear line would be a good predictor twice every cycle. You will need to do something about the seasonal fluctuations in order to make better forecasts; otherwise, they will simply be straight lines with no account of the seasonality. With seasonality in mind, there are functions in R that can break apart the trend, seasonality, and random components of a time series. The decompose() function found in the forecast package shows how each of these three components influence the data. You can think of this technique as being similar to creating the correlogram plot during exploratory data analysis. It captures a greater understanding of the data in a single plot: library(forecast); plot(decompose(airpass)) The output of this code is shown here: This decomposition capability is nice as it gives you insights about approaches you may want to take with the data, and with reference to the previous output, they are described as follows: The top panel provides a view of the original data for context. The next panel shows the trend. It smooths the data and removes the seasonal component. In this case, you will see that over time, air passenger volume has increased steadily and in the same direction. The third plot shows the seasonal component. Removing the trend helps reveal any seasonal cycles. This data shows a regular and repeated seasonal cycle through the years. The final plot is the randomness—everything else in the data. It is like the error term in linear regression. You will see less error in the middle of the series. The stationary assumption There is an assumption for creating time series models. The data must be stationary. Stationary data exists when its mean and variance do not change as a function of time. If you decompose a time series and witness a trend, seasonal component, or both, then you have non-stationary data. You can transform them into stationary data in order to meet the required assumption. Using a linear model for comparison, there is randomness around a mean—represented by data points scattered randomly around a fitted line. The data is independent of time and it does not follow other data in a cycle. This means that the data is stationary. Not all the data lies on the fitted line, but it is not moving. In order to analyze time series data, you need your data points to stay still. Imagine trying to count a class of primary school students while they are on the playground during recess. They are running about back and forth. In order to count them, you need them to stay still—be stationary. Transforming non-stationary data into stationary data allows you to analyze it. You can transform non-stationary data into stationary data using a technique called differencing. The differencing techniques Differencing subtracts each data point from the data point that is immediately in front of it in the series. This is done with the diff() function. Mathematically, it works as follows: Seasonal differencing is similar, but it subtracts each data point from its related data point in the next cycle. This is done with the diff() function, along with a lag parameter set to the number of data points in a cycle. Mathematically, it works as follows: Look at the results of differencing in this toy example. Build a small sample dataset of 36 data points that include an upward trend and seasonal component, as shown here: seq_down <- seq(.625, .125, -0.125) seq_up <- seq(0, 1.5, 0.25) y <- c(seq_down, seq_up, seq_down + .75, seq_up + .75, seq_down + 1.5, seq_up + 1.5) Then, plot the original data and the results obtained after calling the diff() function: par(mfrow = c(1, 3)) plot(y, type = "b", ylim = c(-.1, 3)) plot(diff(y), ylim = c(-.1, 3), xlim = c(0, 36)) plot(diff(diff(y), lag = 12), ylim = c(-.1, 3), xlim = c(0, 36)) par(mfrow = c(1, 1)) detach(package:TSA, unload=TRUE) These three panels show the results of differencing and seasonal differencing. Detach the TSA package to avoid conflicts with other functions in the forecast library we will use, as follows: These three panes are described as follows: The left pane shows n = 36 data points, with 12 points in each of the three cycles. It also shows a steadily increasing trend. Either of these characteristics breaks the stationary data assumption. The center pane shows the results of differencing. Plotting the difference between each point and its next neighbor removes the trend. Also, notice that you get one less data point. With differencing, you get (n - 1) results. The right pane shows seasonal differencing with a lag of 12. The data is stationary. Notice that the trend differencing is now the data in the seasonal differencing. Also note that you will lose a cycle of data, getting (n - lag) results. Summary Congratulations, you truly deserve recognition for getting through a very tough topic. You now have more awareness about time series analysis than some people with formal statistical training.  Resources for Article:   Further resources on this subject: Managing Oracle Business Intelligence [article] Self-service Business Intelligence, Creating Value from Data [article] Business Intelligence and Data Warehouse Solution - Architecture and Design [article]
Read more
  • 0
  • 0
  • 12277

article-image-getting-started-android-development
Packt
08 Aug 2016
14 min read
Save for later

Getting started with Android Development

Packt
08 Aug 2016
14 min read
In this article by Raimon Ràfols Montané, author of the book Learning Android Application Development, we will go through all the steps required to start developing Android devices. We have to be aware that Android is an evolving platform and so are its development tools. We will show how to download and install Android Studio and how to create a new project and run it on either an emulator or a real device. (For more resources related to this topic, see here.) Setting up Android Studio Before being able to build an Android application, we have to download and install Android Studio on our computer. It is still possible to download and use Eclipse with the Android Development Tools (ADT) plugin, but Google no longer supports it and they recommend that we migrate to Android Studio. In order to be aligned with this, we will only focus on Android Studio in this article. for more information on this, visit http://android-developers.blogspot.com.es/2015/06/an-update-on-eclipse-android-developer.html. Getting the right version of Android Studio The latest stable version of Android Studio can be found at http://developer.android.com/sdk/index.html. If you are among the bravest developers, and you are not afraid of bugs, you can always go to the Canary channel and download the latest version. The Canary channel is one of the preview channels available on the Android tools download page (available at http://tools.android.com/download/studio) and contains weekly builds. The following are other preview channels available at that URL: The Canary channel contains weekly builds. These builds are tested but they might contain some issues. Just use a build from this channel if you need or want to see the latest features. The Dev channel contains selected Canary builds. The beta channel contains the beta milestones for the next version of Android Studio. The stable channel contains the most recent stable builds of Android Studio. The following screenshot illustrates the Android tools download page: It is not recommended to use an unstable version for production. To be on the safer side, always use the latest stable version. In this article, we will use the version 2.2 preview. Although it is a beta version at this moment, we will have the main version quite soon. Installing Android Studio Android Studio requires JDK 6 or higher and, at least, JDK 7 is required if you aim to develop for Android 5.0 and higher. You can easily check which version you have installed by running this on your command line: javac –version If you don't have any version of the JDK or you have an unsupported version, please install or update your JDK before proceeding to install Android Studio. Refer to the official documentation for a more comprehensive installation guide and details on all platforms (Windows, Linux, and Mac OSX): http://developer.android.com/sdk/installing/index.html?pkg=studio Once you have JDK installed, unpack the package you have just downloaded from the Internet and proceed with the installation. For example, let's use Mac OSX. If you download the latest stable version, you will get a .dmg file that can be mounted on your filesystem. Once mounted, a new finder window that appears will ask us to drag the Android Studio icon to the Applications folder. Just doing this simple step will complete the basic installation. If you have downloaded a preview version, you will have a ZIP file that once unpacked will contain the Android Studio Application directly (can be just dragged to the Applications folder using finder). For other platforms, refer to the official installation guide provided by Google at the web address mentioned earlier. First run Once you have finished installing Android Studio, it is time to run it for the first time. On the first execution (at least if you have downloaded version 2.2), will let you configure some options and install some SDK components if you choose the custom installation type. Otherwise, both these settings and SDK components can be later configured or installed. The first option you will be able to choose is the UI theme. We have the default UI theme or the Darcula theme, which basically is a choice of light or dark background, respectively. After this step, the next window will show the SDK Components Setup where the installation process will let you choose some components to automatically download and install. On Mac OS, there is a bug in some versions of Android Studio 2.0 that sometimes does not allow selecting any option if the target folder does not exist. If that happens, follow these steps for a quick fix: Copy the contents of the Android SDK Location field, just the path or something like /Users/<username>/Library/Android/sdk, to the clipboard. Open the terminal application. Create the folder manually as mkdir /Users/<username>/Library/Android/sdk. Go back to Android Studio, press the Previous button and then the Next button to come back to this screen. Now, you will be able to select the components that you would like to install. If that still does not work, cancel the installation process, ensuring that you checked the option to rerun the setup on the next installation. Quit Android Studio and rerun it. Creating a sample project We will introduce some of the most common elements on Android Studio by creating a sample project, building it and running it on an android emulator or on a real android device. It is better to dispaly those elements when you need them rather than just enumerate a long list without a real use behind. Starting a new project Just click on the Start a new Android Studio project button to start a project from scratch. Android Studio will ask you to make some project configuration settings, and you will be able to launch your project. If you have an already existing project and would like to import it to Android Studio, you could do it now as well. Any projects based on Eclipse, Ant, or Gradle build can be easily imported into Android Studio. Projects can be also checked out from Version Control software such as Subversion or Git directly from Android Studio. When creating a new project, it will ask for the application name and the company domain name, which will be reversed into the application package name. Once this information is filled out, Android Studio will ask the type of devices or form factors your application will target. This includes not only phone and tablet, but also Android Wear, Android TV, Android Auto, or Google Glass. In this example, we will target only phone and tablet and require a minimum SDK API level of 14 (Android 4.0 or Ice Cream Sandwich). By setting the minimum required level to 14, we make sure that the app will run on approximately 96.2% of devices accessing Google Play Store, which is good enough. If we set 23 as the minimum API level (Android 6.0 Marshmallow), our application will only run on Android Marshmallow devices, which is less than 1% of active devices on Google Play right now. Unless we require a very specific feature available on a specific API level, we should use common sense and try to aim for as many devices as we can. Having said that, we should not waste time supporting very old devices (or very old versions of the Android), as they might be, for example, only 5% of the active devices but may imply lots and lots of work to make your application support them. In addition to the minimum SDK version, there is also the target SDK version. The target SDK version should be, ideally, set to the latest stable version of Android available to allow your application to take advantage of all the new features, styles, and behaviors from newer versions. As a rule of thumb, Google gives you the percentage of active devices on Google Play, not the percentage of devices out there in the wild. So, unless we need to build an enterprise application for a closed set of devices and installed ad hoc, we should not mind those people not even accessing Google Play, as they will not the users of our application because they do not usually download applications, unless we are targeting countries where Google Play is not available. In that case, we should analyze our requirements with real data from the available application stores in those countries. To see the Android OS version distribution, always check the Android's developer dashboard at http://developer.android.com/about/dashboards/index.html. Alternatively, when creating a new project from Android Studio, there is a link to help you choose the version that you would like to target, which will open a new screen with the cumulative percentage of coverage. If you click on each version, it will give you more details about that Android OS version and the features that were introduced. After this step, and to simplify our application creation process, Android Studio will allow us to add an Activity class to the project out from some templates. In this case, we can add an empty Activity class for the moment being. Let's not worry for the name of the Activity class and layout file at this moment; we can safely proceed with the prefilled values. As defined by Android developer documentation an: Activity is a single, focused thing that the user can do.  http://developer.android.com/reference/android/app/Activity.html To simplify further, we can consider an Activity class as every single screen of our application where the user can interact with it. If we take into consideration the MVC pattern, we can assume the activity to be the controller, as it will receive all the user inputs and events from the views, and the layouts XML and UI widgets to be the views. To know more about the MVC pattern, visit https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller. So, we have just added one activity to our application; let's see what else the Android Studio wizard created for us. Running your project Android Studio project wizard not only created an empty Activity class for us, but it also created an AndroidManifest, a layout file (activity_main.xml) defining the view controlled by the Activity class, an application icon placed carefully into different mipmaps (https://en.wikipedia.org/wiki/Mipmap) so that the most appropriate will be used depending on the screen resolution, some Gradle scripts, and and some other .xml files containing colors, dimensions, strings, and style definitions. We can have multiple resources, and even repeated resources, depending on screen resolution, screen orientation, night mode, layout direction, or even mobile country code of the SIM card. Take a look at the next topic to understand how to add qualifiers and filters to resources. For the time being, let's just try to run this example by pressing the play button next to our build configuration named app at the top of the screen. Android Studio will show us a small window where we can select the deployment target: a real device or emulator where our application will be installed and launched. If we have not connected any device or created any emulator, we can do it from the following screen. Let's press the Create New Emulator button. From this new screen, we can easily select a device and create an emulator that looks like that device. A Nexus 5X will suit us. After choosing the device, we can choose which version of the Android OS and architecture on which the platform will run. For instance, if we want to select Android Marshmallow (API level 23), we can choose from armeabi-v7a, x86 (Intel processors) and x86_64 (Intel 64bit processors). As we previously installed HAXM during our first run (https://software.intel.com/en-us/android/articles/intel-hardware-accelerated-execution-manager), we should install an Intel image, so emulator will be a lot faster than having to emulate an ARM processor. If we do not have the Android OS image downloaded to our computer, we can do it from this screen as well. Note that you can have an image of the OS with Google APIs or without them. We will use one image or another depending on whether the application uses any of the Google-specific libraries (Google Play Services) or only the Android core libraries. Once the image is selected (and downloaded and installed, if needed), we can proceed to finish the Android Virtual Device (AVD) configuration. On the last configuration screen, we can fine-tune some elements of our emulator, such as the default orientation (portrait or landscape), the screen scale, the SD card(if we enable the advanced settings), the amount of physical RAM, network latency, and we can use the webcam in our computer as emulator's camera. You are now ready to run your application on the Android emulator that you just created. Just select it as the deployment target and wait for it to load and install the app. If everything goes as it should, you should see this screen on the Android emulator: If you want to use a real device instead of an emulator, make sure that your device has the developer options enabled and it is connected to your computer using a USB cable (to enable development mode on your device or get information on how to develop and debug applications over the network, instead of having the device connected through an USB cable; check out the following links: http://developer.android.com/tools/help/adb.html http://developer.android.com/tools/device.html) If these steps are performed correctly, your device will appear as a connected device on the deployment target selection window. Resource configuration qualifiers As we introduced in the previous section, we can have multiple resources depending on the screen resolution or any other device configuration, and Android will choose the most appropriate resource in runtime. In order to do that, we have to use what is called configuration qualifiers. These qualifiers are only strings appended to the resource folder. Consider the following example: drawable drawable-hdpi drawable-mdpi drawable-en-rUS-land layout layout-en layout-sw600dp layout-v7 Qualifiers can be combined, but they must always follow the order specified by Google in the Providing Resource documentation, available at http://developer.android.com/guide/topics/resources/providing-resources.html. This allows us, for instance, to target multiple resolutions and have the best experience for each of them. It can be also used to have different images based on the country, in which the application is executed, or language. We have to be aware that putting too many resources (basically, images or any other media) will make our application grow in size. It is always good to apply common sense. And, in the case of having too many different resources or configurations, do not bloat the application and produce different binaries that can be deployed selectively to different devices on Google Play. We will briefly explain on the Gradle build system topic in this article, how to produce different binaries from one single source code. It will add some complexity on our development but will make our application smaller and more convenient for end users. For more information on multiple APK support, visit http://developer.android.com/google/play/publishing/multiple-apks.html. Summary In this article, we covered how to install Android Studio and get started with it. We also introduced some of the most common elements on Android Studio by creating a sample project, building it and running it on an android emulator or on a real android device. %MCEPASTEBIN% Resources for Article: Further resources on this subject: Hacking Android Apps Using the Xposed Framework [article] Speeding up Gradle builds for Android [article] The Art of Android Development Using Android Studio [article]
Read more
  • 0
  • 0
  • 19276

article-image-introduction-nagios
Packt
04 Aug 2016
16 min read
Save for later

Introduction to Nagios

Packt
04 Aug 2016
16 min read
In this article by Wojciech Kocjan and Piotr Beltowski, the authors of the book Learning Nagios - Third Edition, we will look at the brief introduction of Nagios, the IT monitoring and management software for problem solving. (For more resources related to this topic, see here.) Imagine you're an administrator of a large IT infrastructure. You have just started receiving e-mails that a web application has suddenly stopped working. When you try to access the same page, it just does not load. What are the possibilities? Is it the router? Maybe the firewall? Perhaps the machine hosting the page is down? The server process has crashed? Before you even start thinking rationally about what to do, your boss calls about the critical situation and demands explanations. In all this panic, you'll probably start plugging everything in and out of the network, rebooting the machine… and it still doesn't help. After hours of nervous digging into the issue, you've finally found the root cause: although the web server was working properly, it continuously timed out during communication with the database server. This is because the machine with the database did not get an IP address assigned. Your organization requires all IP addresses to be configured using the DHCP procotol and the local DHCP server ran out of memory and killed several processes, including the dhcpd process responsible for assigning IP addresses. Imagine how much time it would take to determine all this manually! To make things worse, the database server could be located in another branch of the company or in a different time zone, and it could be the middle of the night over there. But what if you had Nagios up and running across your entire company? You would just go to the web interface and see that there are no problems with the web server and the machine on which it is running. There would also be a list of issues—the machine serving IP addresses to the entire company does not do its job and the database is down. If the setup also monitored the DHCP server, you'd get a warning e-mail that little swap memory is available or too many processes are running. Maybe it would even have an event handler for such cases to just kill or restart non critical processes. Also, Nagios would try to restart the dhcpd process over the network in case it is down. In the worst case, Nagios would reduce hours of investigation to ten minutes. Ideally, you would just get an e-mail that there was such a problem and another e-mail that it's already fixed. You would just disable a few services and increase the swap size for the DHCP machine and solve the problem permanently. Hopefully, it would be solved fast enough so that nobody would notice that there was a problem in the first place! Understanding the basics of Nagios Nagios is an open source tool for system monitoring. It means that it watches servers or other devices on your network and makes sure that they are working properly. Nagios constantly checks if other machines are working properly. It also verifies that various services on those machines are working properly. In addition, Nagios can accept information from other processes or machines regarding their status; for example, a web server can send information to Nagios if it is overloaded. The main purpose of system monitoring is to detect if any system is not working properly as soon as possible and notify the appropriate staff, and if possible, try to resolve the error—such as by restarting system services if needed. System monitoring in Nagios is split into two categories of objects, hosts and services: Hosts represent a physical or virtual device on your network (servers, routers, workstations, printers, and so on) Services are particular functionalities, for example, a Secure Shell (SSH) server (sshd process on the machine) can be defined as a service to be monitored Each service is associated with a host on which it is running. In addition, machines can be grouped into host groups, as shown here: A major benefit of Nagios' performance checks is that it only uses four distinct states—Ok, Warning, Critical, and Unknown. Another advantage is that it is based on the framework of plugins, allowing you to develop your own plugin; this means if you want to check something that's not yet possible to do, you just need to write a simple piece of code and that's it! The approach to offer only three states (excluding Unknown as it informs about not being able to determine the current state) allows administrators to ignore monitoring values and just decide on what the warning/critical limits are. This is a proven concept, and is far more efficient than monitoring graphs and analyzing trends. This is similar to traffic lights, where green indicates Ok and means a service is working correctly, a Warning state is same as the yellow light, and a Critical state is same as the red traffic light. For example, system administrators tend to ignore things such as gradually declining storage space. People often simply ignore early warnings until a critical process runs out of disk space. Having a strict limit to watch is much better, because you always catch a problem regardless of whether it turns from warning to critical in fifteen minutes or in a week. This approach is exactly what Nagios does. Each check performed by Nagios is turned from numerical values (such as the amount of disk space or CPU usage) to one of the three possible states. Another benefit is a clear report stating that X services are up and running, Y are in the warning state, and Z are currently critical, which is much more readable than a matrix of values. It saves you the time of analyzing what's working and what's failing. It can also help prioritize what needs to be handled first and which problems can be handled later. Introducing plugins Nagios performs all of its checks using plugins. These are external components for which Nagios passes information on what should be checked and what the warning and critical limits are. Plugins are responsible for performing the checks and analyzing the results. The output from such a check is the status (Ok, Warning, Critical, or Unknown) and additional text describing information on the service in detail. This text is mainly intended for system administrators to be able to read the detailed status of a service. Nagios comes with a set of standard plugins that allow performance checks for almost all the services your company might use or offer. Moreover, if you need to perform a specific check (for example, connect to a web service and invoke methods), it is very easy to write your own plugins. And that's not all—they can be written in any language and it takes less than 15 minutes to write a complete check command! Benefits of monitoring There are many reasons for you to ensure that all your resources are working as expected. If you're still not convinced after reading the introduction to this article, here are a few important points why it is important to monitor your infrastructure. The main reason is quality improvement. If your IT staff can notice failures quicker by using a monitoring tool, they will also be able to respond to them much faster. Sometimes it takes hours or days to get the first report of a failure even if many users bump into errors. Nagios ensures that if something is not working, you'll know about it. In some cases, event handling can even be done so that Nagios can switch to the backup solution until the primary process is fixed. A typical case would be to start a dial-up connection and use it as a primary connection in cases when the company VPN is down. Another reason is, much better problem determination. Very often what the users report as a failure is far from the root cause of the problem, such as when an e-mail system is down due to the LDAP service not working correctly. If you define dependencies between hosts correctly, then Nagios will point out that the IMAP or POP3 e-mail server is assumed to be not working because the LDAP service that it depends upon has a problem. Nagios will start checking the e-mail server as soon as the problem with LDAP has been resolved. Nagios is also very flexible when it comes to notifying people of what isn't functioning correctly. In most cases, your company has a large IT team or multiple teams. Usually, you want some people to handle servers and others to handle network switches/routers/modems. There might also be a team responsible for network printers, or a division is made based on geographical location. You can instruct Nagios about who is responsible for particular machines or groups of machines, so that when something is wrong, the right people will get to know about it. You can also use Nagios' web interface to manage who is working on which issue. Monitoring resources is not only useful for identifying problems, but it also saves you from having them, as Nagios handles warnings and critical situations differently. This means that it's possible to be aware of situations that may become problems really soon. For example, if your disk storage on an e-mail server is running out, it's better to be aware of this situation before it becomes a critical issue. Monitoring can also be set up on multiple machines across various locations. These machines will then communicate all their results to a central Nagios server so that information on all hosts and services in your system can be accessed from a single machine. This gives you a more accurate picture of your IT infrastructure as well as allows testing more complex systems such as firewalls. For example, it is vital that a testing environment is accessible from a production environment, but not the other way around. It is also possible to set up a Nagios server outside the company's intranet (for example, over a dedicated DSL) to make sure that traffic from the Internet is properly blocked. It can be used to check if only certain services are available, for example, to verify that only SSH and Hypertext Transfer Protocol (HTTP) are accessible from external IP addresses, and that services such as databases are inaccessible to users. Main features Nagios' main strength is flexibility—it can be configured to monitor your IT infrastructure in the way you want it. It also has a mechanism to react automatically to problems and has a powerful notification system. All of this is based on a clear object definition system, which in turn is based on a few types of object, as follows: Commands: These are definitions of how Nagios should perform particular types of check. They are an abstraction layer on top of actual plugins that allow you to group similar types of operation. Time periods: Date and time spans during which an operation should or should not be performed. For example, Monday–Friday, 09:00 A.M. – 5:00 P.M. Hosts and host groups: These are devices along with the possibility of group hosts. A single host might be a member of more than one group. Services: Various functionalities or resources to monitor on a specific host. For example, CPU usage, storage space, or web server. Contacts and contact groups: People that should be notified with information on how and when they should be contacted; contacts can be grouped, and a single contact might be a member of more than one group. Notifications: These define who should be notified of what, for example, all errors related to the linux-servers host group should go to the linux-admins contact group during working hours and to the critsit-team (critical situations handling team) contact group outside of working hours. Notifications are not strictly an object, but a combination of all the preceding objects and are an essential part of Nagios. Escalations: Extension to notifications; escalations define these after an object is in the same state for a specific period of time, other people should get notified of certain events. For example, a critical server being down for more than four hours should alert IT management so that they track the issue. A beneficial feature of using Nagios is that it is a mature dependency system. For any administrator, it is obvious that if your router is down then all machines accessed via it will fail. Some systems don't take that into account, and in such cases, you get a list of several failing machines and services. Nagios allows you to define dependencies between hosts to reflect actual network topology. For example, if a router that connects you to the rest of your network is down, Nagios will not perform checks for the subsequent parts and machines that are dependent on the router. This is illustrated in the following image: You can also define that a particular service depends on another service, either on the same host or a different host. If one of the dependent services is down, a check for a service is not even performed. For example, in order for your company's intranet application to function properly, both the underlying web and database servers must be running properly. So, if a database service is not working properly, Nagios will not perform checks and/or not send notifications that your application is not working, because the root cause of the problem is that the database is not working properly. The database server might be on the same or on a different host. If the database is not working properly or the dependent machine is down or not accessible, all services dependent on the database service will not be checked as well. Nagios offers a consistent system of macro definitions. These are variables that can be put into all object definitions and depend on the context. They can be put inside commands, and depending on the host, service, and many other parameters, macro definitions are substituted accordingly. For example, a command definition might use an IP address of the host it is currently checking in all remote tests. It also makes it possible to put information such as the previous and current status of a service in a notification e-mail. Nagios also offers various extensions to macro definitions, which makes it an even more powerful mechanism. Additionally, there is a built-in mechanism for scheduling planned downtimes. This is mainly used when maintenance of the IT infrastructure is to be carried out, and servers and/or services they provide are out of order for a period of time. You can let Nagios know that such an event will occur, and it will not send notifications about problems with hosts and/or services that have a scheduled downtime. In such cases, dependencies are also taken into consideration—if a database has a scheduled downtime, notifications for the services that depend on it will not be sent out. Nagios can also notify people of planned downtimes automatically. This allows creating an integrated process of scheduling downtimes that will also handle informing users. Soft and hard states Nagios works by checking if a particular host or service is working correctly and storing its status. Because the status is only one of four possible values, it is crucial that it precisely reflects the actual situation. In order to avoid detecting random and temporary failures, Nagios uses soft and hard states to describe what the current status is for a host or service. Imagine that an administrator is restarting a web server, which in turn makes web pages inaccessible for around five seconds. Since such restarts are usually done at night to lower the number of users affected, this is an acceptable period of time. However, a problem might be that Nagios will try to connect to the server and notice that it is actually down. If it only relies on a single result, Nagios could trigger an alert that a web server is down. It would actually be up and running again in a few seconds, but Nagios would require another couple of minutes to find that out. To handle situations where a service is down for a very short time, or the test has temporarily failed, soft states were introduced. When a previous status of a check is unknown or is different from the previous one, Nagios will retest the host or service a couple of times to make sure the change is permanent. Nagios assumes that the new result is a soft state. After additional tests have verified that the new state is permanent, it is considered a hard one. Each host and service check defines the number of retries to perform before assuming a change to be permanent. This allows more flexibility over how many failures should be treated as an actual problem instead of a temporary one. Setting the number of checks to 1 will cause all changes to be treated as hard instantly. The following figure is an illustration of soft and hard state changes, assuming that number of checks to be performed is set to 3: This feature is very useful for checks that should skip short outages of a service or use a protocol that might fail in case of extensive traffic—such as ICMP or UDP. Monitoring devices over SNMP is also an example of a check that can fail in cases where a single check fails; nevertheless, the check will eventually succeed during the second or third check. Summary In this article, you learned the basics of Nagios as a tool for performing system monitoring. It can be used to ensure that services are working correctly, problems are detected earlier, and appropriate people are aware when something's wrong. You learned the basic types of objects in Nagios—commands, hosts, services, time periods, contacts, as well as object grouping. You also found out about notifications and escalations, which can be used to notify administrators about problems. The article also introduced the concept of dependencies that helps in understanding the root cause of complex problems. Resources for Article: Further resources on this subject: Passive Checks and NSCA (Nagios Service Check Acceptor)[article] Notifications and Events in Nagios 3.0-part1 [article] Notifications and Events in Nagios 3.0- part2 [article]
Read more
  • 0
  • 0
  • 2643
article-image-saying-hello-java-ee
Packt
03 Aug 2016
27 min read
Save for later

Saying Hello to Java EE

Packt
03 Aug 2016
27 min read
To develop a scalable, distributed, well-presented, complex, and multi-layered enterprise application is complicated. The development becomes even worse if the developer is not well aware of the software development fundamentals. Instead of looking at a bigger scenario, if we cut it down into parts and later combine them, it becomes easy for understanding as well as for developing. Each technology has some basics which we cannot overlook. Rather, if we overlook them, it will be the biggest mistake; the same is applicable to Java EE. In this article by TejaswiniMandar Jog, author of the book Learning Modular Java Programming, we are going to explore the following: Java EE technologies Why servlet and JSP? Introduction to Spring MVC Creating a sample application through Spring MVC (For more resources related to this topic, see here.) The enterprise as an application To withstand the high, competitive, and daily increasing requirements, it's becoming more and more difficult nowadays to develop an enterprise application. The difficulty is due to more than one kind of service, requirement of application to be robust and should support concurrency, security, and many more. Along with these things, enterprise applications should provide an easy user interface but good look and feel for different users. In the last article, we discussed enterprise applications. The discussion was more over understanding the terminology or the aspect. Let's now discuss it in terms of development, and what developers look forward to: The very first thing even before starting the development is: what we are we developing and why? Yes, as a developer we need to understand the requirements or the expectations from the application. Developers have to develop an application which will meet the requirements. The application should be efficient and with high quality so as to sustain in the market. The application code should be reliable and bug-free to avoid runtime problems. No application is perfect; it's a continuous process to update it for new demands. Develop an application in such a way that it is easy to update. To meet high expectations, developers write code which becomes complicated to understand as well as to change. Each one of us wants to have a new and different product, different from what is on the market. To achieve this, designers make an over-clumsy design which is not easy to change in the future. Try to avoid over-complexity both in design and business logic. When development starts, developers look forward to providing a solution, but they have to give thought to what they are developing and how the code will be organized in terms of easy maintenance and future extension. Yes, we are thinking about modules which are doing a defined task and those which are less dependent. Try to write a module which will be loosely coupled and highly cohesive. Today we are using enterprise applications through different browsers, such as Internet Explorer, Mozilla, or Firefox. We are even using mobile browsers for the same task. This demands an application that has been developed to withstand the number of platforms and browsers. Going through all this discussion, many technologies come to mind. We will go through one such platform which covers the maximum of the above requirements: the Java Enterprise Edition (Java EE) platform. Let's dive in and explore it!! The Java EE platform Sun Microsystems released the Java EE platform in 2000, which was formerly known as the J2EE specification. It defines the standards for developing component-based enterprise applications easily. The concrete implementation is provided by application servers such as Weblogic and GlassFish, and servlet containers such as Tomcat. Today we have Java EE 8 on the market. Features of the Java EE platform The following are the various features of the Java EE platform: Platform independency: Different types of information which the user needs in day-to-day life is spread all over the network on a wide range of platforms. Java EE is well adapted to support, and use this widely spread multiple format information on different platforms easily. Modularity: The development of enterprise applications is complex and needs to be well organized. The complexity of the application can be reduced by dividing it into different, small modules which perform individual tasks, which allows for easy maintenance and testing. They can be organized in separate layers or tiers. These modules interact with each other to perform a business logic. Reusability: Enterprise applications need frequent updates to match up client requirements. Inheritance, the fundamental aspect of an object-oriented approach, offers reusability of the components with the help of functions. Java EE offers modularity which can be used individually whenever required. Scalability: To meet the demands of the growing market, the enterprise application should keep on providing new functionalities to the users. In order to provide these new functionalities, the developers have to change the application. They may add new modules or make changes in already existing ones. Java EE offers well-managed modules which make scalability easy. The technologies used in Java EE are as follows: Java servlet Java Server Pages Enterprise Java Bean Java Messaging API XML Java Transaction API Java Mail Web Services The world of dotcoms In the 1990s, many people started using computers for a number of reasons. For personal use, it was really good. When it came to enterprise use, it was helpful to speed up the work. But one main drawback was; how to share files, data or information? The computers were in a network but if someone wanted to access the data from any computer then they had to access it personally. Sometimes, they had to learn the programs on that computer, which is not only very time-consuming but also unending. What if we can use the existing network to share the data remotely?? It was a thought put forward by a British computer scientist, Sir Tim Berners-Lee. He thought of a way to share the data through the network by exploring an emerging technology called hypertext. In October 1990, Tim wrote three technologies to fulfill sharing using Hyper Text Markup Language (HTML), Uniform Resource Identifier (URI), and Hyper Text Transfer Protocol (HTTP): HTML is a computer language which is used in website creation. Hypertext facilitates clicking on a link to navigate on the Internet. Markups are HTML tags defining what to do with the text they contain. URIs defines a resource by location or name of resource, or both. URIs generally refer to a text document or images. HTTP is the set of rules for transferring the files on the Web. HTTP runs on the top of TCP/IP. He also wrote the first web page browser (World Wide Webapp) and the first web server (HTTP). The web server is where the application is hosted. This opened the doors to the new amazing world of the "dotcom". This was just the beginning and many more technologies have been added to make the Web more realistic. Using HTTP and HTML, people were able to browse files and get content from remote servers. A little bit of user interaction or dynamicity was only possible through JavaScript. People were using the Web but were not satisfied; they needed something more. Something which was able to generate output in a totally dynamic way, maybe displaying the data which had been obtained from the data store. Something which can manipulate user input and accordingly display the results on the browser. Java developed one technology: Common Gateway Interface (CGI). As CGI was a small Java program, it was capable of manipulating the data at the server side and producing the result. When any user made a request, the server forward the edit to CGI, which was an external program. We got an output but with two drawbacks: Each time the CGI script was called, a new process was created. As we were thinking of a huge number of hits to the server, the CGI became a performance hazard. Being an external script, CGI was not capable of taking advantage of server abilities. To add dynamic content which can overcome the above drawbacks and replace CGI, the servletwas developed by Sun in June 1997. Servlet – the dynamicity Servlets are Java programs that generate dynamic output which will be displayed in the browser and hosted on the server. These servers are normally called servlet containers or web servers. These containers are responsible for managing the lifecycle of the servlets and they can take advantage of the capabilities of servers. A single instance of a servlet handles multiple requests through multithreading. This enhances the performance of the application. Let's discuss servlets in depth to understand them better. The servlet is capable of handling the request (input) from the user and generates the response (output) in HTML dynamically. To create a servlet, we have to write a class which will be extended from GenericServlet or HttpServlet. These classes have service() as a method, to handle request and response. The server manages the lifecycle of a servlet as follows: The servlet will be loaded on arrival of a request by the servers. The instance will be created. The init() will be invoked to do the initialization. The preceding steps will be performed only once in the life cycle of the servlet unless the servlet has not been destroyed. After initialization, the thread will be created separately for each request by the server, and request and response objects will be passed to the servlet thread. The server will call the service() function. The service() function will generate a dynamic page and bind it to the HttpResponse object. Once the response is sent back to the user, the thread will be deallocated. From the preceding steps, it is pretty clear that the servlet is responsible for: Reading the user input Manipulating the received input Generating the response A good developer always keeps a rule of thumb in mind that a module should not have more than one responsibility, but here the servlet is doing much more. So this has addressed the first problem in testing the code, maybe we will find a solution for this. But the second issue is about response generation. We cannot neglect a very significant problem in writing well-designed code to have a nice look and feel for the page from the servlet. That means a programmer has to know or adapt designing skills as well, but, why should a servlet be responsible for presentation? The basic thought of taking presentation out of the servlet leads to Java Server Page (JSP). JSP solves the issue of using highly designed HTML pages. JSP provides the facility of using all HTML tags as well as writing logical code using Java. The designers can create well-designed pages using HTML, where programmers can add code using scriptlet, expression, declaration, or directives. Even standard actions like useBean can be used to take advantage of Java Beans. These JSP's now get transformed, compiled into the servlet by the servers. Now we have three components: Controller, which handles request and response Model, which holds data acquired from handling business logic View, which does the presentation Combining these three we have come across a design pattern—Model-View-Controller (MVC). Using MVC design patterns, we are trying to write modules which have a clear separation of work. These modules can be upgradable for future enhancement. These modules can be easily tested as they are less dependent on other modules. The discussion of MVC is incomplete without knowing two architectural flavors of it: MVC I architecture MVC II architecture MVC I architecture In this model, the web application development is page-centric around JSP pages. In MVC I, JSP performs the functionalities of handling a request and response and manipulating the input, as well as producing the output alone. In such web applications, we find a number of JSP pages, each one of them performing different functionalities. MVC I architecture is good for small web applications where less complexity and maintaining the flow is easy. The JSP performs the dual task of business logic and presentation together, which makes it unsuitable for enterprise applications. MVC I architecture MVC II architecture In MVC II, a more powerful model has been put forward to give a solution to enterprise applications with a clear separation of work. It comprises two components: one is the controller and other the view, as compared to MVC I where view and controller is JSP (view). The servlets are responsible for maintaining the flow (the controller) and JSP to present the data (the view). In MVC II, it's easy for developers to develop business logic- the modules which are reusable. MVC II is more flexible due to responsibility separation. MVC II architecture The practical aspect We have traveled a long way. So, instead of moving ahead, let's first develop a web application to accept data from the user and display that using MVC II architecture. We need to perform the following steps: Create a dynamic web application using the name Ch02_HelloJavaEE. Find the servlet-api.jar file from your tomcat/lib folder. Add servlet-api.jar to the lib folder. Create index.jsp containing the form which will accept data from the user. Create a servlet with the name HelloWorldServlet in the com.packt.ch02.servlets package. Declare the method doGet(HttpServletRequestreq,HttpServletResponsers) to perform the following task: Read the request data using the HttpServletRequest object. Set the MIME type. Get an object of PrintWriter. Perform the business logic. Bind the result to the session, application or request scope. Create the view with name hello.jsp under the folder jsps. Configure the servlet in deployment descriptor (DD) for the URL pattern. Use expression language or Java Tag Library to display the model in the JSP page. Let's develop the code. The filesystem for the project is shown in the following screenshot: We have created a web application and added the JARs. Let's now add index.jsp to accept the data from the user: <form action="HelloWorldServlet"> <tr> <td>NAME:</td> <td><input type="text" name="name"></td> </tr> <tr> <td></td> <td><input type="submit" value="ENTER"></td> </tr> </form> When the user submits the form, the request will be sent to the URL HelloWorldServlet. Let's create the HelloWorldServlet which will get invoked for the above URL, which will have doGet(). Create a model with the name message, which we will display in the view. It is time to forward the request with the help of the RequestDispatcher object. It will be done as follows: protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {     // TODO Auto-generated method stub     //read the request parameter     String name=request.getParameter("name");     //get the writer PrintWriter writer=response.getWriter();       //set the MIME type response.setContentType("text/html");       // create a model and set it to the scope of request request.setAttribute("message","Hello "+name +" From JAVA Enterprise"); RequestDispatcher dispatcher=request.getRequestDispatcher("jsps/hello.jsp"); dispatcher.forward(request, response);     } Now create the page hello.jsp under the folder jsps to display the model message as follows: <h2>${message }</h2> The final step is to configure the servlet which we just have created in DD. The configuration is made for the URL HelloWorldServlet as follows: <servlet> <servlet-name>HelloWorldServlet</servlet-name> <servlet-class>com.packt.ch02.servlets.HelloWorldServlet </servlet-class> </servlet> <servlet-mapping> <servlet-name>HelloWorldServlet</servlet-name> <url-pattern>/HelloWorldServlet</url-pattern></servlet-mapping> Let's deploy the application to check the output: Displaying the home page for a J2EE application The following screenshot shows the output when a name is entered by the user: Showing the output when a name is entered by the user After developing the above application, we now have a sound knowledge of how web development happens, how to manage the flow, and how navigation happens. We can observe one more thing: that whether it's searching data, adding data, or any other kind of operation, there are certain steps which are common, as follows: Reading the request data Binding this data to a domain object in terms of model data Sending the response We need to perform one or more of the above steps as per the business requirement. Obviously, by only performing the above steps, we will not be able to achieve the end effect but there is no alternative. Let's discuss an example. We want to manage our contact list. We want to have the facilities for adding a new contact, updating a contact, searching one or many contacts, and deleting a contact. The required data will be taken from the user by asking them to fill in a form. Then the data will be persisted in the database. Here, for example, we just want to insert the record in the database. We have to start the coding from reading request data, binding it to an object and then our business operation. The programmers have to unnecessarily repeat these steps. Can't they get rid of them? Is it possible to automate this process?? This is the perfect time to discuss frameworks. What is a framework? A framework is software which gives generalized solutions to common tasks which occur in application development. It provides a platform which can be used by the developers to build up their application elegantly. Advantages of frameworks The advantages of using frameworks are as follows: Faster development Easy binding of request data to a domain object Predefined solutions Validations framework In December 1996, Sun Microsystems published a specification for JavaBean. This specification was about the rules, using which developers can develop reusable, less complex Java components. These POJO classes are now going to be used as a basis for developing a lightweight, less complex, flexible framework: the Spring framework. This framework is from the thoughts of Rod Johnson in February 2003. The Spring framework consists of seven modules: Spring modules Though Spring consists of several modules, the developer doesn't have to be always dependent on the framework. They can use any module as per the requirement. It's not even compulsory to develop the code which has been dependent upon Spring API. It is called a non-intrusive framework. Spring works on the basis of dependency injection (DI), which makes it easy for integration. Each class which the developer develops has some dependencies. Take the example of JDBC: to obtain a connection, the developer needs to provide URL, username, and password values. Obtaining the connection is dependent on these values so we can call them dependencies, and injection of these dependencies in objects is called DI. This makes the emerging spring framework the top choice for the middle tier or business tier in enterprise applications. Spring MVC The spring MVC module is a choice when we look forward for developing web applications. The spring MVC helps to simplify development to develop a robust application. This module can also be used to leave common concerns such as reading request data, data binding to domain object, server-side validation and page rendering to the framework and will concentrate on business logic processes. That's what, as a developer we were looking for. The spring MVC can be integrated with technologies such as Velocity, Freemarker, Excel, and PDF. They can even take advantage of other services such as aspect-oriented programming for cross-cutting technologies, transaction management, and security provided by the framework. The components Let's first try to understand the flow of normal web applications in view of the Spring framework so that it will be easy to discuss the component and all other details: On hitting the URL, the web page will be displayed in the browser. The user will fill in the form and submit it. The front controller intercepts the request. The front controller tries to find the Spring MVC controller and pass the request to it. Business logic will be executed and the generated result is bound to the ModelAndView. The ModelAndView will be sent back to the front controller. The front controller, with the help of ViewResolver, will discover the view, bind the data and send it to the browser. Spring MVC The front controller As already seen in servlet JSP to maintain each flow of the application the developer will develop the servlet and data model from servlet will be forwarded to JSP using attributes. There is no single servlet to maintain the application flow completely. This drawback has been overcome in Spring MVC as it depends on the front controller design pattern. In the front controller design pattern, there will be a single entry point to the application. Whatever URLs are hit by the client, it will be handled by a single piece of the code and then it will delegate the request to the other objects in the application. In Spring MVC, the DispatcherServlet acts as front controller. DispatcherServlet takes the decision about which Spring MVC controller the request will be delegated to. In the case of a single Spring MVC controller in the application, the decision is quite easy. But we know in enterprise applications, there are going to be multiple Spring MVC controllers. Here, the front controller needs help to find the correct Spring MVC controller. The helping hand is the configuration file, where the information to discover the Spring MVC controller is configured using handler mapping. Once the Spring MVC controller is found, the front controller will delegate the request to it. Spring MVC controller All processes, such as the actual business logic, decision making or manipulation of data, happen in the Spring MVC controller. Once this module completes the operation, it will send the view and the model encapsulated in the object normally in the form of ModelAndView to the front controller. The front controller will further resolve the location of the view. The module which helps front controller to obtain the view information is ViewResolver. ModelAndView The object which holds information about the model and view is called as ModelAndView. The model represents the piece of information used by the view for display in the browser of different formats. ViewResolver The Spring MVC controller returns ModelAndView to the front controller. The ViewResolver interface helps to map the logical view name to the actual view. In web applications, data can be displayed in a number of formats, from as simple as JSP to complicated formats like JasperReport. Spring provides InternalResourceViewResolver, JspViewResolver, JasperReportsViewResolver, VelocityLayoutViewResolver, and so on, to support different view formats. The configuration file DispatcherServlet needs to discover information about the Spring MVC controller, ViewResolver, and many more. All this information is centrally configured in a file named XXX-servlet.xml where XXX is the name of the front controller. Sometimes the beans will be distributed across multiple configuration files. In this case, extra configuration has to be made, which we will see later in this article. The basic configuration file will be: <beans xsi_schemaLocation="    http://www.springframework.org/schema/beans        http://www.springframework.org/schema/beans/spring-beans-3.0.xsd    http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.0.xsd"> <!—mapping of the controller --> <!—bean to be configured here for view resolver  - -> </beans> The controller configuration file will be named name_of_servlet-servlet.xml. In our project, we will name this HelloWeb-servlet.xml. Let's do the basics of a web application using Spring MVC to accept the data and display it. We need to perform the following steps: Create a web application named Ch02_HelloWorld. Add the required JAR files for Spring (as shown in the following screenshot) and servlets in the lib folder. Create an index page from where the data can be collected from the user and a request sent to the controller. Configure the front controller in DD. Create a SpringMVCcontroller as HelloWorldController. Add a method for accepting requests in the controller which performs business logic, and sends the view name and model name along with its value to the front controller. Create an XML file in WEB-INF as Front_Controller_name-servlet.xml and configure SpringMVCcontroller and ViewResolver. Create a JSP which acts as a view to display the data with the help of Expression Language (EL) and Java Standard Tag Library (JSTL). Let's create the application. The filesystem for the project is as follows: We have already created the dynamic web project Ch02_HelloSpring and added the required JAR files in lib folder. Let's start by creating index.jsp page as: <form action="hello.htm"> <tr> <td>NAME:</td> <td><input type="text" name="name"></td> </tr> <tr> <td></td> <td><input type="submit" value="ENTER"></td> </tr> </form>   When we submit the form, the request is sent to the resource which is mapped for the URL hello.htm. Spring MVC follows the front controller design pattern. So all the requests hitting the application will be first attended by the front controller and then it will send it to the respective Spring controllers. The front controller is mapped in DD as: <servlet> <servlet-name>HelloSpring</servlet-name> <servlet-class>org.springframework.web.servlet.DispatcherServlet </servlet-class> </servlet> <servlet-mapping> <servlet-name>HelloSpring</servlet-name> <url-pattern>*.htm</url-pattern> </servlet-mapping> Now the controller needs help to find the Spring MVC controller. This will be taken care of by the configuration file. This file will have the name XXX-servlet.xml where XXX is replaced by the name of the front controller from DD. Here, in this case HelloSpring-servlet.xml will have the configuration. This file we need to keep in the WEB-INF folder. In the Configuration files section, we saw the structure of the file. In this file, the mapping will be done to find out how the package in which the controllers are kept will be configured. This is done as follows: <context:component-scan base-package="com.packt.ch02.controllers" /> Now the front controller will find the controller from the package specified as a value of base-package attribute. The front controller will now visit HelloController. This class has to be annotated by @Controller: @Controller public class HelloController { //code here } Once the front controller knows what the controller class is, the task of finding the appropriate method starts. This will be done by matching the values of @RequestMapping annotation applied either on the class or on the methods present in the class. In our case, the URL mapping is hello.htm. So the method will be developed as: @RequestMapping(value="/hello.htm") publicModelAndViewsayHello(HttpServletRequest request)   {     String name=request.getParameter("name"); ModelAndView mv=new ModelAndView(); mv.setViewName("hello");     String message="Hello "+name +" From Spring"; mv.addObject("message",message); return mv;   } This method will return a ModelAndView object which contains a view name, model name and value for the model. In our code the view name is hello and the model is presented by message. The Front Controller now again uses HelloSpring-servlet.xml for finding the ViewResolver to get the actual name and location of the view. ViewResolver will provide the directory name (location) where the view is placed with a property prefix. The format of the view is given by the property suffix. Using the view name, prefix and suffix, the front controller gets the page. The ViewResolver will bind the model to be used in the view: <bean id="viewResolver" class="org.springframework.web.servlet.view.InternalResourceViewResolver"> <property name="prefix" value="/WEB-INF/jsps/" /> <property name="suffix" value=".jsp" /> </bean> In our case, it will be /WEB-INF/jsps/ as prefix, hello as the name of page, and .jsp is the suffix value. Combining them, we will get /WEB-INF/jsps/hello.jsp, which acts as our view. The Actual view is written as prefix+view_name from ModelAndView+suffix, for instance: /WEB-INF/jsps/+hello+.jsp The data is bounded by the front controller and the view will be able to use it: <h2>${message}</h2>. This page is now ready to be rendered by the browser, which will give output in the browser as: Displaying the home page for a Spring application Entering the name in the text field (for example, Bob) and submitting the form gives the following output: Showing an output when a name is entered by the user Now we understand the working of spring MVC, let's discuss a few more things required in order to develop the Spring MVC controller. Each class which we want to discover as the controller should be annotated with the @Controller annotation. In this class, there may be number of methods which can be invoked on request. The method which we want to map for URL has to be annotated with the annotation @RequestMapping. There can be more than one method mapped for the same URL but it will be invoked for different HTTP methods. This can be done as follows: @RequestMapping(value="/hello.htm",method= RequestMethod.GET) publicModelAndViewsayHello(HttpServletRequest request)   {       } @RequestMapping(value="/hello.htm",method= RequestMethod.POST) publicModelAndViewsayHello(HttpServletRequest request)   {      } These methods normally accept Request as parameter and will return ModelAndView. But the following return types and parameters are also supported. The following are some of the supported method argument types: HttpServletRequest HttpSession Java.util.Map/ org.springframework.ui.Model/ org.springframework.ui.ModelMap @PathVariable @RequestParam org.springframework.validation.Errors/ org.springframework.validation.BindingResult The following are some of the supported method return types: ModelAndView Model Map View String void Sometimes the bean configuration is scattered in more than one file. For example, we can have controller configuration in one file and database, security-related configuration in a separate file. In that case, we have to add extra configuration in DD to load multiple configuration files, as follows: <servlet> <servlet-name>HelloSpring</servlet-name> <servlet-class>org.springframework.web.servlet.DispatcherServlet </servlet-class> <init-param> <param-name>contextConfigLocation</param-name> <param-value>/WEB-INF/beans.xml</param-value> </init-param> </servlet> <servlet-mapping> <servlet-name>HelloSpring</servlet-name> <url-pattern>*.htm</url-pattern> </servlet-mapping> Summary In this article, we learned how the application will be single-handedly controlled by the front controller, the dispatcher servlet. The actual work of performing business logic and giving the name of the view, data model back will be done by the Spring MVC controller. The view will be resolved by the front controller with the help of the respective ViewResolver. The view will display the data got from the Spring MVC controller. To understand and explore Spring MVC, we need to understand the web layer, business logic layer and data layer in depth using the basics of Spring MVC discussed in this article. Resources for Article:   Further resources on this subject: Working with Spring Tag Libraries [article] Working with JIRA [article] A capability model for microservices [article]
Read more
  • 0
  • 0
  • 2232

article-image-showing-cached-content-first-then-networks
Packt
03 Aug 2016
9 min read
Save for later

Showing cached content first then networks

Packt
03 Aug 2016
9 min read
In this article by Sean Amarasinghe, the author of the book, Service Worker Development Cookbook, we are going to look at the methods that enable us to control cached content by creating a performance art event viewer web app. If you are a regular visitor to a certain website, chances are that you may be loading most of the resources, like CSS and JavaScript files, from your cache, rather than from the server itself. This saves us necessary bandwidth for the server, as well as requests over the network. Having the control over which content we deliver from the cache and server is a great advantage. Server workers provide us with this powerful feature by having programmatic control over the content. (For more resources related to this topic, see here.) Getting ready To get started with service workers, you will need to have the service worker experiment feature turned on in your browser settings. Service workers only run across HTTPS. How to do it... Follow these instructions to set up your file structure. Alternatively, you can download the files from the following location: https://github.com/szaranger/szaranger.github.io/tree/master/service-workers/03/02/ First, we must create an index.html file as follows: <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <title>Cache First, then Network</title> <link rel="stylesheet" href="style.css"> </head> <body> <section id="events"> <h1><span class="nyc">NYC</span> Events TONIGHT</h1> <aside> <img src="hypecal.png" /> <h2>Source</h2> <section> <h3>Network</h3> <input type="checkbox" name="network" id="network- disabled-checkbox"> <label for="network">Disabled</label><br /> <h3>Cache</h3> <input type="checkbox" name="cache" id="cache- disabled-checkbox"> <label for="cache">Disabled</label><br /> </section> <h2>Delay</h2> <section> <h3>Network</h3> <input type="text" name="network-delay" id="network-delay" value="400" /> ms <h3>Cache</h3> <input type="text" name="cache-delay" id="cache- delay" value="1000" /> ms </section> <input type="button" id="fetch-btn" value="FETCH" /> </aside> <section class="data connection"> <table> <tr> <td><strong>Network</strong></td> <td><output id='network-status'></output></td> </tr> <tr> <td><strong>Cache</strong></td> <td><output id='cache-status'></output><td> </tr> </table> </section> <section class="data detail"> <output id="data"></output> </section> <script src="index.js"></script> </body> </html> Create a CSS file called style.css in the same folder as the index.html file. You can find the source code in the following location on GitHub: https://github.com/szaranger/szaranger.github.io/blob/master/service-workers/03/02/style.css Create a JavaScript file called index.js in the same folder as the index.html file. You can find the source code in the following location on GitHub: https://github.com/szaranger/szaranger.github.io/blob/master/service-workers/03/02/index.js Open up a browser and go to index.html. First we are requesting data from the network with the cache enabled. Click on the Fetch button. If you click fetch again, the data has been retrieved first from cache, and then from the network, so you see duplicate data. (See the last line is same as the first.) Now we are going to select the Disabled checkbox under the Network label, and click the Fetch button again, in order to fetch data only from the cache. Select the Disabled checkbox under the Network label, as well as the Cache label, and click the Fetch button again. How it works... In the index.js file, we are setting a page specific name for the cache, as the caches are per origin based, and no other page should use the same cache name: var CACHE_NAME = 'cache-and-then-network'; If you inspect the Resources tab of the development tools, you will find the cache inside the Cache Storage tab. If we have already fetched network data, we don't want the cache fetch to complete and overwrite the data that we just got from the network. We use the networkDataReceived flag to let the cache fetch callbacks to know if a network fetch has already completed: var networkDataReceived = false; We are storing elapsed time for both network and cache in two variables: var networkFetchStartTime; var cacheFetchStartTime; The source URL for example is pointing to a file location in GitHub via RawGit: var SOURCE_URL = 'https://cdn.rawgit.com/szaranger/ szaranger.github.io/master/service-workers/03/02/events'; If you want to set up your own source URL, you can easily do so by creating a gist, or a repository, in GitHub, and creating a file with your data in JSON format (you don't need the .json extension). Once you've done that, copy the URL of the file, head over to https://rawgit.com, and paste the link there to obtain another link with content type header as shown in the following screenshot: Between the time we press the Fetch button, and the completion of receiving data, we have to make sure the user doesn't change the criteria for search, or press the Fetch button again. To handle this situation, we disable the controls: function clear() { outlet.textContent = ''; cacheStatus.textContent = ''; networkStatus.textContent = ''; networkDataReceived = false; } function disableEdit(enable) { fetchButton.disabled = enable; cacheDelayText.disabled = enable; cacheDisabledCheckbox.disabled = enable; networkDelayText.disabled = enable; networkDisabledCheckbox.disabled = enable; if(!enable) { clear(); } } The returned data will be rendered to the screen in rows: function displayEvents(events) { events.forEach(function(event) { var tickets = event.ticket ? '<a href="' + event.ticket + '" class="tickets">Tickets</a>' : ''; outlet.innerHTML = outlet.innerHTML + '<article>' + '<span class="date">' + formatDate(event.date) + '</span>' + ' <span class="title">' + event.title + '</span>' + ' <span class="venue"> - ' + event.venue + '</span> ' + tickets + '</article>'; }); } Each item of the events array will be printed to the screen as rows. The function handleFetchComplete is the callback for both the cache and the network. If the disabled checkbox is checked, we are simulating a network error by throwing an error: var shouldNetworkError = networkDisabledCheckbox.checked, cloned; if (shouldNetworkError) { throw new Error('Network error'); } Because of the reason that request bodies can only be read once, we have to clone the response: cloned = response.clone(); We place the cloned response in the cache using cache.put as a key value pair. This helps subsequent cache fetches to find this update data: caches.open(CACHE_NAME).then(function(cache) { cache.put(SOURCE_URL, cloned); // cache.put(URL, response) }); Now we read the response in JSON format. Also, we make sure that any in-flight cache requests will not be overwritten by the data we have just received, using the networkDataReceived flag: response.json().then(function(data) { displayEvents(data); networkDataReceived = true; }); To prevent overwriting the data we received from the network, we make sure only to update the page in case the network request has not yet returned: result.json().then(function(data) { if (!networkDataReceived) { displayEvents(data); } }); When the user presses the fetch button, they make nearly simultaneous requests of the network and the cache for data. This happens on a page load in a real world application, instead of being the result of a user action: fetchButton.addEventListener('click', function handleClick() { ... } We start by disabling any user input while the network fetch requests are initiated: disableEdit(true); networkStatus.textContent = 'Fetching events...'; networkFetchStartTime = Date.now(); We request data with the fetch API, with a cache busting URL, as well as a no-cache option in order to support Firefox, which hasn't implemented the caching options yet: networkFetch = fetch(SOURCE_URL + '?cacheBuster=' + now, { mode: 'cors', cache: 'no-cache', headers: headers }) In order to simulate network delays, we wait before calling the network fetch callback. In situations where the callback errors out, we have to make sure that we reject the promise we received from the original fetch: return new Promise(function(resolve, reject) { setTimeout(function() { try { handleFetchComplete(response); resolve(); } catch (err) { reject(err); } }, networkDelay); }); To simulate cache delays, we wait before calling the cache fetch callback. If the callback errors out, we make sure that we reject the promise we got from the original call to match: return new Promise(function(resolve, reject) { setTimeout(function() { try { handleCacheFetchComplete(response); resolve(); } catch (err) { reject(err); } }, cacheDelay); }); The formatDate function is a helper function for us to convert the date format we receive in the response into a much more readable format on the screen: function formatDate(date) { var d = new Date(date), month = (d.getMonth() + 1).toString(), day = d.getDate().toString(), year = d.getFullYear(); if (month.length < 2) month = '0' + month; if (day.length < 2) day = '0' + day; return [month, day, year].join('-'); } If you consider a different date format, you can shuffle the position of the array in the return statement to your preferred format. Summary In this article, we have learned how to control cached content by creating a performance art event viewer web app. Resources for Article: Further resources on this subject: AngularJS Web Application Development Cookbook [Article] Being Offline [Article] Consuming Web Services using Microsoft Dynamics AX [Article]
Read more
  • 0
  • 0
  • 8853

article-image-animations-sprites
Packt
03 Aug 2016
10 min read
Save for later

Animations Sprites

Packt
03 Aug 2016
10 min read
In this article by, Abdelrahman Saher and Francesco Sapio, from the book, Unity 5.x 2D Game Development Blueprints, we will learn how to create and play animations for the player character to see as Unity controls the player and other elements in the game. The following is what we will go through: (For more resources related to this topic, see here.) Animating sprites Integrating animations into animators Continuing our platform game Animating sprites Creating and using animation for sprites is a bit easier than other parts of the development stage. By using animations and tools to animate our game, we have the ability to breathe some life into it. Let's start by creating a running animation for our player. There are two ways of creating animations in Unity: automatic clip creation and manual clip creation. Automatic clip creation This is the recommended method for creating 2D animations. Here, Unity is able to create the entire animation for you with a single-click. If you navigate in the Project Panel to Platformer Pack | Player | p1_walk, you can find an animation sheet as a single file p1_walk.png and a folder of a PNG image for each frame of the animation. We will use the latter. The reason for this is because the single sprite sheet will not work perfectly as it is not optimized for Unity. In the Project Panel, create a new folder and rename it to Animations. Then, select all the PNG images in Platformer Pack | Player | p1_walk | PNG and drop them in the Hierarchy Panel: A new window will appear that will give us the possibility to save them as a new animation in a folder that we chose. Let's save the animation in our new folder titled Animations as WalkAnim: After saving the animation, look in the Project Panel next to the animation file. Now, there is another asset with the name of one of the dropped sprites. This is an Animator Controller and, as the name suggests, it is used to control the animation. Let's rename it to PlayerAnimator so that we can distinguish it later on. In the Hierarchy panel, a game object has been automatically created with the original name of our controller. If we select it, the Inspector should look like the following: You can always add an Animator component to a game object by clicking on Add Component | Miscellaneous | Animator. As you can see, below the Sprite Renderer component there is an Animator component. This component will control the animation for the player and is usually accessed through a custom script to change the animations. For now, drag and drop the new controller PlayerAnimator on to our Player object. Manual clip creation Now, we also need a jump animation for our character. However, since we only have one sprite for the player jumping, we will manually create the animation clip for it. To achieve this, select the Player object in the Hierarchy panel and open the Animation window from Window | Animation. The Animation window will appear, as shown in the screenshot below: As you can see, our animation WalkAnim is already selected. To create a new animation clip, click on where the text WalkAnim is. As a result, a dropdown menu appears and here you can select Create New Clip. Save the new animation in the Animations folder as JumpAnim. On the right, you can find the animation timeline. Select from the Project Panel the folder Platformer Pack/Player. Drag and drop the sprite p1_jump on the timeline. You can see that the timeline for the animation has changed. In fact, now it contains the jumping animation, even if it is made out of only one sprite. Finally, save what we have done so far. The Animation window's features are best used to make fine tunes for the animation or even merging two or more animations into one. Now the Animations folder should look like this in the Project panel: By selecting the WalkAnim file, you will be able to see the Preview panel, which is collocated at the bottom of the Inspector when an object that may contain animation is selected. To test the animation, drag the Player object and drop it in the Preview panel and hit play: In the Preview panel, you can check out your animations without having to test them directly from code. In addition, you can easily select the desired animation and then drag the animation into a game object with the corresponding Animator Controller and dropping it in the Preview panel. The Animator In order to display an animation on a game object, you will be using both Animator Components and Animator Controllers. These two work hand in hand to control the animation of any animated object that you might have, and are described below: Animator Controller uses a state-machine to manage the animation states and the transitions between one another, almost like a flow chart of animations. Animator Component uses an Animator Controller to define which animation clips to use and applies them on the game object when needed. It also controls the blending and the transitions between them. Let's start modifying our controller to make it right for our character animations. Click on the Player and then open the Animator window from Window | Animator. We should see something like this: This is a state-machine, although it is automatically generated. To move around the grid, hold the middle mouse button and drag around. First, let's understand how all the different kinds of nodes work: Entry node (marked green): It is used when transitioning into a state machine, provided the required conditions were met. Exit node (marked red): It is used to exit a state machine when the conditions have been changed or completed. By default, it is not present, as there isn't one in the previous image. Default node (marked orange): It is the default state of the Animator and is automatically transitioned to from the entry node. Sub-state nodes (marked grey): They are also called custom nodes. They are used typically to represent a state for an object where an event will occur (in our case, an animation will be played). Transitions (arrows): They allow state machines to switch between one another by setting the conditions that will be used by Animator to decide which state will be activated. To keep things organized, let's reorder the nodes in the grid. Drag the three sub-states just right under the Entry node. Order them from left to right WalkAnim, New Animation, and JumpAnim. Then, right-click on New Animation and choose Set as Layer Default State. Now, our Animator window should look like the following: To edit a node, we need to select it and modify it as needed in the Inspector. So, select New Animation and the Inspector should be like the screenshot below: Here, we can have access to all the properties of the state or node New Animation. Let's change its name to Idle. Next, we need to change the speed of the state machine, which controls how fast the animation will be played. Next, we have Motion which refers to the animation that will be used for this state. After we have changed the name, save the scene, and this is what everything should look like now: We can test what we have done so far, by hitting play. As we can see in the Game view, the character is not animated. This is because the character is always in the Idle state and there are no transitions to let him change state. While the game is in runtime, we can see in the Animator window that the Idle state is running. Stop the game, and right-click on the WalkAnim node in the Animator window. Select from the menu Set as Layer Default State. As a result, the walking animation will be played automatically at the beginning of the game. If we press the play button again, we can notice that the walk animation is played, as shown in the screenshot below: You can experiment with the other states of the Animator. For example, you can try to set JumpAnim as the default animation or even tweak the speed of each state to see how they will be affected. Now that we know the basics of how the Animator works, let's stop the playback and revert the default state to the Idle state. To be able to connect our states together, we need to create transitions. To achieve this, right-click on the Idle state and select Make Transition which turns the mouse cursor into an arrow. By clicking on other states, we can connect them with a transition. In our case, click on the WalkAnim state to make a transition from the Idle state to the WalkAnim state. The animator window should look like the following: If we click on the arrow, we can have access to its properties in the Inspector, as shown in the following screenshot: The main properties that we might want to change are: Name (optional): We can assign a name to the transition. This is useful to keep everything organized and easy to access. In this case, let's name this transition Start Walking. Has Exit Time: Whether or not the animation should be played to the end before exiting its state when the conditions are not being met anymore. Conditions: The conditions that should be met so that the transition takes place. Let's try adding a condition and see what happens: When we try to create a condition for our transition, the following message appears next to Parameter does not exist in Controller which means that we need to add parameters that will be used for our condition. To create a parameter, switch to Parameters in the top left of the Animator window and add a new float using the + button and name it PlayerSpeed, as shown in the following screenshot: Any parameters that are created in the Animator are usually changed from code and those changes affect the state of animation. In the following screenshot, we can see the PlayerSpeed parameter on the left side: Now that we have created a parameter, let's head back to the transition. Click the drop down button next to the condition we created earlier and choose the parameter PlayerSpeed. After choosing the parameter, another option appears next to it. You can either choose Greater or Less, which means that the transition will happen when this parameter is respectively less than X or greater than X. Don't worry, as that X will be changed by our code later on. For now, choose Greater and set the value to 1, which means that when the player speed is more than one, the walk animation starts playing. You can test what we have done so far and change the PlayerSpeed parameter in runtime. Summary This wraps up everything that we will cover in this article. So far, we have added animations to our character to be played according to the player controls. Resources for Article: Further resources on this subject: Animations in Cocos2d-x [Article] Adding Animations [Article] Bringing Your Game to Life with AI and Animations [Article]
Read more
  • 0
  • 0
  • 27850
article-image-responsive-web-design
Packt
03 Aug 2016
32 min read
Save for later

What is Responsive Web Design

Packt
03 Aug 2016
32 min read
In this article by Alex Libby, Gaurav Gupta, and Asoj Talesra, the authors of the book, Responsive Web Design with HTML5 and CSS3 Essentials we will cover the basic elements of responsive web design (RWD). Getting started with Responsive Web Design If one had to describe Responsive Web Design in a sentence, then responsive design describes how the content is displayed across various screens and devices, such as mobiles, tablets, phablets or desktops. To understand what this means, let's use water as an example. The property of water is that it takes the shape of the container in which it is poured. It is an approach in which a website or a webpage adjusts the layout according to the size or resolution of the screen dynamically. This ensures that the users get the best experience while using the website. We develop a single website that uses a single code base. This will contain fluid, flexible images, proportion-based grids, fluid images or videos and CSS3 media queries to work across multiple devices and device resolutions—the key to making them work is the use of percentage values in place of fixed units, such as pixels or ems-based sizes. The best part of this is that we can use this technique without the knowledge or need of server based/backend solutions—to see it in action, we can use Packt's website as an example. Go ahead and browse to https://www.packtpub.com/web-development/mastering-html5-forms; this is what we will see as a desktop view: The mobile view for the same website shows this if viewed on a smaller device: We can clearly see the same core content is being displayed (that is, an image of the book, the buy button, pricing details and information about the book), but element such as the menu have been transformed into a single drop down located in the top left corner. This is what responsive web design is all about—producing a flexible design that adapts according to which device we choose to use in a format that suits the device being used. Understanding the elements of RWD Now that we've been introduced to RWD, it's important to understand some of the elements that make up the philosophy of what we know as flexible design. A key part of this is understanding the viewport or visible screen estate available to us—in addition, there are several key elements that make up RWD. There are several key elements involved—in addition to viewports, these center around viewports, flexible media, responsive text and grids, and media queries. We will cover each in more detail later in the book, but for now, let's have a quick overview of the elements that make up RWD. Controlling the viewport A key part of RWD is working with the viewport, or visible content area on a device. If we're working with desktops, then it is usually the resolution; this is not the case for mobile devices. There is a temptation to reach for JavaScript (or a library, such as jQuery) to set values, such as viewport width or height: there is no need, as we can do this using CSS: <meta name="viewport" content="width=device-width"> Or by using this directive: <meta name="viewport" content="width=device-width, initial-scale=1"> This means that the browser should render the width of the page to the same width as the browser window—if, for example, the latter is 480px, then the width of the page will be 480px. To see what a difference not setting a viewport can have, take a look at this example screenshot: This example was created from displaying some text in Chrome, in iPhone 6 Plus emulation mode, but without a viewport. Now, let's take a look at the same text, but this time with a viewport directive set: Even though this is a simple example, do you notice any difference? Yes, the title color has changed, but more importantly the width of our display has increased. This is all part of setting a viewport—browsers frequently assume we want to view content as if we're on a desktop PC. If we don't tell it that the viewport area has been shrunken in size, it will try to shoe horn all of the content into a smaller size, which doesn't work very well! It's critical therefore that we set the right viewport for our design and that we allow it to scale up or down in size, irrespective of the device—we will explore this in more detail. Creating flexible grids When designing responsive websites, we can either create our own layout or use a grid system already created for use, such as Bootstrap. The key here though is ensuring that the mechanics of our layout sizes and spacing are set according to the content we want to display for our users, and that when the browser is resized in width, it realigns itself correctly. For many developers, the standard unit of measure has been pixel values; a key part of responsive design is to make the switch to using percentage and em (or preferably rem) units. The latter scale better than standard pixels, although there is a certain leap of faith needed to get accustomed to working with the replacements! Making media responsive A key part of our layout is, of course, images and text—the former though can give designers a bit of a headache, as it is not enough to simply use large images and set overflow: hidden to hide the parts that are not visible! Images in a responsive website must be as flexible as the grid used to host them—for some, this may be a big issue if the website is very content-heavy; now is a good time to consider if some of that content is no longer needed, and can be removed from the website. We can, of course simply apply display: none to any image which shouldn't be displayed, according to the viewport set. This isn't a good idea though, as content still has to be downloaded before styles can be applied; it means we're downloading more than is necessary! Instead, we should assess the level of content, make sure it is fully optimized, and apply percentage values so it can be resized automatically to a suitable size when the browser viewport changes. Constructing suitable breakpoints With content and media in place, we must turn our attention to media queries—there is a temptation to create queries that suit specific devices, but this can become a maintenance headache. We can avoid the headache by designing queries based on where the content breaks, rather than for specific devices—the trick to this is to start small and gradually enhance the experience, with the use of media queries: <link rel="stylesheet" media="(max-device-width: 320px)" href="mobile.css" /> <link rel="stylesheet" media="(min-width: 1600px)" href="widescreen.css" /> We should aim for around 75 characters per line, to maintain an optimal length for our content. Introducing flexible grid layouts For many years, designers have built layouts of different types—they may be as simple as a calling card website, right through to a theme for a content management system, such as WordPress or Joomla. The meteoric rise of accessing the Internet through different devices means that we can no longer create layouts that are tied to specific devices or sizes—we must be flexible! To achieve this flexibility requires us to embrace a number of changes in our design process – the first being the type of layout we should create. A key part of this is the use of percentage values to define our layouts; rather than create something from ground up, we can make use of a predefined grid system that has been tried and tested, as a basis for future designs. The irony is that there are lots of grid systems vying for our attention, so without further ado, let's make a start by exploring the different types of layouts, and how they compare to responsive designs. Understanding the different layout types A problem that has been faced by web designers for some years is the type of layout their website should use—should it be fluid, fixed width, have the benefits of being elastic or a hybrid version that draws on the benefits of a mix of these layouts? The type of layout we choose use will of course depend on client requirements—making it a fluid layout means we are effectively one step closer to making it responsive: the difference being that the latter uses media queries to allow resizing of content for different devices, not just normal desktops! To understand the differences, and how responsive layouts compare, let's take a quick look at each in turn: Fixed-Width layouts: These are constrained to a fixed with; a good size is around 960px, as this can be split equally into columns, with no remainder. The downside is the fixed width makes assumptions about the available viewport area, and that if the screen is too small or large, it results in scrolling or lots of which affects the user experience. Fluid layouts: Instead of using static values, we use percentage-based units; it means that no matter what the size of the browser window, our website will adjust accordingly. This removes the problems that surround fixed layouts at a stroke. Elastic layouts: They are similar to fluid layouts, but the constraints are measure by type or font size, using em or rem units; these are based on the defined font size, so 16px is 1 rem, 32px is 2 rem, and so on. These layouts allow for decent readability, with lines of 45-70 characters; font sizes are resized automatically. We may still see scrollbars appear in some instances, or experience some odd effects, if we zoom our page content. Hybrid layouts: They combine a mix of two or more of these different layout types; this allows us to choose static widths for some elements whilst others remain elastic or fluid. In comparison, responsive layouts take fluid layouts a step further, by using media queries to not only make our designs resize automatically, but present different views of our content on multiple devices. Exploring the benefits of flexible grid layouts Now that we've been introduced to grid layouts as a tenet of responsive design, it's a good opportunity to explore why we should use them. Creating a layout from scratch can be time-consuming, and need lots of testing—there are some real benefits from using a grid layout: Grids make for a simpler design: Instead of trying to develop the proverbial wheel, we can focus on providing the content instead; the infrastructure will have already been tested by the developer and other users. They provide for a visually appealing design: Many people prefer content to be displayed in columns, so grid layouts make good use of this concept, to help organize content on the page. Grids can of course adapt to different size viewports: The system they use makes it easier to display a single codebase on multiple devices, which reduces the effort required for developers to maintain and webmasters to manage. Grids help with the display of adverts: Google has been known to favor websites which display genuine content and not those where it believes the sole purpose of the website is for ad generation; we can use the grid to define specific area for adverts, without getting in the way of natural content. All in all, it makes sense to familiarize ourselves with grid layouts—the temptation is of course to use an existing library. There is nothing wrong with this, but to really get the benefit out of using them, it's good to understand some of the basics around the mechanics of grid layouts, and how this can help with the construction of our website. Making media responsive Our journey through the basics of adding responsive capabilities to a website has so far touched on how we make our layouts respond automatically to changes – it's time for us to do the same to media! If your first thought is that we need lots of additional functionality to make media responsive, then I am sorry to disappoint—it's much easier, and requires zero additional software to do it! Yes, all we need is just a text editor and a browser. I'll use my favorite editor, Sublime Text, but you can use whatever works for you. Over the course of this chapter, we will take a look in turn at images, video, audio and text, and we'll see how with some simple changes, we can make each of them responsive. Let's kick off our journey first, with a look at making image content responsive. Creating fluid images It is often said that images speak a thousand words. We can express a lot more with media than we can using words. This is particularly true for website selling products—a clear, crisp image clearly paints a better picture than a poor quality one! When constructing responsive websites, we need our images to adjust in size automatically—to see why this is important, go ahead and extract coffee.html from a copy of the code download that accompanies this book, and run it in a browser. Try resizing the window—we should see something akin to this: It doesn't look great, does it? Leaving aside my predilection for nature's finest bean drink (that is, coffee!), we can't have images that don't resize properly, so let's take a look at what is involved to make this happen: Go ahead and extract a copy of coffee.html and save it to our project area. We also need our image. This is in the img folder; save a copy to the img folder in our project area. In a new text file, add the following code, saving it as coffee.css: img { max-width: 100%; height: auto; } Revert back to coffee.html. You will see line 6 is currently commented out; remove the comment tags. Save the file, then preview it in a browser. If all is well, we will still see the same image as before, but this time try resizing it. This time around, our image grows or shrinks automatically, depending on the size of our browser window: Although our image does indeed fit better, there are a couple of points we should be aware of, when using this method: Sometimes you might see !important set as a property against the height attribute when working with responsive images; this isn't necessary, unless you're setting sizes in a website where image sizes may be overridden at a later date. We've set max-width to 100% as a minimum. You may also need to set a width value too, to be sure that your images do not become too big and break your layout. This is an easy technique to use, although there is a downside that can trip us up—spot what it is? If we use a high quality image, its file size will be hefty. We can't expect users of mobile devices to download it, can we? Don't worry though—there is a great alternative that has quickly gained popularity amongst browsers; we can use the <picture> element to control what is displayed, depending on the size of the available window. Implementing the <picture> element In a nutshell, responsive images are images that are displayed their optimal form on a page, depending on the device your website is being viewed from. This can mean several things: You want to show a separate image asset based on the user's physical screen size—this might be a 13.5 inch laptop, or a 5inch mobile phone screen. You want to show a separate image based on the resolution of the device, or using the device-pixel ratio (which is the ratio of device pixels to CSS pixels). You want to show an image in a specified image format (WebP, for example) if the browser supports it. Traditionally, we might have used simple scripting to achieve this, but it is at the risk of potentially downloading multiple images or none at all, if the script loads after images have loaded, or if we don't specify any image in our HTML and want the script to take care of loading images. Making video responsive Flexible videos are somewhat more complex than images. The HTML5 <video> maintains its aspect ratio just like images, and therefore we can apply the same CSS principle to make it responsive: video { max-width: 100%; height: auto !important; } Until relatively recently, there have been issues with HTML5 video—this is due in the main to split support for codecs, required to run HTML video. The CSS required to make a HTML5 video is very straightforward, but using it directly presents a few challenges: Hosting video is bandwidth intensive and expensive Streaming requires complex hardware support in addition to video It is not easy to maintain a consistent look and feel across different formats and platforms For many, a better alternative is to host the video through a third-party service such as YouTube—we can let them worry about bandwidth issues and providing a consistent look and feel; we just have to make it fit on the page! This requires a little more CSS styling to make it work, so let's dig in and find out what is involved. We clearly need a better way to manage responsive images! A relatively new tag for HTML5 is perfect for this job: <picture>. We can use this in one of three different ways, depending on whether we want to resize an existing image, display a larger one, or show a high-resolution version of the image. Implementing the <picture> element. In a nutshell, responsive images are images that are displayed their optimal form on a page, depending on the device your website is being viewed from. This can mean several things: You want to show a separate image asset based on the user's physical screen size—this might be a 13.5 inch laptop, or a 5inch mobile phone screen You want to show a separate image based on the resolution of the device, or using the device-pixel ratio (which is the ratio of device pixels to CSS pixels) You want to show an image in a specified image format (WebP, for example) if the browser supports it Traditionally, we might have used simple scripting to achieve this, but it is at the risk of potentially downloading multiple images or none at all, if the script loads after images have loaded, or if we don't specify any image in our HTML and want the script to take care of loading images. We clearly need a better way to manage responsive images! A relatively new tag for HTML5 is perfect for this job: <picture>. We can use this in one of three different ways, depending on whether we want to resize an existing image, display a larger one, or show a high-resolution version of the image. Making text fit on screen When building websites, it goes without saying but our designs clearly must start somewhere—this is usually with adding text. It's therefore essential that we allow for this in our responsive designs at the same time. Now is a perfect opportunity to explore how to do this—although text is not media in the same way as images or video, it is still content that has to be added at some point to our pages! With this in mind, let's dive in and explore how we can make our text responsive. Sizing with em units When working on non-responsive websites, it's likely that sizes will be quoted in pixel values – it's a perfectly acceptable way of working. However, if we begin to make our websites responsive, then content won't resize well using pixel values—we have to use something else. There are two alternatives - em or rem units. The former is based on setting a base font size that in most browsers defaults to 16px; in this example, the equivalent pixel sizes are given in the comments that follow each rule: h1 { font-size: 2.4em; } /* 38px */ p { line-height: 1.4em; } /* 22px */ Unfortunately there is an inherent problem with using em units—if we nest elements, then font sizes will be compounded, as em units are calculated relative to its parent. For example, if the font size of a list element is set at 1.4em (22px), then the font size of a list within a list becomes 30.8em (1.4 x 22px). To work around these issues, we can use rem values as a replacement—these are calculated from the root element, in place of the parent element. If you look carefully throughout many of the demos created for this book, you will see rem units being used to define the sizes of elements in that demo. Using rem units as a replacement The rem (or root em) unit is set to be relative to the root, instead of the parent – it means that we eliminate any issue with compounding at a stroke, as our reference point remains constant, and is not affected by other elements on the page. The downside of this is support—rem units are not supported in IE7 or 8, so if we still have to support these browsers, then we must fall back to using pixel or em values instead. This of course raises the question—should we still support these browsers, or is their usage of our website so small, as to not be worth the effort required to update our code? If the answer is that we must support IE8 or below, then we can take a hybrid approach—we can set both pixel/em and rem values at the same time in our code, thus: .article-body { font-size: 1.125rem; /* 18 / 16 */ font-size: 18px; } .caps, figure, footer { font-size: 0.875rem; /* 14 / 16 */ font-size: 14px; } Notice how we set rem values first? Browsers which support rem units will use these first; any that don't can automatically fall back to using pixel or em values instead. Exploring some examples Open a browser—let's go and visit some websites. Now, you may think I've lost my marbles, but stay with me: I want to show you a few examples. Let's take a look at a couple of example websites at different screen widths—how about this example, from my favorite coffee company, Starbucks: Try resizing the browser window—if you get small enough, you will see something akin to this: Now, what was the point of all that, I hear you ask? Well, it's simple—all of them use media queries in some form or other; CSS Tricks uses the queries built into WordPress, Packt's website is hosted using Drupal, and Starbuck's website is based around the Handlebars template system. The key here is that all use media queries to determine what should be displayed—throughout the course of this chapter, we'll explore using them in more detail, and see how we can use them to better manage content in responsive websites. Let's make a start with exploring their make up in more detail. Understanding media queries The developer Bruce Lee sums it up perfectly, when liking the effects of media queries to how water acts in different containers: "Empty your mind, be formless, shapeless - like water. Now you put water in a cup, it becomes the cup; you put water into a bottle it becomes the bottle; you put it in a teapot it becomes the teapot. Now water can flow or it can crash. Be water, my friend." We can use media queries to apply different CSS styles, based on available screen estate or specific device characteristics. These might include, but not be limited to the type of display, screen resolution or display density. Media queries work on the basis of testing to see if certain conditions are true, using this format: @media [not|only] [mediatype] and ([media feature]) { // CSS code; } We can use a similar principle to determine if entire style sheets should be loaded, instead of individual queries: <link rel="stylesheet" media="mediatype and|only|not (media feature)" href="myStyle.css"> Seems pretty simple, right? The great thing about media queries is that we don't need to download or install any additional software to use or create them – we can build most of them in the browser directly. Removing the need for breakpoints Up until now, we've covered how we can use breakpoints to control what is displayed, and when, according to which device is being used. Let's assume you're working on a project for a client, and have created a series of queries that use values such as 320px, 480px, 768px, and 1024px to cover support for a good range of devices. No matter what our design looks like, we will always be faced with two issues, if we focus on using specific screen viewports as the basis for controlling our designs: Keeping up with the sheer number of devices that are available The inflexibility of limiting our screen width So hold on: we're creating breakpoints, yet this can end up causing us more problems? If we're finding ourselves creating lots of media queries that address specific problems (in addition to standard ones), then we will start to lose the benefits of a responsive website—instead we should re-examine our website, to understand why the design isn't working and see if we can't tweak it so as to remove the need for the custom query. Ultimately our website and target devices will dictate what is required—a good rule of thumb is if we are creating more custom queries than a standard bunch of 4-6 breakpoints, then perhaps it is time to recheck our design! As an alternative to working with specific screen sizes, there is a different approach we can take, which is to follow the principle of adaptive design, and not responsive design. Instead of simply specifying a number of fixed screen sizes (such as for the iPhone 6 Plus or a Samsung Galaxy unit), we build our designs around the point at which the design begins to fail. Why? The answer is simple—the idea here is to come up with different bands, where designs will work between a lower and upper value, instead of simply specifying a query that checks for fixed screen sizes that are lower or above certain values. Understanding the importance of speed The advent of using different devices to access the internet means speed is critical – the time it takes to download content from hosting servers, and how quickly the user can interact with the website are key to the success of any website. Why it is important to focus on the performance of our website on the mobile devices or those devices with lesser screen resolution? There are several reasons for this—they include: 80 percent of internet users owns a smartphone Around 90 percent of users go online through a mobile device, with 48% of users using search engines to research new products Approximately 72 percent users abandon a website if the loading time is more than 5-6 seconds Mobile digital media time is now significantly higher than compared to desktop use If we do not consider statistics such as these, then we may go ahead and construct our website, but end up with a customer losing both income and market share, if we have not fully considered the extent of where our website should work. Coupled with this is the question of performance – if our website is slow, then this will put customers off, and contribute to lost sales. A study performed by San Francisco-based Kissmetrics shows that mobile users wait between 6 to 10 seconds before they close the website and lose faith in it. At the same time, tests performed by Guy Podjarny for the Mediaqueri.es website (http://mediaqueri.es) indicate that we're frequently downloading the same content for both large and small screens—this is entirely unnecessary, when with some simple changes, we can vary content to better suit desktop PCs or mobile devices! So what can we do? Well, before we start exploring where to make changes, let's take a look at some of the reasons why websites run slowly. Understanding why pages load slowly Although we may build a great website that works well across multiple devices, it's still no good if it is slow! Every website will of course operate differently, but there are a number of factors to allow for, which can affect page (and website) speed: Downloading data unnecessarily: On a responsive website, we may hide elements that are not displayed on smaller devices; the use of display: none in code means that we still download content, even though we're not showing it on screen, resulting in slower websites and higher bandwidth usage. Downloading images before shrinking them: If we have not optimized our website with properly sized images, then we may end up downloading images that are larger than necessary on a mobile device. We can of course make them fluid by using percentage-based size values, but this places extra demand on the server and browser to resize them. A complicated DOM in use on the website: When creating a responsive website, we have to add in a layer of extra code to manage different devices; this makes the DOM more complicated, and slow our website down. It is therefore imperative that we're not adding in any unnecessary elements that require additional parsing time by the browser. Downloading media or feeds from external sources: It goes without saying that these are not under our control; if our website is dependent on them, then the speed of our website will be affected if these external sources fail. Use of Flash: Websites that rely on heavy use of Flash will clearly be slower to access than those that don't use the technology. It is worth considering if our website really needs to use it; recent changes by Adobe mean that Flash as a technology is being retired in favor of animation using other means such as HTML5 Canvas or WebGL. There is one point to consider that we've not covered in this list—the average size of a page has significantly increased since the dawn of the Internet in the mid-nineties. Although these figures may not be 100% accurate, they still give a stark impression of how things have changed: 1995: At that time the average page size used to be around 14.1 KB in size. The reason for it can be that it contained around 2 or 3 objects. That means just 2 or 3 calls to server on which the website was hosted. 2008: The average page size increased to around 498 KB in size, with an average use of around 70 objects that includes changes to CSS, images and JavaScript. Although this is tempered with the increased use of broadband, not everyone can afford fast access, so we will lose customers if our website is slow to load. All is not lost though—there are some tricks we can use to help optimize the performance of our websites. Testing website compatibility At this stage, our website would be optimized, and tested for performance—but what about compatibility? Although the wide range of available browsers has remained relatively static (at least for the ones in mainstream use), the functionality they offer is constantly changing—it makes it difficult for developers and designers to handle all of the nuances required to support each browser. In addition, the wide range makes it costly to support—in an ideal world, we would support every device available, but this is impossible; instead, we must use analytical software to determine which devices are being used, and therefore worthy of support. Working out a solution If we test our website on a device such as an iPhone 6, then there is a good chance it will work as well on other Apple devices, such as iPads. The same can be said for testing on a mobile device such as a Samsung Galaxy S4—we can use this principle to help prioritize support for particular mobile devices, if they require more tweaks to be made than for other devices. Ultimately though, we must use analytical software to determine who visits our website; the information such as browser, source, OS and device used will help determine what our target audience should be. This does not mean we completely neglect other devices; we can ensure they work with our website, but this will not be a priority during development. A key point of note is that we should not attempt to support every device – this is too costly to manage, and we would never keep up with all of the devices available for sale! Instead, we can use our analytics software to determine which devices are being used by our visitors; we can then test a number of different properties: Screen size: This should encompass a variety of different resolutions for desktop and mobile devices. Connection speed: Testing across different connection speeds will help us understand how the website behaves, and identify opportunities or weaknesses where we may need to effect changes. Pixel density: Some devices will support higher a pixel density, which allows them to display higher resolution images or content; this will make it easier to view and fix any issues with displaying content. Interaction style: The ability to view the Internet across different devices means that we should consider how our visitors interact with the website: is it purely on a desktop, or do they use tablets, smartphones or gaming-based devices? It's highly likely that the former two will be used to an extent, but the latter is not likely to feature as highly. Once we've determined which devices we should be supporting, then there are a range of tools available for us to use, to test browser compatibility. These include physical devices (ideal, but expensive to maintain), emulators or online services (these can be commercial, or free). Let's take a look at a selection of what is available, to help us test compatibility. Exploring tools available for testing When we test a mobile or responsive website, there are factors which we need to consider before we start testing, to help deliver a website which looks consistent across all the devices and browsers. These factors include: Does the website look good? Are there any bugs or defects? Is our website really responsive? To help test our websites, we can use any one of several tools (either paid or free)—a key point to note though is that we can already get a good idea of how well our websites work, by simply using the Developer toolbar that is available in most browsers! Viewing with Chrome We can easily emulate a mobile device within Chrome, by pressing Ctrl + Shift + M; Chrome displays a toolbar at the top of the window, which allows us to select different devices: If we click on the menu entry (currently showing iPhone 6 Plus), and change it to Edit, we can add new devices; this allows us to set specific dimensions, user agent strings and whether the device supports high-resolution images: Although browsers can go some way to providing an indication of how well our website works, they can only provide a limited view – sometimes we need to take things a step further and use commercial solutions to test our websites across multiple browsers at the same time. Let's take a look at some of the options available commercially. Exploring our options If you've spent any time developing code, then there is a good chance you may already be aware of Browserstack (from https://www.browserstack.com)—other options include the following: GhostLab: https://www.vanamco.com/ghostlab/ Muir: http://labs.iqfoundry.com/ CrossBrowserTesting: http://www.crossbrowsertesting.com/ If however all we need to do is check our website for its level of responsiveness, then we don't need to use paid options – there are a number of websites that allow us to check, without needing to installing plugins or additional tools: Am I Responsive: http://ami.responsive.is ScreenQueries: http://screenqueri.es Cybercrab's screen check facility: http://cybercrab.com/screencheck Remy Sharp's check website: http://responsivepx.com We can also use bookmarklets to check to see how well our websites work on different devices – a couple of examples to try are at http://codebomber.com/jquery/resizer and http://responsive.victorcoulon.fr/; it is worth noting that current browsers already include this functionality, making the bookmarklets less attractive as an option. We have now reached the end of our journey through the essentials of creating responsive websites with nothing more than plain HTML and CSS code. We hope you have enjoyed it as much as we have with writing, and that it helps you make a start into the world of responsive design using little more than plain HTML and CSS. Summary This article covers the elements of RWD and introduces us to the different flexible grid layouts. Resources for Article: Responsive Web Design with WordPress Web Design Principles in Inkscape Top Features You Need to Know About – Responsive Web Design
Read more
  • 0
  • 1
  • 43398

article-image-scripting-animation-maya
Packt
02 Aug 2016
28 min read
Save for later

Scripting for Animation in Maya

Packt
02 Aug 2016
28 min read
This article, written by Adrian Herbez, author of Maya Programming with Python Cookbook, will cover various recipes related to animating objects with scripting: Querying animation data Working with animation layers Copying animation from one object to another Setting keyframes Creating expressions via script (For more resources related to this topic, see here.) In this article, we'll be looking at how to use scripting to create animation and set keyframes. We'll also see how to work with animation layers and create expressions from code. Querying animation data In this example, we'll be looking at how to retrieve information about animated objects, including which attributes are animated and both the location and value of keyframes. Although this script is unlikely to be useful by itself, knowing the number, time, and values of keyframes is sometimes a prerequisite for more complex animation tasks. Getting ready To make get the most out of this script, you'll need to have an object with some animation curves defined. Either load up a scene with animation or skip ahead to the recipe on setting keyframes. How to do it... Create a new file and add the following code: import maya.cmds as cmds def getAnimationData(): objs = cmds.ls(selection=True) obj = objs[0] animAttributes = cmds.listAnimatable(obj); for attribute in animAttributes: numKeyframes = cmds.keyframe(attribute, query=True, keyframeCount=True) if (numKeyframes > 0): print("---------------------------") print("Found ", numKeyframes, " keyframes on ", attribute) times = cmds.keyframe(attribute, query=True, index=(0,numKeyframes), timeChange=True) values = cmds.keyframe(attribute, query=True, index=(0,numKeyframes), valueChange=True) print('frame#, time, value') for i in range(0, numKeyframes): print(i, times[i], values[i]) print("---------------------------") getAnimationData() If you select an object with animation curves and run the script, you should see a readout of the time and value for each keyframe on each animated attribute. For example, if we had a simple bouncing ball animation with the following curves: We would see something like the following output in the script editor: --------------------------- ('Found ', 2, ' keyframes on ', u'|bouncingBall.translateX') frame#, time, value (0, 0.0, 0.0) (1, 190.0, 38.0) --------------------------- --------------------------- ('Found ', 20, ' keyframes on ', u'|bouncingBall.translateY') frame#, time, value (0, 0.0, 10.0) (1, 10.0, 0.0) (2, 20.0, 8.0) (3, 30.0, 0.0) (4, 40.0, 6.4000000000000004) (5, 50.0, 0.0) (6, 60.0, 5.120000000000001) (7, 70.0, 0.0) (8, 80.0, 4.096000000000001) (9, 90.0, 0.0) (10, 100.0, 3.276800000000001) (11, 110.0, 0.0) (12, 120.0, 2.6214400000000011) (13, 130.0, 0.0) (14, 140.0, 2.0971520000000008) (15, 150.0, 0.0) (16, 160.0, 1.6777216000000008) (17, 170.0, 0.0) (18, 180.0, 1.3421772800000007) (19, 190.0, 0.0) --------------------------- How it works... We start out by grabbing the selected object, as usual. Once we've done that, we'll iterate over all the keyframeable attributes, determine if they have any keyframes and, if they do, run through the times and values. To get the list of keyframeable attributes, we use the listAnimateable command: objs = cmds.ls(selection=True) obj = objs[0] animAttributes = cmds.listAnimatable(obj) This will give us a list of all the attributes on the selected object that can be animated, including any custom attributes that have been added to it. If you were to print out the contents of the animAttributes array, you would likely see something like the following: |bouncingBall.rotateX |bouncingBall.rotateY |bouncingBall.rotateZ Although the bouncingBall.rotateX part likely makes sense, you may be wondering about the | symbol. This symbol is used by Maya to indicate hierarchical relationships between nodes in order to provide fully qualified node and attribute names. If the bouncingBall object was a child of a group named ballGroup, we would see this instead: |ballGroup|bouncingBall.rotateX Every such fully qualified name will contain at least one pipe (|) symbol, as we see in the first, nongrouped example, but there can be many more—one for each additional layer of hierarchy. While this can lead to long strings for attribute names, it allows Maya to make use of objects that may have the same name, but under different parts of a larger hierarchy (to have control objects named handControl for each hand of a character, for example). Now that we have a list of all of the possibly animated attributes for the object, we'll next want to determine if there are any keyframes set on it. To do this, we can use the keyframe command in the query mode. for attribute in animAttributes: numKeyframes = cmds.keyframe(attribute, query=True, keyframeCount=True) At this point, we have a variable (numKeyframes) that will be greater than zero for any attribute with at least one keyframe. Getting the total number of keyframes on an attribute is only one of the things that the keyframe command can do; we'll also use it to grab the time and value for each of the keyframes. To do this, we'll call it two more times, both in the query mode—once to get the times and once to get the values: times = cmds.keyframe(attribute, query=True, index=(0,numKeyframes), timeChange=True) values = cmds.keyframe(attribute, query=True, index=(0,numKeyframes), valueChange=True) These two lines are identical in everything except what type of information we're asking for. The important thing to note here is the index flag, which is used to tell Maya which keyframes we're interested in. The command requires a two-element argument representing the first (inclusive) and last (exclusive) index of keyframes to examine. So, if we had total 20 keyframes, we would pass in (0,20), which would examine the keys with indices from 0 to 19. The flags we're using to get the values likely look a bit odd—both valueChange and timeChange might lead you to believe that we would be getting relative values, rather than absolute. However, when used in the previously mentioned manner, the command will give us what we want—the actual time and value for each keyframe, as they appear in the graph editor. If you want to query information on a single keyframe, you still have to pass in a pair of values- just use the index that you're interested in twice- to get the fourth frame, for example, use (3,3). At this point, we have two arrays—the times array, which contains the time value for each keyframe, and the values array that contains the actual attribute value. All that's left is to print out the information that we've found: print('frame#, time, value') for i in range(0, numKeyframes): print(i, times[i], values[i]) There's more... Using the indices to get data on keyframes is an easy way to run through all of the data for a curve, but it's not the only way to specify a range. The keyframe command can also accept time values. If we wanted to know how many keyframes existed on a given attribute between frame 1 and frame 100, for example, we could do the following: numKeyframes = cmds.keyframe(attributeName, query=True, time=(1,100) keyframeCount=True) Also, if you find yourself with highly nested objects and need to extract just the object and attribute names, you may find Python's built-in split function helpful. You can call split on a string to have Python break it up into a list of parts. By default, Python will break up the input string by spaces, but you can specify a particular string or character to split on. Assume that you have a string like the following: |group4|group3|group2|group1|ball.rotateZ Then, you could use split to break it apart based on the | symbol. It would give you a list, and using −1 as an index would give you just ball.rotateZ. Putting that into a function that can be used to extract the object/attribute names from a full string would be easy, and it would look something like the following: def getObjectAttributeFromFull(fullString): parts = fullString.split("|") return parts[-1] Using it would look something like this: inputString = "|group4|group3|group2|group1|ball.rotateZ" result = getObjectAttributeFromFull(inputString) print(result) # outputs "ball.rotateZ" Working with animation layers Maya offers the ability to create multiple layers of animation in a scene, which can be a good way to build up complex animation. The layers can then be independently enabled or disabled, or blended together, granting the user a great deal of control over the end result. In this example, we'll be looking at how to examine the layers that exist in a scene, and building a script will ensure that we have a layer of a given name. For example, we might want to create a script that would add additional randomized motion to the rotations of selected objects without overriding their existing motion. To do this, we would want to make sure that we had an animation layer named randomMotion, which we could then add keyframes to. How to do it... Create a new script and add the following code: import maya.cmds as cmds def makeAnimLayer(layerName): baseAnimationLayer = cmds.animLayer(query=True, root=True) foundLayer = False if (baseAnimationLayer != None): childLayers = cmds.animLayer(baseAnimationLayer, query=True, children=True) if (childLayers != None) and (len(childLayers) > 0): if layerName in childLayers: foundLayer = True if not foundLayer: cmds.animLayer(layerName) else: print('Layer ' + layerName + ' already exists') makeAnimLayer("myLayer") Run the script, and you should see an animation layer named myLayer appear in the Anim tab of the channel box. How it works... The first thing that we want to do is to find out if there is already an animation layer with the given name present in the scene. To do this, we start by grabbing the name of the root animation layer: baseAnimationLayer = cmds.animLayer(query=True, root=True) In almost all cases, this should return one of two possible values—either BaseAnimation or (if there aren't any animation layers yet) Python's built-in None value. We'll want to create a new layer in either of the following two possible cases: There are no animation layers yet There are animation layers, but none with the target name In order to make the testing for the above a bit easier, we first create a variable to hold whether or not we've found an animation layer and set it to False: foundLayer = False Now we need to check to see whether it's true that both animation layers exist and one of them has the given name. First off, we check that there was, in fact, a base animation layer: if (baseAnimationLayer != None): If this is the case, we want to grab all the children of the base animation layer and check to see whether any of them have the name we're looking for. To grab the children animation layers, we'll use the animLayer command again, again in the query mode: childLayers = cmds.animLayer(baseAnimationLayer, query=True, children=True) Once we've done that, we'll want to see if any of the child layers match the one we're looking for. We'll also need to account for the possibility that there were no child layers (which could happen if animation layers were created then later deleted, leaving only the base layer): if (childLayers != None) and (len(childLayers) > 0): if layerName in childLayers: foundLayer = True If there were child layers and the name we're looking for was found, we set our foundLayer variable to True. If the layer wasn't found, we create it. This's easily done by using the animLayer command one more time, with the name of the layer we're trying to create: if not foundLayer: cmds.animLayer(layerName) Finally, we finish off by printing a message if the layer was found to let the user know. There's more... Having animation layers is great, in that we can make use of them when creating or modifying keyframes. However, we can't actually add animation to layers without first adding the objects in question to the animation layer. Let's say that we had an object named bouncingBall, and we wanted to set some keyframes on its translateY attribute, in the bounceLayer animation layer. The actual command to set the keyframe(s) would look something like this: cmds.setKeyframe("bouncingBall.translateY", value=yVal, time=frame, animLayer="bounceLayer") However, this would only work as expected if we had first added the bouncingBall object to the bounceLayer animation layer. To do it, we could use the animLayer command in the edit mode, with the addSelectedObjects flag. Note that because the flag operates on the currently selected objects, we would need to first select the object we want to add: cmds.select("bouncingBall", replace=True) cmds.animLayer("bounceLayer", edit=True, addSelectedObjects=True) Adding the object will, by default, add all of its animatable attributes. You can also add specific attributes, rather than entire objects. For example, if we only wanted to add the translateY attribute to our animation layer, we could do the following: cmds.animLayer("bounceLayer", edit=True, attribute="bouncingBall.translateY") Copying animation from one object to another In this example, we'll create a script that will copy all of the animation data on one object to one or more additional objects, which could be useful to duplicate motion across a range of objects. Getting ready For the script to work, you'll need an object with some keyframes set. Either create some simple animation or skip ahead to the example on creating keyframes with script, later in this article. How to do it... Create a new script and add the following code: import maya.cmds as cmds def getAttName(fullname): parts = fullname.split('.') return parts[-1] def copyKeyframes(): objs = cmds.ls(selection=True) if (len(objs) < 2): cmds.error("Please select at least two objects") sourceObj = objs[0] animAttributes = cmds.listAnimatable(sourceObj); for attribute in animAttributes: numKeyframes = cmds.keyframe(attribute, query=True, keyframeCount=True) if (numKeyframes > 0): cmds.copyKey(attribute) for obj in objs[1:]: cmds.pasteKey(obj, attribute=getAttName(attribute), option="replace") copyKeyframes() Select the animated object, shift-select at least one other object, and run the script. You'll see that all of the objects have the same motion. How it works... The very first part of our script is a helper function that we'll be using to strip the attribute name off a full object name/attribute name string. More on it will be given later. Now on to the bulk of the script. First off, we run a check to make sure that the user has selected at least two objects. If not, we'll display a friendly error message to let the user know what they need to do: objs = cmds.ls(selection=True) if (len(objs) < 2): cmds.error("Please select at least two objects") The error command will also stop the script from running, so if we're still going, we know that we had at least two objects selected. We'll set the first one to be selected to be our source object. We could just as easily use the second-selected object, but that would mean using the first selected object as the destination, limiting us to a single target:     sourceObj = objs[0] Now we're ready to start copying animation, but first, we'll need to determine which attributes are currently animated, through a combination of finding all the attributes that can be animated, and checking each one to see whether there are any keyframes on it: animAttributes = cmds.listAnimatable(sourceObj); for attribute in animAttributes: numKeyframes = cmds.keyframe(attribute, query=True, keyframeCount=True) If we have at least one keyframe for the given attribute, we move forward with the copying: if (numKeyframes > 0): cmds.copyKey(attribute) The copyKey command will cause the keyframes for a given object to be temporarily held in memory. If used without any additional flags, it will grab all of the keyframes for the specified attribute, exactly what we want in this case. If we wanted only a subset of the keyframes, we could use the time flag to specify a range. We're passing in each of the values that were returned by the listAnimatable function. These will be full names (both object name and attribute). That's fine for the copyKey command, but will require a bit of additional work for the paste operation. Since we're copying the keys onto a different object than the one that we copied them from, we'll need to separate out the object and attribute names. For example, our attribute value might be something like this: |group1|bouncingBall.rotateX From this, we'll want to trim off just the attribute name (rotateX) since we're getting the object name from the selection list. To do this, we created a simple helper function that takes a full-length object/attribute name and returns just the attribute name. That's easy enough to do by just breaking the name/attribute string apart on the . and returning the last element, which in this case is the attribute: def getAttName(fullname): parts = fullname.split('.') return parts[-1] Python's split function breaks apart the string into an array of strings, and using a negative index will count back from the end, with −1 giving us the last element. Now we can actually paste our keys. We'll run through all the remaining selected objects, starting with the second, and paste our copied keyframes: for obj in objs[1:]: cmds.pasteKey(obj, attribute=getAttName(attribute), option="replace") Note that we're using the nature of Python's for loops to make the code a bit more readable. Rather than using an index, as would be the case in most other languages, we can just use the for x in y construction. In this case, obj will be a temporary variable, scoped to the for loop, that takes on the value of each item in the list. Also note that instead of passing in the entire list, we use objs[1:] to indicate the entire list, starting at index 1 (the second element). The colon allows us to specify a subrange of the objs list, and leaving the right-hand side blank will cause Python to include all the items to the end of the list. We pass in the name of the object (from our original selection), the attribute (stripped from full name/attribute string via our helper function), and we use option="replace" to ensure that the keyframes we're pasting in replace anything that's already there. Original animation (top). Here, we see the result of pasting keys with the default settings (left) and with the replace option (right). Note that the default results still contain the original curves, just pushed to later frames If we didn't include the option flag, Maya would default to inserting the pasted keyframes while moving any keyframes already present forward in the timeline. There's more... There are a lot of other options for the option flag, each of which handles possible conflicts with the keys you're pasting and the ones that may already exist in a slightly different way. Be sure to have a look at the built-in documentation for the pasteKeys command for more information. Another, and perhaps better option to control how pasted keys interact with existing one is to paste the new keys into a separate animation layer. For example, if we wanted to make sure that our pasted keys end up in an animation layer named extraAnimation, we could modify the call to pasteKeys as follows: cmds.pasteKey(objs[i], attribute=getAttName(attribute), option="replace", animLayer="extraAnimation") Note that if there was no animation layer named extraAnimation present, Maya would fail to copy the keys. See the section on working with animation layers for more information on how to query existing layers and create new ones. Setting keyframes While there are certainly a variety of ways to get things to move in Maya, the vast majority of motion is driven by keyframes. In this example, we'll be looking at how to create keyframes with code by making that old animation standby—a bouncing ball. Getting ready The script we'll be creating will animate the currently selected object, so make sure that you have an object—either the traditional sphere or something else you'd like to make bounce. How to do it... Create a new file and add the following code: import maya.cmds as cmds def setKeyframes(): objs = cmds.ls(selection=True) obj = objs[0] yVal = 0 xVal = 0 frame = 0 maxVal = 10 for i in range(0, 20): frame = i * 10 xVal = i * 2 if i % 2 == 1: yVal = 0 else: yVal = maxVal maxVal *= 0.8 cmds.setKeyframe(obj + '.translateY', value=yVal, time=frame) cmds.setKeyframe(obj + '.translateX', value=xVal, time=frame) setKeyframes() Run the preceding script with an object selected and trigger playback. You should see the object move up and down. How it works... In order to get our object to bounce, we'll need to set keyframes such that the object alternates between a Y-value of zero and an ever-decreasing maximum so that the animation mimics the way a falling object loses velocity with each bounce. We'll also make it move forward along the x-axis as it bounces. We start by grabbing the currently selected object and setting a few variables to make things easier to read as we run through our loop. Our yVal and xVal variables will hold the current value that we want to set the position of the object to. We also have a frame variable to hold the current frame and a maxVal variable, which will be used to hold the Y-value of the object's current height. This example is sufficiently simple that we don't really need separate variables for frame and the attribute values, but setting things up this way makes it easier to swap in more complex math or logic to control where keyframes get set and to what value. This gives us the following: yVal = 0 xVal = 0 frame = 0 maxVal = 10 The bulk of the script is a single loop, in which we set keyframes on both the X and Y positions. For the xVal variable, we'll just be multiplying a constant value (in this case, 2 units). We'll do the same thing for our frame. For the yVal variable, we'll want to alternate between an ever-decreasing value (for the successive peaks) and zero (for when the ball hits the ground). To alternate between zero and non-zero, we'll check to see whether our loop variable is divisible by two. One easy way to do this is to take the value modulo (%) 2. This will give us the remainder when the value is divided by two, which will be zero in the case of even numbers and one in the case of odd numbers. For odd values, we'll set yVal to zero, and for even ones, we'll set it to maxVal. To make sure that the ball bounces a little less each time, we set maxVal to 80% of its current value each time we make use of it. Putting all of that together gives us the following loop: for i in range(0, 20): frame = i * 10 xVal = i * 2 if (i % 2) == 1: yVal = 0 else: yVal = maxVal maxVal *= 0.8 Now we're finally ready to actually set keyframes on our object. This's easily done with the setKeyframe command. We'll need to specify the following three things: The attribute to keyframe (object name and attribute) The time at which to set the keyframe The actual value to set the attribute to In this case, this ends up looking like the following: cmds.setKeyframe(obj + '.translateY', value=yVal, time=frame) cmds.setKeyframe(obj + '.translateX', value=xVal, time=frame) And that's it! A proper bouncing ball (or other object) animated with pure code. There's more... By default, the setKeyframe command will create keyframes with both in tangent and out tangent being set to spline. That's fine for a lot of things, but will result in overly smooth animation for something that's supposed to be striking a hard surface. We can improve our bounce animation by keeping smooth tangents for the keyframes when the object reaches its maximum height, but setting the tangents at its minimum to be linear. This will give us a nice sharp change every time the ball strikes the ground. To do this, all we need to do is to set both the inTangentType and outTangentType flags to linear, as follows: cmds.setKeyframe(obj + ".translateY", value=animVal, time=frame, inTangentType="linear", outTangentType="linear") To make sure that we only have linear tangents when the ball hits the ground, we could set up a variable to hold the tangent type, and set it to one of two values in much the same way that we set the yVal variable. This would end up looking like this: tangentType = "auto" for i in range(0, 20): frame = i * 10 if i % 2 == 1: yVal = 0 tangentType = "linear" else: yVal = maxVal tangentType = "spline" maxVal *= 0.8 cmds.setKeyframe(obj + '.translateY', value=yVal, time=frame, inTangentType=tangentType, outTangentType=tangentType) Creating expressions via script While most animation in Maya is created manually, it can often be useful to drive attributes directly via script, especially for mechanical objects or background items. One way to approach this is through Maya's expression editor. In addition to creating expressions via the expression editor, it is also possible to create expressions with scripting, in a beautiful example of code-driven code. In this example, we'll be creating a script that can be used to create a sine wave-based expression to smoothly alter a given attribute between two values. Note that expressions cannot actually use Python code directly; they require the code to be written in the MEL syntax. But this doesn't mean that we can't use Python to create expressions, which is what we'll do in this example. Getting ready Before we dive into the script, we'll first need to have a good handle on the kind of expression we'll be creating. There are a lot of different ways to approach expressions, but in this instance, we'll keep things relatively simple and tie the attribute to a sine wave based on the current time. Why a sine wave? Sine waves are great because they alter smoothly between two values, with a nice easing into and out of both the minimum and maximums. While the minimum and maximum values range from −1 to 1, it's easy enough to alter the output to move between any two numbers we want. We'll also make things a bit more flexible by setting up the expression to rely on a custom speed attribute that can be used to control the rate at which the attribute animates. The end result will be a value that varies smoothly between any two numbers at a user-specified (and keyframeable) rate. How to do it... Create a new script and add the following code: import maya.cmds as cmds def createExpression(att, minVal, maxVal, speed): objs = cmds.ls(selection=True) obj = objs[0] cmds.addAttr(obj, longName="speed", shortName="speed", min=0, keyable=True) amplitude = (maxVal – minVal)/2.0 offset = minVal + amplitude baseString = "{0}.{1} = ".format(obj, att) sineClause = '(sin(time * ' + obj + '.speed)' valueClause = ' * ' + str(amplitude) + ' + ' + str(offset) + ')' expressionString = baseString + sineClause + valueClause cmds.expression(string=expressionString) createExpression('translateY', 5, 10, 1) How it works... The first that we do is to add a speed attribute to our object. We'll be sure to make it keyable for later animation: cmds.addAttr(obj, longName="speed", shortName="speed", min=0, keyable=True) It's generally a good idea to include at least one keyframeable attribute when creating expressions. While math-driven animation is certainly a powerful technique, you'll likely still want to be able to alter the specifics. Giving yourself one or more keyframeable attributes is an easy way to do just that. Now we're ready to build up our expression. But first, we'll need to understand exactly what we want; in this case, a value that smoothly varies between two extremes, with the ability to control its speed. We can easily build an expression to do that using the sine function, with the current time as the input. Here's what it looks like in a general form: animatedValue = (sin(time * S) * M) + O; Where: S is a value that will either speed up (if greater than 1) or slow down (if less) the rate at which the input to the sine function changes M is a multiplier to alter the overall range through which the value changes O is an offset to ensure that the minimum and maximum values are correct You can also think about it visually—S will cause our wave to stretch or shrink along the horizontal (time) axis, M will expand or contract it vertically, and O will move the entire shape of the curve either up or down. S is already taken care of; it's our newly created "speed" attribute. M and O will need to be calculated, based on the fact that sine functions always produce values ranging from −1 to 1. The overall range of values should be from our minVal to our maxVal, so you might think that M should be equal to (maxVal – minVal). However, since it gets applied to both −1 and 1, this would leave us with double the desired change. So, the final value we want is instead (maxVal – minVal)/2. We store that into our amplitude variable as follows: amplitude = (maxVal – minVal)/2.0 Next up is the offset value O. We want to move our graph such that the minimum and maximum values are where they should be. It might seem like that would mean just adding our minVal, but if we left it at that, our output would dip below the minimum for 50% of the time (anytime the sine function is producing negative output). To fix it, we set O to (minVal + M) or in the case of our script: offset = minVal + amplitude This way, we move the 0 position of the wave to be midway between our minVal and maxVal, which is exactly what we want. To make things clearer, let's look at the different parts we're tacking onto sin(), and the way they effect the minimum and maximum values the expression will output. We'll assume that the end result we're looking for is a range from 0 to 4. Expression Additional component Minimum Maximum sin(time) None- raw sin function −1 1 sin(time * speed) Multiply input by "speed" −1 (faster) 1 (faster) sin(time * speed) * 2 Multiply output by 2 −2 2 (sin(time * speed) * 2) + 2 Add 2 to output 0 4   Note that 2 = (4-0)/2 and 2 = 0 + 2. Here's what the preceding progression looks like when graphed:   Four steps in building up an expression to var an attribute from 0 to 4 with a sine function. Okay, now that we have the math locked down, we're ready to translate that into Maya's expression syntax. If we wanted an object named myBall to animate along Y with the previous values, we would want to end up with: myBall.translateY = (sin(time * myBall.speed) * 5) + 12; This would work as expected if entered into Maya's expression editor, but we want to make sure that we have a more general-purpose solution that can be used with any object and any values. That's straightforward enough and just requires building up the preceding string from various literals and variables, which is what we do in the next few lines: baseString = "{0}.{1} = ".format(obj, att) sineClause = '(sin(time * ' + obj + '.speed)' valueClause = ' * ' + str(amplitude) + ' + ' + str(offset) + ')' expressionString = baseString + sineClause + valueClause I've broken up the string creation into a few different lines to make things clearer, but it's by no means necessary. The key idea here is that we're switching back and forth between literals (sin(time *, .speed, and so on) and variables (obj, att, amplitude, and offset) to build the overall string. Note that we have to wrap numbers in the str() function to keep Python from complaining when we combine them with strings. At this point, we have our expression string ready to go. All that's left is to actually add it to the scene as an expression, which is easily done with the expression command: cmds.expression(string=expressionString) And that's it! We will now have an attribute that varies smoothly between any two values. There's more... There are tons of other ways to use expressions to drive animation, and all sorts of simple mathematical tricks that can be employed. For example, you can easily get a value to move smoothly to a target value with a nice easing-in to the target by running this every frame: animatedAttribute = animatedAttribute + (targetValue – animatedAttribute) * 0.2; This will add 20% of the current difference between the target and the current value to the attribute, which will move it towards the target. Since the amount that is added is always a percentage of the current difference, the per-frame effect reduces as the value approaches the target, providing an ease-in effect. If we were to combine this with some code to randomly choose a new target value, we would end up with an easy way to, say, animate the heads of background characters to randomly look in different positions (maybe to provide a stadium crowd). Assume that we had added custom attributes for targetX, targetY, and targetZ to our object that would end up looking something like the following: if (frame % 20 == 0) { myCone.targetX = rand(time) * 360; myCone.targetY = rand(time) * 360; myCone.targetZ = rand(time) * 360; } myObject.rotateX += (myObject.targetX - myCone.rotateX) * 0.2; myObject.rotateY += (myObject.targetY - myCone.rotateY) * 0.2; myObject.rotateZ += (myObject.targetZ - myCone.rotateZ) * 0.2; Note that we're using the modulo (%) operator to do something (setting the target) only when the frame is an even multiple of 20. We're also using the current time as the seed value for the rand() function to ensure that we get different results as the animation progresses. The previously mentioned example is how the code would look if we entered it directly into Maya's expression editor; note the MEL-style (rather than Python) syntax. Generating this code via Python would be a bit more involved than our sine wave example, but would use all the same principles—building up a string from literals and variables, then passing that string to the expression command. Summary In this article, we primarily discussed scripting and animation using Maya.  Resources for Article: Further resources on this subject: Introspecting Maya, Python, and PyMEL [article] Discovering Python's parallel programming tools [article] Mining Twitter with Python – Influence and Engagement [article]
Read more
  • 0
  • 0
  • 28343
Modal Close icon
Modal Close icon