Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Newsletter Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials

7019 Articles
article-image-modernizing-our-spring-boot-app
Packt
26 Nov 2014
15 min read
Save for later

Modernizing our Spring Boot app

Packt
26 Nov 2014
15 min read
In this article by Greg L. Turnquist, the author of the book, Learning Spring Boot, we will discuss modernizing our Spring Boot app with JavaScript and adding production-ready support features. (For more resources related to this topic, see here.) Modernizing our app with JavaScript We just saw that, with a single @Grab statement, Spring Boot automatically configured the Thymeleaf template engine and some specialized view resolvers. We took advantage of Spring MVC's ability to pass attributes to the template through ModelAndView. Instead of figuring out the details of view resolvers, we instead channeled our efforts into building a handy template to render data fetched from the server. We didn't have to dig through reference docs, Google, and Stack Overflow to figure out how to configure and integrate Spring MVC with Thymeleaf. We let Spring Boot do the heavy lifting. But that's not enough, right? Any real application is going to also have some JavaScript. Love it or hate it, JavaScript is the engine for frontend web development. See how the following code lets us make things more modern by creating modern.groovy: @Grab("org.webjars:jquery:2.1.1")@Grab("thymeleaf-spring4")@Controllerclass ModernApp {def chapters = ["Quick Start With Groovy","Quick Start With Java","Debugging and Managing Your App","Data Access with Spring Boot","Securing Your App"]@RequestMapping("/")def home(@RequestParam(value="name", defaultValue="World")String n) {new ModelAndView("modern").addObject("name", n).addObject("chapters", chapters)}} A single @Grab statement pulls in jQuery 2.1.1. The rest of our server-side Groovy code is the same as before. There are multiple ways to use JavaScript libraries. For Java developers, it's especially convenient to use the WebJars project (http://webjars.org), where lots of handy JavaScript libraries are wrapped up with Maven coordinates. Every library is found on the /webjars/<library>/<version>/<module> path. To top it off, Spring Boot comes with prebuilt support. Perhaps you noticed this buried in earlier console outputs: ...2014-05-20 08:33:09.062 ... : Mapped URL path [/webjars/**] onto handlerof [...... With jQuery added to our application, we can amp up our template (templates/modern.html) like this: <html><head><title>Learning Spring Boot - Chapter 1</title><script src="webjars/jquery/2.1.1/jquery.min.js"></script><script>$(document).ready(function() {$('p').animate({fontSize: '48px',}, "slow");});</script></head><body><p th_text="'Hello, ' + ${name}"></p><ol><li th_each="chapter : ${chapters}"th:text="${chapter}"></li></ol></body></html> What's different between this template and the previous one? It has a couple extra <script> tags in the head section: The first one loads jQuery from /webjars/jquery/2.1.1/jquery.min.js (implying that we can also grab jquery.js if we want to debug jQuery) The second script looks for the <p> element containing our Hello, world! message and then performs an animation that increases the font size to 48 pixels after the DOM is fully loaded into the browser If we run spring run modern.groovy and visit http://localhost:8080, then we can see this simple but stylish animation. It shows us that all of jQuery is available for us to work with on our application. Using Bower instead of WebJars WebJars isn't the only option when it comes to adding JavaScript to our app. More sophisticated UI developers might use Bower (http://bower.io), a popular JavaScript library management tool. WebJars are useful for Java developers, but not every library has been bundled as a WebJar. There is also a huge community of frontend developers more familiar with Bower and NodeJS that will probably prefer using their standard tool chain to do their jobs. We'll see how to plug that into our app. First, it's important to know some basic options. Spring Boot supports serving up static web resources from the following paths: /META-INF/resources/ /resources/ /static/ /public/ To craft a Bower-based app with Spring Boot, we first need to craft a .bowerrc file in the same folder we plan to create our Spring Boot CLI application. Let's pick public/ as the folder of choice for JavaScript modules and put it in this file, as shown in the following code: {"directory": "public/"} Do I have to use public? No. Again, you can pick any of the folders listed previously and Spring Boot will serve up the code. It's a matter of taste and semantics. Our first step towards a Bower-based app is to define our project by answering a series of questions (this only has to be done once): $ bower init[?] name: app_with_bower[?] version: 0.1.0[?] description: Learning Spring Boot - bower sample[?] main file:[?] what types of modules does this package expose? amd[?] keywords:[?] authors: Greg Turnquist <gturnquist@pivotal.io>[?] license: ASL[?] homepage: http://blog.greglturnquist.com/category/learning-springboot[?] set currently installed components as dependencies? No[?] add commonly ignored files to ignore list? Yes[?] would you like to mark this package as private which prevents it frombeing accidentally published to the registry? Yes...[?] Looks good? Yes Now that we have set our project, let's do something simple such as install jQuery with the following command: $ bower install jquery --savebower jquery#* cached git://github.com/jquery/jquery.git#2.1.1bower jquery#* validate 2.1.1 against git://github.com/jquery/jquery.git#* These two commands will have created the following bower.json file: {"name": "app_with_bower","version": "0.1.0","authors": ["Greg Turnquist <gturnquist@pivotal.io>"],"description": "Learning Spring Boot - bower sample","license": "ASL","homepage": "http://blog.greglturnquist.com/category/learningspring-boot","private": true,"ignore": ["**/.*","node_modules","bower_components","public/","test","tests"],"dependencies": {"jquery": "~2.1.1"}} It will also have installed jQuery 2.1.1 into our app with the following directory structure: public└── jquery├── MIT-LICENSE.txt├── bower.json└── dist├── jquery.js└── jquery.min.js We must include --save (two dashes) whenever we install a module. This ensures that our bower.json file is updated at the same time, allowing us to rebuild things if needed. The altered version of our app with WebJars removed should now look like this: @Grab("thymeleaf-spring4")@Controllerclass ModernApp {def chapters = ["Quick Start With Groovy","Quick Start With Java","Debugging and Managing Your App","Data Access with Spring Boot","Securing Your App"]@RequestMapping("/")def home(@RequestParam(value="name", defaultValue="World")String n) {new ModelAndView("modern_with_bower").addObject("name", n).addObject("chapters", chapters)}} The view name has been changed to modern_with_bower, so it doesn't collide with the previous template if found in the same folder. This version of the template, templates/modern_with_bower.html, should look like this: <html><head><title>Learning Spring Boot - Chapter 1</title><script src="jquery/dist/jquery.min.js"></script><script>$(document).ready(function() {$('p').animate({fontSize: '48px',}, "slow");});</script></head><body><p th_text="'Hello, ' + ${name}"></p><ol><li th_each="chapter : ${chapters}"th:text="${chapter}"></li></ol></body></html> The path to jquery is now jquery/dist/jquery.min.js. The rest is the same as the WebJars example. We just launch the app with spring run modern_with_bower.groovy and navigate to http://localhost:8080. (Might need to refresh the page to ensure loading of the latest HTML.) The animation should work just the same. The options shown in this section can quickly give us a taste of how easy it is to use popular JavaScript tools with Spring Boot. We don't have to fiddle with messy tool chains to achieve a smooth integration. Instead, we can use them the way they are meant to be used. What about an app that is all frontend with no backend? Perhaps we're building an app that gets all its data from a remote backend. In this age of RESTful backends, it's not uncommon to build a single page frontend that is fed data updates via AJAX. Spring Boot's Groovy support provides the perfect and arguably smallest way to get started. We do so by creating pure_javascript.groovy, as shown in the following code: @Controllerclass JsApp { } That doesn't look like much, but it accomplishes a lot. Let's see what this tiny fragment of code actually does for us: The @Controller annotation, like @RestController, causes Spring Boot to auto-configure Spring MVC. Spring Boot will launch an embedded Apache Tomcat server. Spring Boot will serve up static content from resources, static, and public. Since there are no Spring MVC routes in this tiny fragment of code, things will fall to resource resolution. Next, we can create a static/index.html page as follows: <html>Greetings from pure HTML which can, in turn, load JavaScript!</html> Run spring run pure_javascript.groovy and navigate to http://localhost:8080. We will see the preceding plain text shown in our browser as expected. There is nothing here but pure HTML being served up by our embedded Apache Tomcat server. This is arguably the lightest way to serve up static content. Use spring jar and it's possible to easily bundle up our client-side app to be installed anywhere. Spring Boot's support for static HTML, JavaScript, and CSS opens the door to many options. We can add WebJar annotations to JsApp or use Bower to introduce third-party JavaScript libraries in addition to any custom client-side code. We might just manually download the JavaScript and CSS. No matter what option we choose, Spring Boot CLI certainly provides a super simple way to add rich-client power for app development. To top it off, RESTful backends that are decoupled from the frontend can have different iteration cycles as well as different development teams. You might need to configure CORS (http://spring.io/understanding/CORS) to properly handle making remote calls that don't go back to the original server. Adding production-ready support features So far, we have created a Spring MVC app with minimal code. We added views and JavaScript. We are on the verge of a production release. Before deploying our rapidly built and modernized web application, we might want to think about potential issues that might arise in production: What do we do when the system administrator wants to configure his monitoring software to ping our app to see if it's up? What happens when our manager wants to know the metrics of people hitting our app? What are we going to do when the Ops center supervisor calls us at 2:00 a.m. and we have to figure out what went wrong? The last feature we are going to introduce in this article is Spring Boot's Actuator module and CRaSH remote shell support (http://www.crashub.org). These two modules provide some super slick, Ops-oriented features that are incredibly valuable in a production environment. We first need to update our previous code (we'll call it ops.groovy), as shown in the following code: @Grab("spring-boot-actuator")@Grab("spring-boot-starter-remote-shell")@Grab("org.webjars:jquery:2.1.1")@Grab("thymeleaf-spring4")@Controllerclass OpsReadyApp {@RequestMapping("/")def home(@RequestParam(value="name", defaultValue="World")String n) {new ModelAndView("modern").addObject("name", n)}} This app is exactly like the WebJars example with two key differences: it adds @Grab("spring-boot-actuator") and @Grab("spring-boot-starter-remote-shell"). When you run this version of our app, the same business functionality is available that we saw earlier, but there are additional HTTP endpoints available: Actuator endpoint Description /autoconfig This reports what Spring Boot did and didn't auto-configure and why /beans This reports all the beans configured in the application context (including ours as well as the ones auto-configured by Boot) /configprops This exposes all configuration properties /dump This creates a thread dump report /env This reports on the current system environment /health This is a simple endpoint to check life of the app /info This serves up custom content from the app /metrics This shows counters and gauges on web usage /mappings This gives us details about all Spring MVC routes /trace This shows details about past requests Pinging our app for general health Each of these endpoints can be visited using our browser or using other tools such as curl. For example, let's assume we ran spring run ops.groovy and then opened up another shell. From the second shell, let's run the following curl command: $ curl localhost:8080/health{"status":"UP"} This immediately solves our first need listed previously. We can inform the system administrator that he or she can write a management script to interrogate our app's health. Gathering metrics Be warned that each of these endpoints serves up a compact JSON document. Generally speaking, command-line curl probably isn't the best option. While it's convenient on *nix and Mac systems, the content is dense and hard to read. It's more practical to have: A JSON plugin installed in our browser (such as JSONView at http://jsonview.com) A script that uses a JSON parsing library if we're writing a management script (such as Groovy's JsonSlurper at http://groovy.codehaus.org/gapi/groovy/json/JsonSlurper.html or JSONPath at https://code.google.com/p/json-path) Assuming we have JSONView installed, the following screenshot shows a listing of metrics: It lists counters for each HTTP endpoint. According to this, /metrics has been visited four times with a successful 200 status code. Someone tried to access /foo, but it failed with a 404 error code. The report also lists gauges for each endpoint, reporting the last response time. In this case, /metrics took 2 milliseconds. Also included are some memory stats as well as the total CPUs available. It's important to realize that the metrics start at 0. To generate some numbers, you might want to first click on some links before visiting /metrics. The following screenshot shows a trace report: It shows the entire web request and response for curl localhost:8080/health. This provides a basic framework of metrics to satisfy our manager's needs. It's important to understand that metrics gathered by Spring Boot Actuator aren't persistent across application restarts. So to gather long-term data, we have to gather them and then write them elsewhere. With these options, we can perform the following: Write a script that gathers metrics every hour and appends them to a running spreadsheet somewhere else in the filesystem, such as a shared drive. This might be simple, but probably also crude. To step it up, we can dump the data into a Hadoop filesystem for raw collection and configure Spring XD (http://projects.spring.io/spring-xd/) to consume it. Spring XD stands for Spring eXtreme Data. It is an open source product that makes it incredibly easy to chain together sources and sinks comprised of many components, such as HTTP endpoints, Hadoop filesystems, Redis metrics, and RabbitMQ messaging. Unfortunately, there is no space to dive into this subject. With any monitoring, it's important to check that we aren't taxing the system too heavily. The same container responding to business-related web requests is also serving metrics data, so it will be wise to engage profilers periodically to ensure that the whole system is performing as expected. Detailed management with CRaSH So what can we do when we receive that 2:00 a.m. phone call from the Ops center? After either coming in or logging in remotely, we can access the convenient CRaSH shell we configured. Every time the app launches, it generates a random password for SSH access and prints this to the local console: 2014-06-11 23:00:18.822 ... : Configuring property ssh.port=2000 fromproperties2014-06-11 23:00:18.823 ... : Configuring property ssh.authtimeout=600000 fro...2014-06-11 23:00:18.824 ... : Configuring property ssh.idletimeout=600000 fro...2014-06-11 23:00:18.824 ... : Configuring property auth=simple fromproperties2014-06-11 23:00:18.824 ... : Configuring property auth.simple.username=user f...2014-06-11 23:00:18.824 ... : Configuring property auth.simple.password=bdbe4a... We can easily see that there's SSH access on port 2000 via a user if we use this information to log in: $ ssh -p 2000 user@localhostPassword authenticationPassword:. ____ _ __ _ _/\ / ___'_ __ _ _(_)_ __ __ _ ( ( )___ | '_ | '_| | '_ / _' | \/ ___)| |_)| | | | | || (_| | ) ) ) )' |____| .__|_| |_|_| |___, | / / / /=========|_|==============|___/=/_/_/_/:: Spring Boot :: (v1.1.6.RELEASE) on retina> There's a fistful of commands: help: This gets a listing of available commands dashboard: This gets a graphic, text-based display of all the threads, environment properties, memory, and other things autoconfig: This prints out a report of which Spring Boot auto-configuration rules were applied and which were skipped (and why) All of the previous commands have man pages: > man autoconfigNAMEautoconfig - Display auto configuration report fromApplicationContextSYNOPSISautoconfig [-h | --help]STREAMautoconfig <java.lang.Void, java.lang.Object>PARAMETERS[-h | --help]Display this help message... There are many commands available to help manage our application. More details are available at http://www.crashub.org/1.3/reference.html. Summary In this article, we learned about modernizing our Spring Boot app with JavaScript and adding production-ready support features. We plugged in Spring Boot's Actuator module as well as the CRaSH remote shell, configuring it with metrics, health, and management features so that we can monitor it in production by merely adding two lines of extra code. Resources for Article: Further resources on this subject: Getting Started with Spring Security [Article] Spring Roo 1.1: Working with Roo-generated Web Applications [Article] Spring Security 3: Tips and Tricks [Article]
Read more
  • 0
  • 0
  • 3132

Packt
26 Nov 2014
26 min read
Save for later

Ansible – An Introduction

Packt
26 Nov 2014
26 min read
In this article by Madhurranjan Mohan and Ramesh Raithatha, the authors of the book, Learning Ansible, have given an overview of the basic features of Ansible, right from the installation part till the deployment. (For more resources related to this topic, see here.) What is Ansible? Ansible is an orchestration engine in IT, which can be used for several use cases, such as configuration management, orchestration, provisioning, and deployment. Compared to other automation tools, Ansible brings you an easy way to configure your orchestration engine without the overhead of a client or central server setup. That's right! No central server! It comes preloaded with a wide range of modules that make your life simpler. Ansible is an open source tool (with enterprise editions available) developed using Python and runs on Windows, Mac, and Unix-like systems. You can use Ansible for configuration management, orchestration, provisioning, and deployments, which covers many of the problems that are solved under the broad umbrella of DevOps. We won't be talking about culture here as that's a book by itself! You could refer to the book, Continuous Delivery and DevOps – A Quickstart Guide by Packt Publishing for more information at https://www.packtpub.com/virtualization-and-cloud/continuous-delivery-and-devops-quickstart-guide. Let's try to answer some questions that you may have right away. Can I use Ansible if I am starting afresh, have no automation in my system, and would like to introduce that (and as a result, increase my bonus for the next year)? A short answer to this question is Ansible is probably perfect for you. The learning curve with Ansible is way shorter than most other tools currently present in the market. I have other tools in my environment. Can I still use Ansible? Yes, again! If you already have other tools in your environment, you can still augment those with Ansible as it solves many problems in an elegant way. A case in point is a puppet shop that uses Ansible for orchestration and provisioning of new systems but continues to use Puppet for configuration management. I don't have Python in my environment and introducing Ansible would bring in Python. What do I do? You need to remember that, on most Linux systems, a version of Python is present at boot time, and you don't have to explicitly install Python. You should still go ahead with Ansible if it solves particular problems for you. Always question what problems you are trying to solve and then check whether a tool such as Ansible would solve that use case. I have no configuration management at present. Can I start today? The answer is yes! In many of the conferences we presented, the preceding four questions popped up most frequently. Now that these questions are answered, let's dig deeper. The architecture of Ansible is agentless. Yes, you heard that right; you don't have to install any client-side software. It works purely on SSH connections, in which case you can consider SSH as your agent and our previous statement that Ansible is agentless is not entirely right. However, SSH is almost assumed to run on every server that its consider as a separate agent. Hence, they call Ansible agentless. So, if you have a well-oiled SSH setup, then you're ready to roll Ansible into your environment in no time. This also means that you can install it only on one system (either a Linux or Mac machine) and you can control your entire infrastructure from that machine. Yes, we understand that you must be thinking about what happens if this machine goes down. You would probably have multiple such machines in production, but this was just an example to elucidate the simplicity of Ansible. As Ansible works on SSH connections, it would be slower; in order to speedup default SSH connections, you can always enable ControlPersist and the pipeline mode, which makes Ansible faster and secure. Ansible works like any other Unix command that doesn't require any daemon process to be running all the time. So you would either run it via a cron, on demand from a single node, or at startup. Ansible can be push or pull based and you can utilize whatever suits you. When you start with something new, the first aspect you need to pay attention to is the nomenclature. The faster you're able to pick up the terms associated with the tool, the faster you're comfortable with that tool. So, to deploy, let's say, a package on one or more machines in Ansible, you would need to write a playbook that has a single task, which in turn uses the package module that would then go ahead and install the package based on an inventory file that contains a list of these machines. If you feel overwhelmed by the nomenclature, don't worry, you'll soon get used to it. Similar to the package module, Ansible comes loaded with more than 200 modules, purely written in Python. We will talk about modules in detail later. It is now time to install Ansible to start trying out various fun examples. Installing Ansible Installing Ansible is rather quick and simple. You can directly use the source code by cloning it from the GitHub project (https://github.com/ansible/ansible), install it using your system's package manager, or use Python's package management tool (pip). You can use Ansible on any Windows, Mac, or Unix-like system. Ansible doesn't require any database and doesn't need any daemons running. This makes it easier to maintain the Ansible versions and upgrade without any breaks. We'd like to call the machine where we will install Ansible our command center. Many people also refer to it as the Ansible workstation. Note that, as Ansible is developed using Python, you would need Python Version 2.4 or a higher version installed. Python is preinstalled, as specified earlier, on the majority of operating systems. If this is not the case for you, refer to https://wiki.python.org/moin/BeginnersGuide/Download to download/upgrade Python. Installing Ansible from source Installing from source is as easy as cloning a repository. You don't require any root permissions while installing from source. Let's clone a repository and activate virtualenv, which is an isolated environment in Python where you can install packages without interfering with the system's Python packages. The command is as follows: $ git clone git://github.com/ansible/ansible.git $ cd ansible/ $ source ./hacking/env-setup Ansible needs a couple of Python packages, which you can install using pip. If you don't have pip installed on your system, install it using the following command: $ sudo easy_install pip Once you have installed pip, install the paramiko, PyYAML, jinja2, and httplib2 packages using the following command lines: $ sudo pip install paramiko PyYAML jinja2 httplib2 By default, Ansible will be running against the development branch. You might want to check out the latest stable branch. Check what the latest stable version is using the following command line: $ git branch -a Copy the latest version you want to use. Version 1.7.1 was the latest version available at the time of writing. Check the latest version you would like to use using the following command lines: $ git checkout release1.7.1 You now have a working setup of Ansible ready. One of the benefits of running Ansible through source is that you can enjoy the benefits of new features immediately without waiting for your package manager to make them available for you. Installing Ansible using the system's package manager Ansible also provides a way to install itself using the system's package manager. We will look into installing Ansible via Yum, Apt, Homebrew, and pip. Installing via Yum If you are running a Fedora system, you can install Ansible directly. For CentOS- or RHEL-based systems, you should add the EPEL repository first, as follows: $ sudo yum install ansible On CentOS 6 or RHEL 6, you have to run the command rpm -Uvh. Refer to http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm for instructions on how to install EPEL. You can also install Ansible from an RPM file. You need to use the make rpm command against the git checkout of Ansible, as follows: $ git clone git://github.com/ansible/ansible.git $ cd ./ansible $ make rpm $ sudo rpm -Uvh ~/rpmbuild/ansible-*.noarch.rpm You should have rpm-build, make, and python2-devel installed on your system to build an rpm. Installing via Apt Ansible is available for Ubuntu in a Personal Package Archive (PPA). To configure the PPA and install Ansible on your Ubuntu system, use the following command lines: $ sudo apt-get install apt-add-repository $ sudo apt-add-repository ppa:rquillo/ansible $ sudo apt-get update $ sudo apt-get install ansible You can also compile a deb file for Debian and Ubuntu systems, using the following command line: $ make deb Installing via Homebrew You can install Ansible on Mac OSX using Homebrew, as follows: $ brew update $ brew install ansible Installing via pip You can install Ansible via Python's package manager pip. If you don't have pip installed on your system, install it. You can use pip to install Ansible on Windows too, using the following command line: $ sudo easy_install pip You can now install Ansible using pip, as follows: $ sudo pip install ansible Once you're done installing Ansible, run ansible --version to verify that it has been installed: $ ansible –version You will get the following as the output of the preceding command line: ansible 1.7.1 Hello Ansible Let's start by checking if two remote machines are reachable; in other words, let's start by pinging two machines following which we'll echo hello ansible on the two remote machines. The following are the steps that need to be performed: Create an Ansible inventory file. This can contain one or more groups. Each group is defined within square brackets. This example has one group called servers: $ cat inventory [servers] machine1 machine2 Now, we have to ping the two machines. In order to do that, first run ansible --help to view the available options, as shown below (only copying the subset that we need for this example): ansible --help Usage: ansible <host-pattern> [options] Options: -a MODULE_ARGS, --args=MODULE_ARGS          module arguments -i INVENTORY, --inventory-file=INVENTORY          specify inventory host file          (default=/etc/ansible/hosts) -m MODULE_NAME, --module-name=MODULE_NAME          module name to execute          (default=command) We'll now ping the two servers using the Ansible command line, as shown in the following screenshot: Now that we can ping these two servers, let's echo hello ansible!, using the command line shown in the following screenshot: Consider the following command: $ansible servers -i inventory -a '/bin/echo hello ansible!' The preceding command line is the same as the following one: $ansible servers -i inventory -m command -a '/bin/echo hello ansible!'. If you move the inventory file to /etc/ansible/hosts, the Ansible command will become even simpler, as follows: $ ansible servers -a '/bin/echo hello ansible!' There you go. The 'Hello Ansible' program works! Time to tweet! You can also specify the inventory file by exporting it in a variable named ANSIBLE_HOSTS. The preceding command, without the –i option, will work even in that situation. Developing a playbook In Ansible, except for ad hoc tasks that are run using the ansible command, we need to make sure we have playbooks for every other repeatable task. In order to do that, it is important to have a local development environment, especially when a larger team is involved, where people can develop their playbooks and test them before checking them into Git. A very popular tool that currently fits this bill is Vagrant. Vagrant's aim is to help users create and configure lightweight, reproducible, and portable development environments. By default, Vagrant works on VirtualBox, which can run on a local laptop or desktop. To elaborate further, it can be used for the following use cases: Vagrant can be used when creating development environments to constantly check new builds of software, especially when you have several other dependent components. For example, if I am developing service A and it depends on two other services, B and C, and also a database, then the fastest way to test the service locally is to make sure the dependent services and the database are set up (especially if you're testing multiple versions), and every time you compile the service locally, you deploy the module against these services and test them out. Testing teams can use Vagrant to deploy the versions of code they want to test and work with them, and each person in the testing team can have local environments on their laptop or desktop rather than common environments that are shared between teams. If your software is developed for cloud-based platforms and needs to be deployed on AWS and Rackspace (for example), apart from testing it locally on VMware Fusion or VirtualBox, Vagrant is perfect for this purpose. In Vagrant's terms, you can deploy your software on multiple providers with a configuration file that differs only for a provider. For example, the following screenshot shows the VirtualBox configuration for a simple machine: The following is the AWS configuration for a simple machine: As you can see, the provider configuration changes but the rest of the configuration remains more or less the same. (Private IP is virtual-box-specific but it is ignored when run using the AWS provider.) Vagrant also provides provisioners. Vagrant provides users with multiple options to configure new machines as they come up using provisioners. They support shell scripts, tools such as Chef, Puppet, Salt, and Docker, as well as Ansible. By using Ansible with Vagrant, you can develop your Ansible scripts locally, deploy, and redeploy them as many times as needed to get them right, and then check them in. The advantage, from an infrastructure point of view, is that the same code can also be used by other developers and testers when they spawn off their vagrant environments for testing (The software would be configured to work in the expected manner by Ansible playbooks.). The checked-in Ansible code will then flow like the rest of your software, through testing and stage environments, before they are finally deployed into Production. Roles When you start thinking about your infrastructure, you will soon look at the purposes each node in your infrastructure is solving and you will be able to categorize them. You will also start to abstract out information regarding nodes and start thinking at a higher level. For example, if you're running a web application, you'll be able to categorize them broadly as db_servers, app_servers, web servers, and load balancers. If you then talk to your provisioning team, they will tell you which base packages need to be installed on each machine, either for the sake of compliance or to manage them remotely after choosing the OS distribution or for security purposes. Simple examples can be packages such as bind, ntp, collectd, psacct, and so on. Soon you will add all these packages under a category named common or base. As you dig deeper, you might find further dependencies that exist. For example, if your application is written in Java, having some version of JDK is a dependency. So, for what we've discussed so far, we have the following categories: db_servers app_servers web_servers load_balancers common jdk We've taken a top-down approach to come up with the categories listed. Now, depending on the size of your infrastructure, you will slowly start identifying reusable components, and these can be as simple as ntp or collectd. These categories, in Ansible's terms, are called Roles. If you're familiar with Chef, the concept is very similar. Callback plugins One of the features that Ansible provides is a callback mechanism. You can configure as many callback plugins as required. These plugins can intercept events and trigger certain actions. Let's see a simple example where we just print the run results at the end of the playbook run as part of a callback and then take a brief look at how to configure a callback. We first use grep for the location of callback_plugins in ansible.cfg as follows: $ grep callback ansible.cfg callback_plugins   = /usr/share/ansible_plugins/callback_plugins We then create our callback plugin in this location. $ ls -l /usr/share/ansible_plugins/callback_plugins callback_sample.py Let's now look at the contents of the callback_sample file: $ cat /usr/share/ansible_plugins/callback_plugins/callback_sample.py class CallbackModule(object): def on_any(self, *args, **kwargs):    pass def runner_on_failed(self, host, res, ignore_errors=False):    pass def runner_on_ok(self, host, res):    pass def runner_on_error(self, host, msg):    pass def runner_on_skipped(self, host, item=None):    pass def runner_on_unreachable(self, host, res):    pass def runner_on_no_hosts(self):    pass def runner_on_async_poll(self, host, res, jid, clock):    pass def runner_on_async_ok(self, host, res, jid):    pass def runner_on_async_failed(self, host, res, jid):    pass def playbook_on_start(self):    pass def playbook_on_notify(self, host, handler):    pass def playbook_on_no_hosts_matched(self):    pass def playbook_on_no_hosts_remaining(self):    pass def playbook_on_task_start(self, name, is_conditional):    pass def playbook_on_vars_prompt(self, varname, private=True, prompt=None, encrypt=None, confirm=False, salt_size=None, salt=None, default=None):    pass def playbook_on_setup(self):    pass def playbook_on_import_for_host(self, host, imported_file):    pass def playbook_on_not_import_for_host(self, host, missing_file):    pass def playbook_on_play_start(self, pattern):    pass def playbook_on_stats(self, stats):    results = dict([(h, stats.summarize(h)) for h in stats.processed])    print "Run Results: %s" % results As you can see, the callback class, CallbackModule, contains several methods. The methods of this class are called and the Ansible run parameters are provided as parameters to these methods. Playbook activities can be intercepted by using these methods and relevant actions can be taken based on that. Relevant methods are called based on the action, for example, we've used the playbook_on_stats method (in bold) to display statistics regarding the playbook run. Let's run a basic playbook with the callback plugin and view the output as follows: In the preceding screenshot, you can now see the Run Results line right at the end consisting of a dictionary or hash of the actual results. This is due to our custom code. This is just an example of how you can intercept methods and use them to your advantage. You can utilize this information in any number of ways. Isn’t this powerful? Are you getting any ideas around how you can utilize a feature such as this? Do write it down before reading further. If you’re able to write out even two use cases that we’ve not covered here and is relevant to your infrastructure, give yourself a pat on the back! Modules Ansible allows you to extend functionality using custom modules. Arguments , as you have seen can be passed to the modules. The arguments that you pass, provided they are in a key value format, will be forwarded in a separate file along with the module. Ansible expects at least two variables in your module output, that is, result of the module run, whether it passed or failed, and a message for the user and they have to be in JSON format. If you adhere to this simple rule, you can customize as much as you want and the module can be written in any language. Using Bash modules Bash modules in Ansible are no different than any other bash scripts, except the way it prints the data on stdout. Bash modules could be as simple as checking if a process is running on the remote host to running some complex commands. We recommend that you use bash over other languages, such as Python and Ruby only when you're performing simple tasks. In other cases, you should use languages that provide better error handling. Let's see an example for the bash module as follows: The preceding bash module will take the service_name argument and forcefully kill all of the Java processes that belong to that service. As you know, Ansible passes the argument file to the module. We then source the arguments file using source $1. This will actually set the environment variable with the name, service_name. We then access this variable using $service_name as follows: We then check to see if we obtained any PIDs for the service and run a loop over it to forcefully kill all of the Java processes that match service_name. Once they're killed, we exit the module with failed=False and a message with an exit code of 0, as shown in the following screenshot, because terminating the Ansible run might not make sense: Provisioning a machine in the cloud With that, let's jump to the first topic. Teams managing infrastructures have a lot of choices today to run their builds, tests, and deployments. Providers such as Amazon, Rackspace, and DigitalOcean primarily provide Infrastructure as a Service (IAAS). They expose an API via SDKs, which you can invoke in order to create new machines, or use their GUI to set it up. We're more interested in using their SDK as it will play an important part in our automation effort. Setting up new servers and provisioning them is interesting at first but at some stage it can become boring as it's quite repetitive in nature. Each provisioning step would involve several similar steps to get them up-and-running. Imagine one fine morning you receive an e-mail asking for three new customer setups, where each customer setup has three to four instances and a bunch of services and dependencies. This might be an easy task for you, but would require running the same set of repetitive commands multiple times, followed by monitoring the servers once they come up to confirm that everything just went fine. In addition, anything you do manually has a chance of introducing bugs. What if two of the customer setups come up correctly but, due to fatigue, you miss out a step for the third customer and hence introduce a bug? To deal with such situations, there exists automation. Cloud provisioning automation makes it easy for an engineer to build up a new server as quickly as possible, allowing him/her to concentrate on other priorities. Using Ansible, you can easily perform these actions and automate cloud provisioning with minimal effort. Ansible provides you with the power to automate various different cloud platforms, such as Amazon, DigitalOcean, Google Cloud, Rackspace, and so on, with modules for different services available in the Ansible core. Docker provisioning Docker is perhaps the most popular open source tool that has been released in the last year. The following quote can be seen on the Docker website: Docker is an open platform for developers and sysadmins to build, ship, and run distributed applications, whether on laptops, data center VMs, or the cloud. Increasingly, more and more individuals and companies are adopting Docker. The tagline for Docker is Containerization is the new virtualization. At a high level, all Docker allows you to do is prepare lightweight containers using instructions from a Dockerfile and run the container. The same container can be shared or shipped across environments, thereby making sure you run the exact same image and reducing the chance of errors. The Docker image that you build is cached by default; thus, the next time you have similar instructions, the time taken to bring up a similar container is reduced to almost nothing. Let's now look at how Ansible can be used with Docker to make this a powerful working combination. You can use Ansible with Docker to perform the following: Installing Docker on hosts Deploying new Docker images Building or provisioning new Docker images Deployment strategies with RPM In most cases, we already have a certain version of the application that has been deployed and now, either to introduce a new feature or fix bugs, a new version has to be deployed. We'd like to discuss this scenario in greater detail. At a high level, whenever we deploy an application, there are three kinds of changes to take care of: Code changes Config changes Database changes The first two types are ideally handled by the RPM, unless you have very specific values that need to be set up during runtime. Files with passwords can be checked but they should be encrypted with Ansible Vault or dropped into files as templates during run time, just as we did with database.yml. With templates, if the configs are ideally just name-value pairs that can be handled in a Jinja template, you should be good, but if you have other lines in the configuration that do not change, then it's better that those configuration files are checked and appear as part of the RPM. Many teams we've worked with check environment-specific folders that have all the configuration parameters; while starting the application, we provide the environment in which the application should be started. Another way is to deploy the RPM with default values for all configuration properties while writing a different folder with name-value pairs that override the parameters in the default configuration that is part of the RPM. The database changes should be handled carefully. Ideally, you want them to be idempotent for a particular version of the RPM so that, even if someone tries to push database changes multiple times, the database is not really affected. For example, in the preceding case, we run rake db:migrate that is idempotent in nature; even if you run the same command from multiple machines, you shouldn't really face issues. The Rails framework does it by storing the database migration version in a separate table. Having looked at the three types of changes, we can now examine how to perform rpm deployments for each release. When you're pushing new changes, the current version or service is already running. It's recommended that you take the server out of service before performing upgrades. For example, if it's part of a load balancer, make sure it's out of the load balancer and not serving any traffic before performing the upgrades. Primarily, there are the following two ways: Deploying newer versions of rpm in the same directory Deploying the rpm into a version-specific directory Canary deployment The name Canary is used with reference to the canaries that were often sent in to coal mines to alert miners about toxic gases reaching dangerous levels. Canary deployment is a technique that is used to reduce the risk of releasing a new version of software by first releasing it to a small subset of users (demographically, location-wise, and so on), gauging their reaction, and then slowly releasing it to the rest of the users. Whenever possible, keep the first set of users as internal users, since it reduces the impact on the actual customers. This is especially useful when you introduce a new feature that you want to test with a small subset of users before releasing it to the rest of your customer base. If you're running, let's say, an application across multiple data centers and you're sure that certain users would only contact specific data centers when they access your site, you could definitely run a Canary deployment. Deploying Ansible pull The last topic we'd like to cover in this section is Ansible pull. If you have a large number of hosts that you'd like to release software on simultaneously, you will be limited by the number of parallel SSH connections that can be run. At scale, the pull model is preferred to the push model. Ansible supports what is called as Ansible pull. Ansible pull works individually on each node. The prerequisite is that it points to a repository from where it can invoke a special file called localhost.yml or <hostname>.yml. Typically, the ansible-pull option is run either as a cron job or is triggered remotely by some other means. We're going to use our tomcat example again, with the only difference being that the structure of the repository has been changed slightly. Let's look at the structure of the git repository that will work for Ansible pull as follows: As you can see, localhost.yml is present at the top level and the roles folder consists of the tomcat folder, under which is the tasks folder with the main.yml task file. Let's now run the playbook using ansible-pull as follows: Let's look at the preceding run in detail as follows: The ansible-pull command: We invoke ansible-pull with the following options: –o: This option means that the Ansible run takes place only if the remote repository has changes. –C master: This option indicates which branch to refer to in the git repository. –U < >: This option indicates the repository that needs to be checked out. –i localhost: This option indicates the inventory that needs to be considered. In this case, since we're only concerned about one tomcat group, we use -i localhost. However, when there are many more inventory groups, make sure you use an inventory file with the –i option. The localhost | success JSON: This option checks whether the repository has changed and lists the latest commit. The actual Ansible playbook run: The Ansible playbook run is just like before. At the end of the run, we will have the WAR file up and running. Summary In this article, we got an overview of Ansible, we looked at the introduction to Ansible, how to install Ansible in various ways, wrote our very first playbook, learned ways to use the Ansible command line, how to debug playbooks, and learned how to develop a playbook on our own. We also looked into the various aspects of Ansible such as Roles, callback plugins, modules and the Bash module, how to provision a machine in the cloud, Docker provisioning, the deployments strategies with RPM, Canary deployment and deployment using Ansible pull. Resources for Article: Further resources on this subject: Getting Started with Ansible [Article] Module, Facts, Types and Reporting tools in Puppet [Article] Designing Puppet Architectures [Article]
Read more
  • 0
  • 0
  • 2276

article-image-concurrency-practice
Packt
26 Nov 2014
25 min read
Save for later

Concurrency in Practice

Packt
26 Nov 2014
25 min read
This article written by Aleksandar Prokopec, the author of Learning Concurrent Programming in Scala, helps you develop skills that are necessary to write correct and efficient concurrent programs. It teaches you about concurrency in Scala through a sequence of programs. (For more resources related to this topic, see here.) "The best theory is inspired by practice."                                          -Donald Knuth We have studied a plethora of different concurrency facilities in this article. By now, you will have learned about dozens of different ways of starting concurrent computations and accessing shared data. Knowing how to use different styles of concurrency is useful, but it might not yet be obvious when to use which. The goal of this article is to introduce the big picture of concurrent programming. We will study the use cases for various concurrency abstractions, see how to debug concurrent programs, and how to integrate different concurrency libraries in larger applications. In this article, we perform the following tasks: Investigate how to deal with various kinds of bugs appearing in concurrent applications Learn how to identify and resolve performance bottlenecks Apply the previous knowledge about concurrency to implement a larger concurrent application, namely, a remote file browser We start with an overview of the important concurrency frameworks that we learned about in this article, and a summary of when to use each of them. Choosing the right tools for the job In this section, we present an overview of the different concurrency libraries that we learned about. We take a step back and look at the differences between these libraries, and what they have in common. This summary will give us an insight into what different concurrency abstractions are useful for. A concurrency framework usually needs to address several concerns: It must provide a way to declare data that is shared between concurrent executions It must provide constructs for reading and modifying program data It must be able to express conditional execution, triggered when a certain set of conditions are fulfilled It must define a way to start concurrent executions Some of the frameworks from this article address all of these concerns; others address only a subset, and transfer part of the responsibility to another framework. Typically, in a concurrent programming model, we express concurrently shared data differently from data intended to be accessed only from a single thread. This allows the JVM runtime to optimize sequential parts of the program more effectively. So far, we've seen a lot of different ways to express concurrently shared data, ranging from the low-level facilities to advanced high-level abstractions. We summarize different data abstractions in the following table: Data abstraction Datatype or annotation Description Volatile variables (JDK) @volatile Ensure visibility and the happens-before relationship on class fields and local variables that are captured in closures. Atomic variables (JDK) AtomicReference[T] AtomicInteger AtomicLong Provide basic composite atomic operations, such as compareAndSet and incrementAndGet. Futures and promises (scala.concurrent) Future[T] Promise[T] Sometimes called single-assignment variables, these express values that might not be computed yet, but will eventually become available. Observables and subjects (Rx) Observable[T] Subject[T] Also known as first-class event streams, these describe many different values that arrive one after another in time. Transactional references (Scala Software Transactional Memory (STM)) Ref[T] These describe memory locations that can only be accessed from within memory transactions. Their modifications only become visible after the transaction successfully commits. The next important concern is providing access to shared data, which includes reading and modifying shared memory locations. Usually, a concurrent program uses special constructs to express such accesses. We summarize the different data access constructs in the following table: Data abstraction Data access constructs Description Arbitrary data (JDK) synchronized   Uses intrinsic object locks to exclude access to arbitrary shared data. Atomic variables and classes (JDK) compareAndSet Atomically exchanges the value of a single memory location. It allows implementing lock-free programs. Futures and promises (scala.concurrent) value tryComplete Used to assign a value to a promise, or to check the value of the corresponding future. The value method is not a preferred way to interact with a future. Transactional references (ScalaSTM) atomic orAtomic single Atomically modify the values of a set of memory locations. Reduces the risk of deadlocks, but disallow side effects inside the transactional block. Concurrent data access is not the only concern of a concurrency framework. Concurrent computations sometimes need to proceed only after a certain condition is met. In the following table, we summarize different constructs that enable this: Concurrency framework Conditional execution constructs Description JVM concurrency wait notify notifyAll Used to suspend the execution of a thread until some other thread notifies that the conditions are met. Futures and promises onComplete Await.ready Conditionally schedules an asynchronous computation. The Await.ready method suspends the thread until the future completes. Reactive extensions subscribe Asynchronously or synchronously executes a computation when an event arrives. Software transactional memory retry retryFor withRetryTimeout Retries the current memory transaction when some of the relevant memory locations change. Actors receive Executes the actor's receive block when a message arrives. Finally, a concurrency model must define a way to start a concurrent execution. We summarize different concurrency constructs in the following table: Concurrency framework Concurrency constructs Description JVM concurrency Thread.start Starts a new thread of execution. Execution contexts execute Schedules a block of code for execution on a thread pool. Futures and promises Future.apply Schedules a block of code for execution, and returns the future value with the result of the execution. Parallel collections par Allows invoking data-parallel versions of collection methods. Reactive extensions Observable.create observeOn The create method defines an event source. The observeOn method schedules the handling of events on different threads. Actors actorOf Schedules a new actor object for execution. This breakdown shows us that different concurrency libraries focus on different tasks. For example, parallel collections do not have conditional waiting constructs, because a data-parallel operation proceeds on separate elements independently. Similarly, software transactional memory does not come with a construct to express concurrent computations, and focuses only on protecting access to shared data. Actors do not have special constructs for modeling shared data and protecting access to it, because data is encapsulated within separate actors and accessed serially only by the actor that owns it. Having classified concurrency libraries according to how they model shared data and express concurrency, we present a summary of what different concurrency libraries are good for: The classical JVM concurrency model uses threads, the synchronized statement, volatile variables, and atomic primitives for low-level tasks. Uses include implementing a custom concurrency utility, a concurrent data structure, or a concurrency framework optimized for specific tasks. Futures and promises are best suited for referring to concurrent computations that produce a single result value. Futures model latency in the program, and allow composing values that become available later during the execution of the program. Uses include performing remote network requests and waiting for replies, referring to the result of an asynchronous long-running computation, or reacting to the completion of an I/O operation. Futures are usually the glue of a concurrent application, binding the different parts of a concurrent program together. We often use futures to convert single-event callback APIs into a standardized representation based on the Future type. Parallel collections are best suited for efficiently executing data-parallel operations on large datasets. Usages include file searching, text processing, linear algebra applications, numerical computations, and simulations. Long-running Scala collection operations are usually good candidates for parallelization. Reactive extensions are used to express asynchronous event-based programs. Unlike parallel collections, in reactive extensions, data elements are not available when the operation starts, but arrive while the application is running. Uses include converting callback-based APIs, modeling events in user interfaces, modeling events external to the application, manipulating program events with collection-style combinators, streaming data from input devices or remote locations, or incrementally propagating changes in the data model throughout the program. Use STM to protect program data from getting corrupted by concurrent accesses. An STM allows building complex data models and accessing them with the reduced risk of deadlocks and race conditions. A typical use is to protect concurrently accessible data, while retaining good scalability between threads whose accesses to data do not overlap. Actors are suitable for encapsulating concurrently accessible data, and seamlessly building distributed systems. Actor frameworks provide a natural way to express concurrent tasks that communicate by explicitly sending messages. Uses include serializing concurrent access to data to prevent corruption, expressing stateful concurrency units in the system, and building distributed applications like trading systems, P2P networks, communication hubs, or data mining frameworks. Advocates of specific programming languages, libraries, or frameworks might try to convince you that their technology is the best for any task and any situation, often with the intent of selling it. Richard Stallman once said how computer science is the only industry more fashion-driven than women's fashion. As engineers, we need to know better than to succumb to programming fashion and marketing propaganda. Different frameworks are tailored towards specific use cases, and the correct way to choose a technology is to carefully weigh its advantages and disadvantages when applied to a specific situation. There is no one-size-fits-all technology. Use your own best judgment when deciding which concurrency framework to use for a specific programming task. Sometimes, choosing the best-suited concurrency utility is easier said than done. It takes a great deal of experience to choose the correct technology. In many cases, we do not even know enough about the requirements of the system to make an informed decision. Regardless, a good rule of thumb is to apply several concurrency frameworks to different parts of the same application, each best suited for a specific task. Often, the real power of different concurrency frameworks becomes apparent when they are used together. This is the topic of the next section. Putting it all together – a remote file browser In this section, we use our knowledge about different concurrency frameworks to build a remote file browser. This larger application example illustrates how different concurrency libraries work together, and how to apply them to different situations. We will name our remote file browser ScalaFTP. The ScalaFTP browser is divided into two main components: the server and the client process. The server process will run on the machine whose filesystem we want to manipulate. The client will run on our own computer, and comprise of a graphical user interface used to navigate the remote filesystem. To keep things simple, the protocol that the client and the server will use to communicate will not really be FTP, but a custom communication protocol. By choosing the correct concurrency libraries to implement different parts of ScalaFTP, we will ensure that the complete ScalaFTP implementation fits inside just 500 lines of code. Specifically, the ScalaFTP browser will implement the following features: Displaying the names of the files and the directories in a remote filesystem, and allow navigating through the directory structure Copying files between directories in a remote filesystem Deleting files in a remote filesystem To implement separate pieces of this functionality, we will divide the ScalaFTP server and client programs into layers. The task of the server program is to answer to incoming copy and delete requests, and to answer queries about the contents of specific directories. To make sure that its view of the filesystem is consistent, the server will cache the directory structure of the filesystem. We divide the server program into two layers: the filesystem API and the server interface. The filesystem API will expose the data model of the server program, and define useful utility methods to manipulate the filesystem. The server interface will receive requests and send responses back to the client. Since the server interface will require communicating with the remote client, we decide to use the Akka actor framework. Akka comes with remote communication facilities. The contents of the filesystem, that is, its state, will change over time. We are therefore interested in choosing proper constructs for data access. In the filesystem API, we can use object monitors and locking to synchronize access to shared state, but we will avoid these due to the risk of deadlocks. We similarly avoid using atomic variables, because they are prone to race conditions. We could encapsulate the filesystem state within an actor, but note that this can lead to a scalability bottleneck:an actor would serialize all accesses to the filesystem state. Therefore, we decide to use the ScalaSTM framework to model the filesystem contents. An STM avoids the risk of deadlocks and race conditions, and ensures good horizontal scalability. The task of the client program will be to graphically present the contents of the remote filesystem, and communicate with the server. We divide the client program into three layers of functionality. The GUI layer will render the contents of the remote filesystem and register user requests such as button clicks. The client API will replicate the server interface on the client side and communicate with the server. We will use Akka to communicate with the server, but expose the results of remote operations as futures. Finally, the client logic will be a gluing layer, which binds the GUI and the client API together. The architecture of the ScalaFTP browser is illustrated in the following diagram, in which we indicate which concurrency libraries will be used by separate layers. The dashed line represents the communication path between the client and the server: We now start by implementing the ScalaFTP server, relying on the bottom-up design approach. In the next section, we will describe the internals of the filesystem API. Modeling the filesystem We used atomic variables and concurrent collections to implement a non-blocking, thread-safe filesystem API, which allowed copying files and retrieving snapshots of the filesystem. In this section, we repeat this task using STM. We will see that it is much more intuitive and less error-prone to use an STM. We start by defining the different states that a file can be in. The file can be currently created, in the idle state, being copied, or being deleted. We model this with a sealed State trait, and its four cases: sealed trait Statecase object Created extends Statecase object Idle extends Statecase class Copying(n: Int) extends Statecase object Deleted extends State A file can only be deleted if it is in the idle state, and it can only be copied if it is in the idle state or in the copied state. Since a file can be copied to multiple destinations at a time, the Copying state encodes how many copies are currently under way. We add the methods inc and dec to the State trait, which return a new state with one more or one fewer copy, respectively. For example, the implementation of inc and dec for the Copying state is as follows: def inc: State = Copying(n + 1)def dec: State = if (n > 1) Copying(n - 1) else Idle Similar to the File class in the java.io package, we represent both the files and directories with the same entity, and refer to them more generally as files. Each file is represented by the FileInfo class that encodes the path, its name, its parent directory, and the date of the last modification to the file; a Boolean value denoting if the file is a directory, the size of the file, and its State object. The FileInfo class is immutable, and updating the state of the file will require creating a fresh FileInfo object: case class FileInfo(path: String, name: String,parent: String, modified: String, isDir: Boolean,size: Long, state: State) We separately define the factory methods apply and creating that take a File object and return a FileInfo object in the Idle or Created state, respectively. Depending on where the server is started, the root of the ScalaFTP directory structure is a different subdirectory in the actual filesystem. A FileSystem object tracks the files in the given rootpath directory, using a transactional map called files: class FileSystem(val rootpath: String) {val files = TMap[String, FileInfo]()} We introduce a separate init method to initialize the FileSystem object. The init method starts a transaction, clears the contents of the files map, and traverses the files and directories under rootpath using the Apache Commons IO library. For each file and directory, the init method creates a FileInfo object and adds it to the files map, using its path as the key: def init() = atomic { implicit txn =>files.clear()val rootDir = new File(rootpath)val all = TrueFileFilter.INSTANCEval fileIterator =FileUtils.iterateFilesAndDirs(rootDir, all, all).asScalafor (file <- fileIterator) {val info = FileInfo(file)files(info.path) = info} Recall that the ScalaFTP browser must display the contents of the remote filesystem. To enable directory queries, we first add the getFileList method to the FileSystem class, which retrieves the files in the specified dir directory. The getFileList method starts a transaction and filters the files whose direct parent is equal to dir: def getFileList(dir: String): Map[String, FileInfo] =atomic { implicit txn =>files.filter(_._2.parent == dir)} We implement the copying logic in the filesystem API with the copyFile method. This method takes a path to the src source file and the dest destination file, and starts a transaction. After checking whether the dest destination file exists or not, the copyFile method inspects the state of the source file entry, and fails unless the state is Idle or Copying. It then calls inc to create a new state with the increased copy count, and updates the source file entry in the files map with the new state. Similarly, the copyFile method creates a new entry for the destination file in the files map. Finally, the copyFile method calls the afterCommit handler to physically copy the file to disk after the transaction completes. Recall that it is not legal to execute side-effecting operations from within the transaction body, so the private copyOnDisk method is called only after the transaction commits: def copyFile(src: String, dest: String) = atomic { implicit txn =>val srcfile = new File(src)val destfile = new File(dest)val info = files(src)if (files.contains(dest)) sys.error(s"Destination exists.")info.state match {case Idle | Copying(_) =>files(src) = info.copy(state = info.state.inc)files(dest) = FileInfo.creating(destfile, info.size)Txn.afterCommit { _ => copyOnDisk(srcfile, destfile) }src}} The copyOnDisk method calls the copyFile method on the FileUtils class from the Apache Commons IO library. After the file transfer completes, the copyOnDisk method starts another transaction, in which it decreases the copy count of the source file and sets the state of the destination file to Idle: private def copyOnDisk(srcfile: File, destfile: File) = {FileUtils.copyFile(srcfile, destfile)atomic { implicit txn =>val ninfo = files(srcfile.getPath)files(srcfile.getPath) = ninfo.copy(state = ninfo.state.dec)files(destfile.getPath) = FileInfo(destfile)}} The deleteFile method deletes a file in a similar way. It changes the file state to Deleted, deletes the file, and starts another transaction to remove the file entry: def deleteFile(srcpath: String): String = atomic { implicit txn =>val info = files(srcpath)info.state match {case Idle =>files(srcpath) = info.copy(state = Deleted)Txn.afterCommit { _ =>FileUtils.forceDelete(info.toFile)files.single.remove(srcpath)}srcpath}} Modeling the server data model with the STM allows seamlessly adding different concurrent computations to the server program. In the next section, we will implement a server actor that uses the server API to execute filesystem operations. Use STM to model concurrently accessible data, as an STM works transparently with most concurrency frameworks. Having completed the filesystem API, we now proceed to the server interface layer of the ScalaFTP browser. The Server interface The server interface comprises of a single actor called FTPServerActor. This actor will receive client requests and respond to them serially. If it turns out that the server actor is the sequential bottleneck of the system, we can simply add additional server interface actors to improve horizontal scalability. We start by defining the different types of messages that the server actor can receive. We follow the convention of defining them inside the companion object of the FTPServerActor class: object FTPServerActor {sealed trait Commandcase class GetFileList(dir: String) extends Commandcase class CopyFile(src: String, dest: String) extends Commandcase class DeleteFile(path: String) extends Commanddef apply(fs: FileSystem) = Props(classOf[FTPServerActor], fs)} The actor template of the server actor takes a FileSystem object as a parameter. It reacts to the GetFileList, CopyFile, and DeleteFile messages by calling the appropriate methods from the filesystem API: class FTPServerActor(fileSystem: FileSystem) extends Actor {val log = Logging(context.system, this)def receive = {case GetFileList(dir) =>val filesMap = fileSystem.getFileList(dir)val files = filesMap.map(_._2).to[Seq]sender ! filescase CopyFile(srcpath, destpath) =>Future {Try(fileSystem.copyFile(srcpath, destpath))} pipeTo sendercase DeleteFile(path) =>Future {Try(fileSystem.deleteFile(path))} pipeTo sender}} When the server receives a GetFileList message, it calls the getFileList method with the specified dir directory, and sends a sequence collection with the FileInfo objects back to the client. Since FileInfo is a case class, it extends the Serializable interface, and its instances can be sent over the network. When the server receives a CopyFile or DeleteFile message, it calls the appropriate filesystem method asynchronously. The methods in the filesystem API throw exceptions when something goes wrong, so we need to wrap calls to them in Try objects. After the asynchronous file operations complete, the resulting Try objects are piped back as messages to the sender actor, using the Akka pipeTo method. To start the ScalaFTP server, we need to instantiate and initialize a FileSystem object, and start the server actor. We parse the network port command-line argument, and use it to create an actor system that is capable of remote communication. For this, we use the remotingSystem factory method that we introduced. The remoting actor system then creates an instance of the FTPServerActor. This is shown in the following program: object FTPServer extends App {val fileSystem = new FileSystem(".")fileSystem.init()val port = args(0).toIntval actorSystem = ch8.remotingSystem("FTPServerSystem", port)actorSystem.actorOf(FTPServerActor(fileSystem), "server")} The ScalaFTP server actor can run inside the same process as the client application, in another process in the same machine, or on a different machine connected with a network. The advantage of the actor model is that we usually need not worry about where the actor runs until we integrate it into the entire application. When you need to implement a distributed application that runs on different machines, use an actor framework. Our server program is now complete, and we can run it with the run command from SBT. We set the actor system to use the port 12345: run 12345 In the next section, we will implement the file navigation API for the ScalaFTP client, which will communicate with the server interface over the network. Client navigation API The client API exposes the server interfaces to the client program through asynchronous methods that return future objects. Unlike the server's filesystem API, which runs locally, the client API methods execute remote network requests. Futures are a natural way to model latency in the client API methods, and to avoid blocking during the network requests. Internally, the client API maintains an actor instance that communicates with the server actor. The client actor does not know the actor reference of the server actor when it is created. For this reason, the client actor starts in an unconnected state. When it receives the Start message with the URL of the server actor system, the client constructs an actor path to the server actor, sends out an Identify message, and switches to the connecting state. If the actor system is able to find the server actor, the client actor eventually receives the ActorIdentity message with the server actor reference. In this case, the client actor switches to the connected state, and is able to forward commands to the server. Otherwise, the connection fails and the client actor reverts to the unconnected state. The state diagram of the client actor is shown in the following figure: We define the Start message in the client actor's companion object: object FTPClientActor {case class Start(host: String)} We then define the FTPClientActor class and give it an implicit Timeout parameter. The Timeout parameter will be used later in the Akka ask pattern, when forwarding client requests to the server actor. The stub of the FTPClientActor class is as follows: class FTPClientActor(implicit val timeout: Timeout)extends Actor Before defining the receive method, we define behaviors corresponding to different actor states. Once the client actor in the unconnected state receives the Start message with the host string, it constructs an actor path to the server, and creates an actor selection object. The client actor then sends the Identify message to the actor selection, and switches its behavior to connecting. This is shown in the following behavior method, named unconnected: def unconnected: Actor.Receive = {case Start(host) =>val serverActorPath =s"akka.tcp://FTPServerSystem@$host/user/server"val serverActorSel = context.actorSelection(serverActorPath)serverActorSel ! Identify(())context.become(connecting(sender))} The connecting method creates a behavior given an actor reference to the sender of the Start message. We call this actor reference clientApp, because the ScalaFTP client application will send the Start message to the client actor. Once the client actor receives an ActorIdentity message with the ref reference to the server actor, it can send true back to the clientApp reference, indicating that the connection was successful. In this case, the client actor switches to the connected behavior. Otherwise, if the client actor receives an ActorIdentity message without the server reference, the client actor sends false back to the application, and reverts to the unconnected state: def connecting(clientApp: ActorRef): Actor.Receive = {case ActorIdentity(_, Some(ref)) =>clientApp ! truecontext.become(connected(ref))case ActorIdentity(_, None) =>clientApp ! falsecontext.become(unconnected)} The connected state uses the serverActor server actor reference to forward the Command messages. To do so, the client actor uses the Akka ask pattern, which returns a future object with the server's response. The contents of the future are piped back to the original sender of the Command message. In this way, the client actor serves as an intermediary between the application, which is the sender, and the server actor. The connected method is shown in the following code snippet: def connected(serverActor: ActorRef): Actor.Receive = {case command: Command =>(serverActor ? command).pipeTo(sender)} Finally, the receive method returns the unconnected behavior, in which the client actor is created: def receive = unconnected Having implemented the client actor, we can proceed to the client API layer. We model it as a trait with a connected value, the concrete methods getFileList, copyFile, and deleteFile, and an abstract host method. The client API creates a private remoting actor system and a client actor. It then instantiates the connected future that computes the connection status by sending a Start message to the client actor. The methods getFileList, copyFile, and deleteFile are similar. They use the ask pattern on the client actor to obtain a future with the response. Recall that the actor messages are not typed, and the ask pattern returns a Future[Any] object. For this reason, each method in the client API uses the mapTo future combinator to restore the type of the message: trait FTPClientApi {implicit val timeout: Timeout = Timeout(4 seconds)private val props = Props(classOf[FTPClientActor], timeout)private val system = ch8.remotingSystem("FTPClientSystem", 0)private val clientActor = system.actorOf(props)def host: Stringval connected: Future[Boolean] = {val f = clientActor ? FTPClientActor.Startf.mapTo[Boolean]}def getFileList(d: String): Future[(String, Seq[FileInfo])] = {val f = clientActor ? FTPServerActor.GetFileList(d)f.mapTo[Seq[FileInfo]].map(fs => (d, fs))}def copyFile(src: String, dest: String): Future[String] = {val f = clientActor ? FTPServerActor.CopyFile(src, dest)f.mapTo[Try[String]].map(_.get)}def deleteFile(srcpath: String): Future[String] = {val f = clientActor ? FTPServerActor.DeleteFile(srcpath)f.mapTo[Try[String]].map(_.get)}} Note that the client API does not expose the fact that it uses actors for remote communication. Moreover, the client API is similar to the server API, but the return types of the methods are futures instead of normal values. Futures encode the latency of a method without exposing the cause for the latency, so we often find them at the boundaries between different APIs. We can internally replace the actor communication between the client and the server with the remote Observable objects, but that would not change the client API. In a concurrent application, use futures at the boundaries of the layers to express latency. Now that we can programmatically communicate with the remote ScalaFTP server, we turn our attention to the user interface of the client program. Summary This article summarized the different concurrency libraries introduced to us. In this article, you learned how to choose the correct concurrent abstraction to solve a given problem. We learned to combine different concurrency abstractions together when designing larger concurrent applications. Resources for Article: Further resources on this subject: Creating Java EE Applications [Article] Differences in style between Java and Scala code [Article] Integrating Scala, Groovy, and Flex Development with Apache Maven [Article]
Read more
  • 0
  • 0
  • 1782

article-image-high-availability-scenarios
Packt
26 Nov 2014
14 min read
Save for later

High Availability Scenarios

Packt
26 Nov 2014
14 min read
"Live Migration between hosts in a Hyper-V cluster is very straightforward and requires no specific configuration, apart from type and amount of simultaneous Live Migrations. If you add multiple clusters and standalone Hyper-V hosts into the mix, I strongly advise you to configure Kerberos Constrained Delegation for all hosts and clusters involved." Hans Vredevoort – MVP Hyper-V This article written by Benedict Berger, the author of Hyper-V Best Practices, will guide you through the installation of Hyper-V clusters and their best practice configuration. After installing the first Hyper-V host, it may be necessary to add another layer of availability to your virtualization services. With Failover Clusters, you get independence from hardware failures and are protected from planned or unplanned service outages. This article includes prerequirements and implementation of Failover Clusters. (For more resources related to this topic, see here.) Preparing for High Availability Like every project, a High Availability (HA) scenario starts with a planning phase. Virtualization projects are often turning up the question for additional availability for the first time in an environment. In traditional data centers with physical server systems and local storage systems, an outage of a hardware component will only affect one server hosting one service. The source of the outage can be localized very fast and the affected parts can be replaced in a short amount of time. Server virtualization comes with great benefits, such as improved operating efficiency and reduced hardware dependencies. However, a single component failure can impact a lot of virtualized systems at once. By adding redundant systems, these single points of failure can be avoided. Planning a HA environment The most important factor in the decision whether you need a HA environment is your business requirements. You need to find out how often and how long an IT-related production service can be interrupted unplanned, or planned, without causing a serious problem to your business. Those requirements are defined in a central IT strategy of a business as well as in process definitions that are IT-driven. They include Service Level Agreements of critical business services run in the various departments of your company. If those definitions do not exist or are unavailable, talk to the process owners to find out the level of availability needed. High Availability is structured in different classes, measured by the total uptime in a defined timespan, that is 99.999 percent in a year. Every nine in this figure adds a huge amount of complexity and money needed to ensure this availability, so take time to find out the real availability needed by your services and resist the temptation to plan running every service on multi-redundant, geo-spread cluster systems, as it may not fit in the budget. Be sure to plan for additional capacity in a HA environment, so you can lose hardware components without the need to sacrifice application performance. Overview of the Failover Cluster A Hyper-V Failover Cluster consists of two or more Hyper-V Server compute nodes. Technically, it's possible to use a Failover Cluster with just one computing node; however, it will not provide any availability advantages over a standalone host and is typically only used for migration scenarios. A Failover Cluster is hosting roles such as Hyper-V virtual machines on its computing nodes. If one node fails due to a hardware problem, it will not answer any more to cluster heartbeat communication, even though the service interruption is almost instantly detected. The virtual machines running on the particular node are powered off immediately due to the hardware failure on their computing node. The remaining cluster nodes then immediately take over these VMs in an unplanned failover process and start them on their respective own hardware. The virtual machines will be the backup running after a successful boot of their operating systems and applications in just a few minutes. Hyper-V Failover Clusters work under the condition that all compute nodes have access to a shared storage instance, holding the virtual machine configuration data and its virtual hard disks. In case of a planned failover, that is, for patching compute nodes, it's possible to move running virtual machines from one cluster node to another without interrupting the VM. All cluster nodes can run virtual machines at the same time, as long as there is enough failover capacity running all services when a node goes down. Even though a Hyper-V cluster is still called a Failover Cluster—utilizing the Windows Server Failover-Clustering feature—it is indeed capable of running an Active/Active Cluster. To ensure that all these capabilities of a Failover Cluster are indeed working, it demands an accurate planning and implementation process. Failover Cluster prerequirements To successfully implement a Hyper-V Failover Cluster, we need suitable hardware, software, permissions, and network and storage infrastructure as outlined in the following sections. Hardware The hardware used in a Failover Cluster environment needs to be validated against the Windows Server Catalogue. Microsoft will only support Hyper-V clusters when all components are certified for Windows Server 2012 R2. The servers used to run our HA virtual machines should ideally consist of identical hardware models with identical components. It is possible, and supported, to run servers in the same cluster with different hardware components, that is, different size of RAM; however, due to a higher level of complexity, this is not best practice. Special planning considerations are needed to address the CPU requirements of a cluster. To ensure maximum compatibility, all CPUs in a cluster should be exactly the same model. While it's possible from a technical point of view to mix even CPUs from Intel and AMD in the same cluster through to different architecture, you will lose core cluster capabilities such as Live Migration. Choosing a single vendor for your CPUs is not enough, even when using different CPU models your cluster nodes may be using a different set of CPU instruction set extensions. With different instructions sets, Live Migrations won't work either. There is a compatibility mode that disables most of the instruction set on all CPUs on all cluster nodes; however, this leaves you with a negative impact on performance and should be avoided. A better approach to this problem would be creating another cluster from the legacy CPUs running smaller or non-production workloads without affecting your high-performance production workloads. If you want to extend your cluster after some time, you will find yourself with the problem of not having the exact same hardware available to purchase. Choose the current revision of the model or product line you are already using in your cluster and manually compare the CPU instruction sets at http://ark.intel.com/ and http://products.amd.com/, respectively. Choose the current CPU model that best fits the original CPU features of your cluster and have this design validated by your hardware partner. Ensure that your servers are equipped with compatible CPUs, the same amount of RAM, and the same network cards and storage controllers. The network design Mixing different vendors of network cards in a single server is fine and best practice for availability, but make sure all your Hyper-V hosts are using an identical hardware setup. A network adapter should only be used exclusively for LAN traffic or storage traffic. Do not mix these two types of communication in any basic scenario. There are some more advanced scenarios involving converged networking that can enable mixed traffic, but in most cases, this is not a good idea. A Hyper-V Failover Cluster requires multiple layers of communication between its nodes and storage systems. Hyper-V networking and storage options have changed dramatically through the different releases of Hyper-V. With Windows Server 2012 R2, the network design options are endless. In this article, we will work with a typically seen basic set of network designs. We have at least six Network Interface Cards (NICs) available in our servers with a bandwidth of 1 Gb/s. If you have more than five interface cards available per server, use NIC Teaming to ensure the availability of the network or even use converged networking. Converged networking will also be your choice if you have less than five network adapters available. The First NIC will be exclusively used for Host Communication to our Hyper-V host and will not be involved in the VM network traffic or cluster communication at any time. It will ensure Active Directory and management traffic to our Management OS. The second NIC will ensure Live Migration of virtual machines between our cluster nodes. The third NIC will be used for VM traffic. Our virtual machines will be connected to the various production and lab networks through this NIC. The fourth NIC will be used for internal cluster communication. The first four NICs can either be teamed through Windows Server NIC Teaming or can be abstracted from the physical hardware through to Windows Server network virtualization and converged fabric design. The fifth NIC will be reserved for storage communication. As advised, we will be isolating storage and production LAN communication from each other. If you do not use iSCSI or SMB3 storage communication, this NIC will not be necessary. If you use Fibre Channel SAN technology, use a FC-HBA instead. If you leverage Direct Attached Storage (DAS), use the appropriate connector for storage communication. The sixth NIC will also be used for storage communication as a redundancy. The redundancy will be established via MPIO and not via NIC Teaming. There is no need for a dedicated heartbeat network as in older revisions of Windows Server with Hyper-V. All cluster networks will automatically be used for sending heartbeat signals throughout the other cluster members. If you don't have 1 Gb/s interfaces available, or if you use 10 GbE adapters, it’s best practice to implement a converged networking solution. Storage design All cluster nodes must have access to the virtual machines residing on a centrally shared storage medium. This could be a classic setup with a SAN, a NAS, or a more modern concept with Windows Scale Out File Servers hosting Virtual Machine Files SMB3 Fileshares. In this article, we will use a NetApp SAN system that's capable of providing a classic SAN approach with LUNs mapped to our Hosts as well as utilizing SMB3 Fileshares, but any other Windows Server 2012 R2 validated SAN will fulfill the requirements. In our first setup, we will utilize Cluster Shared Volumes (CSVs) to store several virtual machines on the same storage volume. It's not good these days to create a single volume per virtual machine due to a massive management overhead. It's a good rule of thumb to create one CSV per cluster node; in larger environments with more than eight hosts, a CSV per two to four cluster nodes. To utilize CSVs, follow these steps: Ensure that all components (SAN, Firmware, HBAs, and so on) are validated for Windows Server 2012 R2 and are up to date. Connect your SAN physically to all your Hyper-V hosts via iSCSI or Fibre Channel connections. Create two LUNs on your SAN for hosting virtual machines. Activate Hyper-V performance options for these LUNs if possible (that is, on a NetApp, by setting the LUN type to Hyper-V). Size the LUNs for enough capacity to host all your virtual hard disks. Label the LUNs CSV01 and CSV02 with appropriate LUN IDs. Create another small LUN with 1 GB in size and label it Quorum. Make the LUNs available to all Hyper-V hosts in this specified cluster by mapping it on the storage device. Do not make these LUNs available to any other hosts or cluster. Prepare storage DSMs and drivers (that is, MPIO) for Hyper-V host installation. Refresh disk configuration on hosts, install drivers and DSMs, and format volumes as NTFS (quick). Install Microsoft Multipath IO when using redundant storage paths: Install-WindowsFeature -Name Multipath-IO –Computername ElanityHV01, ElanityHV02 In this example, I added the MPIO feature to two Hyper-V hosts with the computer names ElanityHV01 and ElanityHV02. SANs typically are equipped with two storage controllers for redundancy reasons. Make sure to disperse your workloads over both controllers for optimal availability and performance. If you leverage file servers providing SMB3 shares, the preceding steps do not apply to you. Perform the following steps instead: Create a storage space with the desired disk-types, use storage tiering if possible. Create a new SMB3 Fileshare for applications. Customize the Permissions to include all Hyper-V servers from the planned clusters as well as the Hyper-V cluster object itself with full control. Server and software requirements To create a Failover Cluster, you need to install a second Hyper-V host. Use the same unattended file but change the IP address and the hostname. Join both Hyper-V hosts to your Active Directory domain if you have not done this until yet. Hyper-V can be clustered without leveraging Active Directory but it's lacking several key components, such as Live Migration, and shouldn't be done on purpose. The availability to successfully boot up a domain-joined Hyper-V cluster without the need to have any Active Directory domain controller present during boot time is the major benefit from the Active Directory independency of Failover Clusters. Ensure that you create a Hyper-V virtual switch as shown earlier with the same name on both hosts, to ensure cluster compatibility and that both nodes are installed with all updates. If you have System Center 2012 R2 in place, use the System Center Virtual Machine Manager to create a Hyper-V cluster. Implementing Failover Clusters After preparing our Hyper-V hosts, we will now create a Failover Cluster using PowerShell. I'm assuming your hosts are installed, storage and network connections are prepared, and the Hyper-V role is already active utilizing up-to-date drivers and firmware on your hardware. First, we need to ensure that Servername, Date, and Time of our Hosts are correct. Time and Timezone configurations should occur via Group Policy. For automatic network configuration later on, it's important to rename the network connections from default to their designated roles using PowerShell, as seen in the following commands: Rename-NetAdapter -Name "Ethernet" -NewName "Host" Rename-NetAdapter -Name "Ethernet 2" -NewName "LiveMig" Rename-NetAdapter -Name "Ethernet 3" -NewName "VMs" Rename-NetAdapter -Name "Ethernet 4" -NewName "Cluster" Rename-NetAdapter -Name "Ethernet 5" -NewName "Storage" The Network Connections window should look like the following screenshot: Hyper-V host Network Connections Next, IP configuration of the network adapters. If you are not using DHCP for your servers, manually set the IP configuration (different subnets) of the specified network cards. Here is a great blog post on how to automate this step: http://bit.ly/Upa5bJ Next, we need to activate the necessary Failover Clustering features on both of our Hyper-V hosts: Install-WindowsFeature -Name Failover-Clustering-IncludeManagementTools –Computername ElanityHV01, ElanityHV02 Before actually creating the cluster, we are launching a cluster validation cmdlet via PowerShell: Test-Cluster ElanityHV01, ElanityHV02 Test-Cluster cmdlet Open the generated .mht file for more details, as shown in the following screenshot: Cluster validation As you can see, there are some warnings that should be investigated. However, as long as there are no errors, the configuration is ready for clustering and fully supported by Microsoft. However, check out Warnings to be sure you won't run into problems in the long run. After you have fixed potential errors and warnings listed in the Cluster Validation Report, you can finally create the cluster as follows: New-Cluster-Name CN=ElanityClu1,OU=Servers,DC=cloud,DC=local-Node ElanityHV01, ElanityHV02-StaticAddress 192.168.1.49 This will create a new cluster named ElanityClu1 consisting of the nodes ElanityHV01 and ElanityHV02 and using the cluster IP address 192.168.1.49. This cmdlet will create the cluster and the corresponding Active Directory Object in the specified OU. Moving the cluster object to a different OU later on is no problem at all; even renaming is possible when done the right way. After creating the cluster, when you open the Failover Cluster Management console, you should be able to connect to your cluster: Failover Cluster Manager You will see that all your cluster nodes and Cluster Core Resources are online. Rerun the Validation Report and copy the generated .mht files to a secure location if you need them for support queries. Keep in mind that you have to rerun this wizard if any hardware or configuration changes occurring to the cluster components, including any of its nodes. The initial cluster setup is now complete and we can continue with post creation tasks. Summary With the knowledge from this article, you are now able to design and implement Hyper-V Failover Clusters as well as guest clusters. You are aware of the basic concepts of High Availability and the storage and networking options necessary to achieve this. You have seen real-world proven configurations to ensure a stable operating environment. Resources for Article: Further resources on this subject: Planning Desktop Virtualization [Article] Backups in the VMware View Infrastructure [Article] Virtual Machine Design/a> [Article]
Read more
  • 0
  • 0
  • 9389

Packt
25 Nov 2014
26 min read
Save for later

Detecting Beacons – Showing an Advert

Packt
25 Nov 2014
26 min read
In this article, by Craig Gilchrist, author of the book Learning iBeacon, we're going to expand our knowledge and get an in-depth understanding of the broadcasting triplet, and we'll expand on some of the important classes within the Core Location framework. (For more resources related to this topic, see here.) To help demonstrate the more in-depth concepts, we'll build an app that shows different advertisements depending on the major and minor values of the beacon that it detects. We'll be using the context of an imaginary department store called Matey's. Matey's are currently undergoing iBeacon trials in their flagship London store and at the moment are giving offers on their different-themed restaurants and also on their ladies clothing to users who are using their branded app. Uses of the UUID/major/minor broadcasting triplet In the last article, we covered the reasons behind the broadcasting triplet; we're going to use the triplet with a more realistic scenario. Let's go over the three values again in some more detail. UUID – Universally Unique Identifier The UUID is meant to be unique to your app. It can be spoofed, but generally, your app would be the only app looking for that UUID. The UUID identifies a region, which is the maximum broadcast range of a beacon from its center point. Think of a region as a circle of broadcast with the beacon in the middle. If lots of beacons with the same UUID have overlapping broadcasting ranges, then the region is represented by the broadcasting range of all the beacons combined as shown in the following figure. The combined range of all the beacons with the same UUID becomes the region. Broadcast range More specifically, the region is represented by an instance of the CLBeaconRegion class, which we'll cover in more detail later in this article. The following code shows how to configure CLBeaconRegion: NSString * uuidString = @"78BC6634-A424-4E05-A2AE-A59A25CAC4A9";   NSUUID * regionUUID; regionUUID = [[NSUUID alloc] initWithUUIDString:uuidString"];    CLBeaconRegion * region; region = [[CLBeaconRegion alloc] initWithProximityUUID: regionUUID identifier:@"My Region"]; Generally, most apps will be monitoring only for one region. This is normally sufficient since the major and minor values are 16-bit unsigned integers, which means that each value can be a number up to 65,535 giving 4,294,836,225 unique beacon combinations per UUID. Since the major and minor values are used to represent a subsection of the use case, there may be a time when 65,535 combinations of a major value may not be enough and so, this would be the rare time that your app can monitor multiple regions with different UUIDs. Another more likely example is that your app has multiple use cases, which are more logically split by UUID. An example where an app has multiple use cases would be a loyalty app that has offers for many different retailers when the app is within the vicinity of the retail stores. Here you can have a different UUID for every retailer. Major The major value further identifies your use case. The major value should separate your use case along logical categories. This could be sections in a shopping mall or exhibits in a museum. In our example, a use case of the major value represents the different types of service within a department store. In some cases, you may wish to separate logical categories into more than one major value. This would only be if each category has more than 65,535 beacons. Minor The minor value ultimately identifies the beacon itself. If you consider the major value as the category, then the minor value is the beacon within that category. Example of a use case The example laid out in this article uses the following UUID/major/minor values to broadcast different adverts for Matey's: Department Food Women's clothing UUID 8F0C1DDC-11E5-4A07-8910-425941B072F9 Major 1 2 Minor 1 30 percent off on sushi at The Japanese Kitchen 50 percent off on all ladies' clothing   2 Buy one get one free at Tucci's Pizza N/A Understanding Core Location The Core Location framework lets you determine the current location or heading associated with the device. The framework has been around since 2008 and was present in iOS 2.0. Up until the release of iOS 7, the framework was only used for geolocation based on GPS coordinates and so was suitable only for outdoor location. The framework got a new set of classes and new methods were added to the existing classes to accommodate the beacon-based location functionality. Let's explore a few of these classes in more detail. The CLBeaconRegion class Geo-fencing (geofencing) is a feature in a software program that uses the global positioning system (GPS) or radio frequency identification (RFID) to define geographical boundaries. A geofence is a virtual barrier. The CLBeaconRegion class defines a geofenced boundary identified by a UUID and the collective range of all physical beacons with the same UUID. When a device matching the CLBeaconRegion UUID comes in range, the region triggers the delivery of an appropriate notification. CLBeaconRegion inherits CLRegion, which also serves as the superclass of CLCircularRegion. The CLCircularRegion class defines the location and boundaries for a circular geographic region. You can use instances of this class to define geofences for a specific location, but it shouldn't be confused with CLBeaconRegion. The CLCircularRegion class shares many of the same methods but is specifically related to a geographic location based on the GPS coordinates of the device. The following figure shows the CLRegion class and its descendants. The CLRegion class hierarchy The CLLocationManager class The CLLocationManager class defines the interface for configuring the delivery of location-and heading-related events to your application. You use an instance of this class to establish the parameters that determine when location and heading events should be delivered and to start and stop the actual delivery of those events. You can also use a location manager object to retrieve the most recent location and heading data. Creating a CLLocationManager class The CLLocationManager class is used to track both geolocation and proximity based on beacons. To start tracking beacon regions using the CLLocationManager class, we need to do the following: Create an instance of CLLocationManager. Assign an object conforming to the CLLocationManagerDelegate protocol to the delegate property. Call the appropriate start method to begin the delivery of events. All location- and heading-related updates are delivered to the associated delegate object, which is a custom object that you provide. Defining a CLLocationManager class line by line Consider the following steps to define a CLLocationManager class line by line: Every class that needs to be notified about CLLocationManager events needs to first import the Core Location framework (usually in the header file) as shown: #import <CoreLocation/CoreLocation.h> Then, once the framework is imported, the class needs to declare itself as implementing the CLLocationManagerDelegate protocol like the following view controller does: @interface MyViewController :   UIViewController<CLLocationManagerDelegate> Next, you need to create an instance of CLLocationManager and set your class as the instance delegate of CLLocationManager as shown:    CLLocationManager * locationManager =       [[CLLocationManager alloc] init];    locationManager.delegate = self; You then need a region for your location manager to work with: // Create a unique ID to identify our region. NSUUID * regionId = [[NSUUID alloc]   initWithUUIDString:@" AD32373E-9969-4889-9507-C89FCD44F94E"];   // Create a region to monitor. CLBeaconRegion * beaconRegion =   [[CLBeaconRegion alloc] initWithProximityUUID: regionId identifier:@"My Region"]; Finally, you need to call the appropriate start method using the beacon region. Each start method has a different purpose, which we'll explain shortly: // Start monitoring and ranging beacons. [locationManager startMonitoringForRegion:beaconRegion]; [locationManager startRangingBeaconsInRegion:beaconRegion]; Once the class is imported, you need to implement the methods of the CLLocationManagerDelegate protocol. Some of the most important delegate methods are explained shortly. This isn't an exhaustive list of the methods, but it does include all of the important methods we'll be using in this article. locationManager:didEnterRegion Whenever you enter a region that your location manager has been instructed to look for (by calling startRangingBeaconsInRegion), the locationManager:didEnterRegion delegate method is called. This method gives you an opportunity to do something with the region such as start monitoring for specific beacons, shown as follows: -(void)locationManager:(CLLocationManager *) manager didEnterRegion:(CLRegion *)region {    // Do something when we enter a region. } locationManager:didExitRegion Similarly, when you exit the region, the locationManager:didExitRegion delegate method is called. Here you can do things like stop monitoring for specific beacons, shown as follows: -(void)locationManager:(CLLocationManager *)manager   didExitRegion:(CLRegion *)region {    // Do something when we exit a region. } When testing your region monitoring code on a device, realize that region events may not happen immediately after a region boundary is crossed. To prevent spurious notifications, iOS does not deliver region notifications until certain threshold conditions are met. Specifically, the user's location must cross the region boundary and move away from that boundary by a minimum distance and remain at that minimum distance for at least 20 seconds before the notifications are reported. locationManager:didRangeBeacons:inRegion The locationManager:didRangeBeacons:inRegion method is called whenever a beacon (or a number of beacons) change distance from the device. For now, it's enough to know that each beacon that's returned in this array has a property called proximity, which returns a CLProximity enum value (CLProximityUnknown, CLProximityFar, CLProximityNear, and CLProximityImmediate), shown as follows: -(void)locationManager:(CLLocationManager *)manager   didRangeBeacons:(NSArray *)beacons inRegion: (CLBeaconRegion *)region {    // Do something with the array of beacons. } locationManager:didChangeAuthorizationStatus Finally, there's one more delegate method to cover. Whenever the users grant or deny authorization to use their location, locationManager:didChangeAuthorizationStatus is called. This method is passed as a CLAuthorizationStatus enum (kCLAuthorizationStatusNotDetermined, kCLAuthorizationStatusRestricted, kCLAuthorizationStatusDenied, and kCLAuthorizationStatusAuthorized), shown as follows: -(void)locationManager:(CLLocationManager *)manager   didChangeAuthorizationStatus:(CLAuthorizationStatus)status {    // Do something with the array of beacons. } Understanding iBeacon permissions It's important to understand that apps using the Core Location framework are essentially monitoring location, and therefore, they have to ask the user for their permission. The authorization status of a given application is managed by the system and determined by several factors. Applications must be explicitly authorized to use location services by the user, and the current location services must themselves be enabled for the system. A request for user authorization is displayed automatically when your application first attempts to use location services. Requesting the location can be a fine balancing act. Asking for permission at a point in an app, when your user wouldn't think it was relevant, makes it more likely that they will decline it. It makes more sense to tell the users why you're requesting their location and why it benefits them before requesting it so as not to scare away your more squeamish users. Building those kinds of information views isn't covered in this book, but to demonstrate the way a user is asked for permission, our app should show an alert like this: Requesting location permission If your user taps Don't Allow, then the location can't be enabled through the app unless it's deleted and reinstalled. The only way to allow location after denying it is through the settings. Location permissions in iOS 8 Since iOS 8.0, additional steps are required to obtain location permissions. In order to request location in iOS 8.0, you must now provide a friendly message in the app's plist by using the NSLocationAlwaysUsageDescription key, and also make a call to the CLLocationManager class' requestAlwaysAuthorization method. The NSLocationAlwaysUsageDescription key describes the reason the app accesses the user's location information. Include this key when your app uses location services in a potentially nonobvious way while running in the foreground or the background. There are two types of location permission requests as of iOS 8 as specified by the following plist keys: NSLocationWhenInUseUsageDescription: This plist key is required when you use the requestAlwaysAuthorization method of the CLLocationManager class to request authorization for location services. If this key is not present and you call the requestAlwaysAuthorization method, the system ignores your request and prevents your app from using location services. NSLocationAlwaysUsageDescription: This key is required when you use the requestWhenInUseAuthorization method of the CLLocationManager class to request authorization for location services. If the key is not present when you call the requestWhenInUseAuthorization method without including this key, the system ignores your request. Since iBeacon requires location services in the background, we will only ever use the NSLocationAlwaysUsageDescription key with the call to the CLLocationManager class' requestAlwaysAuthorization. Enabling location after denying it If a user denies enabling location services, you can follow the given steps to enable the service again on iOS 7: Open the iOS device settings and tap on Privacy. Go to the Location Services section. Turn location services on for your app by flicking the switch next to your app name. When your device is running iOS 8, you need to follow these steps: Open the iOS device settings and tap on Privacy. Go to your app in the Settings menu. Tap on Privacy. Tap on Location Services. Set the Allow Location Access to Always. Building the tutorial app To demonstrate the knowledge gained in this article, we're going to build an app for our imaginary department store Matey's. Matey's is trialing iBeacons with their app Matey's offers. People with the app get special offers in store as we explained earlier. For the app, we're going to start a single view application containing two controllers. The first is the default view controller, which will act as our CLLocationManagerDelegate, the second is a view controller that will be shown modally and shows the details of the offer relating to the beacon we've come into proximity with. The final thing to consider is that we'll only show each offer once in a session and we can only show an offer if one isn't showing. Shall we begin? Creating the app Let's start by firing up Xcode and choosing a new single view application just as we did in the previous article. Choose these values for the new project: Product Name: Matey's Offers Organization Name: Learning iBeacon Company Identifier: com.learning-iBeacon Class Prefix: LI Devices: iPhone Your project should now contain your LIAppDelegate and LIViewController classes. We're not going to touch the app delegate this time round, but we'll need to add some code to the LIViewController class since this is where all of our CLLocationManager code will be running. For now though, let's leave it to come back to later. Adding CLOfferViewController Our offer view controller will be used as a modal view controller to show the offer relating to the beacon that we come in contact with. Each of our offers is going to be represented with a different background color, a title, and an image to demonstrate the offer. Be sure to download the code relating to this article and add the three images contained therein to your project by dragging the images from finder into the project navigator: ladiesclothing.jpg pizza.jpg sushi.jpg Next, we need to create the view controller. Add a new file and be sure to choose the template Objective-c class from the iOS Cocoa Touch menu. When prompted, name this class LIOfferViewController and make it a subclass of UIViewController. Setting location permission settings We need to add our permission message to the applications so that when we request permission for the location, our dialog appears: Click on the project file in the project navigator to show the project settings. Click the Info tab of the Matey's Offers target. Under the Custom iOS Target Properties dictionary, add the NSLocationAlwaysUsageDescription key with the value. This app needs your location to give you wonderful offers. Adding some controls The offer view controller needs two controls to show the offer the view is representing, an image view and a label. Consider the following steps to add some controls to the view controller: Open the LIOfferViewController.h file and add the following properties to the header: @property (nonatomic, strong) UILabel * offerLabel; @property (nonatomic, strong) UIImageView * offerImageView; Now, we need to create them. Open the LIOfferViewController.m file and first, let's synthesize the controls. Add the following code just below the @implementation LIOfferViewController line: @synthesize offerLabel; @synthesize offerImageView; We've declared the controls; now, we need to actually create them. Within the viewDidLoad method, we need to create the label and image view. We don't need to set the actual values or images of our controls. This will be done by LIViewController when it encounters a beacon. Create the label by adding the following code below the call to [super viewDidLoad]. This will instantiate the label making it 300 points wide and appear 10 points from the left and top: UILabel * label = [[UILabel alloc]   initWithFrame:CGRectMake(10, 10, 300, 100)]; Now, we need to set some properties to style the label. We want our label to be center aligned, white in color, and with bold text. We also want it to auto wrap when it's too wide to fit the 300 point width. Add the following code: label setTextAlignment:NSTextAlignmentCenter]; [label setTextColor:[UIColor whiteColor]]; [label setFont:[UIFont boldSystemFontOfSize:22.f]]; label.numberOfLines = 0; // Allow the label to auto wrap. Now, we need to add our new label to the view and assign it to our property: [self.view addSubview:label]; self.offerLabel = label; Next, we need to create an image. Our image needs a nice border; so to do this, we need to add the QuartzCore framework. Add the QuartzCore framework like we did with CoreLocation in the previous article, and come to mention it, we'll need CoreLocation; so, add that too. Once that's done add #import <QuartzCore/QuartzCore.h> to the top of the LIOfferViewController.m file. Now, add the following code to instantiate the image view and add it to our view: UIImageView * imageView = [[UIImageView alloc]   initWithFrame:CGRectMake(10, 120, 300, 300)]; [imageView.layer setBorderColor:[[UIColor   whiteColor] CGColor]]; [imageView.layer setBorderWidth:2.f]; imageView.contentMode = UIViewContentModeScaleToFill; [self.view addSubview:imageView]; self.offerImageView = imageView; Setting up our root view controller Let's jump to LIViewController now and start looking for beacons. We'll start by telling LIViewController that LIOfferViewController exists and also that the view controller should act as a location manager delegate. Consider the following steps: Open LIViewController.h and add an import to the top of the file: #import <CoreLocation/CoreLocation.h> #import "LIOfferViewController.h" Now, add the CLLocationManagerDelegate protocol to the declaration: @interface LIViewController :   UIViewController<CLLocationManagerDelegate> LIViewController also needs three things to manage its roll: A reference to the current offer on display so that we know to show only one offer at a time An instance of CLLocationManager for monitoring beacons A list of offers seen so that we only show each offer once Let's add these three things to the interface in the CLViewController.m file (as they're private instances). Change the LIViewController interface to look like this: @interface LIViewController ()    @property (nonatomic, strong) CLLocationManager *       locationManager;    @property (nonatomic, strong) NSMutableDictionary *       offersSeen;    @property (nonatomic, strong) LIOfferViewController *       currentOffer; @end Configuring our location manager Our location manager needs to be configured when the root view controller is first created, and also when the app becomes active. It makes sense therefore that we put this logic into a method. Our reset beacon method needs to do the following things: Clear down our list of offers seen Request permission to the user's location Create a region and set our LIViewController instance as the delegate Create a beacon region and tell CLLocationManager to start ranging beacons Let's add the code to do this now: -(void)resetBeacons { // Initialize the location manager. self.locationManager = [[CLLocationManager alloc] init]; self.locationManager.delegate = self;   // Request permission. [self.locationManager requestAlwaysAuthorization];   // Clear the offers seen. self.offersSeen = [[NSMutableDictionary alloc]   initWithCapacity:3];    // Create a region. NSUUID * regionId = [[NSUUID alloc] initWithUUIDString: @"8F0C1DDC-11E5-4A07-8910-425941B072F9"];   CLBeaconRegion * beaconRegion = [[CLBeaconRegion alloc]   initWithProximityUUID:regionId identifier:@"Mateys"];   // Start monitoring and ranging beacons. [self.locationManager stopRangingBeaconsInRegion:beaconRegion]; [self.locationManager startMonitoringForRegion:beaconRegion]; [self.locationManager startRangingBeaconsInRegion:beaconRegion]; } Now, add the two calls to the reset beacon to ensure that the location manager is reset when the app is first started and then every time the app becomes active. Let's add this code now by changing the viewDidLoad method and adding the applicationDidBecomeActive method: -(void)viewDidLoad {    [super viewDidLoad];    [self resetBeacons]; }   - (void)applicationDidBecomeActive:(UIApplication *)application {    [self resetBeacons]; } Wiring up CLLocationManagerDelegate Now, we need to wire up the delegate methods of the CLLocationManagerDelegate protocol so that CLViewController can show the offer view when the beacons come into proximity. The first thing we need to do is to set the background color of the view to show whether or not our app has been authorized to use the device location. If the authorization has not yet been determined, we'll use orange. If the app has been authorized, we'll use green. Finally, if the app has been denied, we'll use red. We'll be using the locationManager:didChangeAuthorizationStatus delegate method to do this. Let's add the code now: -(void)locationManager:(CLLocationManager *)manager   didChangeAuthorizationStatus:(CLAuthorizationStatus) status {    switch (status) {        case kCLAuthorizationStatusNotDetermined:        {            // Set a lovely orange background            [self.view setBackgroundColor:[UIColor               colorWithRed:255.f/255.f green:147.f/255.f               blue:61.f/255.f alpha:1.f]];            break;        }        case kCLAuthorizationStatusAuthorized:        {             // Set a lovely green background.            [self.view setBackgroundColor:[UIColor               colorWithRed:99.f/255.f green:185.f/255.f               blue:89.f/255.f alpha:1.f]];            break;        }        default:        {             // Set a dark red background.            [self.view setBackgroundColor:[UIColor               colorWithRed:188.f/255.f green:88.f/255.f               blue:88.f/255.f alpha:1.f]];            break;        }    } } The next thing we need to do is to save the battery life by stopping and starting the ranging of beacons when we're within the region (except for when the app first starts). We do this by calling the startRangingBeaconsInRegion method with the locationManager:didEnterRegion delegate method and calling the stopRangingBeaconsInRegion method within the locationManager:didExitRegion delegate method. Add the following code to do what we've just described: -(void)locationManager:(CLLocationManager *)manager   didEnterRegion:(CLRegion *)region {    [self.locationManager       startRangingBeaconsInRegion:(CLBeaconRegion*)region]; } -(void)locationManager:(CLLocationManager *)manager   didExitRegion:(CLRegion *)region {    [self.locationManager       stopRangingBeaconsInRegion:(CLBeaconRegion*)region]; } Showing the advert To actually show the advert, we need to capture when a beacon is ranged by adding the locationManager:didRangeBeacons:inRegion delegate method to LIViewController. This method will be called every time the distance changes from an already discovered beacon in our region or when a new beacon is found for the region. The implementation is quite long so I'm going to explain each part of the method as we write it. Start by creating the method implementation as follows: -(void)locationManager:(CLLocationManager *)manager   didRangeBeacons:(NSArray *)beacons inRegion: (CLBeaconRegion *)region {   } We only want to show an offer associated with the beacon if we've not seen it before and there isn't a current offer being shown. We do this by checking the currentOffer property. If this property isn't nil, it means an offer is already being displayed and so, we need to return from the method. The locationManager:didRangeBeacons:inRegion method gets called by the location manager and gets passed to the region instance and an array of beacons that are currently in range. We only want to see each advert once in a session and so need to loop through each of the beacons to determine if we've seen it before. Let's add a for loop to iterate through the beacons and in the beacon looping do an initial check to see if there's an offer already showing: for (CLBeacon * beacon in beacons) {    if (self.currentOffer) return; >} Our offersSeen property is NSMutableDictionary containing all the beacons (and subsequently offers) that we've already seen. The key consists of the major and minor values of the beacon in the format {major|minor}. Let's create a string using the major and minor values and check whether this string exists in our offersSeen property by adding the following code to the loop: NSString * majorMinorValue = [NSString stringWithFormat: @"%@|%@", beacon.major, beacon.minor]; if ([self.offersSeen objectForKey:majorMinorValue]) continue; If offersSeen contains the key, then we continue looping. If the offer hasn't been seen, then we need to add it to the offers that are seen, before presenting the offer. Let's start by adding the key to our offers that are seen in the dictionary and then preparing an instance of LIOfferViewController: [self.offersSeen setObject:[NSNumber numberWithBool:YES]   forKey:majorMinorValue]; LIOfferViewController * offerVc = [[LIOfferViewController alloc]   init]; offerVc.modalPresentationStyle = UIModalPresentationFullScreen; Now, we're going prepare some variables to configure the offer view controller. Food offers show with a blue background while clothing offers show with a red background. We use the major value of the beacon to determine the color and then find out the image and label based on the minor value: UIColor * backgroundColor; NSString * labelValue; UIImage * productImage;        // Major value 1 is food, 2 is clothing. if ([beacon.major intValue] == 1) {       // Blue signifies food.    backgroundColor = [UIColor colorWithRed:89.f/255.f       green:159.f/255.f blue:208.f/255.f alpha:1.f];       if ([beacon.minor intValue] == 1) {        labelValue = @"30% off sushi at the Japanese Kitchen.";        productImage = [UIImage imageNamed:@"sushi.jpg"];    }    else {        labelValue = @"Buy one get one free at           Tucci's Pizza.";        productImage = [UIImage imageNamed:@"pizza.jpg"];    } } else {    // Red signifies clothing.    backgroundColor = [UIColor colorWithRed:188.f/255.f       green:88.f/255.f blue:88.f/255.f alpha:1.f];    labelValue = @"50% off all ladies clothing.";    productImage = [UIImage imageNamed:@"ladiesclothing.jpg"]; } Finally, we need to set these values on the view controller and present it modally. We also need to set our currentOffer property to be the view controller so that we don't show more than one color at the same time: [offerVc.view setBackgroundColor:backgroundColor]; [offerVc.offerLabel setText:labelValue]; [offerVc.offerImageView setImage:productImage]; [self presentViewController:offerVc animated:YES  completion:nil]; self.currentOffer = offerVc; Dismissing the offer Since LIOfferViewController is a modal view, we're going to need a dismiss button; however, we also need some way of telling it to our root view controller (LIViewController). Consider the following steps: Add the following code to the LIViewController.h interface to declare a public method: -(void)offerDismissed; Now, add the implementation to LIViewController.h. This method simply clears the currentOffer property as the actual dismiss is handled by the offer view controller: -(void)offerDismissed {    self.currentOffer = nil; } Now, let's jump back to LIOfferViewController. Add the following code to the end of the viewDidLoad method of LIOfferViewController to create a dismiss button: UIButton * dismissButton = [[UIButton alloc]   initWithFrame:CGRectMake(60.f, 440.f, 200.f, 44.f)]; [self.view addSubview:dismissButton]; [dismissButton setTitle:@"Dismiss"   forState:UIControlStateNormal]; [dismissButton setTitleColor:[UIColor whiteColor] forState:UIControlStateNormal]; [dismissButton addTarget:self   action:@selector(dismissTapped:)   forControlEvents:UIControlEventTouchUpInside]; As you can see, the touch up event calls @selector(dismissTapped:), which doesn't exist yet. We can get a handle of LIViewController through the app delegate (which is an instance of LIAppDelegate). In order to use this, we need to import it and LIViewController. Add the following imports to the top of LIOfferViewController.m: #import "LIViewController.h" #import "LIAppDelegate.h" Finally, let's complete the tutorial by adding the dismissTapped method: -(void)dismissTapped:(UIButton*)sender {    [self dismissViewControllerAnimated:YES completion:^{        LIAppDelegate * delegate =           (LIAppDelegate*)[UIApplication           sharedApplication].delegate;        LIViewController * rootVc =           (LIViewController*)delegate.         window.rootViewController;        [rootVc offerDismissed];    }]; } Now, let's run our app. You should be presented with the location permission request as shown in the Requesting location permission figure, from the Understanding iBeacon permissions section. Tap on OK and then fire up the companion app. Play around with the Chapter 2 beacon configurations by turning them on and off. What you should see is something like the following figure: Our app working with the companion OS X app Remember that your app should only show one offer at a time and your beacon should only show each offer once per session. Summary Well done on completing your first real iBeacon powered app, which actually differentiates between beacons. In this article, we covered the real usage of UUID, major, and minor values. We also got introduced to the Core Location framework including the CLLocationManager class and its important delegate methods. We introduced the CLRegion class and discussed the permissions required when using CLLocationManager. Resources for Article: Further resources on this subject: Interacting with the User [Article] Physics with UIKit Dynamics [Article] BSD Socket Library [Article]
Read more
  • 0
  • 0
  • 8060

article-image-machine-learning-examples-applicable-businesses
Packt
25 Nov 2014
7 min read
Save for later

Machine Learning Examples Applicable to Businesses

Packt
25 Nov 2014
7 min read
The purpose of this article by Michele Usuelli, author of the book, R Machine Learning Essentials, is to show how you machine learning helps in solving a business problem. (For more resources related to this topic, see here.) Predicting the output The past marketing campaign targeted part of the customer base. Among other 1,000 clients, how do we identify the 100 that are keener to subscribe? We can build a model that learns from the data and estimates which clients are more similar to the ones that subscribed in the previous campaign. For each client, the model estimates a score that is higher if the client is more likely to subscribe. There are different machine learning models determining the scores and we use two well-performing techniques, as follows: Logistic regression: This is a variation of the linear regression to predict a binary output Random forest: This is an ensemble based on a decision tree that works well in presence of many features In the end, we need to choose one out of the two techniques. There are cross-validation methods that allow us to estimate model accuracy. Starting from that, we can measure the accuracy of both the options and pick the one performing better. After choosing the most proper machine learning algorithm, we can optimize it using cross validation. However, in order to avoid overcomplicating the model building, we don't perform any feature selection or parameter optimization. These are the steps to build and evaluate the models: Load the randomForest package containing the random forest algorithm:library('randomForest') Define the formula defining the output and the variable names. The formula is in the format output ~ feature1 + feature2 + ...: arrayFeatures <- names(dtBank)arrayFeatures <- arrayFeatures[arrayFeatures != 'output']formulaAll <- paste('output', '~')formulaAll <- paste(formulaAll, arrayFeatures[1])for(nameFeature in arrayFeatures[-1]){formulaAll <- paste(formulaAll, '+', nameFeature)}formulaAll <- formula(formulaAll) Initialize the table containing all the testing sets: dtTestBinded <- data.table() Define the number of iterations: nIter <- 10 Start a for loop: for(iIter in 1:nIter){ Define the training and the test datasets: indexTrain <- sample(x = c(TRUE, FALSE),size = nrow(dtBank),replace = T,prob = c(0.8, 0.2))dtTrain <- dtBank[indexTrain]dtTest <- dtBank[!indexTrain] Select a subset from the test set in such a way that we have the same number of output == 0 and output == 1. First, we split dtTest in two parts (dtTest0 and dtTest1) on the basis of the output and we count the number of rows of each part (n0 and n1). Then, as dtTest0 has more rows, we randomly select n1 rows. In the end, we redefine dtTest binding dtTest0 and dtTest1, as follows: dtTest1 <- dtTest[output == 1]dtTest0 <- dtTest[output == 0]n0 <- nrow(dtTest0)n1 <- nrow(dtTest1)dtTest0 <- dtTest0[sample(x = 1:n0, size = n1)]dtTest <- rbind(dtTest0, dtTest1) Build the random forest model using randomForest. The formula argument defines the relationship between variables and the data argument defines the training dataset. In order to avoid overcomplicating the model, all the other parameters are left as their defaults: modelRf <- randomForest(formula = formulaAll,data = dtTrain) Build the logistic regression model using glm, which is a function used to build Generalized Linear Models (GLM). GLMs are a generalization of the linear regression and they allow to define a link function that connects the linear predictor with the outputs. The input is the same as the random forest, with the addition of family = binomial(logit) defining that the regression is logistic: modelLr <- glm(formula = formulaAll,data = dtTest,family = binomial(logit)) Predict the output of the random forest. The function is predict and its main arguments are object defining the model and newdata defining the test set, as follows: dtTest[, outputRf := predict(object = modelRf, newdata = dtTest, type='response')] Predict the output of the logistic regression, using predict similar to the random forest. The other argument is type='response' and it is necessary in the case of the logistic regression: dtTest[, outputLr := predict(object = modelLr, newdata = dtTest, type='response')] Add the new test set to dtTestBinded: dtTestBinded <- rbind(dtTestBinded, dtTest) End the for loop: } We built dtTestBinded containing the output column that defines which clients subscribed and the scores estimated by the models. Comparing the scores with the real output, we can validate the model performances. In order to explore dtTestBinded, we can build a chart showing how the scores of the non-subscribing clients are distributed. Then, we add the distribution of the subscribing clients to the chart and compare them. In this way, we can see the difference between the scores of the two groups. Since we use the same chart for the random forest and for the logistic regression, we define a function building the chart by following the given steps: Define the function and its input that includes the data table and the name of the score column: plotDistributions <- function(dtTestBinded, colPred){ Compute the distribution density for the clients that didn't subscribe. With output == 0, we extract the clients not subscribing, and using density, we define a density object. The adjust parameter defines the smoothing bandwidth that is a parameter of the way we build the curve starting from the data. The bandwidth can be interpreted as the level of detail: densityLr0 <- dtTestBinded[   output == 0,   density(get(colPred), adjust = 0.5)   ] Compute the distribution density for the clients that subscribed: densityLr1 <- dtTestBinded[   output == 1,   density(get(colPred), adjust = 0.5)   ] Define the colors in the chart using rgb. The colors are transparent red and transparent blue: col0 <- rgb(1, 0, 0, 0.3)col1 <- rgb(0, 0, 1, 0.3) Build the plot with the density of the clients not subscribing. Here, polygon is a function that adds the area to the chart: plot(densityLr0, xlim = c(0, 1), main = 'density')polygon(densityLr0, col = col0, border = 'black') Add the clients that subscribed to the chart: polygon(densityLr1, col = col1, border = 'black') Add the legend: legend(   'top',   c('0', '1'),   pch = 16,   col = c(col0, col1)) End the function: return()} Now, we can use plotDistributions on the random forest output: par(mfrow = c(1, 1))plotDistributions(dtTestBinded, 'outputRf') The histogram obtained is as follows:   The x-axis represents the score and the y-axis represents the density that is proportional to the number of clients that subscribed for similar scores. Since we don't have a client for each possible score, assuming a level of detail of 0.01, the density curve is smoothed in the sense that the density of each score is the average between the data with a similar score. The red and blue areas represent the non-subscribing and subscribing clients respectively. As can be easily noticed, the violet area comes from the overlapping of the two curves. For each score, we can identify which density is higher. If the highest curve is red, the client will be more likely to subscribe, and vice versa. For the random forest, most of the non-subscribing client scores are between 0 and 0.2 and the density peak is around 0.05. The subscribing clients have a more spread score, although higher, and their peak is around 0.1. The two distributions overlap a lot, so it's not easy to identify which clients will subscribe starting from their scores. However, if the marketing campaign targets all customers with a score higher than 0.3, they will likely belong to the blue cluster. In conclusion, using random forest, we are able to identify a small set of customers that will subscribe very likely. Summary In this article, you learned how to predict your output using proper machine learning techniques. Resources for Article: Further resources on this subject: Using R for Statistics, Research, and Graphics [article] Machine Learning in Bioinformatics [article] Learning Data Analytics with R and Hadoop [article]
Read more
  • 0
  • 0
  • 1349
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at $19.99/month. Cancel anytime
article-image-no-nodistinct
Packt
25 Nov 2014
4 min read
Save for later

No to nodistinct

Packt
25 Nov 2014
4 min read
This article is written by Stephen Redmond, the author of Mastering QlikView. There is a great skill in creating the right expression to calculate the right answer. Being able to do this in all circumstances relies on having a good knowledge of creating advanced expressions. Of course, the best path to mastery in this subject is actually getting out and doing it, but there is a great argument here for regularly practicing with dummy or test datasets. (For more resources related to this topic, see here.) When presented with a problem that needs to be solved, all the QlikView masters will not necessarily know immediately how to answer it. What they will have though is a very good idea of where to start, that is, what to try and what not to try. This is what I hope to impart to you here. Knowing how to create many advanced expressions will arm you to know where to apply them—and where not to apply them. This is one area of QlikView that is alien to many people. For some reason, they fear the whole idea of concepts such as Aggr. However, the reality is that these concepts are actually very simple and supremely logical. Once you get your head around them, you will wonder what all the fuss was about. No to nodistinct The Aggr function has as an optional clause, that is, the possibility of stating that the aggregation will be either distinct or nodistinct. The default option is distinct, and as such, is rarely ever stated. In this default operation, the aggregation will only produce distinct results for every combination of dimensions—just as you would expect from a normal chart or straight table. The nodistinct option only makes sense within a chart, one that has more dimensions than are in the Aggr statement. In this case, the granularity of the chart is lower than the granularity of Aggr, and therefore, QlikView will only calculate that Aggr for the first occurrence of lower granularity dimensions and will return null for the other rows. If we specify nodistinct, the same result will be calculated across all of the lower granularity dimensions. This can be difficult to understand without seeing an example, so let's look at a common use case for this option. We will start with a dataset: ProductSales:Load * Inline [Product, Territory, Year, SalesProduct A, Territory A, 2013, 100Product B, Territory A, 2013, 110Product A, Territory B, 2013, 120Product B, Territory B, 2013, 130Product A, Territory A, 2014, 140Product B, Territory A, 2014, 150Product A, Territory B, 2014, 160Product B, Territory B, 2014, 170]; We will build a report from this data using a pivot table: Now, we want to bring the value in the Total column into a new column under each year, perhaps to calculate a percentage for each year. We might think that, because the total is the sum for each Product and Territory, we might use an Aggr in the following manner: Sum(Aggr(Sum(Sales), Product, Territory)) However, as stated previously, because the chart includes an additional dimension (Year) than Aggr, the expression will only be calculated for the first occurrence of each of the lower granularity dimensions (in this case, for Year = 2013): The commonly suggested fix for this is to use Aggr without Sum and with nodistinct as shown: Aggr(NoDistinct Sum(Sales), Product, Territory) This will allow the Aggr expression to be calculated across all the Year dimension values, and at first, it will appear to solve the problem: The problem occurs when we decide to have a total row on this chart: As there is no aggregation function surrounding Aggr, it does not total correctly at the Product or Territory dimensions. We can't add an aggregation function, such as Sum, because it will break one of the other totals. However, there is something different that we can do; something that doesn't involve Aggr at all! We can use our old friend Total: Sum(Total<Product, Territory> Sales) This will calculate correctly at all the levels: There might be other use cases for using a nodistinct clause in Aggr, but they should be reviewed to see whether a simpler Total will work instead. Summary We discussed an important function, the Aggr function. We now know that the Aggr function is extremely useful, but we don't need to apply it in all circumstances where we have vertical calculations. Resources for Article: Further resources on this subject: Common QlikView script errors [article] Introducing QlikView elements [article] Creating sheet objects and starting new list using Qlikview 11 [article]
Read more
  • 0
  • 0
  • 2023

Packt
25 Nov 2014
7 min read
Save for later

Creating an Apache JMeter™ test workbench

Packt
25 Nov 2014
7 min read
This article is written by Colin Henderson, the author of Mastering GeoServer. This article will give you a brief introduction about how to create an Apache JMeter™ test workbench. (For more resources related to this topic, see here.) Before we can get into the nitty-gritty of creating a test workbench for Apache JMeter™, we must download and install it. Apache JMeter™ is a 100 percent Java application, which means that it will run on any platform provided there is a Java 6 or higher runtime environment present. The binaries can be downloaded from http://jmeter.apache.org/download_jmeter.cgi, and at the time of writing, the latest version is 2.11. No installation is required; just download the ZIP file and decompress it to a location you can access from a command-line prompt or shell environment. To launch JMeter on Linux, simply open shell and enter the following command: $ cd <path_to_jmeter>/bin$ ./jmeter To launch JMeter on Windows, simply open a command prompt and enter the following command: C:> cd <path_to_jmeter>\binC:> jmeter After a short time, JMeter GUI should appear, where we can construct our test plan. For ease and convenience, consider setting your system's PATH environment variable to the location of the JMeter bin directory. In future, you will be able to launch JMeter from the command line without having to CD first. The JMeter workbench will open with an empty configuration ready for us to construct our test strategy: The first thing we need to do is give our test plan a name; for now, let's call it GeoServer Stress Test. We can also provide some comments, which is good practice as it will help us remember for what reason we devised the test plan in future. To demonstrate the use of JMeter, we will create a very simple test plan. In this test plan, we will simulate a certain number of users hitting our GeoServer concurrently and requesting maps. To set this up, we first need to add Thread Group to our test plan. In a JMeter test, a thread is equivalent to a user: In the left-hand side menu, we need to right-click on the GeoServer Stress Test node and choose the Add | Threads (Users) | Thread Group menu option. This will add a child node to the test plan that we right-clicked on. The right-hand side panel provides options that we can set for the thread group to control how the user requests are executed. For example, we can name it something meaningful, such as Web Map Requests. In this test, we will simulate 30 users, making map requests over a total duration of 10 minutes, with a 10-second delay between each user starting. The number of users is set by entering a value for Number of Threads; in this case, 30. The Ramp-Up Period option controls the delay in starting each user by specifying the duration in which all the threads must start. So, in our case, we enter a duration of 300 seconds, which means all 30 users will be started by the end of 300 seconds. This equates to a 10-second delay between starting threads (300 / 30 = 10). Finally, we will set a duration for the test to run over by ticking the box for Scheduler, and then specifying a value of 600 seconds for Duration. By specifying a duration value, we override the End Time setting. Next, we need to provide some basic configuration elements for our test. First, we need to set the default parameters for all web requests. Right-click on the Web Map Requests thread group node that we just created, and then navigate to Add | Config Element | User Defined Variables. This will add a new node in which we can specify the default HTTP request parameters for our test: In the right-hand side panel, we can specify any number of variables. We can use these as replacement tokens later when we configure the web requests that will be sent during our test run. In this panel, we specify all the standard WMS query parameters that we don't anticipate changing across requests. Taking this approach is a good practice as it means that we can create a mix of tests using the same values, so if we change one, we don't have to change all the different test elements. To execute requests, we need to add Logic Controller. JMeter contains a lot of different logic controllers, but in this instance, we will use Simple Controller to execute a request. To add the controller, right-click on the Web Map Requests node and navigate to Add | Logic Controller | Simple Controller. A simple controller does not require any configuration; it is merely a container for activities we want to execute. In our case, we want the controller to read some data from our CSV file, and then execute an HTTP request to WMS. To do this, we need to add a CSV dataset configuration. Right-click on the Simple Controller node and navigate to Add | Config Element | CSV Data Set Config. The settings for the CSV data are pretty straightforward. The filename is set to the file that we generated previously, containing the random WMS request properties. The path can be specified as relative or absolute. The Variable Names property is where we specify the structure of the CSV file. The Recycle on EOF option is important as it means that the CSV file will be re-read when the end of the file is reached. Finally, we need to set Sharing mode to All threads to ensure the data can be used across threads. Next, we need to add a delay to our requests to simulate user activity; in this case, we will introduce a small delay of 5 seconds to simulate a user performing a map-pan operation. Right-click on the Simple Controller node, and then navigate to Add | Timer | Constant Timer: Simply specify the value we want the thread to be paused for in milliseconds. Finally, we need to add a JMeter sampler, which is the unit that will actually perform the HTTP request. Right-click on the Simple Controller node and navigate to Add | Sampler | HTTP Request. This will add an HTTP Request sampler to the test plan: There is a lot of information that goes into this panel; however, all it does is construct an HTTP request that the thread will execute. We specify the server name or IP address along with the HTTP method to use. The important part of this panel is the Parameters tab, which is where we need to specify all the WMS request parameters. Notice that we used the tokens that we specified in the CSV Data Set Config and WMS Request Defaults configuration components. We use the ${token_name} token, and JMeter replaces the token with the appropriate value of the referenced variable. We configured our test plan, but before we execute it, we need to add some listeners to the plan. A JMeter listener is the component that will gather the information from all of the test runs that occur. We add listeners by right-clicking on the thread group node and then navigating to the Add | Listeners menu option. A list of available listeners is displayed, and we can select the one we want to add. For our purposes, we will add the Graph Results, Generate Summary Results, Summary Report, and Response Time Graph listeners. Each listener can have its output saved to a datafile for later review. When completed, our test plan structure should look like the following: Before executing the plan, we should save it for use later. Summary In this article, we looked at how Apache JMeter™ can be used to construct and execute test plans to place loads on our servers so that we can analyze the results and gain an understanding of how well our servers perform. Resources for Article: Further resources on this subject: Geo-Spatial Data in Python: Working with Geometry [article] Working with Geo-Spatial Data in Python [article] Getting Started with GeoServer [article]
Read more
  • 0
  • 0
  • 3458

article-image-audio-processing-and-generation-maxmsp
Packt
25 Nov 2014
19 min read
Save for later

Audio Processing and Generation in Max/MSP

Packt
25 Nov 2014
19 min read
In this article, by Patrik Lechner, the author of Multimedia Programming Using Max/MSP and TouchDesigner, focuses on the audio-specific examples. We will take a look at the following audio processing and generation techniques: Additive synthesis Subtractive synthesis Sampling Wave shaping Nearly every example provided here might be understood very intuitively or taken apart in hours of math and calculation. It's up to you how deep you want to go, but in order to develop some intuition; we'll have to be using some amount of Digital Signal Processing (DSP) theory. We will briefly cover the DSP theory, but it is highly recommended that you study its fundamentals deeper to clearly understand this scientific topic in case you are not familiar with it already. (For more resources related to this topic, see here.) Basic audio principles We already saw and stated that it's important to know, see, and hear what's happening along a signal way. If we work in the realm of audio, there are four most important ways to measure a signal, which are conceptually partly very different and offer a very broad perspective on audio signals if we always have all of them in the back of our heads. These are the following important ways: Numbers (actual sample values) Levels (such as RMS, LUFS, and dB FS) Transversal waves (waveform displays, so oscilloscopes) Spectra (an analysis of frequency components) There are many more ways to think about audio or signals in general, but these are the most common and important ones. Let's use them inside Max right away to observe their different behavior. We'll feed some very basic signals into them: DC offset, a sinusoid, and noise. The one that might surprise you the most and get you thinking is the constant signal or DC offset (if it's digital-analog converted). In the following screenshot, you can see how the different displays react: In general, one might think, we don't want any constant signals at all; we don't want any DC offset. However, we will use audio signals a lot to control things later, say, an LFO or sequencers that should run with great timing accuracy. Also, sometimes, we just add a DC offset to our audio streams by accident. You can see in the preceding screenshot, that a very slowly moving or constant signal can be observed best by looking at its value directly, for example, using the [number~] object. In a level display, the [meter~] or [levelmeter~] objects will seem to imply that the incoming signal is very loud, in fact, it should be at -6 dB Full Scale (FS). As it is very loud, we just can't hear anything since the frequency is infinitely low. This is reflected by the spectrum display too; we see a very low frequency at -6 dB. In theory, we should just see an infinitely thin spike at 0 Hz, so everything else can be considered an (inevitable but reducible) measuring error. Audio synthesis Awareness of these possibilities of viewing a signal and their constraints, and knowing how they actually work, will greatly increase our productivity. So let's get to actually synthesizing some waveforms. A good example of different views of a signal operation is Amplitude Modulation (AM); we will also try to formulate some other general principles using the example of AM. Amplitude modulation Amplitude modulation means the multiplication of a signal with an oscillator. This provides a method of generating sidebands, which is partial in a very easy, intuitive, and CPU-efficient way. Amplitude modulation seems like a word that has a very broad meaning and can be used as soon as we change a signal's amplitude by another signal. While this might be true, in the context of audio synthesis, it very specifically means the multiplication of two (most often sine) oscillators. Moreover, there is a distinction between AM and Ring Modulation. But before we get to this distinction, let's look at the following simple multiplication of two sine waves, and we are first going to look at the result in an oscilloscope as a wave: So in the preceding screenshot, we can see the two sine waves and their product. If we imagine every pair of samples being multiplied, the operation seems pretty intuitive as the result is what we would expect. But what does this resulting wave really mean besides looking like a product of two sine waves? What does it sound like? The wave seems to have stayed in there certainly, right? Well, viewing the product as a wave and looking at the whole process in the time domain rather than the frequency domain is helpful but slightly misleading. So let's jump over to the following frequency domain and look what's happening with the spectrum: So we can observe here that if we multiply a sine wave a with a sine wave b, a having a frequency of 1000 Hz and b a frequency of 100 Hz, we end up with two sine waves, one at 900 Hz and another at 1100 Hz. The original sine waves have disappeared. In general, we can say that the result of multiplying a and b is equal to adding and subtracting the frequencies. This is shown in the Equivalence to Sum and difference subpatcher (in the following screenshot, the two inlets to the spectrum display overlap completely, which might be hard to see): So in the preceding screenshot, you see a basic AM patcher that produces sidebands that we can predict quite easily. Multiplication is commutative; you will say, 1000 + 100 = 1100, 1000 - 100 = 900; that's alright, but what about 100 - 1000 and 100 + 1000? We get -900 and 1100 once again? It still works out, and the fact that it does has to do with negative frequencies, or the symmetry of a real frequency spectrum around 0. So you can see that the two ways of looking at our signal and thinking about AM lend different opportunities and pitfalls. Here is another way to think about AM: it's the convolution of the two spectra. We didn't talk about convolution yet; we will at a later point. But keep it in mind or do a little research on your own; this aspect of AM is yet another interesting one. Ring modulation versus amplitude modulation The difference between ring modulation and what we call AM in this context is that the former one uses a bipolar modulator and the latter one uses a unipolar one. So actually, this is just about scaling and offsetting one of the factors. The difference in the outcome is yet a big one; if we keep one oscillator unipolar, the other one will be present in the outcome. If we do so, it starts making sense to call one oscillator on the carrier and the other (unipolar) on the modulator. Also, it therefore introduces a modulation depth that controls the amplitude of the sidebands. In the following screenshot, you can see the resulting spectrum; we have the original signal, so the carrier plus two sidebands, which are the original signals, are shifted up and down: Therefore, you can see that AM has a possibility to roughen up our spectrum, which means we can use it to let through an original spectrum and add sidebands. Tremolo Tremolo (from the Latin word tremare, to shake or tremble) is a musical term, which means to change a sound's amplitude in regular short intervals. Many people confuse it with vibrato, which is a modulating pitch at regular intervals. AM is tremolo and FM is vibrato, and as a simple reminder, think that the V of vibrato is closer to the F of FM than to the A of AM. So multiplying the two oscillators' results in a different spectrum. But of course, we can also use multiplication to scale a signal and to change its amplitude. If we wanted to have a sine wave that has a tremolo, that is an oscillating variation in amplitude, with, say, a frequency of 1 Hertz, we would again multiply two sine waves, one with 1000 Hz for example and another with a frequency of 0.5 Hz. Why 0.5 Hz? Think about a sine wave; it has two peaks per cycle, a positive one and a negative one. We can visualize all that very well if we think about it in the time domain, looking at the result in an oscilloscope. But what about our view of the frequency domain? Well, let's go through it; when we multiply a sine with 1000 Hz and one with 0.5 Hz, we actually get two sine waves, one with 999.5 Hz and one with 100.5 Hz. Frequencies that close create beatings, since once in a while, their positive and negative peaks overlap, canceling out each other. In general, the frequency of the beating is defined by the difference in frequency, which is 1 Hz in this case. So we see, if we look at it this way, we come to the same result again of course, but this time, we actually think of two frequencies instead of one being attenuated. Lastly, we could have looked up trigonometric identities to anticipate what happens if we multiply two sine waves. We find the following: Here, φ and θ are the two angular frequencies multiplied by the time in seconds, for example: This is the equation for the 1000 Hz sine wave. Feedback Feedback always brings the complexity of a system to the next level. It can be used to stabilize a system, but can also make a given system unstable easily. In a strict sense, in the context of DSP, stability means that for a finite input to a system, we get finite output. Obviously, feedback can give us infinite output for a finite input. We can use attenuated feedback, for example, not only to make our AM patches recursive, adding more and more sidebands, but also to achieve some surprising results as we will see in a minute. Before we look at this application, let's quickly talk about feedback in general. In the digital domain, feedback always demands some amount of delay. This is because the evaluation of the chain of operations would otherwise resemble an infinite amount of operations on one sample. This is true for both the Max message domain (we get a stack overflow error if we use feedback without delaying or breaking the chain of events) and the MSP domain; audio will just stop working if we try it. So the minimum network for a feedback chain as a block diagram looks something like this: In the preceding screenshot, X is the input signal and x[n] is the current input sample; Y is the output signal and y[n] is the current output sample. In the block marked Z-m, i is a delay of m samples (m being a constant). Denoting a delay with Z-m comes from a mathematical construct named the Z-transform. The a term is also a constant used to attenuate the feedback circle. If no feedback is involved, it's sometimes helpful to think about block diagrams as processing whole signals. For example, if you think of a block diagram that consists only of multiplication with a constant, it would make a lot of sense to think of its output signal as a scaled version of the input signal. We wouldn't think of the network's processing or its output sample by sample. However, as soon as feedback is involved, without calculation or testing, this is the way we should think about the network. Before we look at the Max version of things, let's look at the difference equation of the network to get a better feeling of the notation. Try to find it yourself before looking at it too closely! In Max, or rather in MSP, we can introduce feedback as soon as we use a [tapin~] [tapout~] pair that introduces a delay. The minimum delay possible is the signal vector size. Another way is to simply use a [send~] and [receive~] pair in our loop. The [send~] and [receive~] pair will automatically introduce this minimum amount of delay if needed, so the delay will be introduced only if there is a feedback loop. If we need shorter delays and feedback, we have to go into the wonderful world of gen~. Here, our shortest delay time is one sample, and can be introduced via the [history] object. In the Fbdiagram.maxpat patcher, you can find a Max version, an MSP version, and a [gen~] version of our diagram. For the time being, let's just pretend that the gen domain is just another subpatcher/abstraction system that allows shorter delays with feedback and has a more limited set of objects that more or less work the same as the MSP ones. In the following screenshot, you can see the difference between the output of the MSP and the [gen~] domain. Obviously, the length of the delay time has quite an impact on the output. Also, don't forget that the MSP version's output will vary greatly depending on our vector size settings. Let's return to AM now. Feedback can, for example, be used to duplicate and shift our spectrum again and again. In the following screenshot, you can see a 1000 Hz sine wave that has been processed by a recursive AM to be duplicated and shifted up and down with a 100 Hz spacing: In the maybe surprising result, we can achieve with this technique is this: if we the modulating oscillator and the carrier have the same frequency, we end up with something that almost sounds like a sawtooth wave. Frequency modulation Frequency modulation or FM is a technique that allows us to create a lot of frequency components out of just two oscillators, which is why it was used a lot back in the days when oscillators were a rare, expensive good, or CPU performance was low. Still, especially when dealing with real-time synthesis, efficiency is a crucial factor, and the huge variety of sounds that can be achieved with just two oscillators and very few parameters can be very useful for live performance and so on. The idea of FM is of course to modulate an oscillator's frequency. The basic, admittedly useless form is depicted in the following screenshot: While trying to visualize what happens with the output in the time domain, we can imagine it as shown in the following screenshot. In the preceding screenshot, you can see the signal that is controlling the frequency. It is a sine wave with a frequency of 50 Hz, scaled and offset to range from -1000 to 5000, so the center or carrier frequency is 2000 Hz, which is modulated to an amount of 3000 Hz. You can see the output of the modulated oscillator in the following screenshot: If we extend the upper patch slightly, we end up with this: Although you can't see it in the screenshot, the sidebands are appearing with a 100 Hz spacing here, that is, with a spacing equal to the modulator's frequency. Pretty similar to AM right? But depending on the modulation amount, we get more and more sidebands. Controlling FM If the ratio between F(c) and F(m) is an integer, we end up with a harmonic spectrum, therefore, it may be more useful to rather control F(m) indirectly via a ratio parameter as it's done inside the SimpleRatioAndIndex subpatcher. Also, an Index parameter is typically introduced to make an FM patch even more controllable. The modulation index is defined as follows: Here, I is the index, Am is the amplitude of the modulation, what we called amount before, and fm is the modulator's frequency. So finally, after adding these two controls, we might arrive here: FM offers a wide range of possibilities, for example, the fact that we have a simple control for how harmonic/inharmonic our spectrum is can be useful to synthesize the mostly noisy attack phase of many instruments if we drive the ratio and index with an envelope as it's done in the SimpleEnvelopeDriven subpatcher. However, it's also very easy to synthesize very artificial, strange sounds. This basically has the following two reasons: Firstly, the partials appearing have amplitudes governed by Bessel functions that may seem quite unpredictable; the partials sometimes seem to have random amplitudes. Secondly, negative frequencies and fold back. If we generate partials with frequencies below 0 Hz, it is equivalent to creating the same positive frequency. For frequencies greater than the sample rate/2 (sample rate/2 is what's called the Nyquist rate), the frequencies reflect back into the spectrum that can be described by our sampling rate (this is an effect also called aliasing). So at a sampling rate of 44,100 Hz, a partial with a frequency of -100 Hz will appear at 100 Hz, and a partial with a frequency of 43100 kHz will appear at 1000 Hz, as shown in the following screenshot: So, for frequencies between the Nyquist frequency and the sampling frequency, what we hear is described by this: Here, fs is the sampling rate, f0 is the frequency we hear, and fi is the frequency we are trying to synthesize. Since FM leads to many partials, this effect can easily come up, and can both be used in an artistically interesting manner or sometimes appear as an unwanted error. In theory, an FM signal's partials extend to even infinity, but the amplitudes become negligibly small. If we want to reduce this behavior, the [poly~] object can be used to oversample the process, generating a bit more headroom for high frequencies. The phenomenon of aliasing can be understood by thinking of a real (in contrast to imaginary) digital signal as having a symmetrical and periodical spectrum; let's not go into too much detail here and look at it in the time domain: In the previous screenshot, we again tried to synthesize a sine wave with 43100 Hz (the dotted line) at a sampling rate of 44100 Hz. What we actually get is the straight black line, a sine with 1000 Hz. Each big black dot represents an actual sample, and there is only one single band-limited signal connecting them: the 1000 Hz wave that is only partly visible here (about half its wavelength). Feedback It is very common to use feedback with FM. We can even frequency modulate one oscillator with itself, making the algorithm even cheaper since we have only one table lookup. The idea of feedback FM quickly leads us to the idea of making networks of oscillators that can be modulated by each other, including feedback paths, but let's keep it simple for now. One might think that modulating one oscillator with itself should produce chaos; FM being a technique that is not the easiest to control, one shouldn't care for playing around with single operator feedback FM. But the opposite is the case. Single operator FM yields very predictable partials, as shown in the following screenshot, and in the Single OP FBFM subpatcher: Again, we are using a gen~ patch, since we want to create a feedback loop and are heading for a short delay in the loop. Note that we are using the [param] object to pass a message into the gen~ object. What should catch your attention is that although the carrier frequency has been adjusted to 1000 Hz, the fundamental frequency in the spectrum is around 600 Hz. What can help us here is switching to phase modulation. Phase modulation If you look at the gen~ patch in the previous screenshot, you see that we are driving our sine oscillator with a phasor. The cycle object's phase inlet assumes an input that ranges from 0 to 1 instead of from 0 to 2π, as one might think. To drive a sine wave through one full cycle in math, we can use a variable ranging from 0 to 2π, so in the following formula, you can imagine t being provided by a phasor, which is the running phase. The 2π multiplication isn't necessary in Max since if we are using [cycle~], we are reading out a wavetable actually instead of really computing the sine or cosine of the input. This is the most common form of denoting a running sinusoid with frequency f0 and phase φ. Try to come up with a formula that describes frequency modulation! Simplifying the phases by setting it to zero, we can denote FM as follows: This can be shown to be nearly identical to the following formula: Here, f0 is the frequency of the carrier, fm is the frequency of the modulator, and A is the modulation amount. Welcome to phase modulation. If you compare it, the previous formula actually just inserts a scaled sine wave where the phase φ used to be. So phase modulation is nearly identical to frequency modulation. Phase modulation has some advantages though, such as providing us with an easy method of synchronizing multiple oscillators. But let's go back to the Max side of things and look at a feedback phase modulation patch right away (ignoring simple phase modulation, since it really is so similar to FM): This gen~ patcher resides inside the One OP FBPM subpatcher and implements phase modulation using one oscillator and feedback. Interestingly, the spectrum is very similar to the one of a sawtooth wave, with the feedback amount having a similar effect of a low-pass filter, controlling the amount of partials. If you take a look at the subpatcher, you'll find the following three sound sources: Our feedback FM gen~ patcher A [saw~] object for comparison A poly~ object We have already mentioned the problem of aliasing and the [poly~] object has already been proposed to treat the problem. However, it allows us to define the quality of parts of patches in general, so let's talk about the object a bit before moving on since we will make great use of it. Before moving on, I would like to tell you that you can double-click on it to see what is loaded inside, and you will see that the subpatcher we just discussed contains a [poly~] object that contains yet another version of our gen~ patcher. Summary In this article, we've finally come to talking about audio. We've introduced some very common techniques and thought about refining them and getting things done properly and efficiently (think about poly~). By now, you should feel quite comfortable building synths that mix techniques such as FM, subtractive synthesis, and feature modulation, as well as using matrices for routing both audio and modulation signals where you need them. Further resources on this subject: Moodle for Online Communities [Article] Techniques for Creating a Multimedia Database [Article] Moodle 2.0 Multimedia: Working with 2D and 3D Maps [Article]
Read more
  • 0
  • 0
  • 14435

article-image-components
Packt
25 Nov 2014
14 min read
Save for later

Components

Packt
25 Nov 2014
14 min read
This article by Timothy Moran, author of Mastering KnockoutJS, teaches you how to use the new Knockout components feature. (For more resources related to this topic, see here.) In Version 3.2, Knockout added components using the combination of a template (view) with a viewmodel to create reusable, behavior-driven DOM objects. Knockout components are inspired by web components, a new (and experimental, at the time of writing this) set of standards that allow developers to define custom HTML elements paired with JavaScript that create packed controls. Like web components, Knockout allows the developer to use custom HTML tags to represent these components in the DOM. Knockout also allows components to be instantiated with a binding handler on standard HTML elements. Knockout binds components by injecting an HTML template, which is bound to its own viewmodel. This is probably the single largest feature Knockout has ever added to the core library. The reason we started with RequireJS is that components can optionally be loaded and defined with module loaders, including their HTML templates! This means that our entire application (even the HTML) can be defined in independent modules, instead of as a single hierarchy, and loaded asynchronously. The basic component registration Unlike extenders and binding handlers, which are created by just adding an object to Knockout, components are created by calling the ko.components.register function: ko.components.register('contact-list, { viewModel: function(params) { }, template: //template string or object }); This will create a new component named contact-list, which uses the object returned by the viewModel function as a binding context, and the template as its view. It is recommended that you use lowercase, dash-separated names for components so that they can easily be used as custom elements in your HTML. To use this newly created component, you can use a custom element or the component binding. All the following three tags produce equivalent results: <contact-list params="data: contacts"><contact-list> <div data-bind="component: { name: 'contact-list', params: { data: contacts }"></div> <!-- ko component: { name: 'contact-list', params: { data: contacts } --><!-- /ko --> Obviously, the custom element syntax is much cleaner and easier to read. It is important to note that custom elements cannot be self-closing tags. This is a restriction of the HTML parser and cannot be controlled by Knockout. There is one advantage of using the component binding: the name of the component can be an observable. If the name of the component changes, the previous component will be disposed (just like it would if a control flow binding removed it) and the new component will be initialized. The params attribute of custom elements work in a manner that is similar to the data-bind attribute. Comma-separated key/value pairs are parsed to create a property bag, which is given to the component. The values can contain JavaScript literals, observable properties, or expressions. It is also possible to register a component without a viewmodel, in which case, the object created by params is directly used as the binding context. To see this, we'll convert the list of contacts into a component: <contact-list params="contacts: displayContacts, edit: editContact, delete: deleteContact"> </contact-list> The HTML code for the list is replaced with a custom element with parameters for the list as well as callbacks for the two buttons, which are edit and delete: ko.components.register('contact-list', { template: '<ul class="list-unstyled" data-bind="foreach: contacts">'    +'<li>'      +'<h3>'        +'<span data-bind="text: displayName"></span> <small data-          bind="text: phoneNumber"></small> '        +'<button class="btn btn-sm btn-default" data-bind="click:          $parent.edit">Edit</button> '        +'<button class="btn btn-sm btn-danger" data-bind="click:          $parent.delete">Delete</button>'      +'</h3>'    +'</li>' +'</ul>' }); This component registration uses an inline template. Everything still looks and works the same, but the resulting HTML now includes our custom element. Custom elements in IE 8 and higher IE 9 and later versions as well as all other major browsers have no issue with seeing custom elements in the DOM before they have been registered. However, older versions of IE will remove the element if it hasn't been registered. The registration can be done either with Knockout, with ko.components.register('component-name'), or with the standard document.createElement('component-name') expression statement. One of these must come before the custom element, either by the script containing them being first in the DOM, or by the custom element being added at runtime. When using RequireJS, being in the DOM first won't help as the loading is asynchronous. If you need to support older IE versions, it is recommended that you include a separate script to register the custom element names at the top of the body tag or in the head tag: <!DOCTYPE html> <html> <body>    <script>      document.createElement('my-custom-element');    </script>    <script src='require.js' data-main='app/startup'></script>      <my-custom-element></my-custom-element> </body> </html> Once this has been done, components will work in IE 6 and higher even with custom elements. Template registration The template property of the configuration sent to register can take any of the following formats: ko.components.register('component-name', { template: [OPTION] }); The element ID Consider the following code statement: template: { element: 'component-template' } If you specify the ID of an element in the DOM, the contents of that element will be used as the template for the component. Although it isn't supported in IE yet, the template element is a good candidate, as browsers do not visually render the contents of template elements. The element instance Consider the following code statement: template: { element: instance } You can pass a real DOM element to the template to be used. This might be useful in a scenario where the template was constructed programmatically. Like the element ID method, only the contents of the elements will be used as the template: var template = document.getElementById('contact-list-template'); ko.components.register('contact-list', { template: { element: template } }); An array of DOM nodes Consider the following code statement: template: [nodes] If you pass an array of DOM nodes to the template configuration, then the entire array will be used as a template and not just the descendants: var template = document.getElementById('contact-list-template') nodes = Array.prototype.slice.call(template.content.childNodes); ko.components.register('contact-list', { template: nodes }); Document fragments Consider the following code statement: template: documentFragmentInstance If you pass a document fragment, the entire fragment will be used as a template instead of just the descendants: var template = document.getElementById('contact-list-template'); ko.components.register('contact-list', { template: template.content }); This example works because template elements wrap their contents in a document fragment in order to stop the normal rendering. Using the content is the same method that Knockout uses internally when a template element is supplied. HTML strings We already saw an example for HTML strings in the previous section. While using the value inline is probably uncommon, supplying a string would be an easy thing to do if your build system provided it for you. Registering templates using the AMD module Consider the following code statement: template: { require: 'module/path' } If a require property is passed to the configuration object of a template, the default module loader will load the module and use it as the template. The module can return any of the preceding formats. This is especially useful for the RequireJS text plugin: ko.components.register('contact-list', { template: { require: 'text!contact-list.html'} }); Using this method, we can extract the HTML template into its own file, drastically improving its organization. By itself, this is a huge benefit to development. The viewmodel registration Like template registration, viewmodels can be registered using several different formats. To demonstrate this, we'll use a simple viewmodel of our contacts list components: function ListViewmodel(params) { this.contacts = params.contacts; this.edit = params.edit; this.delete = function(contact) {    console.log('Mock Deleting Contact', ko.toJS(contact)); }; }; To verify that things are getting wired up properly, you'll want something interactive; hence, we use the fake delete function. The constructor function Consider the following code statement: viewModel: Constructor If you supply a function to the viewModel property, it will be treated as a constructor. When the component is instantiated, new will be called on the function, with the params object as its first parameter: ko.components.register('contact-list', { template: { require: 'text!contact-list.html'}, viewModel: ListViewmodel //Defined above }); A singleton object Consider the following code statement: viewModel: { instance: singleton } If you want all your component instances to be backed by a shared object—though this is not recommended—you can pass it as the instance property of a configuration object. Because the object is shared, parameters cannot be passed to the viewmodel using this method. The factory function Consider the following code statement: viewModel: { createViewModel: function(params, componentInfo) {} } This method is useful because it supplies the container element of the component to the second parameter on componentInfo.element. It also provides you with the opportunity to perform any other setup, such as modifying or extending the constructor parameters. The createViewModel function should return an instance of a viewmodel component: ko.components.register('contact-list', { template: { require: 'text!contact-list.html'}, viewModel: { createViewModel: function(params, componentInfo) {    console.log('Initializing component for',      componentInfo.element);    return new ListViewmodel(params); }} }); Registering viewmodels using an AMD module Consider the following code statement: viewModel: { require: 'module-path' } Just like templates, viewmodels can be registered with an AMD module that returns any of the preceding formats. Registering AMD In addition to registering the template and the viewmodel as AMD modules individually, you can register the entire component with a require call: ko.components.register('contact-list', { require: 'contact-list' }); The AMD module will return the entire component configuration: define(['knockout', 'text!contact-list.html'], function(ko, templateString) {   function ListViewmodel(params) {    this.contacts = params.contacts;    this.edit = params.edit;    this.delete = function(contact) {      console.log('Mock Deleting Contact', ko.toJS(contact));    }; }   return { template: templateString, viewModel: ListViewmodel }; }); As the Knockout documentation points out, this method has several benefits: The registration call is just a require path, which is easy to manage. The component is composed of two parts: a JavaScript module and an HTML module. This provides both simple organization and clean separation. The RequireJS optimizer, which is r.js, can use the text dependency on the HTML module to bundle the HTML code with the bundled output. This means your entire application, including the HTML templates, can be a single file in production (or a collection of bundles if you want to take advantage of lazy loading). Observing changes in component parameters Component parameters will be passed via the params object to the component's viewmodel in one of the following three ways: No observable expression evaluation needs to occur, and the value is passed literally: <component params="name: 'Timothy Moran'"></component> <component params="name: nonObservableProperty"> </component> <component params="name: observableProperty"></component> <component params="name: viewModel.observableSubProperty "></component> In all of these cases, the value is passed directly to the component on the params object. This means that changes to these values will change the property on the instantiating viewmodel, except for the first case (literal values). Observable values can be subscribed to normally. An observable expression needs to be evaluated, so it is wrapped in a computed observable: <component params="name: name() + '!'"></component> In this case, params.name is not the original property. Calling params.name() will evaluate the computed wrapper. Trying to modify the value will fail, as the computed value is not writable. The value can be subscribed to normally. An observable expression evaluates an observable instance, so it is wrapped in an observable that unwraps the result of the expression: <component params="name: isFormal() ? firstName : lastName"></component> In this example, firstName and lastName are both observable properties. If calling params.name() returned the observable, you will need to call params.name()() to get the actual value, which is rather ugly. Instead, Knockout automatically unwraps the expression so that calling params.name() returns the actual value of either firstName or lastName. If you need to access the actual observable instances to, for example, write a value to them, trying to write to params.name will fail, as it is a computed observable. To get the unwrapped value, you can use the params.$raw object, which provides the unwrapped values. In this case, you can update the name by calling params.$raw.name('New'). In general, this case should be avoided by removing the logic from the binding expression and placing it in a computed observable in the viewmodel. The component's life cycle When a component binding is applied, Knockout takes the following steps. The component loader asynchronously creates the viewmodel factory and template. This result is cached so that it is only performed once per component. The template is cloned and injected into the container (either the custom element or the element with the component binding). If the component has a viewmodel, it is instantiated. This is done synchronously. The component is bound to either the viewmodel or the params object. The component is left active until it is disposed. The component is disposed. If the viewmodel has a dispose method, it is called, and then the template is removed from the DOM. The component's disposal If the component is removed from the DOM by Knockout, either because of the name of the component binding or a control flow binding being changed (for example, if and foreach), the component will be disposed. If the component's viewmodel has a dispose function, it will be called. Normal Knockout bindings in the components view will be automatically disposed, just as they would in a normal control flow situation. However, anything set up by the viewmodel needs to be manually cleaned up. Some examples of viewmodel cleanup include the following: The setInterval callbacks can be removed with clearInterval. Computed observables can be removed by calling their dispose method. Pure computed observables don't need to be disposed. Computed observables that are only used by bindings or other viewmodel properties also do not need to be disposed, as garbage collection will catch them. Observable subscriptions can be disposed by calling their dispose method. Event handlers can be created by components that are not part of a normal Knockout binding. Combining components with data bindings There is only one restriction of data-bind attributes that are used on custom elements with the component binding: the binding handlers cannot use controlsDescendantBindings. This isn't a new restriction; two bindings that control descendants cannot be on a single element, and since components control descendant bindings that cannot be combined with a binding handler that also controls descendants. It is worth remembering, though, as you might be inclined to place an if or foreach binding on a component; doing this will cause an error. Instead, wrap the component with an element or a containerless binding: <ul data-bind='foreach: allProducts'> <product-details params='product: $data'></product-details> </ul> It's also worth noting that bindings such as text and html will replace the contents of the element they are on. When used with components, this will potentially result in the component being lost, so it's not a good idea. Summary In this article, we learned that the Knockout components feature gives you a powerful tool that will help you create reusable, behavior-driven DOM elements. Resources for Article: Further resources on this subject: Deploying a Vert.x application [Article] The Dialog Widget [Article] Top features of KnockoutJS [Article]
Read more
  • 0
  • 0
  • 2586
article-image-understanding-hbase-ecosystem
Packt
24 Nov 2014
11 min read
Save for later

Understanding the HBase Ecosystem

Packt
24 Nov 2014
11 min read
This article by Shashwat Shriparv, author of the book, Learning HBase, will introduce you to the world of HBase. (For more resources related to this topic, see here.) HBase is a horizontally scalable, distributed, open source, and a sorted map database. It runs on top of Hadoop file system that is Hadoop Distributed File System (HDFS). HBase is a NoSQL nonrelational database that doesn't always require a predefined schema. It can be seen as a scaling flexible, multidimensional spreadsheet where any structure of data is fit with on-the-fly addition of new column fields, and fined column structure before data can be inserted or queried. In other words, HBase is a column-based database that runs on top of Hadoop distributed file system and supports features such as linear scalability (scale out), automatic failover, automatic sharding, and more flexible schema. HBase is modeled on Google BigTable. It was inspired by Google BigTable, which is a compressed, high-performance, proprietary data store built on the Google filesystem. HBase was developed as a Hadoop subproject to support storage of structural data, which can take advantage of most distributed files systems (typically, the Hadoop Distributed File System known as HDFS). The following table contains key information about HBase and its features: Features Description Developed by Apache Written in Java Type Column oriented License Apache License Lacking features of relational databases SQL support, relations, primary, foreign, and unique key constraints, normalization Website http://hbase.apache.org Distributions Apache, Cloudera Download link http://mirrors.advancedhosters.com/apache/hbase/ Mailing lists The user list: user-subscribe@hbase.apache.org The developer list: dev-subscribe@hbase.apache.org Blog http://blogs.apache.org/hbase/ HBase layout on top of Hadoop The following figure represents the layout information of HBase on top of Hadoop: There is more than one ZooKeeper in the setup, which provides high availability of master status; a RegionServer may contain multiple rations. The RegionServers run on the machines where DataNodes run. There can be as many RegionServers as DataNodes. RegionServers can have multiple HRegions; one HRegion can have one HLog and multiple HFiles with its associate's MemStore. HBase can be seen as a master-slave database where the master is called HMaster, which is responsible for coordination between client application and HRegionServer. It is also responsible for monitoring and recording metadata changes and management. Slaves are called HRegionServers, which serve the actual tables in form of regions. These regions are the basic building blocks of the HBase tables, which contain distribution of tables. So, HMaster and RegionServer work in coordination to serve the HBase tables and HBase cluster. Usually, HMaster is co-hosted with Hadoop NameNode daemon process on a server and communicates to DataNode daemon for reading and writing data on HDFS. The RegionServer runs or is co-hosted on the Hadoop DataNodes. Comparing architectural differences between RDBMs and HBase Let's list the major differences between relational databases and HBase: Relational databases HBase Uses tables as databases Uses regions as databases Filesystems supported are FAT, NTFS, and EXT Filesystem supported is HDFS The technique used to store logs is commit logs The technique used to store logs is Write-Ahead Logs (WAL) The reference system used is coordinate system The reference system used is ZooKeeper Uses the primary key Uses the row key Partitioning is supported Sharding is supported Use of rows, columns, and cells Use of rows, column families, columns, and cells HBase features Let's see the major features of HBase that make it one of the most useful databases for the current and future industry: Automatic failover and load balancing: HBase runs on top of HDFS, which is internally distributed and automatically recovered using multiple block allocation and replications. It works with multiple HMasters and region servers. This failover is also facilitated using HBase and RegionServer replication. Automatic sharding: An HBase table is made up of regions that are hosted by RegionServers and these regions are distributed throughout the RegionServers on different DataNodes. HBase provides automatic and manual splitting of these regions to smaller subregions, once it reaches a threshold size to reduce I/O time and overhead. Hadoop/HDFS integration: It's important to note that HBase can run on top of other filesystems as well. While HDFS is the most common choice as it supports data distribution and high availability using distributed Hadoop, for which we just need to set some configuration parameters and enable HBase to communicate to Hadoop, an out-of-the-box underlying distribution is provided by HDFS. Real-time, random big data access: HBase uses log-structured merge-tree (LSM-tree) as data storage architecture internally, which merges smaller files to larger files periodically to reduce disk seeks. MapReduce: HBase has a built-in support of Hadoop MapReduce framework for fast and parallel processing of data stored in HBase. You can search for the Package org.apache.hadoop.hbase.mapreduce for more details. Java API for client access: HBase has a solid Java API support (client/server) for easy development and programming. Thrift and a RESTtful web service: HBase not only provides a thrift and RESTful gateway but also web service gateways for integrating and accessing HBase besides Java code (HBase Java APIs) for accessing and working with HBase. Support for exporting metrics via the Hadoop metrics subsystem: HBase provides Java Management Extensions (JMX) and exporting matrix for monitoring purposes with tools such as Ganglia and Nagios. Distributed: HBase works when used with HDFS. It provides coordination with Hadoop so that distribution of tables, high availability, and consistency is supported by it. Linear scalability (scale out): Scaling of HBase is not scale up but scale out, which means that we don't need to make servers more powerful but we add more machines to its cluster. We can add more nodes to the cluster on the fly. As soon as a new RegionServer node is up, the cluster can begin rebalancing, start the RegionServer on the new node, and it is scaled up, it is as simple as that. Column oriented: HBase stores each column separately in contrast with most of the relational databases, which uses stores or are row-based storage. So in HBase, columns are stored contiguously and not the rows. More about row- and column-oriented databases will follow. HBase shell support: HBase provides a command-line tool to interact with HBase and perform simple operations such as creating tables, adding data, and scanning data. This also provides full-fledged command-line tool using which we can interact with HBase and perform operations such as creating table, adding data, removing data, and a few other administrative commands. Sparse, multidimensional, sorted map database: HBase is a sparse, multidimensional, sorted map-based database, which supports multiple versions of the same record. Snapshot support: HBase supports taking snapshots of metadata for getting the previous or correct state form of data. HBase in the Hadoop ecosystem Let's see where HBase sits in the Hadoop ecosystem. In the Hadoop ecosystem, HBase provides a persistent, structured, schema-based data store. The following figure illustrates the Hadoop ecosystem: HBase can work as a separate entity on the local filesystem (which is not really effective as no distribution is provided) as well as in coordination with Hadoop as a separate but connected entity. As we know, Hadoop provides two services, a distributed files system (HDFS) for storage and a MapReduce framework for processing in a parallel mode. When there was a need to store structured data (data in the form of tables, rows and columns), which most of the programmers are already familiar with, the programmers were finding it difficult to process the data that was stored on HDFS as an unstructured flat file format. This led to the evolution of HBase, which provided a way to store data in a structural way. Consider that we have got a CSV file stored on HDFS and we need to query from it. We would need to write a Java code for this, which wouldn't be a good option. It would be better if we could specify the data key and fetch the data from that file. So, what we can do here is create a schema or table with the same structure of CSV file to store the data of the CSV file in the HBase table and query using HBase APIs, or HBase shell using key. Data representation in HBase Let's look into the representation of rows and columns in HBase table: An HBase table is divided into rows, column families, columns, and cells. Row keys are unique keys to identify a row, column families are groups of columns, columns are fields of the table, and the cell contains the actual value or the data. So, we have been through the introduction of HBase; now, let's see what Hadoop and its components are in brief. It is assumed here that you are already familiar with Hadoop; if not, following a brief introduction about Hadoop will help you to understand it. Hadoop Hadoop is an underlying technology of HBase, providing high availability, fault tolerance, and distribution. It is an Apache-sponsored, free, open source, Java-based programming framework which supports large dataset storage. It provides distributed file system and MapReduce, which is a distributed programming framework. It provides a scalable, reliable, distributed storage and development environment. Hadoop makes it possible to run applications on a system with tens to tens of thousands of nodes. The underlying distributed file system provides large-scale storage, rapid data access. It has the following submodules: Hadoop Common: This is the core component that supports the other Hadoop modules. It is like the master components facilitating communication and coordination between different Hadoop modules. Hadoop distributed file system: This is the underlying distributed file system, which is abstracted on the top of the local filesystem that provides high throughput of read and write operations of data on Hadoop. Hadoop YARN: This is the new framework that is shipped with newer releases of Hadoop. It provides job scheduling and job and resource management. Hadoop MapReduce: This is the Hadoop-based processing system that provides parallel processing of large data and datasets. Other Hadoop subprojects are HBase, Hive, Ambari, Avro, Cassandra (Cassandra isn't a Hadoop subproject, it's a related project; they solve similar problems in different ways), Mahout, Pig, Spark, ZooKeeper (ZooKeeper isn't a Hadoop subproject. It's a dependency shared by many distributed systems), and so on. All of these have different usability and the combination of all these subprojects forms the Hadoop ecosystem. Core daemons of Hadoop The following are the core daemons of Hadoop: NameNode: This stores and manages all metadata about the data present on the cluster, so it is the single point of contact to Hadoop. In the new release of Hadoop, we have an option of more than one NameNode for high availability. JobTracker: This runs on the NameNode and performs the MapReduce of the jobs submitted to the cluster. SecondaryNameNode: This maintains the backup of metadata present on the NameNode, and also records the file system changes. DataNode: This will contain the actual data. TaskTracker: This will perform tasks on the local data assigned by the JobTracker. The preceding are the daemons in the case of Hadoop v1 or earlier. In newer versions of Hadoop, we have ResourceManager instead of JobTracker, the node manager instead of TaskTrackers, and the YARN framework instead of a simple MapReduce framework. The following is the comparison between daemons in Hadoop 1 and Hadoop 2: Hadoop 1 Hadoop 2 HDFS NameNode Secondary NameNode DataNode   NameNode (more than one active/standby) Checkpoint node DataNode Processing MapReduce v1 JobTracker TaskTracker   YARN (MRv2) ResourceManager NodeManager Application Master Comparing HBase with Hadoop As we now know what HBase and what Hadoop are, let's have a comparison between HDFS and HBase for better understanding: Hadoop/HDFS HBase This provide filesystem for distributed storage This provides tabular column-oriented data storage This is optimized for storage of huge-sized files with no random read/write of these files This is optimized for tabular data with random read/write facility This uses flat files This uses key-value pairs of data The data model is not flexible Provides a flexible data model This uses file system and processing framework This uses tabular storage with built-in Hadoop MapReduce support This is mostly optimized for write-once read-many This is optimized for both read/write many Summary So in this article, we discussed the introductory aspects of HBase and it's features. We have also discussed HBase's components and their place in the HBase ecosystem. Resources for Article: Further resources on this subject: The HBase's Data Storage [Article] HBase Administration, Performance Tuning [Article] Comparative Study of NoSQL Products [Article]
Read more
  • 0
  • 0
  • 15533

article-image-configuring-vshield-app
Packt
24 Nov 2014
10 min read
Save for later

Configuring vShield App

Packt
24 Nov 2014
10 min read
In this article by Mike Greer, the author of vSphere Security Cookbook, we will cover the following topics: Installing vShield App Configuring vShield App Configuring vShield App Flow Monitoring (For more resources related to this topic, see here.) Introduction In most modern operating systems exists the capability to install a firewall on the host itself. The rules configured in a host-based firewall manage the traffic at the host level, and provide an additional layer of defense along with network firewalls and intrusion detection systems. Multiple layers of security provide a complete defense-in-depth architecture. The concept of defense-in-depth builds layers of security providing protection, should another layer fail or be compromised. The second component of the vShield family to be configured that we'll discuss is vShield App. vShield App is a host-based layer 2 firewall that is implemented at the vNIC level of the hypervisor. vShield App presents itself as a virtual appliance in the vCenter management tool. For each protected ESXi host, there is an associated vShield App virtual machine that runs on the said host. To protect the entire virtualization environment managed by vCenter, it is important to install vShield App on each ESXi host in the datacenter. Failure to protect each host will allow the opportunity for virtual machines to be moved to an unprotected host either by vMotion or manually. In the event where DRS is being used, it is very likely that virtual machines will be moved to an unprotected host, assuming that it has resources available and there is a high load on adjacent hosts. Installing vShield App The vShield App is required to provide host-level security and firewall services to each individual ESXi host. This process must be completed on each ESXi host individually. Getting ready A Core Infrastructure Suite (CIS) or vCloud Networking and Security (vCNS) license must be installed prior to installing vShield App and vShield Edge. vShield App is installed per ESXi host, and vShield Manager must have been previously installed as a prerequisite. In order to proceed, we require access to vShield Web Console. The client can be run on any modern Windows or Mac desktop operating system or server operating system. vShield Web Console requires Adobe Flash, which is not supported on Linux operating systems at this time. Ensure the account used for login has administrative rights to vShield Manager. How to do it… Perform the following steps: We'll open the vShield Manager Web Console and log in with an administrative account. Navigate to Datacenter | Lab Cluster | esx5501.training.lab within vShield Manager. Ensure the ESXi host targeted for installation is not hosting the VM running vCenter. A connection is required to vCenter and the installation of vShield App will disrupt the network connection to the ESXi host. Locate vShield App from the Summary tab. Click on Install next to vShield App. Select the Datastore to hold the vShield App service information. In our example, we'll use datastore1. Select the available Management Port Group that can communicate with the vShield Manager installed previously. In our example, we'll use Internal Network. Enter the vShield App IP address. In our example, we'll use 192.168.10.30. The IP address of the vShield App must be unique and not previously assigned. Enter the Netmask. In our example, we'll use 255.255.255.0. Enter the Default Gateway. In our example, we'll use 192.168.10.1. Ensure the vShield Endpoint checkbox is cleared. Click on the Install button. The status will be shown during the installation, as shown in the following screenshot: Verify the completion of the setup with no errors. If an error occurs, the details regarding the error will be highlighted in yellow and begin with vShield App installation encountered error while installing service VM <error details>. Repeat this process for additional ESXi hosts. If vCenter is running on an ESXi host, use vMotion to migrate that VM to another host prior to installation. How it works… vShield App provides firewall functionality to an ESXi host by installing a virtual appliance that is tied to the local host. The virtual appliance is stored on the local datastore to the host. Each firewall appliance is named with the host name included for clarity. In our example, we installed on the ESXi host named esx5501.training.lab and the corresponding firewall appliance is named vShield-FW-esx5501.training.lab. One important point to consider is if the vShield App fails on a particular ESXi host and the default rule is set to deny, then all traffic will be denied to the host which can make troubleshooting difficult. If a vShield App installation fails and requires manual removal, the process to remove the failed install will require the ESXi host to be rebooted in the process. As a result, all virtual machines running on the host will need to be migrated to other nodes of the cluster or powered down. Configuring vShield App There are several important configurations to set in the vShield App management Web Console. Configuring Fail Safe Mode sets the actions that will be taken if the vShield App fails or is down. The Fail Safe Mode can either be set to allow or block. Excluding virtual machines such as vCenter is key to allow proper functionality since it will exclude any virtual machine from firewall rules. Getting ready In order to proceed, we require access to vShield Web Console. The client can be run on any modern Windows or Mac desktop operating system or server operating system. vShield Web Console requires Adobe Flash, which is not supported on Linux operating systems at this time. Ensure the account used for log in has administrative rights to vShield Manager. How to do it… Viewing current status is the first step in assessing the state of the vShield App. Once the app is verified to be in a healthy state, additional configurations can be accomplished by performing the following steps: Launch vSphere Client using an account with administrative rights. For our example, view the following: Choose Home | Inventory | Hosts and Clusters from the menu bar. Navigate to Datacenter | Lab Cluster | esx5501.training.lab. Select the vShield tab. Expand vShield-FW-esx5501.training.lab (192.168.10.30). Note the Status: In Sync status, there are two options to either Force Sync or Restart. Current Management Port Information is displayed including the packet, byte, and error information. Syslog Servers can be added by an IP Address. Configuring the Fail Safe Policy allows traffic to flow or be blocked should the vShield App firewall be down or offline for any reason. Navigate to Settings & Reports | vShield App within vShield Manager. Click on Change under Fail Safe. This step will lead to the following screen: Click on Yes. Note that Default Fail Safe Configuration set to Block is now changed to Allow. In a production situation, there are few times the fail safe setting will be changed to Allow. In a small test environment, should the vShield App be unavailable; all connectivity to the ESXi host will be blocked by default. Configuring the Exclusion List allows certain virtual machines to function without host based firewall rules being applied to them. This can be done by performing the following steps: Navigate to Settings & Reports | vShield App within vShield Manager. Click on Add under Exclusion List. Select a virtual machine to exclude from vShield App (in our case, it is a Linked vCenter server). Click on Add. Click on OK. Click on OK in the next dialog box to confirm. The selected virtual machine is now excluded from protection How it works… The vShield App host firewall is installed per ESXi host and automatically named by the installation program to include the name of the host. It is also important to note that when a host is put into the maintenance mode, the vShield firewall must be shut down in order to let the host successfully achieve the maintenance mode. Current Status provides a single view of the firewall associated with the host including the traffic status displayed by packet count, link, and admin status. One important status check is if the firewall is in sync with vShield Manager. Should the firewall fall out of sync, it can be forced to sync and if that fails, the option for restart is also present on the status page. Fail Safe Policy is an important consideration should the vShield App virtual appliance fail for any reason. The default setting is to block all traffic and this might seem like a good idea at the outset. Careful consideration should be given on setting this flag, depending on what type of virtual machines are running on a specific host or cluster. In the situation where mission critical applications are running on virtual machines within an internal cluster or host, it will make sense to allow traffic if the vShield App were down. The average time it will take to identify and remediate the failure could cause a significant impact on the amount of business lost. Exclusion List, as the name implies, allows certain virtual machines to remain outside the protection of the vShield App firewall. Critical infrastructure such as DNS or Domain Controllers are good candidates to be added to the exclusion list. vCenter servers should always be added to the exclusion list. Configuring vShield App Flow Monitoring The vShield App Flow Monitoring is a traffic analysis tool that provides statistics and graphs of the traffic on the virtual network as it passes through a host running vShield App. The information collected and displayed by Flow Monitoring is detailed to the protocol level and is very useful in spotting unwanted traffic flows. Getting ready In order to proceed, we require access to the vShield App through the vSphere Client plugin. The plugin can be enabled through the Plug-ins menu in the vSphere Client. This client can be run on any modern Windows desktop operating system or server operating system. The vShield vSphere Client plugin requires Adobe Flash, which is not supported on Linux operating systems at this time. Ensure the vCenter account used for login has administrative rights to vShield Manager. How to do it… To view the current traffic flow, launch vSphere Client using an account with administrative rights. For our example view the following: Navigate to Home | Inventory | Hosts and Clusters from the menu bar. Navigate to Datacenter andclick on the vShield tab. Select Flow Monitoring. Note that the Summary information is displayed by default. Summary Information Click on Details to view detailed information. Allowed Flows and Blocked Flows are available to view. Select DNS-UDP to identify the host lookup traffic on port 53. Click on Add Rule for Rule Id 1002. By changing Action from Allow to Block, a firewall rule can be modified to block the DNS traffic from the web server to the DNS Server. Click on Cancel. How it works… The Flow Monitoring component of vShield App, in addition to providing great detail, is able to create vShield App firewall rules on the fly. As shown in the preceding example, we were able to identify the DNS traffic from our web server accessing an internal DNS server. Due to our governance rules, servers accessed in the DMZ are not allowed to request DNS from internal servers. Implementing a firewall rule adds a control to this policy and gets easily implemented once the administrator noticed the request. The ability to view traffic by Top Flows, Top Destinations, and Top Sources is very valuable when troubleshooting a problem or tracking down a virus or trojan that is attempting to send valuable information outside the organization during a breach. Summary This article has thus covered how to set up and configure vShield App. Resources for Article: Further resources on this subject: Introduction to Veeam® Backup & Replication for VMware [article] VMware vCenter Operations Manager Essentials - Introduction to vCenter Operations Manager [article] Introduction to vSphere Distributed switches [article]
Read more
  • 0
  • 0
  • 3805

article-image-decoupling-units-unittestmock
Packt
24 Nov 2014
27 min read
Save for later

Decoupling Units with unittest.mock

Packt
24 Nov 2014
27 min read
In this article by Daniel Arbuckle, author of the book Learning Python Testing, you'll learn how by using the unittest.mock package, you can easily perform the following: Replace functions and objects in your own code or in external packages. Control how replacement objects behave. You can control what return values they provide, whether they raise an exception, even whether they make any calls to other functions, or create instances of other objects. Check whether the replacement objects were used as you expected: whether functions or methods were called the correct number of times, whether the calls occurred in the correct order, and whether the passed parameters were correct. (For more resources related to this topic, see here.) Mock objects in general All right, before we get down to the nuts and bolts of unittest.mock, let's spend a few moments talking about mock objects overall. Broadly speaking, mock objects are any objects that you can use as substitutes in your test code, to keep your tests from overlapping and your tested code from infiltrating the wrong tests. However, like most things in programming, the idea works better when it has been formalized into a well-designed library that you can call on when you need it. There are many such libraries available for most programming languages. Over time, the authors of mock object libraries have developed two major design patterns for mock objects: in one pattern, you can create a mock object and perform all of the expected operations on it. The object records these operations, and then you put the object into playback mode and pass it to your code. If your code fails to duplicate the expected operations, the mock object reports a failure. In the second pattern, you can create a mock object, do the minimal necessary configuration to allow it to mimic the real object it replaces, and pass it to your code. It records how the code uses it, and then you can perform assertions after the fact to check whether your code used the object as expected. The second pattern is slightly more capable in terms of the tests that you can write using it but, overall, either pattern works well. Mock objects according to unittest.mock Python has several mock object libraries; as of Python 3.3, however, one of them has been crowned as a member of the standard library. Naturally that's the one we're going to focus on. That library is, of course, unittest.mock. The unittest.mock library is of the second sort, a record-actual-use-and-then-assert library. The library contains several different kinds of mock objects that, between them, let you mock almost anything that exists in Python. Additionally, the library contains several useful helpers that simplify assorted tasks related to mock objects, such as temporarily replacing real objects with mocks. Standard mock objects The basic element of unittest.mock is the unittest.mock.Mock class. Even without being configured at all, Mock instances can do a pretty good job of pretending to be some other object, method, or function. There are many mock object libraries for Python; so, strictly speaking, the phrase "mock object" could mean any object that was created by any of these libraries. Mock objects can pull off this impersonation because of a clever, somewhat recursive trick. When you access an unknown attribute of a mock object, instead of raising an AttributeError exception, the mock object creates a child mock object and returns that. Since mock objects are pretty good at impersonating other objects, returning a mock object instead of the real value works at least in the common case. Similarly, mock objects are callable; when you call a mock object as a function or method, it records the parameters of the call and then, by default, returns a child mock object. A child mock object is a mock object in its own right, but it knows that it's connected to the mock object it came from—its parent. Anything you do to the child is also recorded in the parent's memory. When the time comes to check whether the mock objects were used correctly, you can use the parent object to check on all of its descendants. Example: Playing with mock objects in the interactive shell (try it for yourself!): $ python3.4 Python 3.4.0 (default, Apr 2 2014, 08:10:08) [GCC 4.8.2] on linux Type "help", "copyright", "credits" or "license" for more information. >>> from unittest.mock import Mock, call >>> mock = Mock() >>> mock.x <Mock name='mock.x' id='140145643647832'> >>> mock.x <Mock name='mock.x' id='140145643647832'> >>> mock.x('Foo', 3, 14) <Mock name='mock.x()' id='140145643690640'> >>> mock.x('Foo', 3, 14) <Mock name='mock.x()' id='140145643690640'> >>> mock.x('Foo', 99, 12) <Mock name='mock.x()' id='140145643690640'> >>> mock.y(mock.x('Foo', 1, 1)) <Mock name='mock.y()' id='140145643534320'> >>> mock.method_calls [call.x('Foo', 3, 14), call.x('Foo', 3, 14), call.x('Foo', 99, 12), call.x('Foo', 1, 1), call.y(<Mock name='mock.x()' id='140145643690640'>)] >>> mock.assert_has_calls([call.x('Foo', 1, 1)]) >>> mock.assert_has_calls([call.x('Foo', 1, 1), call.x('Foo', 99, 12)]) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib64/python3.4/unittest/mock.py", line 792, in assert_has_ calls ) from cause AssertionError: Calls not found. Expected: [call.x('Foo', 1, 1), call.x('Foo', 99, 12)] Actual: [call.x('Foo', 3, 14), call.x('Foo', 3, 14), call.x('Foo', 99, 12), call.x('Foo', 1, 1), call.y(<Mock name='mock.x()' id='140145643690640'>)] >>> mock.assert_has_calls([call.x('Foo', 1, 1), ... call.x('Foo', 99, 12)], any_order = True) >>> mock.assert_has_calls([call.y(mock.x.return_value)]) There are several important things demonstrated in this interactive session. First, notice that the same mock object was returned each time that we accessed mock.x. This always holds true: if you access the same attribute of a mock object, you'll get the same mock object back as the result. The next thing to notice might seem more surprising. Whenever you call a mock object, you get the same mock object back as the return value. The returned mock isn't made new for every call, nor is it unique for each combination of parameters. We'll see how to override the return value shortly but, by default, you get the same mock object back every time you call a mock object. This mock object can be accessed using the return_value attribute name, as you might have noticed from the last statement of the example. The unittest.mock package contains a call object that helps to make it easier to check whether the correct calls have been made. The call object is callable, and takes note of its parameters in a way similar to mock objects, making it easy to compare it to a mock object's call history. However, the call object really shines when you have to check for calls to descendant mock objects. As you can see in the previous example, while call('Foo', 1, 1) will match a call to the parent mock object, if the call used these parameters, call.x('Foo', 1, 1), it matches a call to the child mock object named x. You can build up a long chain of lookups and invocations. For example: >>> mock.z.hello(23).stuff.howdy('a', 'b', 'c') <Mock name='mock.z.hello().stuff.howdy()' id='140145643535328'> >>> mock.assert_has_calls([ ... call.z.hello().stuff.howdy('a', 'b', 'c') ... ]) >>> Notice that the original invocation included hello(23), but the call specification wrote it simply as hello(). Each call specification is only concerned with the parameters of the object that was finally called after all of the lookups. The parameters of intermediate calls are not considered. That's okay because they always produce the same return value anyway unless you've overridden that behavior, in which case they probably don't produce a mock object at all. You might not have encountered an assertion before. Assertions have one job, and one job only: they raise an exception if something is not as expected. The assert_has_calls method, in particular, raises an exception if the mock object's history does not include the specified calls. In our example, the call history matches, so the assertion method doesn't do anything visible. You can check whether the intermediate calls were made with the correct parameters, though, because the mock object recorded a call immediately to mock.z.hello(23) before it recorded a call to mock.z.hello().stuff.howdy('a', 'b', 'c'): >>> mock.mock_calls.index(call.z.hello(23)) 6 >>> mock.mock_calls.index(call.z.hello().stuff.howdy('a', 'b', 'c')) 7 This also points out the mock_calls attribute that all mock objects carry. If the various assertion functions don't quite do the trick for you, you can always write your own functions that inspect the mock_calls list and check whether things are or are not as they should be. We'll discuss the mock object assertion methods shortly. Non-mock attributes What if you want a mock object to give back something other than a child mock object when you look up an attribute? It's easy; just assign a value to that attribute: >>> mock.q = 5 >>> mock.q 5 There's one other common case where mock objects' default behavior is wrong: what if accessing a particular attribute is supposed to raise an AttributeError? Fortunately, that's easy too: >>> del mock.w >>> mock.w Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib64/python3.4/unittest/mock.py", line 563, in __getattr__ raise AttributeError(name) AttributeError: w Non-mock return values and raising exceptions Sometimes, actually fairly often, you'll want mock objects posing as functions or methods to return a specific value, or a series of specific values, rather than returning another mock object. To make a mock object always return the same value, just change the return_value attribute: >>> mock.o.return_value = 'Hi' >>> mock.o() 'Hi' >>> mock.o('Howdy') 'Hi' If you want the mock object to return different value each time it's called, you need to assign an iterable of return values to the side_effect attribute instead, as follows: >>> mock.p.side_effect = [1, 2, 3] >>> mock.p() 1 >>> mock.p() 2 >>> mock.p() 3 >>> mock.p() Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib64/python3.4/unittest/mock.py", line 885, in __call__ return _mock_self._mock_call(*args, **kwargs) File "/usr/lib64/python3.4/unittest/mock.py", line 944, in _mock_call result = next(effect) StopIteration If you don't want your mock object to raise a StopIteration exception, you need to make sure to give it enough return values for all of the invocations in your test. If you don't know how many times it will be invoked, an infinite iterator such as itertools.count might be what you need. This is easily done: >>> mock.p.side_effect = itertools.count() If you want your mock to raise an exception instead of returning a value, just assign the exception object to side_effect, or put it into the iterable that you assign to side_effect: >>> mock.e.side_effect = [1, ValueError('x')] >>> mock.e() 1 >>> mock.e() Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib64/python3.4/unittest/mock.py", line 885, in __call__ return _mock_self._mock_call(*args, **kwargs) File "/usr/lib64/python3.4/unittest/mock.py", line 946, in _mock_call raise result ValueError: x The side_effect attribute has another use, as well that we'll talk about. Mocking class or function details Sometimes, the generic behavior of mock objects isn't a close enough emulation of the object being replaced. This is particularly the case when it's important that they raise exceptions when used improperly, since mock objects are usually happy to accept any usage. The unittest.mock package addresses this problem using a technique called speccing. If you pass an object into unittest.mock.create_autospec, the returned value will be a mock object, but it will do its best to pretend that it's the same object you passed into create_autospec. This means that it will: Raise an AttributeError if you attempt to access an attribute that the original object doesn't have, unless you first explicitly assign a value to that attribute Raise a TypeError if you attempt to call the mock object when the original object wasn't callable Raise a TypeError if you pass the wrong number of parameters or pass a keyword parameter that isn't viable if the original object was callable Trick isinstance into thinking that the mock object is of the original object's type Mock objects made by create_autospec share this trait with all of their children as well, which is usually what you want. If you really just want a specific mock to be specced, while its children are not, you can pass the template object into the Mock constructor using the spec keyword. Here's a short demonstration of using create_autospec: >>> from unittest.mock import create_autospec >>> x = Exception('Bad', 'Wolf') >>> y = create_autospec(x) >>> isinstance(y, Exception) True >>> y <NonCallableMagicMock spec='Exception' id='140440961099088'> Mocking function or method side effects Sometimes, for a mock object to successfully take the place of a function or method means that the mock object has to actually perform calls to other functions, or set variable values, or generally do whatever a function can do. This need is less common than you might think, and it's also somewhat dangerous for testing purposes because, when your mock objects can execute arbitrary code, there's a possibility that they stop being a simplifying tool for enforcing test isolation, and become a complex part of the problem instead. Having said that, there are still times when you need a mocked function to do something more complex than simply returning a value, and we can use the side_effect attribute of mock objects to achieve this. We've seen side_effect before, when we assigned an iterable of return values to it. If you assign a callable to side_effect, this callable will be called when the mock object is called and passed the same parameters. If the side_effect function raises an exception, this is what the mock object does as well; otherwise, the side_effect return value is returned by the mock object. In other words, if you assign a function to a mock object's side_effect attribute, this mock object in effect becomes that function with the only important difference being that the mock object still records the details of how it's used. The code in a side_effect function should be minimal, and should not try to actually do the job of the code the mock object is replacing. All it should do is perform any expected externally visible operations and then return the expected result.Mock object assertion methods As we saw in the Standard mock objects section, you can always write code that checks the mock_calls attribute of mock objects to see whether or not things are behaving as they should. However, there are some particularly common checks that have already been written for you, and are available as assertion methods of the mock objects themselves. As is normal for assertions, these assertion methods return None if they pass, and raise an AssertionError if they fail. The assert_called_with method accepts an arbitrary collection of arguments and keyword arguments, and raises an AssertionError unless these parameters were passed to the mock the last time it was called. The assert_called_once_with method behaves like assert_called_with, except that it also checks whether the mock was only called once and raises AssertionError if that is not true. The assert_any_call method accepts arbitrary arguments and keyword arguments, and raises an AssertionError if the mock object has never been called with these parameters. We've already seen the assert_has_calls method. This method accepts a list of call objects, checks whether they appear in the history in the same order, and raises an exception if they do not. Note that "in the same order" does not necessarily mean "next to each other." There can be other calls in between the listed calls as long as all of the listed calls appear in the proper sequence. This behavior changes if you assign a true value to the any_order argument. In that case, assert_has_calls doesn't care about the order of the calls, and only checks whether they all appear in the history. The assert_not_called method raises an exception if the mock has ever been called. Mocking containers and objects with a special behavior One thing the Mock class does not handle is the so-called magic methods that underlie Python's special syntactic constructions: __getitem__, __add__, and so on. If you need your mock objects to record and respond to magic methods—in other words, if you want them to pretend to be container objects such as dictionaries or lists, or respond to mathematical operators, or act as context managers or any of the other things where syntactic sugar translates it into a method call underneath—you're going to use unittest.mock.MagicMock to create your mock objects. There are a few magic methods that are not supported even by MagicMock, due to details of how they (and mock objects) work: __getattr__, __setattr__, __init__ , __new__, __prepare__, __instancecheck__, __subclasscheck__, and __del__. Here's a simple example in which we use MagicMock to create a mock object supporting the in operator: >>> from unittest.mock import MagicMock >>> mock = MagicMock() >>> 7 in mock False >>> mock.mock_calls [call.__contains__(7)] >>> mock.__contains__.return_value = True >>> 8 in mock True >>> mock.mock_calls [call.__contains__(7), call.__contains__(8)] Things work similarly with the other magic methods. For example, addition: >>> mock + 5 <MagicMock name='mock.__add__()' id='140017311217816'> >>> mock.mock_calls [call.__contains__(7), call.__contains__(8), call.__add__(5)] Notice that the return value of the addition is a mock object, a child of the original mock object, but the in operator returned a Boolean value. Python ensures that some magic methods return a value of a particular type, and will raise an exception if that requirement is not fulfilled. In these cases, MagicMock's implementations of the methods return a best-guess value of the proper type, instead of a child mock object. There's something you need to be careful of when it comes to the in-place mathematical operators, such as += (__iadd__) and |= (__ior__), and that is the fact that MagicMock handles them somewhat strangely. What it does is still useful, but it might well catch you by surprise: >>> mock += 10 >>> mock.mock_calls [] What was that? Did it erase our call history? Fortunately, no, it didn't. What it did was assign the child mock created by the addition operation to the variable called mock. That is entirely in accordance with how the in-place math operators are supposed to work. Unfortunately, it has still cost us our ability to access the call history, since we no longer have a variable pointing at the parent mock object. Make sure that you have the parent mock object set aside in a variable that won't be reassigned, if you're going to be checking in-place math operators. Also, you should make sure that your mocked in-place operators return the result of the operation, even if that just means return self.return_value, because otherwise Python will assign None to the left-hand variable. There's another detailed way in which in-place operators work that you should keep in mind: >>> mock = MagicMock() >>> x = mock >>> x += 5 >>> x <MagicMock name='mock.__iadd__()' id='139845830142216'> >>> x += 10 >>> x <MagicMock name='mock.__iadd__().__iadd__()' id='139845830154168'> >>> mock.mock_calls [call.__iadd__(5), call.__iadd__().__iadd__(10)] Because the result of the operation is assigned to the original variable, a series of in-place math operations builds up a chain of child mock objects. If you think about it, that's the right thing to do, but it is rarely what people expect at first. Mock objects for properties and descriptors There's another category of things that basic Mock objects don't do a good job of emulating: descriptors. Descriptors are objects that allow you to interfere with the normal variable access mechanism. The most commonly used descriptors are created by Python's property built-in function, which simply allows you to write functions to control getting, setting, and deleting a variable. To mock a property (or other descriptor), create a unittest.mock.PropertyMock instance and assign it to the property name. The only complication is that you can't assign a descriptor to an object instance; you have to assign it to the object's type because descriptors are looked up in the type without first checking the instance. That's not hard to do with mock objects, fortunately: >>> from unittest.mock import PropertyMock >>> mock = Mock() >>> prop = PropertyMock() >>> type(mock).p = prop >>> mock.p <MagicMock name='mock()' id='139845830215328'> >>> mock.mock_calls [] >>> prop.mock_calls [call()] >>> mock.p = 6 >>> prop.mock_calls [call(), call(6)] The thing to be mindful of here is that the property is not a child of the object named mock. Because of this, we have to keep it around in its own variable because otherwise we'd have no way of accessing its history. The PropertyMock objects record variable lookup as a call with no parameters, and variable assignment as a call with the new value as a parameter. You can use a PropertyMock object if you actually need to record variable accesses in your mock object history. Usually you don't need to do that, but the option exists. Even though you set a property by assigning it to an attribute of a type, you don't have to worry about having your PropertyMock objects bleed over into other tests. Each Mock you create has its own type object, even though they all claim to be of the same class: >>> type(Mock()) is type(Mock()) False Thanks to this feature, any changes that you make to a mock object's type object are unique to that specific mock object. Mocking file objects It's likely that you'll occasionally need to replace a file object with a mock object. The unittest.mock library helps you with this by providing mock_open, which is a factory for fake open functions. These functions have the same interface as the real open function, but they return a mock object that's been configured to pretend that it's an open file object. This sounds more complicated than it is. See for yourself: >>> from unittest.mock import mock_open >>> open = mock_open(read_data = 'moose') >>> with open('/fake/file/path.txt', 'r') as f: ... print(f.read()) ... moose If you pass a string value to the read_data parameter, the mock file object that eventually gets created will use that value as the data source when its read methods get called. As of Python 3.4.0, read_data only supports string objects, not bytes. If you don't pass read_data, read method calls will return an empty string. The problem with the previous code is that it makes the real open function inaccessible, and leaves a mock object lying around where other tests might stumble over it. Read on to see how to fix these problems. Replacing real code with mock objects The unittest.mock library gives a very nice tool for temporarily replacing objects with mock objects, and then undoing the change when our test is done. This tool is unittest.mock.patch. There are a lot of different ways in which that patch can be used: it works as a context manager, a function decorator, and a class decorator; additionally, it can create a mock object to use for the replacement or it can use the replacement object that you specify. There are a number of other optional parameters that can further adjust the behavior of the patch. Basic usage is easy: >>> from unittest.mock import patch, mock_open >>> with patch('builtins.open', mock_open(read_data = 'moose')) as mock: ... with open('/fake/file.txt', 'r') as f: ... print(f.read()) ... moose >>> open <built-in function open> As you can see, patch dropped the mock open function created by mock_open over the top of the real open function; then, when we left the context, it replaced the original for us automatically. The first parameter of patch is the only one that is required. It is a string describing the absolute path to the object to be replaced. The path can have any number of package and subpackage names, but it must include the module name and the name of the object inside the module that is being replaced. If the path is incorrect, patch will raise an ImportError, TypeError, or AttributeError, depending on what exactly is wrong with the path. If you don't want to worry about making a mock object to be the replacement, you can just leave that parameter off: >>> import io >>> with patch('io.BytesIO'): ... x = io.BytesIO(b'ascii data') ... io.BytesIO.mock_calls [call(b'ascii data')] The patch function creates a new MagicMock for you if you don't tell it what to use for the replacement object. This usually works pretty well, but you can pass the new parameter (also the second parameter, as we used it in the first example of this section) to specify that the replacement should be a particular object; or you can pass the new_callable parameter to make patch use the value of that parameter to create the replacement object. We can also force the patch to use create_autospec to make the replacement object, by passing autospec=True: >>> with patch('io.BytesIO', autospec = True): ... io.BytesIO.melvin Traceback (most recent call last): File "<stdin>", line 2, in <module> File "/usr/lib64/python3.4/unittest/mock.py", line 557, in __getattr__ raise AttributeError("Mock object has no attribute %r" % name) AttributeError: Mock object has no attribute 'melvin' The patch function will normally refuse to replace an object that does not exist; however, if you pass it create=True, it will happily drop a mock object wherever you like. Naturally, this is not compatible with autospec=True. The patch function covers the most common cases. There are a few related functions that handle less common but still useful cases. The patch.object function does the same thing as patch, except that, instead of taking the path string, it accepts an object and an attribute name as its first two parameters. Sometimes this is more convenient than figuring out the path to an object. Many objects don't even have valid paths (for example, objects that exist only in a function local scope), although the need to patch them is rarer than you might think. The patch.dict function temporarily drops one or more objects into a dictionary under specific keys. The first parameter is the target dictionary; the second is a dictionary from which to get the key and value pairs to put into the target. If you pass clear=True, the target will be emptied before the new values are inserted. Notice that patch.dict doesn't create the replacement values for you. You'll need to make your own mock objects, if you want them. Mock objects in action That was a lot of theory interspersed with unrealistic examples. Let's take a look at what we've learned and apply it for a more realistic view of how these tools can help us. Better PID tests The PID tests suffered mostly from having to do a lot of extra work to patch and unpatch time.time, and had some difficulty breaking the dependence on the constructor. Patching time.time Using patch, we can remove a lot of the repetitiveness of dealing with time.time; this means that it's less likely that we'll make a mistake somewhere, and saves us from spending time on something that's kind of boring and annoying. All of the tests can benefit from similar changes: >>> from unittest.mock import Mock, patch >>> with patch('time.time', Mock(side_effect = [1.0, 2.0, 3.0, 4.0, 5.0])): ... import pid ... controller = pid.PID(P = 0.5, I = 0.5, D = 0.5, setpoint = 0, ... initial = 12) ... assert controller.gains == (0.5, 0.5, 0.5) ... assert controller.setpoint == [0.0] ... assert controller.previous_time == 1.0 ... assert controller.previous_error == -12.0 ... assert controller.integrated_error == 0.0 Apart from using patch to handle time.time, this test has been changed. We can now use assert to check whether things are correct instead of having doctest compare the values directly. There's hardly any difference between the two approaches, except that we can place the assert statements inside the context managed by patch. Decoupling from the constructor Using mock objects, we can finally separate the tests for the PID methods from the constructor, so that mistakes in the constructor cannot affect the outcome: >>> with patch('time.time', Mock(side_effect = [2.0, 3.0, 4.0, 5.0])): ... pid = imp.reload(pid) ... mock = Mock() ... mock.gains = (0.5, 0.5, 0.5) ... mock.setpoint = [0.0] ... mock.previous_time = 1.0 ... mock.previous_error = -12.0 ... mock.integrated_error = 0.0 ... assert pid.PID.calculate_response(mock, 6) == -3.0 ... assert pid.PID.calculate_response(mock, 3) == -4.5 ... assert pid.PID.calculate_response(mock, -1.5) == -0.75 ... assert pid.PID.calculate_response(mock, -2.25) == -1.125 What we've done here is set up a mock object with the proper attributes, and pass it into calculate_response as the self-parameter. We could do this because we didn't create a PID instance at all. Instead, we looked up the method's function inside the class and called it directly, allowing us to pass whatever we wanted as the self-parameter instead of having Python's automatic mechanisms handle it. Never invoking the constructor means that we're immune to any errors it might contain, and guarantees that the object state is exactly what we expect here in our calculate_response test. Summary In this article, we've learned about a family of objects that specialize in impersonating other classes, objects, methods, and functions. We've seen how to configure these objects to handle corner cases where their default behavior isn't sufficient, and we've learned how to examine the activity logs that these mock objects keep, so that we can decide whether the objects are being used properly or not. Resources for Article: Further resources on this subject: Installing NumPy, SciPy, matplotlib, and IPython [Article] Machine Learning in IPython with scikit-learn [Article] Python 3: Designing a Tasklist Application [Article]
Read more
  • 0
  • 0
  • 7112
article-image-managing-heroku-command-line
Packt
20 Nov 2014
27 min read
Save for later

Managing Heroku from the Command Line

Packt
20 Nov 2014
27 min read
In this article by Mike Coutermarsh, author of Heroku Cookbook, we will cover the following topics: Viewing application logs Searching logs Installing add-ons Managing environment variables Enabling the maintenance page Managing releases and rolling back Running one-off tasks and dynos Managing SSH keys Sharing and collaboration Monitoring load average and memory usage (For more resources related to this topic, see here.) Heroku was built to be managed from its command-line interface. The better we learn it, the faster and more effective we will be in administering our application. The goal of this article is to get comfortable with using the CLI. We'll see that each Heroku command follows a common pattern. Once we learn a few of these commands, the rest will be relatively simple to master. In this article, we won't cover every command available in the CLI, but we will focus on the ones that we'll be using the most. As we learn each command, we will also learn a little more about what is happening behind the scenes so that we get a better understanding of how Heroku works. The more we understand, the more we'll be able to take advantage of the platform. Before we start, let's note that if we ever need to get a list of the available commands, we can run the following command: $ heroku help We can also quickly display the documentation for a single command: $ heroku help command_name Viewing application logs Logging gets a little more complex for any application that is running multiple servers and several different types of processes. Having visibility into everything that is happening within our application is critical to maintaining it. Heroku handles this by combining and sending all of our logs to one place, the Logplex. The Logplex provides us with a single location to view a stream of our logs across our entire application. In this recipe, we'll learn how to view logs via the CLI. We'll learn how to quickly get visibility into what's happening within our application. How to do it… To start, let's open up a terminal, navigate to an existing Heroku application, and perform the following steps: First, to view our applications logs, we can use the logs command: $ heroku logs2014-03-31T23:35:51.195150+00:00 app[web.1]:   Rendered pages/about.html.slim within layouts/application (25.0ms) 2014-03-31T23:35:51.215591+00:00 app[web.1]:   Rendered layouts/_navigation_links.html.erb (2.6ms)2014-03-31T23:35:51.230010+00:00 app[web.1]:   Rendered layouts/_messages.html.slim (13.0ms)2014-03-31T23:35:51.215967+00:00 app[web.1]:   Rendered layouts/_navigation.html.slim (10.3ms)2014-03-31T23:35:51.231104+00:00 app[web.1]: Completed 200 OK in 109ms (Views: 65.4ms | ActiveRecord: 0.0ms)2014-03-31T23:35:51.242960+00:00 heroku[router]: at=info method=GET path= Heroku logs anything that our application sends to STDOUT or STDERR. If we're not seeing logs, it's very likely our application is not configured correctly.  We can also watch our logs in real time. This is known as tailing: $ heroku logs --tail Instead of --tail, we can also use -t. We'll need to press Ctrl + C to end the command and stop tailing the logs. If we want to see the 100 most recent lines, we can use -n: $ heroku logs -n 100 The Logplex stores a maximum of 1500 lines. To view more lines, we'll have to set up a log storage. We can filter the logs to only show a specific process type. Here, we will only see logs from our web dynos: $ heroku logs -p web If we want, we can be as granular as showing the logs from an individual dyno. This will show only the logs from the second web dyno: $ heroku logs -p web.2 We can use this for any process type; we can try it for our workers if we'd like: $ heroku logs -p worker The Logplex contains more than just logs from our application. We can also view logs generated by Heroku or the API. Let's try changing the source to Heroku to only see the logs generated by Heroku. This will only show us logs related to the router and resource usage: $ heroku logs --source heroku To view logs for only our application, we can set the source to app: $ heroku logs --source app We can also view logs from the API. These logs will show any administrative actions we've taken, such as scaling dynos or changing configuration variables. This can be useful when multiple developers are working on an application: $ heroku logs --source api We can even combine the different flags. Let's try tailing the logs for only our web dynos: $ heroku logs -p web --tail That's it! Remember that if we ever need more information on how to view logs via the CLI, we can always use the help command: $ heroku help logs How it works Under the covers, the Heroku CLI is simply passes our request to Heroku's API and then uses Ruby to parse and display our logs. If you're interested in exactly how it works, the code is open source on GitHub at https://github.com/heroku/heroku/blob/master/lib/heroku/command/logs.rb. Viewing logs via the CLI is most useful in situations where we need to see exactly what our application is doing right now. We'll find that we use it a lot around deploys and when debugging issues. Since the Logplex has a limit of 1500 lines, it's not meant to view any historical data. For this, we'll need to set up log drains and enable a logging add-on. Searching logs Heroku does not have the built-in capability to search our logs from the command line. We can get around this limitation easily by making use of some other command-line tools. In this recipe, we will learn how to combine Heroku's logs with Grep, a command-line tool to search text. This will allow us to search our recent logs for keywords, helping us track down errors more quickly. Getting ready For this recipe, we'll need to have Grep installed. For OS X and Linux machines, it should already be installed. We can install Grep using the following steps: To check if we have Grep installed, let's open up a terminal and type the following: $ grepusage: grep [-abcDEFGHhIiJLlmnOoPqRSsUVvwxZ] [-A num] [-B num] [-C[num]]       [-e pattern] [-f file] [--binary-files=value] [--color=when]       [--context[=num]] [--directories=action] [--label] [--line-buffered]       [--null] [pattern] [file ...] If we do not see usage instructions, we can visit http://www.gnu.org/software/grep/ for the download and installation instructions. How to do it… Let's start searching our logs by opening a terminal and navigating to one of our Heroku applications using the following steps: To search for a keyword in our logs, we need to pipe our logs into Grep. This simply means that we will be passing our logs into Grep and having Grep search them for us. Let's try this now. The following command will search the output of heroku logs for the word error: $ heroku logs | grep error Sometimes, we might want to search for a longer string that includes special characters. We can do this by surrounding it with quotes: $ heroku logs | grep "path=/pages/about host" It can be useful to also see the lines surrounding the line that matched our search. We can do this as well. The next command will show us the line that contains an error as well as the three lines above and below it: $ heroku logs | grep error -C 3 We can even search with regular expressions. The next command will show us every line that matches a number that ends with MB. So, for example, lines with 100 MB, 25 MB, or 3 MB will all appear: $ heroku logs | grep 'd*MB' To learn more about regular expressions, visit http://regex.learncodethehardway.org/. How it works… Like most Unix-based tools, Grep was built to accomplish a single task and to do it well. Global regular expression print (Grep) is built to search a set of files for a pattern and then print all of the matches. Grep can also search anything it receives through standard input; this is exactly how we used it in this recipe. By piping the output of our Heroku logs into Grep, we are passing our logs to Grep as standard input. See also To learn more about Grep, visit http://www.tutorialspoint.com/unix_commands/grep.htm Installing add-ons Our application needs some additional functionality provided by an outside service. What should we do? In the past, this would have involved creating accounts, managing credentials, and, maybe, even bringing up servers and installing software. This whole process has been simplified by the Heroku add-on marketplace. For any additional functionality that our application needs, our first stop should always be Heroku add-ons. Heroku has made attaching additional resources to our application a plug-and-play process. If we need an additional database, caching, or error logging, they can be set up with a single command. In this recipe, we will learn the ins and outs of using the Heroku CLI to install and manage our application's add-ons. How to do it... To begin, let's open a terminal and navigate to one of our Heroku applications using the following steps: Let's start by taking a look at all of the available Heroku add-ons. We can do this with the addons:list command: $ heroku addons:list There are so many add-ons that viewing them through the CLI is pretty difficult. For easier navigation and search, we should take a look at https://addons.heroku.com/. If we want to see the currently installed add-ons for our application, we can simply type the following: $ heroku addons=== load-tester-rails Configured Add-onsheroku-postgresql:dev       HEROKU_POSTGRESQL_MAROONheroku-postgresql:hobby-dev HEROKU_POSTGRESQL_ONYXlibrato:developmentnewrelic:stark Remember that for any command, we can always add --app app_name to specify the application. Alternatively, our application's add-ons are also listed through the Heroku Dashboard available at https://dashboard.heroku.com. The installation of a new add-on is done with addons:add. Here, we are going to install the error logging service, Rollbar: $ heroku addons:add rollbarheroku addons:add rollbarAdding rollbar on load-tester-rails... done, v22 (free)Use `heroku addons:docs rollbar` to view documentation. We can quickly open up the documentation for an add-on with addons:docs: $ heroku addons:docs rollbar Removing an add-on is just as simple. We'll need to type our application name to confirm. For this example, our application is called load-tester-rails: $ heroku addons:remove rollbar!   WARNING: Destructive Action!   This command will affect the app: load-tester-rails!   To proceed, type "load-tester-rails" or re-run this command with --confirm load-tester-rails > load-tester-railsRemoving rollbar on load-tester-rails... done, v23 (free) Each add-on comes with different tiers of service. Let's try upgrading our rollbar add-on to the starter tier: $ heroku addons:upgrade rollbar:starterUpgrading to rollbar:starter on load-tester-rails... done, v26 ($12/mo)Plan changed to starterUse `heroku addons:docs rollbar` to view documentation. Now, if we want, we can downgrade back to its original level with addons:downgrade: $ heroku addons:downgrade rollbarDowngrading to rollbar on load-tester-rails... done, v27 (free)Plan changed to freeUse `heroku addons:docs rollbar` to view documentation. If we ever forget any of the commands, we can always use help to quickly see the documentation: $ heroku help addons Some add-ons might charge you money. Before continuing, let's double check that we only have the correct ones enabled, using the $ heroku addons command. How it works… Heroku has created a standardized process for all add-on providers to follow. This ensures a consistent experience when provisioning any add-on for our application. It starts when we request the creation of an add-on. Heroku sends an HTTP request to the provider, asking them to provision an instance of their service. The provider must then respond to Heroku with the connection details for their service in the form of environment variables. For example, if we were to provision Redis To Go, we will get back our connection details in a REDISTOGO_URL variable: REDISTOGO_URL: redis://user:pass@server.redistogo.com:9652 Heroku adds these variables to our application and restarts it. On restart, the variables are available for our application, and we can connect to the service using them. The specifics on how to connect using the variables will be in the add-ons documentation. Installation will depend on the specific language or framework we're using. See also For details on creating our own add-ons, the process is well documented on Heroku's website at https://addons.heroku.com/provider Check out Kensa, the CLI to create Heroku add-ons, at https://github.com/heroku/kensa Managing environment variables Our applications will often need access to various credentials in the form of API tokens, usernames, and passwords for integrations with third-party services. We can store this information in our Git repository, but then, anyone with access to our code will also have a copy of our production credentials. We should instead use environment variables to store any configuration information for our application. Configuration information should be separate from our application's code and instead be tied to the specific deployment of the application. Changing our application to use environment variables is simple. Let's look at an example in Ruby; let's assume that we currently have secret_api_token defined in our application's code: secret_api_token = '123abc' We can remove the token and replace it with an environment variable: secret_api_token = ENV['SECRET_TOKEN'] In addition to protecting our credentials, using environment variables makes our application more configurable. We'll be able to quickly make configuration changes without having to change code and redeploy. The terms "configuration variable" and "environment variable" are interchangeable. Heroku usually uses "configuration" due to how tightly the variables are coupled with the state of the application. How to do it... Heroku makes it easy to set our application's environment variables through the config command. Let's launch a terminal and navigate to an existing Heroku project to try it out, using the following steps: We can use the config command to see a list of all our existing environment variables: $ heroku config To view only the value of a specific variable, we can use get: $ heroku config:get DATABASE_URL To set a new variable, we can use set: $ heroku config:set VAR_NAME=var_valueSetting config vars and restarting load-tester-rails... done, v28VAR_NAME: var_value Each time we set a config variable, Heroku will restart our application. We can set multiple values at once to avoid multiple restarts: $ heroku config:set SECRET=value SECRET2=valueSetting config vars and restarting load-tester-rails... done, v29SECRET: valueSECRET2: value To delete a variable, we use unset: $ heroku config:unset SECRETUnsetting SECRET and restarting load-tester-rails... done, v30 If we want, we can delete multiple variables with a single command: $ heroku config:unset VAR_NAME SECRET2Unsetting VAR_NAME and restarting load-tester-rails... done, v31Unsetting SECRET2 and restarting load-tester-rails... done, v32 Heroku tracks each configuration change as a release. This makes it easy for us to roll back changes if we make a mistake. How it works… Environment variables are used on Unix-based operating systems to manage and share configuration information between applications. As they are so common, changing our application to use them does not lock us into deploying only to Heroku. Heroku stores all of our configuration variables in one central location. Each change to these variables is tracked, and we can view the history by looking through our past releases. When Heroku spins up a new dyno, part of the process is taking all of our configuration settings and setting them as environment variables on the dyno. This is why whenever we make a configuration change, Heroku restarts our dynos. As configuration variables are such a key part of our Heroku application, any change to them will also be included in our Heroku logs. See also Read about the Twelve-Factor app's rule on configuration at http://12factor.net/config Enabling the maintenance page Occasionally, we will need to make changes to our application that requires downtime. The proper way to do this is to put up a maintenance page that displays a friendly message and respond to all the incoming HTTP requests with a 503 Service Unavailable status. Doing this will keep our users informed and also avoid any negative SEO effects. Search engines understand that when they receive a 503 response, they should come back later to recrawl the site. If we didn't use a maintenance page and our application returned a 404 or 500 errors instead, it's possible that a search engine crawler might remove the page from their index. How to do it... Let's open up a terminal and navigate to one of our Heroku projects to begin with, using the following steps: We can view if our application's maintenance page is currently enabled with the maintenance command: $ heroku maintenanceoff Let's try turning it on. This will stop traffic from being routed to our dynos and show the maintenance page as follows: $ heroku maintenance:onEnabling maintenance mode for load-tester-rails... done Now, if we visit our application, we'll see the default Heroku maintenance page: To disable the maintenance page and resume sending users to our application, we can use the maintenance:off command: $ heroku maintenance:offDisabling maintenance mode for load-tester-rails... done Managing releases and rolling back What do we do if disaster strikes and our newly released code breaks our application? Luckily for us, Heroku keeps a copy of every deploy and configuration change to our application. This enables us to roll back to a previous version while we work to correct the errors in our latest release. Heads up! Rolling back only affects application code and configuration variables. Add-ons and our database will not be affected by a rollback. In this recipe, we will learn how to manage our releases and roll back code from the CLI. How to do it... In this recipe, we'll view and manage our releases from the Heroku CLI, using the releases command. Let's open up a terminal now and navigate to one of our Heroku projects by performing the following steps: Heroku tracks every deploy and configuration change as a release. We can view all of our releases from both the CLI and the web dashboard with the releases command: $ heroku releases=== load-tester-rails Releasesv33 Add WEB_CON config vars coutermarsh.mike@gmail.com 2014/03/30 11:18:49 (~ 5h ago)v32 Remove SEC config vars       coutermarsh.mike@gmail.com 2014/03/29 19:38:06 (~ 21h ago)v31 Remove VAR config vars     coutermarsh.mike@gmail.com 2014/03/29 19:38:05 (~ 21h ago)v30 Remove config vars       coutermarsh.mike@gmail.com 2014/03/29 19:27:05 (~ 21h ago)v29 Deploy 9218c1c vars coutermarsh.mike@gmail.com 2014/03/29 19:24:29 (~ 21h ago) Alternatively, we can view our releases through the Heroku dashboard. Visit https://dashboard.heroku.com, select one of our applications, and click on Activity: We can view detailed information about each release using the info command. This shows us everything about the change and state of the application during this release: $ heroku releases:info v33=== Release v33Addons: librato:development       newrelic:stark       rollbar:free       sendgrid:starterBy:     coutermarsh.mike@gmail.comChange: Add WEB_CONCURRENCY config varsWhen:   2014/03/30 11:18:49 (~ 6h ago)=== v33 Config VarsWEB_CONCURRENCY: 3 We can revert to the previous version of our application with the rollback command: $ heroku rollbackRolling back load-tester-rails... done, v32!   Warning: rollback affects code and config vars; it doesn't add or remove addons. To undo, run: heroku rollback v33 Rolling back creates a new version of our application in the release history. We can also specify a specific version to roll back to: $ heroku rollback v30Rolling back load-tester-rails... done, v30 The version we roll back to does not have to be an older version. Although it sounds contradictory, we can also roll back to newer versions of our application. How it works… Behind the scenes, each Heroku release is tied to a specific slug and set of configuration variables. As Heroku keeps a copy of each slug that we deploy, we're able to quickly roll back to previous versions of our code without having to rebuild our application. For each deploy release created, it will include a reference to the Git SHA that was pushed to master. The Git SHA is a reference to the last commit made to our repository before it was deployed. This is useful if we want to know exactly what code was pushed out in that release. On our local machine, we can run the $ git checkout git-sha-here command to view our application's code in the exact state it was when deployed. Running one-off tasks and dynos In more traditional hosting environments, developers will often log in to servers to perform basic administrative tasks or debug an issue. With Heroku, we can do this by launching one-off dynos. These are dynos that contain our application code but do not serve web requests. For a Ruby on Rails application, one-off dynos are often used to run database migrations or launch a Rails console. How to do it... In this recipe, we will learn how to execute commands on our Heroku applications with the heroku run command. Let's launch a terminal now to get started with the following steps: To have Heroku start a one-off dyno and execute any single command, we will use heroku run. Here, we can try it out by running a simple command to print some text to the screen: $ heroku run echo "hello heroku"Running `echo "hello heroku"` attached to terminal... up, run.7702"hello heroku" One-off dynos are automatically shut down after the command has finished running. We can see that Heroku is running this command on a dyno with our application's code. Let's run ls to see a listing of the files on the dyno. They should look familiar: $ heroku run lsRunning `ls` attached to terminal... up, run.5518app bin config config.ru db Gemfile Gemfile.lock lib log Procfile     public Rakefile README README.md tmp If we want to run multiple commands, we can start up a bash session. Type exit to close the session: $ heroku run bashRunning `bash` attached to terminal... up, run.2331~ $ lsapp bin config config.ru db Gemfile Gemfile.lock      lib log Procfile public Rakefile README README.md tmp~ $ echo "hello"hello~ $ exitexit We can run tasks in the background using the detached mode. The output of the command goes to our logs rather than the screen: $ heroku run:detached echo "hello heroku"Running `echo hello heroku` detached... up, run.4534Use `heroku logs -p run.4534` to view the output. If we need more power, we can adjust the size of the one-off dynos. This command will launch a bash session in a 2X dyno: $ heroku run --size=2X bash If we are running one-off dynos in the detached mode, we can view their status and stop them in the same way we would stop any other dyno: $ heroku ps=== run: one-off processesrun.5927 (1X): starting 2014/03/29 16:18:59 (~ 6s ago)$ heroku ps:stop run.5927 How it works… When we issue the heroku run command, Heroku spins up a new dyno with our latest slug and runs the command. Heroku does not start our application; the only command that runs is the command that we explicitly pass to it. One-off dynos act a little differently than standard dynos. If we create one dyno in the detached mode, it will run until we stop it manually, or it will shut down automatically after 24 hours. It will not restart like a normal dyno will. If we run bash from a one-off dyno, it will run until we close the connection or reach an hour of inactivity. Managing SSH keys Heroku manages access to our application's Git repository with SSH keys. When we first set up the Heroku Toolbelt, we had to upload either a new or existing public key to Heroku's servers. This key allows us to access our Heroku Git repositories without entering our password each time. If we ever want to deploy our Heroku applications from another computer, we'll either need to have the same key on that computer or provide Heroku with an additional one. It's easy enough to do this via the CLI, which we'll learn in this recipe. How to do it… To get started, let's fire up a terminal. We'll be using the keys command in this recipe by performing the following steps: First, let's view all of the existing keys in our Heroku account: $ heroku keys=== coutermarsh.mike@gmail.com Keysssh-rsa AAAAB3NzaC...46hEzt1Q== coutermarsh.mike@gmail.comssh-rsa AAAAB3NzaC...6EU7Qr3S/v coutermarsh.mike@gmail.comssh-rsa AAAAB3NzaC...bqCJkM4w== coutermarsh.mike@gmail.com To remove an existing key, we can use keys:remove. To the command, we need to pass a string that matches one of the keys: $ heroku keys:remove "7Qr3S/v coutermarsh.mike@gmail.com"Removing 7Qr3S/v coutermarsh.mike@gmail.com SSH key... done To add our current user's public key, we can use keys:add. This will look on our machine for a public key (~/.ssh/id_rsa.pub) and upload it: $ heroku keys:addFound existing public key: /Users/mike/.ssh/id_rsa.pubUploading SSH public key /Users/mike/.ssh/id_rsa.pub… done To create a new SSH key, we can run $ ssh-keygen -t rsa. If we'd like, we can also specify where the key is located if it is not in the default /.ssh/ directory: $ heroku keys:add /path/to/key.pub How it works… SSH keys are the standard method for password-less authentication. There are two parts to each SSH key. There is a private key, which stays on our machine and should never be shared, and there is a public key, which we can freely upload and share. Each key has its purpose. The public key is used to encrypt messages. The private key is used to decrypt messages. When we try to connect to our Git repositories, Heroku's server uses our public key to create an encrypted message that can only be decrypted by our private key. The server then sends the message to our machine; our machine's SSH client decrypts it and sends the response to the server. Sending the correct response successfully authenticates us. SSH keys are not used for authentication to the Heroku CLI. The CLI uses an authentication token that is stored in our ~/.netrc file. Sharing and collaboration We can invite collaborators through both the web dashboard and the CLI. In this recipe, we'll learn how to quickly invite collaborators through the CLI. How to do it… To start, let's open a terminal and navigate to the Heroku application that we would like to share, using the following steps: To see the current users who have access to our application, we can use the sharing command: $ heroku sharing=== load-tester-rails Access Listcoutermarsh.mike@gmail.com ownermike@form26.com             collaborator To invite a collaborator, we can use sharing:add: $ heroku sharing:add coutermarshmike@gmail.com Adding coutermarshmike@gmail.com to load-tester-rails as collaborator... done Heroku will send an e-mail to the user we're inviting, even if they do not already have a Heroku account. If we'd like to revoke access to our application, we can do so with sharing:remove:$ heroku sharing:remove coutermarshmike@gmail.comRemoving coutermarshmike@gmail.com from load-tester-rails collaborators... done How it works… When we add another collaborator to our Heroku application, they are granted the same abilities as us, except that they cannot manage paid add-ons or delete the application. Otherwise, they have full control to administrate the application. If they have an existing Heroku account, their SSH key will be immediately added to the application's Git repository. See also Interested in using multiple Heroku accounts on a single machine? Take a look at the Heroku-accounts plugin at https://github.com/ddollar/heroku-accounts. Monitoring load average and memory usage We can monitor the resource usage of our dynos from the command line using the log-runtime-metrics plugin. This will give us visibility into the CPU and memory usage of our dynos. With this data, we'll be able to determine if our dynos are correctly sized, detect problems earlier, and determine whether we need to scale our application. How to do it… Let's open up a terminal; we'll be completing this recipe with the CLI by performing the following steps: First, we'll need to install the log-runtime-metrics plugin via the CLI. We can do this easily through heroku labs: $ heroku labs:enable log-runtime-metrics Now that the runtime metrics plugin is installed, we'll need to restart our dynos for it to take effect: $ heroku restart Now that the plugin is installed and running, our dynos' resource usage will be printed to our logs. Let's view them now: $ heroku logsheroku[web.1]: source=web.1 dyno=heroku.21 sample#load_avg_1m=0.00 sample#load_avg_5m=0.00heroku[web.1]: source=web.1 dyno=heroku.21sample#memory_total=105.28MB sample#memory_rss=105.28MBsample#memory_cache=0.00MBsample#memory_swap=0.00MBsample#memory_pgpgin=31927pagessample#memory_pgpgout=4975pages From the logs, we can see that for this application, our load average is 0, and this dyno is using a total of 105 MB of RAM. How it works… Now that we have some insight into how our dynos are using resources, we need to learn how to interpret these numbers. Understanding the utilization of our dynos will be key for us if we ever need to diagnose a performance-related issue. In our logs, we will now see load_avg_1m and load_avg_5m. This is our dynos' load average over a 1-minute and 5-minute period. The timeframes are helpful in determining whether we're experiencing a brief spike in activity or it is more sustained. Load average is the amount of total computational work that the CPU has to complete. The 1X and 2X dynos have access to four virtual cores. A load average of four means that the dynos' CPU is fully utilized. Any value above four is a warning sign that the dyno might be overloaded, and response times could begin to suffer. Web applications are typically not CPU-intensive applications, seeing low load averages for web dynos should be expected. If we start seeing high load averages, we should consider either adding more dynos or using larger dynos to handle the load. Our memory usage is also shown in the logs. The key value that we want to keep track of is memory_rrs, which is the total amount of RAM being utilized by our application. It's best to keep this value no higher than 50 to 70 percent of the total RAM available on the dyno. For a 1X dyno with 512 MB of memory, this would mean keeping our memory usage no greater than 250 to 350 MB. This allows our application's room to grow under load and helps us avoid any memory swapping. Seeing values above 70 percent is an indication that we need to either adjust our application's memory usage or scale up. Memory swap occurs when our dyno runs out of RAM. To compensate, our dyno will begin using its hard drive to store data that will normally be stored in RAM. For any web application, any swap should be considered evil. This value should always be zero. If our dyno starts swapping, we can expect that it will significantly slow down our application's response times. Seeing any swap is an immediate indication that we must either reduce our application's memory consumption or start scaling. See also Load average and memory usage are particularly useful when performing application load tests. Summary In this article, we learned various commands on how to view application logs, installing add-ons, viewing application logs, enabling the maintenance page, managing SSH keys, sharing and collaboration, and so on. Resources for Article: Further resources on this subject: Securing vCloud Using the vCloud Networking and Security App Firewall [article] vCloud Networks [article] Apache CloudStack Architecture [article]
Read more
  • 0
  • 0
  • 33833

article-image-amazon-web-services
Packt
20 Nov 2014
16 min read
Save for later

Amazon Web Services

Packt
20 Nov 2014
16 min read
 In this article, by Prabhakaran Kuppusamy and Uchit Vyas, authors of AWS Development Essentials, you will learn different tools and methods available to perform the same operation with different, varying complexities. Various options are available, depending on the user's level of experience. In this article, we will start with an overview of each service, learn about the various tools available for programmer interaction, and finally see the troubleshooting and best practices to be followed while using these services. AWS provides a handful of services in every area. In this article, we will cover the following topics: Navigate through the AWS Management Console Describe the security measures that AWS provides AWS interaction through the SDK and IDE tools (For more resources related to this topic, see here.) Background of AWS and its needs AWS is based on an idea presented by Chris Pinkham and Benjamin Black with a vision towards Amazon's retail computing infrastructure. The first Amazon offering was SQS, in the year 2004. Officially, AWS was launched and made available online in 2006, and within a year, 200,000 developers signed up for these services. Later, due to a natural disaster (June 29, 2012 storm in North Virginia, which brought down most of the servers residing at this location) and technical events, AWS faced a lot of challenges. A similar event happened on December 2012, after which AWS has been providing services as stated. AWS learned from these events and made sure that the same kind of outage didn't occur even if the same event occurred again. AWS is an idea born in a single room, but the idea is now made available and used by almost all the cloud developers and IT giants. AWS is greatly loved by all kinds of technology admirers. Irrespective of the user's expertise, AWS has something for various types of users. For an expert programmer, AWS has SDKs for each service. Using these SDKs, the programmer can perform operations by entering commands in the command-line interface. However an end user with limited knowledge of programming can still perform similar operations using the graphical user interface of the AWS Management Console, which is accessible through a web browser. If the programmers need interactions between a low-level (SDK) and a high-level (Management Console), they can go for the integrated development environment (IDE) tools, for which AWS provides plugins and add-ons. One such commonly used IDE for which AWS has provided add-ons is the Eclipse IDE. As of now, we will start with the AWS Management Console. The AWS Management Console The most popular method of accessing AWS is via the Management Console because of its simplicity of usage and power. Another reason why the end user prefers the Management Console is that it doesn't require any software to start with; having an Internet connection and a browser is sufficient. As the name suggests, the Management Console is a place where administrative and advanced operations can be performed on your AWS account details or AWS services. The Management Console mainly focuses on the following features: One-click access to AWS's services AWS account administration AWS management using handheld devices AWS infrastructure management across the globe One-click access to the AWS services To access the Management Console, all you need to do is first sign up with AWS. Once done, the Management Console will be available at https://console.aws.amazon.com/. Once you have signed up, you will be directed to the following page: Each and every icon on this page is an Amazon Web Service. Two or more services will be grouped under a category. For example, in the Analytics category, you can see three services, namely, Data Pipeline, Elastic MapReduce, and Kinesis. Starting with any of these services is very easy. Have a look at the description of the service at the bottom of the service icon. As soon as you click on the service icon, it will take you to the Getting started page of the corresponding service, where brief as well as detailed guidelines are available. In order to start with any of the services, only two things are required. The first one is an AWS account and the second one is the supported browser. The Getting started section usually will have a video, which explains the specialty and use cases of the service that you selected. Once you finish reading the Getting started section, optionally you can go through the DOC files specific to the service to know more about the syntaxes and usage of the service operations. AWS account administration The account administration is one of the most important things to make note of. To do this, click on your displayed name (in this case, Prabhakar) at the top of the page, and then click on the My Account option, as shown in the preceding screenshot. At the beginning of every month, you don't want AWS to deduct all your salary by stating that you have used these many services costing this much money; hence, all this management information is available in the Management Console. Using the Management Console, you can infer the following information: The monthly billing in brief as well as the detailed manner (cost split-up of each service) along with a provision to view VAT and tax exemption Account details, such as the display name and contact information Provision to close the AWS account All the preceding operations and much more are possible. AWS management using handheld devices Managing and accessing the AWS services is through (but not limited to) PC. AWS provides a handful of applications almost for all or most of the mobile platforms, such as Android, iOS, and so on. Using these applications, you can perform all the AWS operations on the move. You won't believe that having a 7-inch Android tablet with the installed AWS Console application from Google Play will enable you to ask for any Elastic Compute Cloud (EC2) instance from Amazon and control it (start, stop, and terminate) very easily. You can install an SSH client in the tablet and connect to the Linux terminal. However, if you wish to make use of the Windows instance from EC2, you might use the Graphics User Interface (GUI) more frequently than a command line. A few more sophisticated software and hardware might be needed, for example, you should have a VNC viewer or remote desktop connection software to get the GUI of the EC2 instance borrowed. As you are making use of the GUI in addition to the keyboard, you will need a pointer device, such as a mouse. As a result, you will almost get addicted to the concept of cloud computing going mobile. AWS infrastructure management across the globe At this point, you might be aware that you can get all of these AWS services from servers residing at any of the following locations. To control these services used by you in different regions, you don't have to go anywhere else. You can control it right here in the same Management Console. Using the same Management Console, just by clicking on N.Virginia and choosing the location (at the top of the Management Console), you can make the service available in that region, as shown in the following screenshot: You can choose the server location at which you want the service (data and machine) to be made available based on the following two factors: The first factor is the distance between the server's location and the client's location. For example, if you have deployed a web application for a client from North California at a Tokyo location, obviously the latency will be high while accessing the application. Therefore, choosing the optimum service location is the primary factor. The second factor is the charge for the service in a specific location. AWS charges more for certain crowded servers. Just for illustration, assume that the server for North California is used by many critical companies. So this might cost you twice if you create your servers at North California compared to the other locations. Hence, you should always consider the tradeoff between the location and cost and then decide on the server location. Whenever you click on any of the services, AWS will always select the location that costs you less money as the default. AWS security measures Whenever you think of moving your data center to a public cloud, the first question that arises in your mind is about data security. In a public cloud, through virtualization technology, multiple users might be using the same hardware (server) in which your data is available. You will learn in detail about how AWS ensures data security. Instance isolation Before learning about instance isolation, you must know how AWS EC2 provisions the instances to the user. This service allows you to rent virtual machines (AWS calls it instances) with whatever configurations you ask. Let's assume that you requested AWS to provision a 2 GB RAM, a 100 GB HDD, and an Ubuntu instance. Within a minute, you will be given the instance's connection details (public DNS, private IP, and so on), and the instance starts running. Does this mean that AWS assembled a 2*1 GB RAM and 100 GB HDD into a CPU cabinet and then installed Ubuntu OS in it and gave you the access? The answer is no. The provisioned instance is not a single PC (or bare metal) with an OS installed in it. The instance is the outcome of a virtual machine provisioned by Amazon's private cloud. The following diagram shows how a virtual machine can be provisioned by a private cloud: Let's examine the diagram from bottom to top. First, we will start with the underlying Hardware/Host. Hardware is the server, which usually has a very high specification. Here, assume that your hardware has the configuration of a 99 GB RAM, a 450 TB HDD, and a few other elements, such as NIC, which you need not consider now. The next component in your sights is the Hypervisor. A hypervisor or virtual machine monitor (VMM) is used to create and run virtual machines on the hardware. In private cloud terms, whichever machine runs a hypervisor on it is called the host machine. Three users can request each of them need instances with a 33 GB RAM and 150 TB HDD space. This request goes to the hypervisor and it then starts creating those VMs. After creating the VMs, a notification about the connection parameters will be sent to each user. In the preceding diagram, you can see the three virtual machines (VMs) created by the hypervisor. All the three VMs are running on different operating systems. Even if all the three virtual machines are used by different users, each will feel that only he/she has access to the single piece of hardware, which is only used by them; user 1 might not know that the same hardware is also being used by user 2, and so on. The process of creating a virtual version of a machine or storage or network is called virtualization. The funny part is that none of the virtual machines knows that it is being virtualized (that is, all the VMs are created on the same host). After getting this information about your instances, some users may feel deceived, and some will be even disappointed and cry out loud, has your instance been created on a shared disc or resource? Even though the disc (or hardware) is shared, one instance (or owner of the instance) is isolated from the other instances on the same disc through a firewall. This concept is termed as instance isolation. The following diagram demonstrates instance isolation in AWS: The preceding diagram clearly demonstrates how EC2 provides instances to every user. Even though all the instances are lying in the same disc, they are isolated by hypervisor. Hypervisor has a firewall that does this isolation. So, the physical interface will not interact with the underlying hardware (machine or disc where instances are available) or virtual interface directly. All these interactions will be through hypervisor's firewall. This way AWS ensures that no user can directly access the disc, and no instance can directly interact with another instance even if both instances are running on the same hardware. In addition to the firewall, during the creation of the EC2 instance, the user can specify the permitted and denied security groups of the instance. These two ideologies provide instance isolation. In the preceding diagram, Customer 1, Customer 2, and so on are virtualized discs since the customer instances have no access to raw or actual disc devices. As an added security measure, the user can encrypt his/her disc so that other users cannot access the disc content (even if someone gets in contact with the disc). Isolated GovCloud Similar to North California or Asia Pacific, GovCloud is also a location where you can get your AWS services. This location is specifically designed only for government and agencies whose data is very confidential and valuable, and disclosing this data might result in disaster. By default, this location will not be available to the user. If you want access to this location, then you need to raise a compliance request at http://aws.amazon.com/compliance/contact/ submit the FedRAMP Package Request Form downloadable at http://cloud.cio.gov/document/fedramp-package-request-form. From these two URLs, you can understand how secured the cloud location really is. CloudTrail CloudTrail is an AWS service that performs the user activity and changes tracking. Enabling CloudTrail will log all the API request information into your S3 bucket, which you have created solely for this purpose. CloudTrail also allows you to create an SNS topic as soon as a new logfile is created by CloudTrail. CloudTrail, in hand with SNS, provides real-time user activity as messages to the user. Password This might sound funny. After looking at CloudTrail, if you feel that someone else is accessing your account, the best option is to change the password. Never let anyone look at your password, as this could easily comprise an entire account. Sharing the password is like leaving your treasury door open. Multi-Factor Authentication Until now, to access AWS through a browser, you had to log in at http://aws.amazon.com and enter your username and password. However, enabling Multi-Factor Authentication (MFA) will add another layer of security and ask you to provide an authentication code sent to the device configured with this account. In the security credential page at https://console.aws.amazon.com/iam/home?#security_credential, there is a provision to enable MFA. Clicking on Enable will display the following window: Selecting the first option A virtual MFA device will not cost you money, but this requires a smartphone (with an Android OS), and you need to download an app from the App Store. After this, during every login, you need to look at your smartphone and enter the authentication token. More information is available at https://youtu.be/MWJtuthUs0w. Access Keys (Access Key ID and Secret Access Key) In the same security credentials page, next to MFA, these access keys will be made available. AWS will not allow you to have more than two access keys. However, you can delete and create as many access keys as possible, as shown in the following screenshot: This access key ID is used while accessing the service via the API and SDK. During this time, you must provide this ID. Otherwise, you won't be able to perform any operation. To put it in other words, if someone else gets or knows this ID, they could pretend to be you through the SDK and API. In the preceding screenshot, the first key is inactive and the second key is active. The Create New Access Key button is disabled because I already have a maximum number of allowed access keys. As an added measure, I forged my actual IDs. It is a very good practice to delete a key and create a new key every month using the Delete command link and toggle the active keys every week (by making it active and inactive) by clicking on the Make Active or Make Inactive command links. Never let anyone see these IDs. If you are ever in doubt, delete the ID and create a new one. Clicking on Create New Access Key button (assuming that you have less than two IDs) will display the following window, asking you to download the new access key ID as a CSV file: The CloudFront key pairs The CloudFront key pairs are very similar to the access-key IDs. Without these keys, you will not be able to perform any operation on CloudFront. Unlike the access key ID (which has only access key ID and secret access key), here you will have a private key and a public key along with the access key ID, as shown in the following screenshot: If you lose these keys once, then you need to delete the key pair and create a new key pair. This is also an added security measure. X.509 certificates X.509 certificates are mandatory if you wish to make any SOAP requests on any AWS service. Clicking on Create new certificate will display the following window, which performs exactly the same function: Account identifiers There are two IDs that are used to identify ourselves when accessing the service via the API or SDK. These are the AWS account ID and the canonical user ID. These two IDs are unique. Just as with the preceding parameters, never share these IDs or let anyone see them. If someone has your access ID or key pair, the best option is generate a new one. But it is not possible to generate a new account ID or canonical user ID. Summary In this article, you learned the AWS Management Console and its commonly used SDKs and IDEs. You also learned how AWS secures your data. Then, you looked at the AWS plugin configuration on the Eclipse IDE. The first part made the user familiar with the AWS Management Console. After that, you explored a few of the important security aspects of AWS and learned how AWS handles it. Finally, you learned about the different AWS tools available to the programmer to make his development work easier. In the end, you examined the common SDKs and IDE tools of AWS. Resources for Article: Further resources on this subject: Amazon DynamoDB - Modelling relationships, Error handling [article] A New Way to Scale [article] Deployment and Post Deployment [article]
Read more
  • 0
  • 0
  • 23415
Modal Close icon
Modal Close icon