How-To Tutorials

In this article by Fernando Monteiro, author of the book Learning Single-page Web Application Development, we will see how to build a solid foundation for our API. Our main aim is to discuss the techniques to build rich web applications, with the SPA approach. We will be covering the following topics in this article: The working of an API Boilerplates and generators The speakers API concept Creating the package.json file The Node server with server.js The model with the Mongoose schema Defining the API routes Using MongoDB in the cloud Inserting data with the Postman Chrome extension (For more resources related to this topic, see here.) The working of an API An API works through communication between different codes, thus defining specific behavior of certain objects on an interface. That is, the API will connect several functions on one website (such as search, images, news, authentications, and so on) to enable it to be used in other applications. Operating systems also have APIs, and they still have the same function. Windows, for example, has APIs such as the Win16 API, Win32 API, or Telephony API, in all its versions. When you run a program that involves some process of the operating system, it is likely that we make a connection with one or more Windows APIs. To clarify the concept of an API, we will give go through some examples of how it works. On Windows, it works on an application that uses the system clock to display the same function within the program. It then associates a behavior to a given clock time in another application, for example, using the Time/Clock API from Windows to use the clock functionality on your own application. Another example, is when you use the Android SDK to build mobile applications. When you use the device GPS, you are interacting with the API (android.location) to display the user location on the map through another API, in this case, Google Maps API. The following is the API example: When it comes to web APIs, the functionality can be even greater. There are many services that provide their code, so that they can be used on other websites. Perhaps, the best example is the Facebook API. Several other websites use this service within their pages, for instance a like button, share, or even authentication. An API is a set of programming patterns and instructions to access a software application based on the Web. So, when you access a page of a beer store in your town, you can log in with your Facebook account. This is accomplished through the API. Using it, software developers and web programmers can create beautiful programs and pages filled with content for their users. Boilerplates and generators On a MEAN stack environment, our ecosystem is infinitely diverse, and we can find excellent alternatives to start the construction of our API. At hand, we have simple boilerplates to complex code generators that can be used with other tools in an integrated way, or even alone. Boilerplates are usually a group of tested code that provides the basic structure to the main goal, that is to create a foundation of a web project. Besides saving us from common tasks such as assembling the basic structure of the code and organizing the files, boilerplates already have a number of scripts to make life easier for the frontend. Let's describe some alternatives that we consider as good starting points for the development of APIs with the Express framework, MongoDB database, Node server, and AngularJS for the frontend. Some more accentuated knowledge of JavaScript might be necessary for the complete understanding of the concepts covered here; so we will try to make them as clearly as possible. It is important to note that everything is still very new when we talk about Node and all its ecosystems, and factors such as scalability, performance, and maintenance are still major risk factors. Bearing in mind also that languages such as Ruby on Rails, Scala, and the Play framework have a higher reputation in building large and maintainable web applications, but without a doubt, Node and JavaScript will conquer your space very soon. That being said, we present some alternatives for the initial kickoff with MEAN, but remember that our main focus is on SPA and not directly on MEAN stack. Hackathon starter Hackathon is highly recommended for a quick start to develop with Node. This is because the boilerplate has the main necessary characteristics to develop applications with the Express framework to build RESTful APIs, as it has no MVC/MVVM frontend framework as a standard but just the Bootstrap UI framework. Thus, you are free to choose the framework of your choice, as you will not need to refactor it to meet your needs. Other important characteristics are the use of the latest version of the Express framework, heavy use of Jade templates and some middleware such as Passport - a Node module to manage authentication with various social network sites such as Twitter, Facebook, APIs for LinkedIn, Github, Last.fm, Foursquare, and many more. They provide the necessary boilerplate code to start your projects very fast, and as we said before, it is very simple to install; just clone the Git open source repository: git clone --depth=1 https://github.com/sahat/hackathon-starter.git myproject Run the NPM install command inside the project folder: npm install Then, start the Node server: node app.js Remember, it is very important to have your local database up and running, in this case MongoDB, otherwise the command node app.js will return the error: Error connecting to database: failed to connect to [localhost: 27017] MEAN.io or MEAN.JS This is perhaps the most popular and currently available boilerplate. MEAN.JS is a fork of the original project MEAN.io; both are open source, with a very peculiar similarity, both have the same author. You can check for more details at http://meanjs.org/. However, there are some differences. We consider MEAN.JS to be a more complete and robust environment. It has a structure of directories, better organized, subdivided modules, and better scalability by adopting a vertical modules development. To install it, follow the same steps as previously: Clone the repository to your machine: git clone https://github.com/meanjs/mean.git Go to the installation directory and type on your terminal: npm install Finally, execute the application; this time with the Grunt.js command: grunt If you are on Windows, type the following command: grunt.cmd Now, you have your app up and running on your localhost. The most common problem when we need to scale a SPA is undoubtedly the structure of directories and how we manage all of the frontend JavaScript files and HTML templates using MVC/MVVM. Later, we will see an alternative to deal with this on a large-scale application; for now, let's see the module structure adopted by MEAN.JS: Note that MEAN.JS leaves more flexibility to the AngularJS framework to deal with the MVC approach for the frontend application, as we can see inside the public folder. Also, note the modules approach; each module has its own structure, keeping some conventions for controllers, services, views, config, and tests. This is very useful for team development, so keep all the structure well organized. It is a complete solution that makes use of additional modules such as passport, swig, mongoose, karma, among others. The Passport module Some things about the Passport module must be said; it can be defined as a simple, unobtrusive authentication module. It is a powerful middleware to use with Node; it is very flexible and also modular. It can also adapt easily within applications that use the Express. It has more than 140 alternative authentications and support session persistence; it is very lightweight and extremely simple to be implemented. It provides us with all the necessary structure for authentication, redirects, and validations, and hence it is possible to use the username and password of social networks such as Facebook, Twitter, and others. The following is a simple example of how to use local authentication: var passport = require('passport'), LocalStrategy = require('passport-local').Strategy, User = require('mongoose').model('User');   module.exports = function() { // Use local strategy passport.use(new LocalStrategy({ usernameField: 'username', passwordField: 'password' }, function(username, password, done) { User.findOne({    username: username }, function(err, user) { if (err) { return done(err); } if (!user) {    return done(null, false, {    message: 'Unknown user'    }); } if (!user.authenticate(password)) {    return done(null, false, {    message: 'Invalid password'    }); } return done(null, user); }); } )); }; Here's a sample screenshot of the login page using the MEAN.JS boilerplate with the Passport module: Back to the boilerplates topic; most boilerplates and generators already have the Passport module installed and ready to be configured. Moreover, it has a code generator so that it can be used with Yeoman, which is another essential frontend tool to be added to your tool belt. Yeoman is the most popular code generator for scaffold for modern web applications; it's easy to use and it has a lot of generators such as Backbone, Angular, Karma, and Ember to mention a few. More information can be found at http://yeoman.io/. Generators Generators are for the frontend as gem is for Ruby on Rails. We can create the foundation for any type of application, using available generators. Here's a console output from a Yeoman generator: It is important to bear in mind that we can solve almost all our problems using existing generators in our community. However, if you cannot find the generator you need, you can create your own and make it available to the entire community, such as what has been done with RubyGems by the Rails community. RubyGem, or simply gem, is a library of reusable Ruby files, labeled with a name and a version (a file called gemspec). Keep in mind the Don't Repeat Yourself (DRY) concept; always try to reuse an existing block of code. Don't reinvent the wheel. One of the great advantages of using a code generator structure is that many of the generators that we have currently, have plenty of options for the installation process. With them, you can choose whether or not to use many alternatives/frameworks that usually accompany the generator. The Express generator Another good option is the Express generator, which can be found at https://github.com/expressjs/generator. In all versions up to Express Version 4, the generator was already pre-installed and served as a scaffold to begin development. However, in the current version, it was removed and now must be installed as a supplement. They provide us with the express command directly in terminal and are quite useful to start the basic settings for utilization of the framework, as we can see in the following commands: create : . create : ./package.json create : ./app.js create : ./public create : ./public/javascripts create : ./public/images create : ./public/stylesheets create : ./public/stylesheets/style.css create : ./routes create : ./routes/index.js create : ./routes/users.js create : ./views create : ./views/index.jade create : ./views/layout.jade create : ./views/error.jade create : ./bin create : ./bin/www   install dependencies:    $ cd . && npm install   run the app:    $ DEBUG=express-generator ./bin/www Very similar to the Rails scaffold, we can observe the creation of the directory and files, including the public, routes, and views folders that are the basis of any application using Express. Note the npm install command; it installs all dependencies provided with the package.json file, created as follows: { "name": "express-generator", "version": "0.0.1", "private": true, "scripts": {    "start": "node ./bin/www" }, "dependencies": {    "express": "~4.2.0",    "static-favicon": "~1.0.0",    "morgan": "~1.0.0",    "cookie-parser": "~1.0.1",    "body-parser": "~1.0.0",    "debug": "~0.7.4",    "jade": "~1.3.0" } } This has a simple and effective package.json file to build web applications with the Express framework. The speakers API concept Let's go directly to build the example API. To be more realistic, let's write a user story similar to a backlog list in agile methodologies. Let's understand what problem we need to solve by the API. The user history We need a web application to manage speakers on a conference event. The main task is to store the following speaker information on an API: Name Company Track title Description A speaker picture Schedule presentation For now, we need to add, edit, and delete speakers. It is a simple CRUD function using exclusively the API with JSON format files. Creating the package.json file Although not necessarily required at this time, we recommend that you install the Webstorm IDE, as we'll use it throughout the article. Note that we are using the Webstorm IDE with an integrated environment with terminal, Github version control, and Grunt to ease our development. However, you are absolutely free to choose your own environment. From now on, when we mention terminal, we are referring to terminal Integrated WebStorm, but you can access it directly by the chosen independent editor, terminal for Mac and Linux and Command Prompt for Windows. Webstorm is very useful when you are using a Windows environment, because Windows Command Prompt does not have the facility to copy and paste like Mac OS X on the terminal window. Initiating the JSON file Follow the steps to initiate the JSON file: Create a blank folder and name it as conference-api, open your terminal, and place the command: npm init This command will walk you through creating a package.json file with the baseline configuration for our application. Also, this file is the heart of our application; we can control all the dependencies' versions and other important things like author, Github repositories, development dependencies, type of license, testing commands, and much more. Almost all commands are questions that guide you to the final process, so when we are done, we'll have a package.json file very similar to this: { "name": "conference-api", "version": "0.0.1", "description": "Sample Conference Web Application", "main": "server.js", "scripts": {    "test": "test" }, "keywords": [    "api" ], "author": "your name here", "license": "MIT" } Now, we need to add the necessary dependencies, such as Node modules, which we will use in our process. You can do this in two ways, either directly via terminal as we did here, or by editing the package.json file. Let's see how it works on the terminal first; let's start with the Express framework. Open your terminal in the api folder and type the following command: npm install express@4.0.0 –-save This command installs the Express module, in this case, Express Version 4, and updates the package.json file and also creates dependencies automatically, as we can see: { "name": "conference-api", "version": "0.0.1", "description": "Sample Conference Web Application", "main": "server.js", "scripts": {    "test": "test" }, "keywords": [    "api" ], "author": "your name here", "license": "MIT", "dependencies": {    "express": "^4.0.0" } } Now, let's add more dependencies directly in the package.json file. Open the file in your editor and add the following lines: { "name": "conference-api", "version": "0.0.1", "description": "Sample Conference Web Application", "main": "server.js", "scripts": {    "test": "test" }, "keywords": [    "api" ], "author": "your name here", "license": "MIT", "engines": {        "node": "0.8.4",        "npm": "1.1.49" }, "dependencies": {    "body-parser": "^1.0.1",    "express": "^4.0.0",    "method-override": "^1.0.0",    "mongoose": "^3.6.13",    "morgan": "^1.0.0",    "nodemon": "^1.2.0" }, } It's very important when you deploy your application using some services such as Travis Cl or Heroku hosting company. It's always good to set up the Node environment. Open the terminal again and type the command: npm install You can actually install the dependencies in two different ways, either directly into the directory of your application or globally with the -g command. This way, you will have the modules installed to use them in any application. When using this option, make sure that you are the administrator of the user machine, as this command requires special permissions to write to the root directory of the user. At the end of the process, we'll have all Node modules that we need for this project; we just need one more action. Let's place our code over a version control, in our case Git. More information about the Git can be found at http://git-scm.com however, you can use any version control as subversion or another. We recommend using Git, as we will need it later to deploy our application in the cloud, more specificly, on Heroku cloud hosting. At this time, our project folder must have the same structure as that of the example shown here: We must point out the utilization of an important module called the Nodemon module. Whenever a file changes it restarts the server automatically; otherwise, you will have to restart the server manually every time you make a change to a file, especially in a development environment that is extremely useful, as it constantly updates our files. Node server with server.js With this structure formed, we will start the creation of the server itself, which is the creation of a main JavaScript file. The most common name used is server.js, but it is also very common to use the app.js name, especially in older versions. Let's add this file to the root folder of the project and we will start with the basic server settings. There are many ways to configure our server, and probably you'll find the best one for yourself. As we are still in the initial process, we keep only the basics. Open your editor and type in the following code: // Import the Modules installed to our server var express   = require('express'); var bodyParser = require('body-parser');   // Start the Express web framework var app       = express();   // configure app app.use(bodyParser());   // where the application will run var port     = process.env.PORT || 8080;   // Import Mongoose var mongoose   = require('mongoose');   // connect to our database // you can use your own MongoDB installation at: mongodb://127.0.0.1/databasename mongoose.connect('mongodb://username:password@kahana.mongohq.com:10073/node-api');   // Start the Node Server app.listen(port); console.log('Magic happens on port ' + port); Realize that the line-making connection with MongoDB on our localhost is commented, because we are using an instance of MongoDB in the cloud. In our case, we use MongoHQ, a MongoDB-hosting service. Later on, will see how to connect with MongoHQ. Model with the Mongoose schema Now, let's create our model, using the Mongoose schema to map our speakers on MongoDB. // Import the Mongoose module. var mongoose     = require('mongoose'); var Schema       = mongoose.Schema;   // Set the data types, properties and default values to our Schema. var SpeakerSchema   = new Schema({    name:           { type: String, default: '' },    company:       { type: String, default: '' },    title:         { type: String, default: '' },    description:   { type: String, default: '' },    picture:       { type: String, default: '' },    schedule:       { type: String, default: '' },    createdOn:     { type: Date,   default: Date.now} }); module.exports = mongoose.model('Speaker', SpeakerSchema); Note that on the first line, we added the Mongoose module using the require() function. Our schema is pretty simple; on the left-hand side, we have the property name and on the right-hand side, the data type. We also we set the default value to nothing, but if you want, you can set a different value. The next step is to save this file to our project folder. For this, let's create a new directory named server; then inside this, create another folder called models and save the file as speaker.js. At this point, our folder looks like this: The README.md file is used for Github; as we are using the Git version control, we host our files on Github. Defining the API routes One of the most important aspects of our API are routes that we take to create, read, update, and delete our speakers. Our routes are based on the HTTP verb used to access our API, as shown in the following examples: To create record, use the POST verb To read record, use the GET verb To update record, use the PUT verb To delete records, use the DELETE verb So, our routes will be as follows: Routes Verb and Action /api/speakers GET retrieves speaker's records /api/speakers/ POST inserts speakers' record /api/speakers/:speaker_id GET retrieves a single record /api/speakers/:speaker_id PUT updates a single record /api/speakers/:speaker_id DELETE deletes a single record Configuring the API routes: Let's start defining the route and a common message for all requests: var Speaker     = require('./server/models/speaker');   // Defining the Routes for our API   // Start the Router var router = express.Router();   // A simple middleware to use for all Routes and Requests router.use(function(req, res, next) { // Give some message on the console console.log('An action was performed by the server.'); // Is very important using the next() function, without this the Route stops here. next(); });   // Default message when access the API folder through the browser router.get('/', function(req, res) { // Give some Hello there message res.json({ message: 'Hello SPA, the API is working!' }); }); Now, let's add the route to insert the speakers when the HTTP verb is POST: // When accessing the speakers Routes router.route('/speakers')   // create a speaker when the method passed is POST .post(function(req, res) {   // create a new instance of the Speaker model var speaker = new Speaker();   // set the speakers properties (comes from the request) speaker.name = req.body.name; speaker.company = req.body.company; speaker.title = req.body.title; speaker.description = req.body.description; speaker.picture = req.body.picture; speaker.schedule = req.body.schedule;   // save the data received speaker.save(function(err) {    if (err)      res.send(err);   // give some success message res.json({ message: 'speaker successfully created!' }); }); }) For the HTTP GET method, we need this: // get all the speakers when a method passed is GET .get(function(req, res) { Speaker.find(function(err, speakers) {    if (err)      res.send(err);      res.json(speakers); }); }); Note that in the res.json() function, we send all the object speakers as an answer. Now, we will see the use of different routes in the following steps: To retrieve a single record, we need to pass speaker_id, as shown in our previous table, so let's build this function: // on accessing speaker Route by id router.route('/speakers/:speaker_id')   // get the speaker by id .get(function(req, res) { Speaker.findById(req.params.speaker_id, function(err,    speaker) {    if (err)      res.send(err);      res.json(speaker);    }); }) To update a specific record, we use the PUT HTTP verb and then insert the function: // update the speaker by id .put(function(req, res) { Speaker.findById(req.params.speaker_id, function(err,     speaker) {      if (err)      res.send(err);   // set the speakers properties (comes from the request) speaker.name = req.body.name; speaker.company = req.body.company; speaker.title = req.body.title; speaker.description = req.body.description; speaker.picture = req.body.picture; speaker.schedule = req.body.schedule;   // save the data received speaker.save(function(err) {    if (err)      res.send(err);      // give some success message      res.json({ message: 'speaker successfully       updated!'}); });   }); }) To delete a specific record by its id: // delete the speaker by id .delete(function(req, res) { Speaker.remove({    _id: req.params.speaker_id }, function(err, speaker) {    if (err)      res.send(err);   // give some success message res.json({ message: 'speaker successfully deleted!' }); }); }); Finally, register the Routes on our server.js file: // register the route app.use('/api', router); All necessary work to configure the basic CRUD routes has been done, and we are ready to run our server and begin creating and updating our database. Open a small parenthesis here, for a quick step-by-step process to introduce another tool to create a database using MongoDB in the cloud. There are many companies that provide this type of service but we will not go into individual merits here; you can choose your preference. We chose Compose (formerly MongoHQ) that has a free sandbox for development, which is sufficient for our examples. Using MongoDB in the cloud Today, we have many options to work with MongoDB, from in-house services to hosting companies that provide Platform as a Service (PaaS) and Software as a Service (SaaS). We will present a solution called Database as a Service (DbaaS) that provides database services for highly scalable web applications. Here's a simple step-by-step process to start using a MongoDB instance with a cloud service: Go to https://www.compose.io/. Create your free account. On your dashboard panel, click on add Database. On the right-hand side, choose Sandbox Database. Name your database as node-api. Add a user to your database. Go back to your database title, click on admin. Copy the connection string. The string connection looks like this: mongodb://<user>:<password>@kahana.mongohq.com:10073/node-api. Let's edit the server.js file using the following steps: Place your own connection string to the Mongoose.connect() function. Open your terminal and input the command: nodemon server.js Open your browser and place http://localhost:8080/api. You will see a message like this in the browser: { Hello SPA, the API is working! } Remember the api folder was defined on the server.js file when we registered the routes: app.use('/api', router); But, if you try to access http://localhost:8080/api/speakers, you must have something like this: [] This is an empty array, because we haven't input any data into MongoDB. We use an extension for the Chrome browser called JSONView. This way, we can view the formatted and readable JSON files. You can install this for free from the Chrome Web Store. Inserting data with Postman To solve our empty database and before we create our frontend interface, let's add some data with the Chrome extension Postman. By the way, it's a very useful browser interface to work with RESTful APIs. As we already know that our database is empty, our first task is to insert a record. To do so, perform the following steps: Open Postman and enter http://localhost:8080/api/speakers. Select the x-www-form-urlencoded option and add the properties of our model: var SpeakerSchema   = new Schema({ name:           { type: String, default: '' }, company:       { type: String, default: '' }, title:         { type: String, default: '' }, description:   { type: String, default: '' }, picture:       { type: String, default: '' }, schedule:       { type: String, default: '' }, createdOn:     { type: Date,   default: Date.now} }); Now, click on the blue button at the end to send the request. With everything going as expected, you should see message: speaker successfully created! at the bottom of the screen, as shown in the following screenshot: Now, let's try http://localhost:8080/api/speakers in the browser again. Now, we have a JSON file like this, instead of an empty array: { "_id": "53a38ffd2cd34a7904000007", "__v": 0, "createdOn": "2014-06-20T02:20:31.384Z", "schedule": "10:20", "picture": "fernando.jpg", "description": "Lorem ipsum dolor sit amet, consectetur     adipisicing elit, sed do eiusmod...", "title": "MongoDB", "company": "Newaeonweb", "name": "Fernando Monteiro" } When performing the same action on Postman, we see the same result, as shown in the following screenshot: Go back to Postman, copy _id from the preceding JSON file and add to the end of the http://localhost:8080/api/speakers/53a38ffd2cd34a7904000005 URL and click on Send. You will see the same object on the screen. Now, let's test the method to update the object. In this case, change the method to PUT on Postman and click on Send. The output is shown in the following screenshot: Note that on the left-hand side, we have three methods under History; now, let's perform the last operation and delete the record. This is very simple to perform; just keep the same URL, change the method on Postman to DELETE, and click on Send. Finally, we have the last method executed successfully, as shown in the following screenshot: Take a look at your terminal, you can see four messages that are the same: An action was performed by the server. We configured this message in the server.js file when we were dealing with all routes of our API. router.use(function(req, res, next) { // Give some message on the console console.log('An action was performed by the server.'); // Is very important using the next() function, without this the Route stops here. next(); }); This way, we can monitor all interactions that take place at our API. Now that we have our API properly tested and working, we can start the development of the interface that will handle all this data. Summary In this article, we have covered almost all modules of the Node ecosystem to develop the RESTful API. Resources for Article: Further resources on this subject: Web Application Testing [article] A look into responsive design frameworks [article] Top Features You Need to Know About – Responsive Web Design [article]

In this article by Vangel Krstevski, author of Mastering System Center Configuration manager, we will learn that a cloud distribution point is a fallback distribution point for the Configuration Manager clients and supports most of the content types. To create a cloud distribution point, you need a Windows Azure subscription, a DNS server, and certificates. For your production environment, you can use the Azure pricing calculator to calculate your subscription fee at http://azure.microsoft.com/en-us/pricing/calculator/?scenario=full. (For more resources related to this topic, see here.) Starting with System Center Configuration Manager SP1, you can use a Windows Azure cloud service to host a distribution point server. When you deploy a cloud-based distribution point server, you configure the client settings and through them, enable users and devices to access the content. You also have to specify a primary site that will manage the content transfer to the cloud-based distribution point. Additionally, you need to specify the thresholds for the amount of content that you want to store on the distribution point and the amount of content that you want to enable clients to transfer from the distribution point. Based on these thresholds, the Configuration Manager can raise alerts that warn you when the combined amount of content that you have stored on the distribution point is near the specified storage amount, or when the transfer of data by the clients is close to the threshold that you defined. The following features are supported by both on-premise and cloud-based distribution points: The management of cloud-based distribution points individually or as members of distribution point groups A cloud-based distribution point can be used as a fallback content location You receive support for both intranet- and Internet-based clients A cloud-based distribution point provides the following additional benefits: The content that is sent to the cloud-based distribution point is encrypted by Configuration Manager before the Configuration Manager sends it to Windows Azure In Windows Azure, you can manually scale the cloud service to meet the changing demands for content request by clients, without the requirement to install and provision additional distribution points The cloud-based distribution point supports the download of content by clients that are configured for Windows BranchCache A cloud-based distribution point has the following limitations: You cannot use a cloud-based distribution point to host software update packages. You cannot use a cloud-based distribution point for PXE-enabled or multicast-enabled deployments. Clients are not offered a cloud-based distribution point as a content location for a task sequence that is deployed using the Download content locally when needed by running task sequence deployment option. However, task sequences that are deployed using the Download all content locally before starting task sequence deployment option can use a cloud-based distribution point as a valid content location. A cloud-based distribution point does not support packages that run from the distribution point. All content must be downloaded by the client and then run locally. A cloud-based distribution point does not support the streaming of applications by using Application Virtualization or similar programs. Prestaged content is not supported. The primary site Distribution Manager that is used for distribution point management does all the content transfers to the distribution point. A cloud-based distribution point cannot be configured as a pull-distribution point. To configure a cloud-based distribution point, follow these steps: Create a management certificate and install it on the site server. This certificate establishes a trust relationship between the site server and Windows Azure. Create a cloud distribution point service certificate and install it on the site server. Create a Windows Azure subscription and import the previously created management certificate in Windows Azure through the management portal. Install a cloud distribution point role in Configuration Manager. Set up the client settings to allow Configuration Manager clients to use the cloud-based distribution point. Create a record in your DNS with the IP address of the cloud distribution point. Cloud distribution points – prerequisites A cloud-based distribution point has the following prerequisites: A Windows Azure subscription. A self-signed or PKI management certificate for communication between the Configuration Manager primary site server and the Windows Azure Cloud Service. A service certificate (PKI) that Configuration Manager clients will use in order to connect to the cloud-based distribution points and also to download content from these distribution points using secure transfer or HTTPS. Before users and devices can access the content on a cloud-based distribution point, a device or a user has to receive the client setting for cloud services of Allow access to cloud distribution points set to Yes. By default, this value is set to No. A client must be able to resolve the name of the cloud service, which requires a Domain Name System (DNS) alias and CNAME record, in your DNS namespace. A client must have Internet access in order to use the cloud-based distribution point. Creating certificates Use the following link to create the needed certificates for the Cloud distribution point creation: http://technet.microsoft.com/en-us/library/230dfec0-bddb-4429-a5db-30020e881f1e#BKMK_clouddp2008_cm2012 Importing the certificates in Windows Azure First, what you need to do is log in to your Windows Azure subscription. To do this, you have to perform the following steps: Go to https://manage.windowsazure.com. After you log in, go to SETTINGS from the menu on the left-hand side, as shown in the following screenshot: Click on MANAGEMENT CERTIFICATES, as shown here: Upload the management certificate that you created for the site server, as shown in the following screenshot: After the import, you will be able to see the certificate in the list of imported MANAGEMENT CERTIFICATES, as shown here: Creating the cloud distribution point In order to create the cloud distribution point, you have to do the following: Start the System Center Configuration Manager console. Navigate to Administration | Hierarchy Configuration | Cloud Services | Cloud Distribution Points, as shown in the following screenshot: From the ribbon bar, click on Create Cloud Distribution Point. On the General page, you have to enter the Windows Azure subscription ID. You can find your Windows Azure subscription ID in the Settings section of the Windows Azure management portal. Click on Browse… to select the certificate that you created for the site server, as shown here: On the Settings page, select the region, for example, West Europe. Click on Browse… and import the cloud distribution point service certificate, as shown in the next screenshot: On the Alerts page, you can configure the settings about the threshold levels of your cloud distribution point. These levels are important because they can alert you when levels drop below a certain level that you have defined. For the purpose of this project, just click on Next: Review all the settings in the Summary page and click on Next to start the cloud distribution point's installation process. After the Cloud distribution point is created, you will be able to see it in the list of Cloud Distribution Points in the System Center Configuration Manager console, as shown here: Configuring DNS for the cloud distribution point For clients to download content from a cloud distribution point, a DNS record must exist for the cloud distribution point's IP address. You can do this by adding a CNAME record in your DNS server that points to the site URL of the Windows Azure Cloud Service. The FQDN of your Windows Azure Cloud Service can be found by proceeding with the following steps: Log in to your Windows Azure subscription. Select Cloud Services from the menu on the left-hand side. From the list of cloud services, click on the service name that represents your cloud distribution point. This will open the cloud service dashboard. The site URL information can be found on the right-hand side of the dashboard, as shown in the following screenshot: Open your DNS server and create the CNAME record. For the alias name, enter CloudDP and for the FQDN of the target host, enter the site URL of your Windows Azure, shown as follows: Summary In this article we saw that the main benefit of a cloud distribution point is that it can work as a backup distribution point. We also saw how we can use System Center Configuration Manager 2012 R2 to deliver applications to different mobile device platforms. We also learned how to connect the Configuration Manager to Windows Intune in order to provide mobile device management and application deployment and to ensure secure and managed access to company resources. Resources for Article: Further resources on this subject: Configuring Roles for High Availability [article] Configuring and Managing the Mailbox Server Role [article] Integration with System Center Operations Manager 2012 SP1 [article]

In this article created by Rodney Ringler, the author of C# Multithreaded and Parallel Programming, we will explore two popular design patterns to solve concurrent problems—Pipeline and producer-consumer, which are used in developing parallel applications using the TPL. A Pipeline design is one where an application is designed with multiple tasks or stages of functionality with queues of work items between them. So, for each stage, the application will read from a queue of work to be performed, execute the work on that item, and then queue the results for the next stage. By designing the application this way, all of the stages can execute in parallel. Each stage just reads from its work queue, performs the work, and puts the results of the work into the queue for the next stage. Each stage is a task and can run independently of the other stages or tasks. They continue executing until their queue is empty and marked completed. They also block and wait for more work items if the queue is empty but not completed. The producer-consumer design pattern is a similar concept but different. In this design, we have a set of functionality that produces data that is then consumed by another set of functionality. Each set of functionality is a TPL task. So, we have a producer task and a consumer task, with a buffer between them. Each of these tasks can run independently of each other. We can also have multiple producer tasks and multiple consumer tasks. The producers run independently and produce queue results to the buffer. The consumers run independently and dequeue from the buffer and perform work on the item. The producer can block if the buffer is full and wait for room to become available before producing more results. Also, the consumer can block if the buffer is empty, waiting on more results to be available to consume. In this article, you will learn the following: Designing an application with a Pipeline design Designing an application with a producer-consumer design Learning how to use BlockingCollection Learning how to use BufferedBlocks Understanding the classes of the System.Threading.Tasks.Dataflow library (For more resources related to this topic, see here.) Pipeline design pattern The Pipeline design is very useful in parallel design when you can divide an application up into series of tasks to be performed in such a way that each task can run concurrently with other tasks. It is important that the output of each task is in the same order as the input. If the order does not matter, then a parallel loop can be performed. When the order matters and we don't want to wait until all items have completed task A before the items start executing task B, then a Pipeline implementation is perfect. Some applications that lend themselves to pipelining are video streaming, compression, and encryption. In each of these examples, we need to perform a set of tasks on the data and preserve the data's order, but we do not want to wait for each item of data to perform a task before any of the data can perform the next task. The key class that .NET has provided for implementing this design pattern is BlockingCollection of the System.Collections.Concurrent namespace. The BlockingCollection class was introduced with .NET 4.5. It is a thread-safe collection specifically designed for producer-consumer and Pipeline design patterns. It supports concurrently adding and removing items by multiple threads to and from the collection. It also has methods to add and remove that block when the collection is full or empty. You can specify a maximum collection size to ensure a producing task that outpaces a consuming task does not make the queue too large. It supports cancellation tokens. Finally, it supports enumerations so that you can use the foreach loop when processing items of the collection. A producer of items to the collection can call the CompleteAdding method when the last item of data has been added to the collection. Until this method is called if a consumer is consuming items from the collection with a foreach loop and the collection is empty, it will block until an item is put into the collection instead of ending the loop. Next, we will see a simple example of a Pipeline design implementation using an encryption program. This program will implement three stages in our pipeline. The first stage will read a text file character-by-character and place each character into a buffer (BlockingCollection). The next stage will read each character out of the buffer and encrypt it by adding 1 to its ASCII number. It will then place the new character into our second buffer and write it to an encryption file. Our final stage will read the character out of the second buffer, decrypt it to its original character, and write it out to a new file and to the screen. As you will see, stages 2 and 3 will start processing characters before stage 1 has finished reading all the characters from the input file. And all of this will be done while maintaining the order of the characters so that the final output file is identical to the input file: Let's get started. How to do it First, let's open up Visual Studio and create a new Windows Presentation Foundation (WPF) application named PipeLineApplication and perform the following steps: Create a new class called Stages.cs. Next, make sure it has the following using statements. using System; using System.Collections.Concurrent; using System.Collections.Generic; using System.IO; using System.Linq; using System.Text; using System.Threading.Tasks; using System.Threading; In the MainWindow.xaml.cs file, make sure the following using statements are present: using System; using System.Collections.Concurrent; using System.Collections.Generic; using System.IO; using System.Linq; using System.Text; using System.Threading.Tasks; using System.Threading; Next, we will add a method for each of the three stages in our pipeline. First, we will create a method called FirstStage. It will take two parameters: one will be a BlockingCollection object that will be the output buffer of this stage, and the second will be a string pointing to the input data file. This will be a text file containing a couple of paragraphs of text to be encrypted. We will place this text file in the projects folder on C:. The FirstStage method will have the following code: public void FirstStage(BlockingCollection<char> output, String PipelineInputFile)        {            String DisplayData = "";            try            {                foreach (char C in GetData(PipelineInputFile))                { //Displayed characters read in from the file.                   DisplayData = DisplayData + C.ToString();   // Add each character to the buffer for the next stage.                    output.Add(C);                  }            }            finally            {                output.CompleteAdding();             }      } Next, we will add a method for the second stage called StageWorker. This method will not return any values and will take three parameters. One will be a BlockingCollection value that will be its input buffer, the second one will be the output buffer of the stage, and the final one will be a file path to store the encrypted text in a data file. The code for this method will look like this: public void StageWorker(BlockingCollection<char> input, BlockingCollection<char> output, String PipelineEncryptFile)        {            String DisplayData = "";              try            {                foreach (char C in input.GetConsumingEnumerable())                {                    //Encrypt each character.                    char encrypted = Encrypt(C);                      DisplayData = DisplayData + encrypted.ToString();   //Add characters to the buffer for the next stage.                    output.Add(encrypted);                  }   //write the encrypted string to the output file.                 using (StreamWriter outfile =                            new StreamWriter(PipelineEncryptFile))                {                    outfile.Write(DisplayData);                }              }            finally            {                output.CompleteAdding();            }        } Now, we will add a method for the third and final stage of the Pipeline design. This method will be named FinalStage. It will not return any values and will take two parameters. One will be a BlockingCollection object that is the input buffer and the other will be a string pointing to an output data file. It will have the following code in it: public void FinalStage(BlockingCollection<char> input, String PipelineResultsFile)        {            String OutputString = "";            String DisplayData = "";              //Read the encrypted characters from the buffer, decrypt them, and display them.            foreach (char C in input.GetConsumingEnumerable())            {                //Decrypt the data.                char decrypted = Decrypt(C);                  //Display the decrypted data.                DisplayData = DisplayData + decrypted.ToString();                  //Add to the output string.                OutputString += decrypted.ToString();              }              //write the decrypted string to the output file.            using (StreamWriter outfile =                        new StreamWriter(PipelineResultsFile))            {                outfile.Write(OutputString);            }        } Now that we have methods for the three stages of our pipeline, let's add a few utility methods. The first of these methods will be one that reads in the input data file and places each character in the data file in a List object. This method will take a string parameter that has a filename and will return a List object of characters. It will have the following code: public List<char> GetData(String PipelineInputFile)        {            List<char> Data = new List<char>();              //Get the Source data.            using (StreamReader inputfile = new StreamReader(PipelineInputFile))            {                while (inputfile.Peek() >= 0)                {                    Data.Add((char)inputfile.Read());                }              }              return Data;        } Now we will need a method to encrypt the characters. This will be a simple encryption method. The encryption method is not really important to this exercise. This exercise is designed to demonstrate the Pipeline design, not implement the world's toughest encryption. This encryption will simply take each character and add one to its ASCII numerical value. The method will take a character type as an input parameter and return a character. The code for it will be as follows: public char Encrypt(char C)        {            //Take the character, convert to an int, add 1, then convert back to a character.            int i = (int)C;            i = i + 1;            C = Convert.ToChar(i);              return C; } Now we will add one final method to the Stages class to decrypt a character value. It will simply do the reverse of the encrypt method. It will take the ASCII numerical value and subtract 1. The code for this method will look like this: public char Decrypt(char C)      {            int i = (int)C;            i = i - 1;            C = Convert.ToChar(i);              return C;        } Now that we are done with the Stages class, let's switch our focus back to the MainWindow.xaml.cs file. First, you will need to add three using statements. They are for the StreamReader, StreamWriter, Threads, and BlockingCollection classes: using System.Collections.Concurrent; using System.IO; using System.Threading; At the top of the MainWindow class, we need four variables available for the whole class. We need three strings that point to our three data files—the input data, encrypted data, and output data. Then we will need a Stages object. These declarations will look like this: private static String PipelineResultsFile = @"c:projectsOutputData.txt";        private static String PipelineEncryptFile = @"c:projectsEncryptData.txt";        private static String PipelineInputFile = @"c:projectsInputData.txt";        private Stages Stage; Then, in the MainWindow constructor method, right after the InitializeComponent call, add a line to instantiate our Stages object: //Create the Stage object and register the event listeners to update the UI as the stages work. Stage = new Stages(); Next, add a button to the MainWindow.xaml file that will initiate the pipeline and encryption. Name this button control butEncrypt, and set its Content property to Encrypt File. Next, add a click event handler for this button in the MainWindow.xaml.cs file. Its event handler method will be butEncrypt_Click and will contain the main code for this application. It will instantiate two BlockingCollection objects for two queues. One queue between stages 1 and 2, and one queue between stages 2 and 3. This method will then create a task for each stage that executes the corresponding methods from the Stages classes. It will then start these three tasks and wait for them to complete. Finally, it will write the output of each stage to the input, encrypted, and results data files and text blocks for viewing. The code for it will look like the following code: private void butEncrpt_Click(object sender, RoutedEventArgs e)        {            //PipeLine Design Pattern              //Create queues for input and output to stages.            int size = 20;            BlockingCollection<char> Buffer1 = new BlockingCollection<char>(size);            BlockingCollection<char> Buffer2 = new BlockingCollection<char>(size);              TaskFactory tasks = new TaskFactory(TaskCreationOptions.LongRunning, TaskContinuationOptions.None);              Task Stage1 = tasks.StartNew(() => Stage.FirstStage(Buffer1, PipelineInputFile));            Task Stage2 = tasks.StartNew(() => Stage.StageWorker(Buffer1, Buffer2, PipelineEncryptFile));            Task Stage3 = tasks.StartNew(() => Stage.FinalStage(Buffer2, PipelineResultsFile));              Task.WaitAll(Stage1, Stage2, Stage3);              //Display the 3 files.            using (StreamReader inputfile = new StreamReader(PipelineInputFile))            {                while (inputfile.Peek() >= 0)                {                    tbStage1.Text = tbStage1.Text + (char)inputfile.Read();                }              }            using (StreamReader inputfile = new StreamReader(PipelineEncryptFile))            {                 while (inputfile.Peek() >= 0)                {                    tbStage2.Text = tbStage2.Text + (char)inputfile.Read();                }              }            using (StreamReader inputfile = new StreamReader(PipelineResultsFile))             {                while (inputfile.Peek() >= 0)                {                    tbStage3.Text = tbStage3.Text + (char)inputfile.Read();                }              }      } One last thing. Let's add three textblocks to display the outputs. We will call these tbStage1, tbStage2, and tbStage3. We will also add three label controls with the text Input File, Encrypted File, and Output File. These will be placed by the corresponding textblocks. Now, the MainWindow.xaml file should look like the following screenshot: Now we will need an input data file to encrypt. We will call this file InputData.txt and put it in the C:projects folder on our computer. For our example, we have added the following text to it: We are all finished and ready to try it out. Compile and run the application and you should have a window that looks like the following screenshot: Now, click on the Encrypt File button and you should see the following output: As you can see, the input and output files look the same and the encrypted file looks different. Remember that Input File is the text we put in the input data text file; this is the input from the end of stage 1 after we have read the file in to a character list. Encrypted File is the output from stage 2 after we have encrypted each character. Output File is the output of stage 3 after we have decrypted the characters again. It should match Input File. Now, let's take a look at how this works. How it works Let's look at the butEncrypt click event handler method in the MainWindow.xaml.cs file, as this is where a lot of the action takes place. Let's examine the following lines of code:            //Create queues for input and output to stages.            int size = 20;            BlockingCollection<char> Buffer1 = new BlockingCollection<char>(size);            BlockingCollection<char> Buffer2 = new BlockingCollection<char>(size);            TaskFactory tasks = new TaskFactory(TaskCreationOptions.LongRunning, TaskContinuationOptions.None);              Task Stage1 = tasks.StartNew(() => Stage.FirstStage(Buffer1, PipelineInputFile));            Task Stage2 = tasks.StartNew(() => Stage.StageWorker(Buffer1, Buffer2, PipelineEncryptFile));            Task Stage3 = tasks.StartNew(() => Stage.FinalStage(Buffer2, PipelineResultsFile)); First, we create two queues that are implemented using BlockingCollection objects. Each of these is set with a size of 20 items. These two queues take a character datatype. Then we create a TaskFactory object and use it to start three tasks. Each task uses a lambda expression that executes one of the stages methods from the Stages class—FirstStage, StageWorker, and FinalStage. So, now we have three separate tasks running besides the main UI thread. Stage1 will read the input data file character by character and place each character in the queue Buffer1. Remember that this queue can only hold 20 items before it will block the FirstStage method waiting on room in the queue. This is how we know that Stage2 starts running before Stage1 completes. Otherwise, Stage1 will only queue the first 20 characters and then block. Once Stage1 has read all of the characters from the input file and placed them into Buffer1, it then makes the following call:            finally            {                output.CompleteAdding();            } This lets the BlockingCollection instance, Buffer1, to know that there are no more items to be put in the queue. So, when Stage2 has emptied the queue after Stage1 has called this method, it will not block but will instead continue until completion. Prior to the CompleteAdding method call, Stage2 will block if Buffer1 is empty, waiting until more items are placed in the queue. This is why a BlockingCollection instance was developed for Pipeline and producer-consumer applications. It provides the perfect mechanism for this functionality. When we created the TaskFactory, we used the following parameter: TaskCreationOptions.LongRunning This tells the threadpool that these tasks may run for a long time and could occasionally block waiting on their queues. In this way, the threadpool can decide how to best manage the threads allocated for these tasks. Now, let's look at the code in Stage2—the StageWorker method. We need a way to remove items in an enumerable way so that we can iterate over the queues items with a foreach loop because we do not know how many items to expect. Also, since BlockingCollection objects support multiple consumers, we need a way to remove items that no other consumer might remove. We use this method of the BlockingCollection class: foreach (char C in input.GetConsumingEnumerable()) This allows multiple consumers to remove items from a BlockingCollection instance while maintaining the order of the items. To further improve performance of this application (assuming we have enough available processing cores), we could create a fourth task that also runs the StageWorker method. So, then we would have two stages and two tasks running. This might be helpful if there are enough processing cores and stage 1 runs faster than stage 2. If this happens, it will continually fill the queue and block until space becomes available. But if we run multiple stage 2 tasks, then we will be able to keep up with stage 1. Then, finally we have this line of code: Task.WaitAll(Stage1, Stage2, Stage3); This tells our button handler to wait until all of the tasks are complete. Once we have called the CompleteAdding method on each BlockingCollection instance and the buffers are then emptied, all of our stages will complete and the TaskFactory.WaitAll command will be satisfied and this method on the UI thread can complete its processing, which in this application is to update the UI and data files:            //Display the 3 files.            using (StreamReader inputfile = new StreamReader(PipelineInputFile))            {                while (inputfile.Peek() >= 0)                {                    tbStage1.Text = tbStage1.Text + (char)inputfile.Read();                }              }            using (StreamReader inputfile = new StreamReader(PipelineEncryptFile))            {                while (inputfile.Peek() >= 0)                {                    tbStage2.Text = tbStage2.Text + (char)inputfile.Read();                }              }            using (StreamReader inputfile = new StreamReader(PipelineResultsFile))            {                while (inputfile.Peek() >= 0)                {                    tbStage3.Text = tbStage3.Text + (char)inputfile.Read();                }              } Next, experiment with longer running, more complex stages and multiple consumer stages. Also, try stepping through the application with the Visual Studio debugger. Make sure you understand the interaction between the stages and the buffers. Explaining message blocks Let's talk for a minute about message blocks and the TPL. There is a new library that Microsoft has developed as part of the TPL, but it does not ship directly with .NET 4.5. This library is called the TPL Dataflow library. It is located in the System.Threading.Tasks.Dataflow namespace. It comes with various dataflow components that assist in asynchronous concurrent applications where messages need to be passed between multiple tasks or the data needs to be passed when it becomes available, as in the case of a web camera streaming video. The Dataflow library's message blocks are very helpful for design patterns such as Pipeline and producer-consumer where you have multiple producers producing data that can be consumed by multiple consumers. The two that we will take a look at are BufferBlock and ActionBlock. The TPL Dataflow library contains classes to assist in message passing and parallelizing I/O-heavy applications that have a lot of throughput. It provides explicit control over how data is buffered and passed. Consider an application that asynchronously loads large binary files from storage and manipulates that data. Traditional programming requires that you use callbacks and synchronization classes, such as locks, to coordinate tasks and have access to data that is shared. By using the TPL Dataflow objects, you can create objects that process image files as they are read in from a disk location. You can set how data is handled when it becomes available. Because the CLR runtime engine manages dependencies between data, you do not have to worry about synchronizing access to shared data. Also, since the CLR engine schedules the work depending on the asynchronous arrival of data, the TPL Dataflow objects can improve performance by managing the threads the tasks run on. In this section, we will cover two of these classes, BufferBlock and ActionBlock. The TPL Dataflow library (System.Threading.Tasks.Dataflow) does not ship with .NET 4.5. To install System.Threading.Tasks.Dataflow, open your project in Visual Studio, select Manage NuGet Packages from under the Project menu and then search online for Microsoft.Tpl.Dataflow. BufferBlock The BufferBlock object in the Dataflow library provides a buffer to store data. The syntax is, BufferBlock<T>. The T indicates that the datatype is generic and can be of any type. All static variables of this object type are guaranteed to be thread-safe. BufferBlock is an asynchronous message structure that stores messages in a first-in-first-out queue. Messages can be "posted" to the queue by multiple producers and "received" from the queue by multiple consumers. The TPL DatafLow library provides interfaces for three types of objects—source blocks, target blocks, and propagator blocks. BufferBlock is a general-purpose message block that can act as both a source and a target message buffer, which makes it perfect for a producer-consumer application design. To act as both a source and a target, it implements two interfaces defined by the TPL Dataflow library—ISourceBlock<TOutput> and ITargetBlock<TOutput>. So, in the application that we will develop in the Producer-consumer design pattern section of this article, you will see that the producer method implements BufferBlock using the ITargetBlock interface and the consumer implements BufferBlock with the ISourceBlock interface. This will be the same BufferBlock object that they will act on but by defining their local objects with a different interface there will be different methods available to use. The producer method will have Post and Complete methods, and the consumer method will use the OutputAvailableAsync and Receive methods. The BufferBlock object only has two properties, namely Count, which is a count of the number of data messages in the queue, and Completion, which gets a task that is an asynchronous operation and completion of the message block. The following is a set of methods for this class: Referenced from http://msdn.microsoft.com/en-us/library/hh160414(v=vs.110).aspx Here is a list of the extension methods provided by the interfaces that it implements: Referenced from http://msdn.microsoft.com/en-us/library/hh160414(v=vs.110).aspx Finally, here are the interface references for this class: Referenced from http://msdn.microsoft.com/en-us/library/hh160414(v=vs.110).aspx So, as you can see, these interfaces make using the BufferBlock object as a general-purpose queue between stages of a pipeline very easy. This technique is also useful between producers and consumers in a producer-consumer design pattern. ActionBlock Another very useful object in the Dataflow library is ActionBlock. Its syntax is ActionBlock<TInput>, where TInput is an Action object. ActionBlock is a target block that executes a delegate when a message of data is received. The following is a very simple example of using an ActionBlock:            ActionBlock<int> action = new ActionBlock<int>(x => Console.WriteLine(x));              action.Post(10); In this sample piece of code, the ActionBlock object is created with an integer parameter and executes a simple lambda expression that does a Console.WriteLine when a message of data is posted to the buffer. So, when the action.Post(10) command is executed, the integer, 10, is posted to the ActionBlock buffer and then the ActionBlock delegate, implemented as a lambda expression in this case, is executed. In this example, since this is a target block, we would then need to call the Complete method to ensure the message block is completed. Another handy method of the BufferBlock is the LinkTo method. This method allows you to link ISourceBlock to ITargetBlock. So, you can have a BufferBlock that is implemented as an ISourceBlock and link it to an ActionBlock since it is an ITargetBlock. In this way, an Action delegate can be executed when a BufferBlock receives data. This does not dequeue the data from the message block. It just allows you to execute some task when data is received into the buffer. ActionBlock only has two properties, namely InputCount, which is a count of the number of data messages in the queue, and Completion, which gets a task that is an asynchronous operation and completion of the message block. It has the following methods: Referenced from http://msdn.microsoft.com/en-us/library/hh194684(v=vs.110).aspx The following extension methods are implemented from its interfaces: Referenced from http://msdn.microsoft.com/en-us/library/hh194684(v=vs.110).aspx Also, it implements the following interfaces: Referenced from http://msdn.microsoft.com/en-us/library/hh194684(v=vs.110).aspx Now that we have examined a little of the Dataflow library that Microsoft has developed, let's use it in a producer-consumer application. Producer-consumer design pattern Now, that we have covered the TPL's Dataflow library and the set of objects it provides to assist in asynchronous message passing between concurrent tasks, let's take a look at the producer-consumer design pattern. In a typical producer-consumer design, we have one or more producers putting data into a queue or message data block. Then we have one or more consumers taking data from the queue and processing it. This allows for asynchronous processing of data. Using the Dataflow library objects, we can create a consumer task that monitors a BufferBlock and pulls items of the data from it when they arrive. If no items are available, the consumer method will block until items are available or the BufferBlock has been set to Complete. Because of this, we can start our consumer at any time, even before the producer starts to put items into the queue. Then we create one or more tasks that produce items and place them into the BufferBlock. Once the producers are finished processing all items of data to the BufferBlock, they can mark the block as Complete. Until then, the BufferBlock object is still available to add items into. This is perfect for long-running tasks and applications when we do not know when the data will arrive. Because the producer task is implementing an input parameter of a BufferBlock as an ITargetBlock object and the consumer task is implementing an input parameter of a BufferBlock as an ISourceBlock, they can both use the same BufferBlock object but have different methods available to them. One has methods to produces items to the block and mark it complete. The other one has methods to receive items and wait for more items until the block is marked complete. In this way, the Dataflow library implements the perfect object to act as a queue between our producers and consumers. Now, let's take a look at the application we developed previously as a Pipeline design and modify it using the Dataflow library. We will also remove a stage so that it just has two stages, one producer and one consumer. How to do it The first thing we need to do is open Visual Studio and create a new console application called ProducerConsumerConsoleApp. We will use a console application this time just for ease. Our main purpose here is to demonstrate how to implement the producer-consumer design pattern using the TPL Dataflow library. Once you have opened Visual Studio and created the project, we need to perform the following steps: First, we need to install and add a reference to the TPL Dataflow library. The TPL Dataflow library (System.Threading.Tasks.Dataflow) does not ship with .NET 4.5. Select Manage NuGet Packages from under the Project menu and then search online for Microsoft.Tpl.Dataflow. Now, we will need to add two using statements to our program. One for StreamReader and StreamWriter and one for the BufferBlock object: using System.Threading.Tasks.Dataflow; using System.IO; Now, let's add two static strings that will point to our input data file and the encrypted data file that we output: private static String PipelineEncryptFile = @"c:projectsEncryptData.txt";        private static String PipelineInputFile = @"c:projectsInputData.txt"; Next, let's add a static method that will act as our producer. This method will have the following code:        // Our Producer method.        static void Producer(ITargetBlock<char> Target)        {            String DisplayData = "";              try            {                foreach (char C in GetData(PipelineInputFile))                {                      //Displayed characters read in from the file.                    DisplayData = DisplayData + C.ToString();                      // Add each character to the buffer for the next stage.                    Target.Post(C);                  }            }              finally            {                Target.Complete();            }          } Then we will add a static method to perform our consumer functionality. It will have the following code:        // This is our consumer method. IT runs asynchronously.        static async Task<int> Consumer(ISourceBlock<char> Source)        {            String DisplayData = "";              // Read from the source buffer until the source buffer has no            // available output data.            while (await Source.OutputAvailableAsync())            {                    char C = Source.Receive();                      //Encrypt each character.                    char encrypted = Encrypt(C);                      DisplayData = DisplayData + encrypted.ToString();              }              //write the decrypted string to the output file.            using (StreamWriter outfile =                         new StreamWriter(PipelineEncryptFile))            {                outfile.Write(DisplayData);            }              return DisplayData.Length;        } Then, let's create a simple static helper method to read our input data file and put it in a List collection character by character. This will give us a character list for our producer to use. The code in this method will look like this:        public static List<char> GetData(String PipelineInputFile)        {            List<char> Data = new List<char>();              //Get the Source data.            using (StreamReader inputfile = new StreamReader(PipelineInputFile))            {                while (inputfile.Peek() >= 0)                {                    Data.Add((char)inputfile.Read());                }              }              return Data;        } Next, we will add a static method to encrypt our characters. This method will work like the one we used in our pipelining application. It will add one to the ASCII numerical value of the character:        public static char Encrypt(char C)        {            //Take the character, convert to an int, add 1, then convert back to a character.            int i = (int)C;            i = i + 1;            C = Convert.ToChar(i);              return C;        } Then, we need to add the code for our Then, we need to add the code for our Main method. This method will start our consumer and producer tasks. Then, when they have completed processing, it will display the results in the console. The code for this method looks like this:        static void Main(string[] args)        {            // Create the buffer block object to use between the producer and consumer.            BufferBlock<char> buffer = new BufferBlock<char>();              // The consumer method runs asynchronously. Start it now.            Task<int> consumer = Consumer(buffer);              // Post source data to the dataflow block.            Producer(buffer);              // Wait for the consumer to process all data.            consumer.Wait();              // Print the count of characters from the input file.            Console.WriteLine("Processed {0} bytes from input file.", consumer.Result);              //Print out the input file to the console.            Console.WriteLine("rnrn");            Console.WriteLine("This is the input data file. rn");            using (StreamReader inputfile = new StreamReader(PipelineInputFile))            {                while (inputfile.Peek() >= 0)                {                    Console.Write((char)inputfile.Read());                }              }              //Print out the encrypted file to the console.            Console.WriteLine("rnrn");            Console.WriteLine("This is the encrypted data file. rn");            using (StreamReader encryptfile = new StreamReader(PipelineEncryptFile))            {                while (encryptfile.Peek() >= 0)                {                    Console.Write((char)encryptfile.Read());                }              }             //Wait before closing the application so we can see the results.            Console.ReadLine();        } That is all the code that is needed. Now, let's build and run the application using the following input data file: Once it runs and completes, your output should look like the following screenshot: Now, try this with your own data files and inputs. Let's examine what happened and how this works. How it works First we will go through the Main method. The first thing Main does is create a BufferBlock object called buffer. This will be used as the queue of items between our producer and consumer. This BufferBlock is defined to accept character datatypes. Next, we start our consumer task using this command: Task<int> consumer = Consumer(buffer); Also, note that when this buffer object goes into the consumer task, it is cast as ISourceBlock. Notice the method header of our consumer: static async Task<int> Consumer(ISourceBlock<char> Source) Next, our Main method starts our producer task using the following command: Producer(buffer); Then we wait until our consumer task finishes, using this command: consumer.Wait(); So, now our Main method just waits. Its work is done for now. It has started both the producer and consumer tasks. Now our consumer is waiting for items to appear in its BufferBlock so it can process them. The consumer will stay in the following loop until all items are removed from the message block and the block has been completed, which is done by someone calling its Complete method:      while (await Source.OutputAvailableAsync())            {                    char C = Source.Receive();                      //Encrypt each character.                    char encrypted = Encrypt(C);                      DisplayData = DisplayData + encrypted.ToString();              } So, now our consumer task will loop asynchronously, removing items from the message queue as they appear. It uses the following command in the while loop to do this: await Source.OutputAvailableAsync()) Likewise, other consumer tasks can run at the same time and do the same thing. If the producer is adding items to the block quicker than the consumer can process them, then adding another consumer will improve performance. Once an item is available, then the consumer calls the following command to get the item from the buffer: char C = Source.Receive(); Since the buffer contains items of type character, we place the item received into a character value. Then the consumer processes it by encrypting the character and appending it to our display string: Now, let's look at the consumer. The consumer first gets its data by calling the following command: GetData(PipelineInputFile) This method returns a List collection of characters that has an item for each character in the input data file. Now the producer iterates through the collection and uses the following command to place each item into the buffer block: Target.Post(C); Also, notice in the method header for our consumer that we cast our buffer as an ITargetBlock type: static void Producer(ITargetBlock<char> Target) Once the producer is done processing characters and adding them to the buffer, it officially closes the BufferBlock object using this command: Target.Complete(); That is it for the producer and consumer. Once the Main method is done waiting on the consumer to finish, it then uses the following code to write out the number of characters processed, the input data, and the encrypted data:      // Print the count of characters from the input file.            Console.WriteLine("Processed {0} bytes from input file.", consumer.Result);              //Print out the input file to the console.            Console.WriteLine("rnrn");            Console.WriteLine("This is the input data file. rn");            using (StreamReader inputfile = new StreamReader(PipelineInputFile))            {                while (inputfile.Peek() >= 0)                {                    Console.Write((char)inputfile.Read());                }              }              //Print out the encrypted file to the console.            Console.WriteLine("rnrn");            Console.WriteLine("This is the encrypted data file. rn");            using (StreamReader encryptfile = new StreamReader(PipelineEncryptFile))            {                while (encryptfile.Peek() >= 0)                {                    Console.Write((char)encryptfile.Read());                }              } Now that you are comfortable implementing a basic producer-consumer design using objects from the TPL Dataflow library, try experimenting with this basic idea but use multiple producers and multiple consumers all with the same BufferBlock object as the queue between them all. Also, try converting our original Pipeline application from the beginning of the article into a TPL Dataflow producer-consumer application with two sets of producers and consumers. The first will act as stage 1 and stage 2, and the second will act as stage 2 and stage 3. So, in effect, stage 2 will be both a consumer and a producer. Summary We have covered a lot in this article. We have learned the benefits and how to implement a Pipeline design pattern and a producer-consumer design pattern. As we saw, these are both very helpful design patterns when building parallel and concurrent applications that require multiple asynchronous processes of data between tasks. In the Pipeline design, we are able to run multiple tasks or stages concurrently even though the stages rely on data being processed and output by other stages. This is very helpful for performance since all functionality doesn't have to wait on each stage to finish processing every item of data. In our example, we are able to start decrypting characters of data while a previous stage is still encrypting data and placing it into the queue. In the Pipeline example, we examined the benefits of the BlockingCollection class in acting as a queue between stages in our pipeline. Next, we explored the new TPL Dataflow library and some of its message block classes. These classes implement several interfaces defined in the library—ISourceBlock, ITargetBlock, and IPropogatorBlock. By implementing these interfaces, it allows us to write generic producer and consumer task functionality that can be reused in a variety of applications. Both of these design patterns and the Dataflow library allow for easy implementations of common functionality in a concurrent manner. You will use these techniques in many applications, and this will become a go-to design pattern when you evaluate a system's requirements and determine how to implement concurrency to help improve performance. Like all programming, parallel programming is made easier when you have a toolbox of easy-to-use techniques that you are comfortable with. Most applications that benefit from parallelism will be conducive to some variation of a producer-consumer or Pipeline pattern. Also, the BlockingCollection and Dataflow message block objects are useful mechanisms for coordinating data between parallel tasks, no matter what design pattern is used in the application. It will be very useful to become comfortable with these messaging and queuing classes. Resources for Article: Further resources on this subject: Parallel Programming Patterns [article] Watching Multiple Threads in C# [article] Clusters, Parallel Computing, and Raspberry Pi – A Brief Background [article]

In this article is written by Mark Kerzner and Sujee Maniyam, the authors of HBase Design Patterns, we will talk about how to write high performance and scalable HBase applications. In particular, will take a look at the following topics: The bulk loading of data into HBase Profiling HBase applications Tips to get good performance on writes Tips to get good performance on reads (For more resources related to this topic, see here.) Loading bulk data into HBase When deploying HBase for the first time, we usually need to import a significant amount of data. This is called initial loading or bootstrapping. There are three methods that can be used to import data into HBase, given as follows: Using the Java API to insert data into HBase. This can be done in a single client, using single or multiple threads. Using MapReduce to insert data in parallel (this approach also uses the Java API), as shown in the following diagram:  Using MapReduce to generate HBase store files in parallel in bulk and then import them into HBase directly. (This approach does not require the use of the API; it does not require code and is very efficient.)  On comparing the three methods speed wise, we have the following order: Java client < MapReduce insert < HBase file import The Java client and MapReduce use HBase APIs to insert data. MapReduce runs on multiple machines and can exploit parallelism. However, both of these methods go through the write path in HBase. Importing HBase files directly, however, skips the usual write path. HBase files already have data in the correct format that HBase understands. That's why importing them is much faster than using MapReduce and the Java client. We covered the Java API earlier. Let's start with how to insert data using MapReduce. Importing data into HBase using MapReduce MapReduce is the distributed processing engine of Hadoop. Usually, programs read/write data from HDFS. Luckily, HBase supports MapReduce. HBase can be the source and the sink for MapReduce programs. A source means MapReduce programs can read from HBase, and sink means results from MapReduce can be sent to HBase. The following diagram illustrates various sources and sinks for MapReduce:     The diagram we just saw can be summarized as follows: Scenario Source Sink Description 1 HDFS HDFS This is a typical MapReduce method that reads data from HDFS and also sends the results to HDFS. 2 HDFS HBase This imports the data from HDFS into HBase. It's a very common method that is used to import data into HBase for the first time. 3 HBase HBase Data is read from HBase and written to it. It is most likely that these will be two separate HBase clusters. It's usually used for backups and mirroring.  Importing data from HDFS into HBase Let's say we have lots of data in HDFS and want to import it into HBase. We are going to write a MapReduce program that reads from HDFS and inserts data into HBase. This is depicted in the second scenario in the table we just saw. Now, we'll be setting up the environment for the following discussion. In addition, you can find the code and the data for this discussion in our GitHub repository at https://github.com/elephantscale/hbase-book. The dataset we will use is the sensor data. Our (imaginary) sensor data is stored in HDFS as CSV (comma-separated values) text files. This is how their format looks: Sensor_id, max temperature, min temperature Here is some sample data: sensor11,90,70 sensor22,80,70 sensor31,85,72 sensor33,75,72 We have two sample files (sensor-data1.csv and sensor-data2.csv) in our repository under the /data directory. Feel free to inspect them. The first thing we have to do is copy these files into HDFS. Create a directory in HDFS as follows: $   hdfs   dfs -mkdir   hbase-import Now, copy the files into HDFS: $   hdfs   dfs   -put   sensor-data*   hbase-import/ Verify that the files exist as follows: $   hdfs   dfs -ls   hbase-import We are ready to insert this data into HBase. Note that we are designing the table to match the CSV files we are loading for ease of use. Our row key is sensor_id. We have one column family and we call it f (short for family). Now, we will store two columns, max temperature and min temperature, in this column family. Pig for MapReduce Pig allows you to write MapReduce programs at a very high level, and inserting data into HBase is just as easy. Here's a Pig script that reads the sensor data from HDFS and writes it in HBase: -- ## hdfs-to-hbase.pigdata = LOAD 'hbase-import/' using PigStorage(',') as (sensor_id:chararray, max:int, min:int);-- describe data;-- dump data; Now, store the data in hbase://sensors using the following line of code: org.apache.pig.backend.hadoop.hbase.HBaseStorage('f:max,f:min'); After creating the table, in the first command, we will load data from the hbase-import directory in HDFS. The schema for the data is defined as follows: Sensor_id : chararray (string)max : intmin : int The describe and dump statements can be used to inspect the data; in Pig, describe will give you the structure of the data object you have, and dump will output all the data to the terminal. The final STORE command is the one that inserts the data into HBase. Let's analyze how it is structured: INTO 'hbase://sensors': This tells Pig to connect to the sensors HBase table. org.apache.pig.backend.hadoop.hbase.HBaseStorage: This is the Pig class that will be used to write in HBase. Pig has adapters for multiple data stores. The first field in the tuple, sensor_id, will be used as a row key. We are specifying the column names for the max and min fields (f:max and f:min, respectively). Note that we have to specify the column family (f:) to qualify the columns. Before running this script, we need to create an HBase table called sensors. We can do this from the HBase shell, as follows: $ hbase shell$ create 'sensors' , 'f'$ quit Then, run the Pig script as follows: $ pig hdfs-to-hbase.pig Now watch the console output. Pig will execute the script as a MapReduce job. Even though we are only importing two small files here, we can insert a fairly large amount of data by exploiting the parallelism of MapReduce. At the end of the run, Pig will print out some statistics: Input(s):Successfully read 7 records (591 bytes) from: "hdfs://quickstart.cloudera:8020/user/cloudera/hbase-import"Output(s):Successfully stored 7 records in: "hbase://sensors" Looks good! We should have seven rows in our HBase sensors table. We can inspect the table from the HBase shell with the following commands: $ hbase shell$ scan 'sensors' This is how your output might look: ROW                      COLUMN+CELL sensor11                 column=f:max, timestamp=1412373703149, value=90 sensor11                 column=f:min, timestamp=1412373703149, value=70 sensor22                 column=f:max, timestamp=1412373703177, value=80 sensor22                column=f:min, timestamp=1412373703177, value=70 sensor31                 column=f:max, timestamp=1412373703177, value=85 sensor31                 column=f:min, timestamp=1412373703177, value=72 sensor33                 column=f:max, timestamp=1412373703177, value=75 sensor33                 column=f:min, timestamp=1412373703177, value=72 sensor44                 column=f:max, timestamp=1412373703184, value=55 sensor44                 column=f:min, timestamp=1412373703184, value=42 sensor45                 column=f:max, timestamp=1412373703184, value=57 sensor45                 column=f:min, timestamp=1412373703184, value=47 sensor55                 column=f:max, timestamp=1412373703184, value=55 sensor55                 column=f:min, timestamp=1412373703184, value=427 row(s) in 0.0820 seconds There you go; you can see that seven rows have been inserted! With Pig, it was very easy. It took us just two lines of Pig script to do the import. Java MapReduce We have just demonstrated MapReduce using Pig, and you now know that Pig is a concise and high-level way to write MapReduce programs. This is demonstrated by our previous script, essentially the two lines of Pig code. However, there are situations where you do want to use the Java API, and it would make more sense to use it than using a Pig script. This can happen when you need Java to access Java libraries or do some other detailed tasks for which Pig is not a good match. For that, we have provided the Java version of the MapReduce code in our GitHub repository. Using HBase's bulk loader utility HBase is shipped with a bulk loader tool called ImportTsv that can import files from HDFS into HBase tables directly. It is very easy to use, and as a bonus, it uses MapReduce internally to process files in parallel. Perform the following steps to use ImportTsv: Stage data files into HDFS (remember that the files are processed using MapReduce). Create a table in HBase if required. Run the import. Staging data files into HDFS The first step to stage data files into HDFS has already been outlined in the previous section. The following sections explain the next two steps to stage data files. Creating an HBase table We will do this from the HBase shell. A note on regions is in order here. Regions are shards created automatically by HBase. It is the regions that are responsible for the distributed nature of HBase. However, you need to pay some attention to them in order to assure performance. If you put all the data in one region, you will cause what is called region hotspotting. What is especially nice about a bulk loader is that when creating a table, it lets you presplit the table into multiple regions. Precreating regions will allow faster imports (because the insert requests will go out to multiple region servers). Here, we are creating a single column family: $ hbase shellhbase> create 'sensors', {NAME => 'f'}, {SPLITS => ['sensor20', 'sensor40', 'sensor60']}0 row(s) in 1.3940 seconds=> Hbase::Table - sensors hbase > describe 'sensors'DESCRIPTION                                       ENABLED'sensors', {NAME => 'f', DATA_BLOCK_ENCODING => true'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE=> '0', VERSIONS => '1', COMPRESSION => 'NONE',MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}1 row(s) in 0.1140 seconds We are creating regions here. Why there are exactly four regions will be clear from the following diagram:   On inspecting the table in the HBase Master UI, we will see this. Also, you can see how Start Key and End Key, which we specified, are showing up. Run the import Ok, now it's time to insert data into HBase. To see the usage of ImportTsv, do the following: $ hbase org.apache.hadoop.hbase.mapreduce.ImportTsv This will print the usage as follows: $ hbase org.apache.hadoop.hbase.mapreduce.ImportTsv -Dimporttsv.separator=, -Dimporttsv.columns=HBASE_ROW_KEY,f:max,f:min sensors   hbase-import/ The following table explains what the parameters mean: Parameter Description -Dimporttsv.separator Here, our separator is a comma (,). The default value is tab (t). -Dimporttsv.columns=HBASE_ROW_KEY,f:max,f:min This is where we map our input files into HBase tables. The first field, sensor_id, is our key, and we use HBASE_ROW_KEY to denote that the rest we are inserting into column family f. The second field, max temp, maps to f:max. The last field, min temp, maps to f:min. sensors This is the table name. hbase-import This is the HDFS directory where the data files are located.  When we run this command, we will see that a MapReduce job is being kicked off. This is how an import is parallelized. Also, from the console output, we can see that MapReduce is importing two files as follows: [main] mapreduce.JobSubmitter: number of splits:2 While the job is running, we can inspect the progress from YARN (or the JobTracker UI). One thing that we can note is that the MapReduce job only consists of mappers. This is because we are reading a bunch of files and inserting them into HBase directly. There is nothing to aggregate. So, there is no need for reducers. After the job is done, inspect the counters and we can see this: Map-Reduce Framework Map input records=7 Map output records=7 This tells us that mappers read seven records from the files and inserted seven records into HBase. Let's also verify the data in HBase: $   hbase shellhbase >   scan 'sensors'ROW                 COLUMN+CELLsensor11           column=f:max, timestamp=1409087465345, value=90sensor11           column=f:min, timestamp=1409087465345, value=70sensor22           column=f:max, timestamp=1409087465345, value=80sensor22           column=f:min, timestamp=1409087465345, value=70sensor31           column=f:max, timestamp=1409087465345, value=85sensor31           column=f:min, timestamp=1409087465345, value=72sensor33           column=f:max, timestamp=1409087465345, value=75sensor33           column=f:min, timestamp=1409087465345, value=72sensor44            column=f:max, timestamp=1409087465345, value=55sensor44           column=f:min, timestamp=1409087465345, value=42sensor45           column=f:max, timestamp=1409087465345, value=57sensor45           column=f:min, timestamp=1409087465345, value=47sensor55           column=f:max, timestamp=1409087465345, value=55sensor55           column=f:min, timestamp=1409087465345, value=427 row(s) in 2.1180 seconds Your output might vary slightly. We can see that seven rows are inserted, confirming the MapReduce counters! Let's take another quick look at the HBase UI, which is shown here:    As you can see, the inserts go to different regions. So, on a HBase cluster with many region servers, the load will be spread across the cluster. This is because we have presplit the table into regions. Here are some questions to test your understanding. Run the same ImportTsv command again and see how many records are in the table. Do you get duplicates? Try to find the answer and explain why that is the correct answer, then check these in the GitHub repository (https://github.com/elephantscale/hbase-book). Bulk import scenarios Here are a few bulk import scenarios: Scenario Methods Notes The data is already in HDFS and needs to be imported into HBase. The two methods that can be used to do this are as follows: If the ImportTsv tool can work for you, then use it as it will save time in writing custom MapReduce code. Sometimes, you might have to write a custom MapReduce job to import (for example, complex time series data, doing data mapping, and so on). It is probably a good idea to presplit the table before a bulk import. This spreads the insert requests across the cluster and results in a higher insert rate. If you are writing a custom MapReduce job, consider using a high-level MapReduce platform such as Pig or Hive. They are much more concise to write than the Java code. The data is in another database (RDBMs/NoSQL) and you need to import it into HBase. Use a utility such as Sqoop to bring the data into HDFS and then use the tools outlined in the first scenario. Avoid writing MapReduce code that directly queries databases. Most databases cannot handle many simultaneous connections. It is best to bring the data into Hadoop (HDFS) first and then use MapReduce. Profiling HBase applications Just like any software development process, once we have our HBase application working correctly, we would want to make it faster. At times, developers get too carried away and start optimizing before the application is finalized. There is a well-known rule that premature optimization is the root of all evil. One of the sources for this rule is Scott Meyers Effective C++. We can perform some ad hoc profiling in our code by timing various function calls. Also, we can use profiling tools to pinpoint the trouble spots. Using profiling tools is highly encouraged for the following reasons: Profiling takes out the guesswork (and a good majority of developers' guesses are wrong). There is no need to modify the code. Manual profiling means that we have to go and insert the instrumentation code all over the code. Profilers work by inspecting the runtime behavior. Most profilers have a nice and intuitive UI to visualize the program flow and time flow. The authors use JProfiler. It is a pretty effective profiler. However, it is neither free nor open source. So, for the purpose of this article, we are going to show you a simple manual profiling, as follows: public class UserInsert {      static String tableName = "users";    static String familyName = "info";      public static void main(String[] args) throws Exception {        Configuration config = HBaseConfiguration.create();        // change the following to connect to remote clusters        // config.set("hbase.zookeeper.quorum", "localhost");        long t1a = System.currentTimeMillis();        HTable htable = new HTable(config, tableName);        long t1b = System.currentTimeMillis();        System.out.println ("Connected to HTable in : " + (t1b-t1a) + " ms");        int total = 100;        long t2a = System.currentTimeMillis();        for (int i = 0; i < total; i++) {            int userid = i;            String email = "user-" + i + "@foo.com";            String phone = "555-1234";              byte[] key = Bytes.toBytes(userid);            Put put = new Put(key);              put.add(Bytes.toBytes(familyName), Bytes.toBytes("email"), Bytes.toBytes(email));            put.add(Bytes.toBytes(familyName), Bytes.toBytes("phone"), Bytes.toBytes(phone));            htable.put(put);          }        long t2b = System.currentTimeMillis();        System.out.println("inserted " + total + " users in " + (t2b - t2a) + " ms");        htable.close();      } } The code we just saw inserts some sample user data into HBase. We are profiling two operations, that is, connection time and actual insert time. A sample run of the Java application yields the following: Connected to HTable in : 1139 msinserted 100 users in 350 ms We spent a lot of time in connecting to HBase. This makes sense. The connection process has to go to ZooKeeper first and then to HBase. So, it is an expensive operation. How can we minimize the connection cost? The answer is by using connection pooling. Luckily, for us, HBase comes with a connection pool manager. The Java class for this is HConnectionManager. It is very simple to use. Let's update our class to use HConnectionManager: Code : File name: hbase_dp.ch8.UserInsert2.java   package hbase_dp.ch8;   import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hbase.HBaseConfiguration; import org.apache.hadoop.hbase.client.HConnection; import org.apache.hadoop.hbase.client.HConnectionManager; import org.apache.hadoop.hbase.client.HTable; import org.apache.hadoop.hbase.client.HTableInterface; import org.apache.hadoop.hbase.client.Put; import org.apache.hadoop.hbase.util.Bytes;   public class UserInsert2 {      static String tableName = "users";    static String familyName = "info";      public static void main(String[] args) throws Exception {        Configuration config = HBaseConfiguration.create();        // change the following to connect to remote clusters        // config.set("hbase.zookeeper.quorum", "localhost");               long t1a = System.currentTimeMillis();        HConnection hConnection = HConnectionManager.createConnection(config);        long t1b = System.currentTimeMillis();        System.out.println ("Connection manager in : " + (t1b-t1a) + " ms");          // simulate the first 'connection'        long t2a = System.currentTimeMillis();        HTableInterface htable = hConnection.getTable(tableName) ;        long t2b = System.currentTimeMillis();        System.out.println ("first connection in : " + (t2b-t2a) + " ms");               // second connection        long t3a = System.currentTimeMillis();        HTableInterface htable2 = hConnection.getTable(tableName) ;        long t3b = System.currentTimeMillis();        System.out.println ("second connection : " + (t3b-t3a) + " ms");          int total = 100;        long t4a = System.currentTimeMillis();        for (int i = 0; i < total; i++) {            int userid = i;            String email = "user-" + i + "@foo.com";            String phone = "555-1234";              byte[] key = Bytes.toBytes(userid);            Put put = new Put(key);              put.add(Bytes.toBytes(familyName), Bytes.toBytes("email"), Bytes.toBytes(email));            put.add(Bytes.toBytes(familyName), Bytes.toBytes("phone"), Bytes.toBytes(phone));            htable.put(put);          }      long t4b = System.currentTimeMillis();        System.out.println("inserted " + total + " users in " + (t4b - t4a) + " ms");        hConnection.close();    } } A sample run yields the following timings: Connection manager in : 98 ms first connection in : 808 ms second connection : 0 ms inserted 100 users in 393 ms The first connection takes a long time, but then take a look at the time of the second connection. It is almost instant ! This is cool! If you are connecting to HBase from web applications (or interactive applications), use connection pooling. More tips for high-performing HBase writes Here we will discuss some techniques and best practices to improve writes in HBase. Batch writes Currently, in our code, each time we call htable.put (one_put), we make an RPC call to an HBase region server. This round-trip delay can be minimized if we call htable.put() with a bunch of put records. Then, with one round trip, we can insert a bunch of records into HBase. This is called batch puts. Here is an example of batch puts. Only the relevant section is shown for clarity. For the full code, see hbase_dp.ch8.UserInsert3.java:        int total = 100;        long t4a = System.currentTimeMillis();        List<Put> puts = new ArrayList<>();        for (int i = 0; i < total; i++) {            int userid = i;            String email = "user-" + i + "@foo.com";            String phone = "555-1234";              byte[] key = Bytes.toBytes(userid);            Put put = new Put(key);              put.add(Bytes.toBytes(familyName), Bytes.toBytes("email"), Bytes.toBytes(email));            put.add(Bytes.toBytes(familyName), Bytes.toBytes("phone"), Bytes.toBytes(phone));                       puts.add(put); // just add to the list        }        htable.put(puts); // do a batch put        long t4b = System.currentTimeMillis();        System.out.println("inserted " + total + " users in " + (t4b - t4a) + " ms"); A sample run with a batch put is as follows: inserted 100 users in 48 ms The same code with individual puts took around 350 milliseconds! Use batch writes when you can to minimize latency. Note that the HTableUtil class that comes with HBase implements some smart batching options for your use and enjoyment. Setting memory buffers We can control when the puts are flushed by setting the client write buffer option. Once the data in the memory exceeds this setting, it is flushed to disk. The default setting is 2 M. Its purpose is to limit how much data is stored in the buffer before writing it to disk. There are two ways of setting this: In hbase-site.xml (this setting will be cluster-wide): <property>  <name>hbase.client.write.buffer</name>    <value>8388608</value>   <!-- 8 M --></property> In the application (only applies for that application): htable.setWriteBufferSize(1024*1024*10); // 10 Keep in mind that a bigger buffer takes more memory on both the client side and the server side. As a practical guideline, estimate how much memory you can dedicate to the client and put the rest of the load on the cluster. Turning off autofush If autoflush is enabled, each htable.put() object incurs a round trip RPC call to HRegionServer. Turning autoflush off can reduce the number of round trips and decrease latency. To turn it off, use this code: htable.setAutoFlush(false); The risk of turning off autoflush is if the client crashes before the data is sent to HBase, it will result in a data loss. Still, when will you want to do it? The answer is: when the danger of data loss is not important and speed is paramount. Also, see the batch write recommendations we saw previously. Turning off WAL Before we discuss this, we need to emphasize that the write-ahead log (WAL) is there to prevent data loss in the case of server crashes. By turning it off, we are bypassing this protection. Be very careful when choosing this. Bulk loading is one of the cases where turning off WAL might make sense. To turn off WAL, set it for each put: put.setDurability(Durability.SKIP_WAL); More tips for high-performing HBase reads So far, we looked at tips to write data into HBase. Now, let's take a look at some tips to read data faster. The scan cache When reading a large number of rows, it is better to set scan caching to a high number (in the 100 seconds or 1,000 seconds range). Otherwise, each row that is scanned will result in a trip to HRegionServer. This is especially encouraged for MapReduce jobs as they will likely consume a lot of rows sequentially. To set scan caching, use the following code: Scan scan = new Scan(); scan.setCaching(1000); Only read the families or columns needed When fetching a row, by default, HBase returns all the families and all the columns. If you only care about one family or a few attributes, specifying them will save needless I/O. To specify a family, use this: scan.addFamily( Bytes.toBytes("familiy1")); To specify columns, use this: scan.addColumn( Bytes.toBytes("familiy1"),   Bytes.toBytes("col1")) The block cache When scanning large rows sequentially (say in MapReduce), it is recommended that you turn off the block cache. Turning off the cache might be completely counter-intuitive. However, caches are only effective when we repeatedly access the same rows. During sequential scanning, there is no caching, and turning on the block cache will introduce a lot of churning in the cache (new data is constantly brought into the cache and old data is evicted to make room for the new data). So, we have the following points to consider: Turn off the block cache for sequential scans Turn off the block cache for random/repeated access Benchmarking or load testing HBase Benchmarking is a good way to verify HBase's setup and performance. There are a few good benchmarks available: HBase's built-in benchmark The Yahoo Cloud Serving Benchmark (YCSB) JMeter for custom workloads HBase's built-in benchmark HBase's built-in benchmark is PerformanceEvaluation. To find its usage, use this: $   hbase org.apache.hadoop.hbase.PerformanceEvaluation To perform a write benchmark, use this: $ hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred randomWrite 5 Here we are using five threads and no MapReduce. To accurately measure the throughput, we need to presplit the table that the benchmark writes to. It is TestTable. $ hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --presplit=3 randomWrite 5 Here, the table is split in three ways. It is good practice to split the table into as many regions as the number of region servers. There is a read option along with a whole host of scan options. YCSB The YCSB is a comprehensive benchmark suite that works with many systems such as Cassandra, Accumulo, and HBase. Download it from GitHub, as follows: $   git clone git://github.com/brianfrankcooper/YCSB.git Build it like this: $ mvn -DskipTests package Create an HBase table to test against: $ hbase shellhbase> create 'ycsb', 'f1' Now, copy hdfs-site.xml for your cluster into the hbase/src/main/conf/ directory and run the benchmark: $ bin/ycsb load hbase -P workloads/workloada -p columnfamily=f1 -p table=ycsb YCSB offers lots of workloads and options. Please refer to its wiki page at https://github.com/brianfrankcooper/YCSB/wiki. JMeter for custom workloads The standard benchmarks will give you an idea of your HBase cluster's performance. However, nothing can substitute measuring your own workload. We want to measure at least the insert speed or the query speed. We also want to run a stress test. So, we can measure the ceiling on how much our HBase cluster can support. We can do a simple instrumentation as we did earlier too. However, there are tools such as JMeter that can help us with load testing. Please refer to the JMeter website and check out the Hadoop or HBase plugins for JMeter. Monitoring HBase Running any distributed system involves decent monitoring. HBase is no exception. Luckily, HBase has the following capabilities: HBase exposes a lot of metrics These metrics can be directly consumed by monitoring systems such as Ganglia We can also obtain these metrics in the JSON format via the REST interface and JMX Monitoring is a big subject and we consider it as part HBase administration. So, in this section, we will give pointers to tools and utilities that allow you to monitor HBase. Ganglia Ganglia is a generic system monitor that can monitor hosts (such as CPU, disk usage, and so on). The Hadoop stack has had a pretty good integration with Ganglia for some time now. HBase and Ganglia integration is set up by modern installers from Cloudera and Hortonworks. To enable Ganglia metrics, update the hadoop-metrics.properties file in the HBase configuration directory. Here's a sample file: hbase.class=org.apache.hadoop.metrics.ganglia.GangliaContext31 hbase.period=10 hbase.servers=ganglia-server:PORT jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext31 jvm.period=10 jvm.servers=ganglia-server:PORT rpc.class=org.apache.hadoop.metrics.ganglia.GangliaContext31 rpc.period=10 rpc.servers=ganglia-server:PORT This file has to be uploaded to all the HBase servers (master servers as well as region servers). Here are some sample graphs from Ganglia (these are Wikimedia statistics, for example): These graphs show cluster-wide resource utilization. OpenTSDB OpenTSDB is a scalable time series database. It can collect and visualize metrics on a large scale. OpenTSDB uses collectors, light-weight agents that send metrics to the open TSDB server to collect metrics, and there is a collector library that can collect metrics from HBase. You can see all the collectors at http://opentsdb.net/docs/build/html/user_guide/utilities/tcollector.html. An interesting factoid is that OpenTSDB is built on Hadoop/HBase. Collecting metrics via the JMX interface HBase exposes a lot of metrics via JMX. This page can be accessed from the web dashboard at http://<hbase master>:60010/jmx. For example, for a HBase instance that is running locally, it will be http://localhost:60010/jmx. Here is a sample screenshot of the JMX metrics via the web UI: Here's a quick example of how to programmatically retrieve these metrics using curl: $ curl 'localhost:60010/jmx' Since this is a web service, we can write a script/application in any language (Java, Python, or Ruby) to retrieve and inspect the metrics. Summary In this article, you learned how to push the performance of our HBase applications up. We looked at how to effectively load a large amount of data into HBase. You also learned about benchmarking and monitoring HBase and saw tips on how to do high-performing reads/writes. Resources for Article:   Further resources on this subject: The HBase's Data Storage [article] Hadoop and HDInsight in a Heartbeat [article] Understanding the HBase Ecosystem [article]

This article is written by Aravind Shenoy and Ulrich Sossou, the authors of the book, Learning Bootstrap. It will introduce you to the concept of deep customization of Bootstrap. (For more resources related to this topic, see here.) Adding your own style sheet works when you are trying to do something quick or when the modifications are minimal. Customizing Bootstrap beyond small changes involves using the uncompiled Bootstrap source code. The Bootstrap CSS source code is written in LESS with variables and mixins to allow easy customization. LESS is an open source CSS preprocessor with cool features used to speed up your development time. LESS allows you to engage an efficient and modular style of working making it easier to maintain your CSS styling in your projects. The advantages of using variables in LESS are profound. You can reuse the same code many times thereby following the write once, use anywhere paradigm. Variables can be globally declared, which allows you to specify certain values in a single place. This needs to be updated only once if changes are required. LESS variables allow you to specify widely-used values such as colors, font family, and sizes in a single file. By modifying a single variable, the changes will be reflected in all the Bootstrap components that use it; for example, to change the background color of the body element to green (#00FF00 is the hexadecimal code for green), all you need to do is change the value of the variable called @body-bg in Bootstrap as shown in the following code: @body-bg: #00FF00; Mixins are similar to variables but for whole classes. Mixins enable you to embed the properties of a class into another. It allows you to group multiple code lines together so that it can be used numerous times across the style sheet. Mixins can also be used alongside variables and functions resulting in multiple inheritances; for example, to add clearfix to an article, you can use the .clearfix mixin as shown in the left column of the following table. It will result in all clearfix declarations included in the compiled CSS code shown in the right column: Mixin CSS code article { .clearfix; }   { article:before, article:after { content: " "; // 1 display: table; // 2 } article:after { clear: both; } }   A clearfix mixin is a way for an element to automatically clear after itself, so that you don't need to add additional markup. It's generally used in float layouts, where elements are floated to be stacked horizontally. Let's look at a pragmatic example to understand how this kind of customization is used in a real-time scenario: Download and unzip the Bootstrap files into a folder. Create an HTML file called bootstrap_example and save it in the same folder where you saved the Bootstrap files. Add the following code to it: <!DOCTYPE html> <html> <head> <title>BootStrap with Packt</title> <meta charset="UTF-8"> <meta name="viewport" content="width=device-width, initial- scale=1.0"> <!-- Downloaded Bootstrap CSS --> <link href="css/bootstrap.css" rel="stylesheet"> <!-- JavaScript plugins (requires jQuery) --> <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/ jquery.min.js"></script> <!-- Include all compiled plugins (below), or include individual files as needed --> <script src="js/bootstrap.min.js"></script> </head> <body> <h1>Welcome to Packt</h1> <button type="button" class="btn btn-default btn-lg" id="packt">PACKT LESSONS</button> </body> </html> The output of this code upon execution will be as follows: The Bootstrap folder includes the following folders and file:     css     fonts     js     bootstrap_example.html This Bootstrap folder is shown in the following screenshot: Since we are going to use the Bootstrap source code now, let's download the ZIP file and keep it at any location. Unzip it, and we can see the contents of the folder as shown in the following screenshot: Let's now create a new folder called bootstrap in the css folder. The contents of our css folder will appear as displayed in the following screenshot: Copy the contents of the less folder from the source code and paste it into the newly created bootstrap folder inside the css folder. Thus, the contents of the same bootstrap folder within the css folder will appear as displayed in the following screenshot: In the bootstrap folder, look for the variable.less file and open it using Notepad or Notepad++. In this example, we are using a simple Notepad, and on opening the variable.less file with Notepad, we can see the contents of the file as shown in the following screenshot: Currently, we can see @body-bg is assigned the default value #fff as the color code. Change the background color of the body element to green by assigning the value #00ff00 to it. Save the file and later on, look for the bootstrap.less file in the bootstrap folder. In the next step, we are going to use WinLess. Open WinLess and add the contents of the bootstrap folder to it. In the folder pane, you will see all the less files loaded as shown in the following screenshot:   Now, we need to uncheck all the files and only select the bootstrap.less file as shown in following screenshot:  Click on Compile. This will compile your bootstrap.less file to bootstrap.css. Copy the newly compiled bootstrap.css file from the bootstrap folder and paste it into the css folder thereby replacing the original bootstrap.css file. Now that we have the updated bootstrap.css file, go back to bootstrap_example.html and execute it. Upon execution, the output of the code would be as follows:  Thus, we can see that the background color of the <body> element turns to green as we have altered it globally in the variables.less file that was linked to the bootstrap.less file, which was later compiled to bootstrap.css by WinLess. We can also use LESS variables and mixins to customize Bootstrap. We can import the Bootstrap files and add our customizations. Let's now create our own less file called styles.less in the css folder. We will now include the Bootstrap files by adding the following line of code in the styles.less file: @import "./bootstrap/bootstrap.less"; We have given the path,./bootstrap/bootstrap.less as per the location of the bootstrap.less file. Remember to give the appropriate path if you have placed it at any other location. Now, let's try a few customizations and add the following code to styles.less: @body-bg: #FFA500; @padding-large-horizontal: 40px; @font-size-base: 7px; @line-height-base: 9px; @border-radius-large: 75px; The next step is to compile the styles.less file to styles.css. We will again use WinLess for this purpose. You have to uncheck all the options and select only styles.less to be compiled:  On compilation, the styles.css file will contain all the CSS declarations from Bootstrap. The next step would be to add the styles.css stylesheet to the bootstrap_example.html file.So your HTML code will look like this: <!DOCTYPE html> <html> <head> <title>BootStrap with Packt</title> <meta charset="UTF-8"> <meta name="viewport" content="width=device-width, initial- scale=1.0"> <!-- Downloaded Bootstrap CSS --> <link href="css/bootstrap.css" rel="stylesheet"> <!-- JavaScript plugins (requires jQuery) --> <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/ jquery.min.js"></script> <!-- Include all compiled plugins (below), or include individual files as needed --> <script src="js/bootstrap.min.js"></script> <link href="css/styles.css" rel="stylesheet"> </head> <body> <h1>Welcome to Packt</h1> <button type="button" class="btn btn-default btn-lg" id="packt">PACKT LESSONS</button> </body> </html> The output of the code is as follows: Since we changed the background color to orange (#ffa500), created a border radius, and defined the font-size-base and line-height-base, the output on execution was as displayed in the preceding screenshot. The LESS variables should be added to the styles.less file after the Bootstrap import so that they override the variables defined in the Bootstrap files. In short, all the custom code you write should be added after the Bootstrap import. Summary Therefore, we had a look at the procedure to implement Deep Customization in Bootstrap. However, we are still at the start of the journey. The learning curve is always steep as there is so much more to learn. Learning is always an ongoing process and it would never cease to exist. Thus, there is still a long way to go and in a pragmatic sense, the journey is the destination. Resources for Article: Further resources on this subject: Creating attention-grabbing pricing tables [article] Getting Started with Bootstrap [article] Bootstrap 3.0 is Mobile First [article]

In this article by Mark Reynolds, author of the book Xamarin Essentials, we will take the next step and look at how the use of design patterns and frameworks can increase the amount of code that can be reused. We will cover the following topics: An introduction to MvvmCross The MVVM design pattern Core concepts Views, ViewModels, and commands Data binding Navigation (ViewModel to ViewModel) The project organization The startup process Creating NationalParks.MvvmCross It's more than a little ambitious to try to cover MvvmCross along with a working example in a single article. Our approach will be to introduce the core concepts at a high level and then dive in and create the national parks sample app using MvvmCross. This will give you a basic understanding of how to use the framework and the value associated with its use. With that in mind, let's get started. (For more resources related to this topic, see here.) Introducing MvvmCross                                            MvvmCross is an open source framework that was created by Stuart Lodge. It is based on the Model-View-ViewModel (MVVM) design pattern and is designed to enhance code reuse across numerous platforms, including Xamarin.Android, Xamarin.iOS, Windows Phone, Windows Store, WPF, and Mac OSX. The MvvmCross project is hosted on GitHub and can be accessed at https://github.com/MvvmCross/MvvmCross. The MVVM pattern MVVM is a variation of the Model-View-Controller pattern. It separates logic traditionally placed in a View object into two distinct objects, one called View and the other called ViewModel. The View is responsible for providing the user interface and the ViewModel is responsible for the presentation logic. The presentation logic includes transforming data from the Model into a form that is suitable for the user interface to work with and mapping user interaction with the View into requests sent back to the Model. The following diagram depicts how the various objects in MVVM communicate: While MVVM presents a more complex implementation model, there are significant benefits of it, which are as follows: ViewModels and their interactions with Models can generally be tested using frameworks (such as NUnit) that are much easier than applications that combine the user interface and presentation layers ViewModels can generally be reused across different user interface technologies and platforms These factors make the MVVM approach both flexible and powerful. Views Views in an MvvmCross app are implemented using platform-specific constructs. For iOS apps, Views are generally implemented as ViewControllers and XIB files. MvvmCross provides a set of base classes, such as MvxViewContoller, that iOS ViewControllers inherit from. Storyboards can also be used in conjunction with a custom presenter to create Views; we will briefly discuss this option in the section titled Implementing the iOS user interface later in this article. For Android apps, Views are generally implemented as MvxActivity or MvxFragment along with their associated layout files. ViewModels ViewModels are classes that provide data and presentation logic to views in an app. Data is exposed to a View as properties on a ViewModel, and logic that can be invoked from a View is exposed as commands. ViewModels inherit from the MvxViewModel base class. Commands Commands are used in ViewModels to expose logic that can be invoked from the View in response to user interactions. The command architecture is based on the ICommand interface used in a number of Microsoft frameworks such as Windows Presentation Foundation (WPF) and Silverlight. MvvmCross provides IMvxCommand, which is an extension of ICommand, along with an implementation named MvxCommand. The commands are generally defined as properties on a ViewModel. For example: public IMvxCommand ParkSelected { get; protected set; } Each command has an action method defined, which implements the logic to be invoked: protected void ParkSelectedExec(NationalPark park){   . . .// logic goes here} The commands must be initialized and the corresponding action method should be assigned: ParkSelected =   new MvxCommand<NationalPark> (ParkSelectedExec); Data binding Data binding facilitates communication between the View and the ViewModel by establishing a two-way link that allows data to be exchanged. The data binding capabilities provided by MvvmCross are based on capabilities found in a number of Microsoft XAML-based UI frameworks such as WPF and Silverlight. The basic idea is that you would like to bind a property in a UI control, such as the Text property of an EditText control in an Android app to a property of a data object such as the Description property of NationalPark. The following diagram depicts this scenario:  The binding modes There are four different binding modes that can be used for data binding: OneWay binding: This mode tells the data binding framework to transfer values from the ViewModel to the View and transfer any updates to properties on the ViewModel to their bound View property. OneWayToSource binding: This mode tells the data binding framework to transfer values from the View to the ViewModel and transfer updates to View properties to their bound ViewModel property. TwoWay binding: This mode tells the data binding framework to transfer values in both directions between the ViewModel and View, and updates on either object will cause the other to be updated. This binding mode is useful when values are being edited. OneTime binding: This mode tells the data binding framework to transfer values from ViewModel to View when the binding is established; in this mode, updates to ViewModel properties are not monitored by the View. The INotifyPropertyChanged interface The INotifyPropertyChanged interface is an integral part of making data binding work effectively; it acts as a contract between the source object and the target object. As the name implies, it defines a contract that allows the source object to notify the target object when data has changed, thus allowing the target to take any necessary actions such as refreshing its display. The interface consists of a single event—the PropertyChanged event—that the target object can subscribe to and that is triggered by the source if a property changes. The following sample demonstrates how to implement INotifyPropertyChanged: public class NationalPark : INotifyPropertyChanged{  public event PropertyChangedEventHandler     PropertyChanged;// rather than use "… code" it is safer to use// the comment formstring _name;public string Name{   get { return _name; }   set   {       if (value.Equals (_name,           StringComparison.Ordinal))      {     // Nothing to do - the value hasn't changed;     return;       }       _name = value;       OnPropertyChanged();   }}. . .void OnPropertyChanged(   [CallerMemberName] string propertyName = null){     var handler = PropertyChanged;if (handler != null){     handler(this,           new PropertyChangedEventArgs(propertyName));}}} Binding specifications Bindings can be specified in a couple of ways. For Android apps, bindings can be specified in layout files. The following example demonstrates how to bind the Text property of a TextView instance to the Description property in a NationalPark instance: <TextView   android_layout_width="match_parent"   android_layout_height="wrap_content"   android_id="@+id/descrTextView"  local:MvxBind="Text Park.Description" /> For iOS, binding must be accomplished using the binding API. CreateBinding() is a method than can be found on MvxViewController. The following example demonstrates how to bind the Description property to a UILabel instance: this.CreateBinding (this.descriptionLabel).   To ((DetailViewModel vm) => vm.Park.Description).   Apply (); Navigating between ViewModels Navigating between various screens within an app is an important capability. Within a MvvmCross app, this is implemented at the ViewModel level so that navigation logic can be reused. MvvmCross supports navigation between ViewModels through use of the ShowViewModel<T>() method inherited from MvxNavigatingObject, which is the base class for MvxViewModel. The following example demonstrates how to navigate to DetailViewModel: ShowViewModel<DetailViewModel>(); Passing parameters In many situations, there is a need to pass information to the destination ViewModel. MvvmCross provides a number of ways to accomplish this. The primary method is to create a class that contains simple public properties and passes an instance of the class into ShowViewModel<T>(). The following example demonstrates how to define and use a parameters class during navigation: public class DetailParams{   public int ParkId { get; set; }} // using the parameters classShowViewModel<DetailViewModel>(new DetailViewParam() { ParkId = 0 }); To receive and use parameters, the destination ViewModel implements an Init() method that accepts an instance of the parameters class: public class DetailViewModel : MvxViewModel{   . . .   public void Init(DetailViewParams parameters)   {       // use the parameters here . . .   }} Solution/project organization Each MvvmCross solution will have a single core PCL project that houses the reusable code and a series of platform-specific projects that contain the various apps. The following diagram depicts the general structure: The startup process MvvmCross apps generally follow a standard startup sequence that is initiated by platform-specific code within each app. There are several classes that collaborate to accomplish the startup; some of these classes reside in the core project and some of them reside in the platform-specific projects. The following sections describe the responsibilities of each of the classes involved. App.cs The core project has an App class that inherits from MvxApplication. The App class contains an override to the Initialize() method so that at a minimum, it can register the first ViewModel that should be presented when the app starts: RegisterAppStart<ViewModels.MasterViewModel>(); Setup.cs Android and iOS projects have a Setup class that is responsible for creating the App object from the core project during the startup. This is accomplished by overriding the CreateApp() method: protected override IMvxApplication CreateApp(){   return new Core.App();} For Android apps, Setup inherits from MvxAndroidSetup. For iOS apps, Setup inherits from MvxTouchSetup. The Android startup Android apps are kicked off using a special Activity splash screen that calls the Setup class and initiates the MvvmCross startup process. This is all done automatically for you; all you need to do is include the splash screen definition and make sure it is marked as the launch activity. The definition is as follows: [Activity(Label="NationalParks.Droid", MainLauncher = true,Icon="@drawable/icon", Theme="@style/Theme.Splash",NoHistory=true,ScreenOrientation = ScreenOrientation.Portrait)]public class SplashScreen : MvxSplashScreenActivity{   public SplashScreen():base(Resource.Layout.SplashScreen)   {   } The iOS startup The iOS app startup is slightly less automated and is initiated from within the FinishedLaunching() method of AppDelegate: public override bool FinishedLaunching (   UIApplication app, NSDictionary options){   _window = new UIWindow (UIScreen.MainScreen.Bounds);    var setup = new Setup(this, _window);   setup.Initialize();   var startup = Mvx.Resolve<IMvxAppStart>();   startup.Start();    _window.MakeKeyAndVisible ();   return true;} Creating NationalParks.MvvmCross Now that we have basic knowledge of the MvvmCross framework, let's put that knowledge to work and convert the NationalParks app to leverage the capabilities we just learned. Creating the MvvmCross core project We will start by creating the core project. This project will contain all the code that will be shared between the iOS and Android app primarily in the form of ViewModels. The core project will be built as a Portable Class Library. To create NationalParks.Core, perform the following steps: From the main menu, navigate to File | New Solution. From the New Solution dialog box, navigate to C# | Portable Library, enter NationalParks.Core for the project Name field, enter NationalParks.MvvmCross for the Solution field, and click on OK. Add the MvvmCross starter package to the project from Nuget. Select the NationalParks.Core project and navigate to Project | Add Packages from the main menu. Enter MvvmCross starter in the search field. Select the MvvmCross – Hot Tuna Starter Pack entry and click on Add Package. A number of things were added to NationalParks.Core as a result of adding the package, and they are as follows: A packages.config file, which contains a list of libraries (dlls) associated with the MvvmCross starter kit package. These entries are links to actual libraries in the Packages folder of the overall solution. A ViewModels folder with a sample ViewModel named FirstViewModel. An App class in App.cs, which contains an Initialize() method that starts the MvvmCross app by calling RegisterAppStart() to start FirstViewModel. We will eventually be changing this to start the MasterViewModel class, which will be associated with a View that lists national parks. Creating the MvvmCross Android app The next step is to create an Android app project in the same solution. To create NationalParks.Droid, complete the following steps: Select the NationalParks.MvvmCross solution, right-click on it, and navigate to Add | New Project. From the New Project dialog box, navigate to C# | Android | Android Application, enter NationalParks.Droid for the Name field, and click on OK. Add the MvvmCross starter kit package to the new project by selecting NationalParks.Droid and navigating to Project | Add Packages from the main menu. A number of things were added to NationalParks.Droid as a result of adding the package, which are as follows: packages.config: This file contains a list of libraries (dlls) associated with the MvvmCross starter kit package. These entries are links to an actual library in the Packages folder of the overall solution, which contains the actual downloaded libraries. FirstView : This class is present in the Views folder, which corresponds to FirstViewModel, which was created in NationalParks.Core. FirstView: This layout is present in Resourceslayout, which is used by the FirstView activity. This is a traditional Android layout file with the exception that it contains binding declarations in the EditView and TextView elements. Setup: This file inherits from MvxAndroidSetup. This class is responsible for creating an instance of the App class from the core project, which in turn displays the first ViewModel via a call to RegisterAppStart(). SplashScreen: This class inherits from MvxSplashScreenActivity. The SplashScreen class is marked as the main launcher activity and thus initializes the MvvmCross app with a call to Setup.Initialize(). Add a reference to NationalParks.Core by selecting the References folder, right-click on it, select Edit References, select the Projects tab, check NationalParks.Core, and click on OK. Remove MainActivity.cs as it is no longer needed and will create a build error. This is because it is marked as the main launch and so is the new SplashScreen class. Also, remove the corresponding Resourceslayoutmain.axml layout file. Run the app. The app will present FirstViewModel, which is linked to the corresponding FirstView instance with an EditView class, and TextView presents the same Hello MvvmCross text. As you edit the text in the EditView class, the TextView class is automatically updated by means of data binding. The following screenshot depicts what you should see: Reusing NationalParks.PortableData and NationalParks.IO Before we start creating the Views and ViewModels for our app, we first need to bring in some code from our previous efforts that can be used to maintain parks. For this, we will simply reuse the NationalParksData singleton and the FileHandler classes that were created previously. To reuse the NationalParksData singleton and FileHandler classes, complete the following steps: Copy NationalParks.PortableData and NationalParks.IO to the NationalParks.MvvmCross solution folder. Add a reference to NationalParks.PortableData in the NationalParks.Droid project. Create a folder named NationalParks.IO in the NationalParks.Droid project and add a link to FileHandler.cs from the NationalParks.IO project. Recall that the FileHandler class cannot be contained in the Portable Class Library because it uses file IO APIs that cannot be references from a Portable Class Library. Compile the project. The project should compile cleanly now. Implementing the INotifyPropertyChange interface We will be using data binding to bind UI controls to the NationalPark object and thus, we need to implement the INotifyPropertyChange interface. This ensures that changes made to properties of a park are reported to the appropriate UI controls. To implement INotifyPropertyChange, complete the following steps: Open NationalPark.cs in the NationalParks.PortableData project. Specify that the NationalPark class implements INotifyPropertyChanged interface. Select the INotifyPropertyChanged interface, right-click on it, navigate to Refactor | Implement interface, and press Enter. Enter the following code snippet: public class NationalPark : INotifyPropertyChanged{   public event PropertyChangedEventHandler       PropertyChanged;   . . .} Add an OnPropertyChanged() method that can be called from each property setter method: void OnPropertyChanged(   CallerMemberName] string propertyName = null){   var handler = PropertyChanged;   if (handler != null)   {       handler(this,           new PropertyChangedEventArgs(propertyName));   }} Update each property definition to call the setter in the same way as it is depicted for the Name property: string _name;public string Name{get { return _name; }set{   if (value.Equals (_name, StringComparison.Ordinal))   {     // Nothing to do - the value hasn't changed;return;   }   _name = value;   OnPropertyChanged();}} Compile the project. The project should compile cleanly. We are now ready to use the NationalParksData singleton in our new project, and it supports data binding. Implementing the Android user interface Now, we are ready to create the Views and ViewModels required for our app. The app we are creating will follow the following flow: A master list view to view national parks A detail view to view details of a specific park An edit view to edit a new or previously existing park The process for creating views and ViewModels in an Android app generally consists of three different steps: Create a ViewModel in the core project with the data and event handlers (commands) required to support the View. Create an Android layout with visual elements and data binding specifications. Create an Android activity, which corresponds to the ViewModel and displays the layout. In our case, this process will be slightly different because we will reuse some of our previous work, specifically, the layout files and the menu definitions. To reuse layout files and menu definitions, perform the following steps: Copy Master.axml, Detail.axml, and Edit.axml from the Resourceslayout folder of the solution to the Resourceslayout folder in the NationalParks.Droid project, and add them to the project by selecting the layout folder and navigating to Add | Add Files. Copy MasterMenu.xml, DetailMenu.xml, and EditMenu.xml from the Resourcesmenu folder of the solution to the Resourcesmenu folder in the NationalParks.Droid project, and add them to the project by selecting the menu folder and navigating to Add | Add Files. Implementing the master list view We are now ready to implement the first of our View/ViewModel combinations, which is the master list view. Creating MasterViewModel The first step is to create a ViewModel and add a property that will provide data to the list view that displays national parks along with some initialization code. To create MasterViewModel, complete the following steps: Select the ViewModels folder in NationalParks.Core, right-click on it, and navigate to Add | New File. In the New File dialog box, navigate to General | Empty Class, enter MasterViewModel for the Name field, and click on New. Modify the class definition so that MasterViewModel inherits from MvxViewModel; you will also need to add a few using directives: . . .using Cirrious.CrossCore.Platform;using Cirrious.MvvmCross.ViewModels;. . .namespace NationalParks.Core.ViewModels{public class MasterViewModel : MvxViewModel{         . . .   }} Add a property that is a list of NationalPark elements to MasterViewModel. This property will later be data-bound to a list view: private List<NationalPark> _parks;public List<NationalPark> Parks{   get { return _parks; }   set { _parks = value;         RaisePropertyChanged(() => Parks);   }} Override the Start() method on MasterViewModel to load the _parks collection with data from the NationalParksData singleton. You will need to add a using directive for the NationalParks.PortableData namespace again: . . .using NationalParks.PortableData;. . .public async override void Start (){   base.Start ();   await NationalParksData.Instance.Load ();   Parks = new List<NationalPark> (       NationalParksData.Instance.Parks);} We now need to modify the app startup sequence so that MasterViewModel is the first ViewModel that's started. Open App.cs in NationalParks.Core and change the call to RegisterAppStart() to reference MasterViewModel: RegisterAppStart<ViewModels.MasterViewModel>(); Updating the Master.axml layout Update Master.axml so that it can leverage the data binding capabilities provided by MvvmCross. To update Master.axml, complete the following steps: Open Master.axml and add a namespace definition to the top of the XML to include the NationalParks.Droid namespace: This namespace definition is required in order to allow Android to resolve the MvvmCross-specific elements that will be specified. Change the ListView element to a Mvx.MvxListView element: <Mvx.MvxListView   android_layout_width="match_parent"   android_layout_height="match_parent"   android_id="@+id/parksListView" /> Add a data binding specification to the MvxListView element, binding the ItemsSource property of the list view to the Parks property of MasterViewModel, as follows:    . . .   android_id="@+id/parksListView" local_MvxBind="ItemsSource Parks" /> Add a list item template attribute to the element definition. This layout controls the content of each item that will be displayed in the list view: local:MvxItemTemplate="@layout/nationalparkitem" Create the NationalParkItem layout and provide TextView elements to display both the name and description of a park, as follows: <LinearLayout    android_orientation="vertical"   android_layout_width="fill_parent"   android_layout_height="wrap_content">   <TextView       android_layout_width="match_parent"       android_layout_height="wrap_content"       android_textSize="40sp"/>   <TextView       android_layout_width="match_parent"       android_layout_height="wrap_content"       android_textSize="20sp"/></LinearLayout> Add data binding specifications to each of the TextView elements: . . .       local_MvxBind="Text Name" />. . .       local_MvxBind="Text Description" />. . . Note that in this case, the context for data binding is an instance of an item in the collection that was bound to MvxListView, for this example, an instance of NationalPark. Creating the MasterView activity Next, create MasterView, which is an MvxActivity instance that corresponds with MasterViewModel. To create MasterView, complete the following steps: Select the ViewModels folder in NationalParks.Core, right-click on it, navigate to Add | New File. In the New File dialog, navigate to Android | Activity, enter MasterView in the Name field, and select New. Modify the class specification so that it inherits from MvxActivity; you will also need to add a few using directives as follows: using Cirrious.MvvmCross.Droid.Views;using NationalParks.Core.ViewModels;. . .namespace NationalParks.Droid.Views{   [Activity(Label = "Parks")]   public class MasterView : MvxActivity   {       . . .   }} Open Setup.cs and add code to initialize the file handler and path for the NationalParksData singleton to the CreateApp() method, as follows: protected override IMvxApplication CreateApp(){   NationalParksData.Instance.FileHandler =       new FileHandler ();   NationalParksData.Instance.DataDir =       System.Environment.GetFolderPath(         System.Environment.SpecialFolder.MyDocuments);   return new Core.App();} Compile and run the app; you will need to copy the NationalParks.json file to the device or emulator using the Android Device Monitor. All the parks in NationalParks.json should be displayed. Implementing the detail view Now that we have the master list view displaying national parks, we can focus on creating the detail view. We will follow the same steps for the detail view as the ones we just completed for the master view. Creating DetailViewModel We start creating DetailViewModel by using the following steps: Following the same procedure as the one that was used to create MasterViewModel, create a new ViewModel named DetailViewModel in the ViewModel folder of NationalParks.Core. Add a NationalPark property to support data binding for the view controls, as follows: protected NationalPark _park;public NationalPark Park{   get { return _park; }   set { _park = value;         RaisePropertyChanged(() => Park);     }} Create a Parameters class that can be used to pass a park ID for the park that should be displayed. It's convenient to create this class within the class definition of the ViewModel that the parameters are for: public class DetailViewModel : MvxViewModel{   public class Parameters   {       public string ParkId { get; set; }   }   . . . Implement an Init() method that will accept an instance of the Parameters class and get the corresponding national park from NationalParkData: public void Init(Parameters parameters){   Park = NationalParksData.Instance.Parks.       FirstOrDefault(x => x.Id == parameters.ParkId);} Updating the Detail.axml layout Next, we will update the layout file. The main changes that need to be made are to add data binding specifications to the layout file. To update the Detail.axml layout, perform the following steps: Open Detail.axml and add the project namespace to the XML file: Add data binding specifications to each of the TextView elements that correspond to a national park property, as demonstrated for the park name: <TextView   android_layout_width="match_parent"   android_layout_height="wrap_content"   android_id="@+id/nameTextView"   local_MvxBind="Text Park.Name" /> Creating the DetailView activity Now, create the MvxActivity instance that will work with DetailViewModel. To create DetailView, perform the following steps: Following the same procedure as the one that was used to create MasterView, create a new view named DetailView in the Views folder of NationalParks.Droid. Implement the OnCreateOptionsMenu() and OnOptionsItemSelected() methods so that our menus will be accessible. Copy the implementation of these methods from the solution. Comment out the section in OnOptionsItemSelect() related to the Edit action for now; we will fill that in once the edit view is completed. Adding navigation The last step is to add navigation so that when an item is clicked on in MvxListView on MasterView, the park is displayed in the detail view. We will accomplish this using a command property and data binding. To add navigation, perform the following steps: Open MasterViewModel and add an IMvxCommand property; this will be used to handle a park that is being selected: protected IMvxCommand ParkSelected { get; protected set; } Create an Action delegate that will be called when the ParkSelected command is executed, as follows: protected void ParkSelectedExec(NationalPark park){   ShowViewModel<DetailViewModel> (       new DetailViewModel.Parameters ()           { ParkId = park.Id });} Initialize the command property in the constructor of MasterViewModel: ParkClicked =   new MvxCommand<NationalPark> (ParkSelectedExec); Now, for the last step, add a data binding specification to MvvListView in Master.axml to bind the ItemClick event to the ParkClicked command on MasterViewModel, which we just created: local:MvxBind="ItemsSource Parks; ItemClick ParkClicked" Compile and run the app. Clicking on a park in the list view should now navigate to the detail view, displaying the selected park. Implementing the edit view We are now almost experts at implementing new Views and ViewModels. One last View to go is the edit view. Creating EditViewModel Like we did previously, we start with the ViewModel. To create EditViewModel, complete the following steps: Following the same process that was previously used in this article to create EditViewModel, add a data binding property and create a Parameters class for navigation. Implement an Init() method that will accept an instance of the Parameters class and get the corresponding national park from NationalParkData in the case of editing an existing park or create a new instance if the user has chosen the New action. Inspect the parameters passed in to determine what the intent is: public void Init(Parameters parameters){   if (string.IsNullOrEmpty (parameters.ParkId))       Park = new NationalPark ();   else       Park =           NationalParksData.Instance.           Parks.FirstOrDefault(           x => x.Id == parameters.ParkId);} Updating the Edit.axml layout Update Edit.axml to provide data binding specifications. To update the Edit.axml layout, you first need to open Edit.axml and add the project namespace to the XML file. Then, add the data binding specifications to each of the EditView elements that correspond to a national park property. Creating the EditView activity Create a new MvxActivity instance named EditView to will work with EditViewModel. To create EditView, perform the following steps: Following the same procedure as the one that was used to create DetailView, create a new View named EditView in the Views folder of NationalParks.Droid. Implement the OnCreateOptionsMenu() and OnOptionsItemSelected() methods so that the Done action will accessible from the ActionBar. You can copy the implementation of these methods from the solution. Change the implementation of Done to call the Done command on EditViewModel. Adding navigation Add navigation to two places: when New (+) is clicked from MasterView and when Edit is clicked in DetailView. Let's start with MasterView. To add navigation from MasterViewModel, complete the following steps: Open MasterViewModel.cs and add a NewParkClicked command property along with the handler for the command. Be sure to initialize the command in the constructor, as follows: protected IMvxCommand NewParkClicked { get; set; }protected void NewParkClickedExec(){ShowViewModel<EditViewModel> ();} Note that we do not pass in a parameter class into ShowViewModel(). This will cause a default instance to be created and passed in, which means that ParkId will be null. We will use this as a way to determine whether a new park should be created. Now, it's time to hook the NewParkClicked command up to the actionNew menu item. We do not have a way to accomplish this using data binding, so we will resort to a more traditional approach—we will use the OnOptionsItemSelected() method. Add logic to invoke the Execute() method on NewParkClicked, as follows: case Resource.Id.actionNew:   ((MasterViewModel)ViewModel).       NewParkClicked.Execute ();   return true; To add navigation from DetailViewModel, complete the following steps: Open DetailViewModel.cs and add a EditParkClicked command property along with the handler for the command. Be sure to initialize the command in the constructor, as shown in the following code snippet: protected IMvxCommand EditPark { get; protected set;}protected void EditParkHandler(){   ShowViewModel<EditViewModel> (       new EditViewModel.Parameters ()          { ParkId = _park.Id });} Note that an instance of the Parameters class is created, initialized, and passed into the ShowViewModel() method. This instance will in turn be passed into the Init() method on EditViewModel. Initialize the command property in the constructor for MasterViewModel, as follows: EditPark =    new MvxCommand<NationalPark> (EditParkHandler); Now, update the OnOptionsItemSelect() method in DetailView to invoke the DetailView.EditPark command when the Edit action is selected: case Resource.Id.actionEdit:   ((DetailViewModel)ViewModel).EditPark.Execute ();   return true; Compile and run NationalParks.Droid. You should now have a fully functional app that has the ability to create new parks and edit the existing parks. Changes made to EditView should automatically be reflected in MasterView and DetailView. Creating the MvvmCross iOS app The process of creating the Android app with MvvmCross provides a solid understanding of how the overall architecture works. Creating the iOS solution should be much easier for two reasons: first, we understand how to interact with MvvmCross and second, all the logic we have placed in NationalParks.Core is reusable, so that we just need to create the View portion of the app and the startup code. To create NationalParks.iOS, complete the following steps: Select the NationalParks.MvvmCross solution, right-click on it, and navigate to Add | New Project. From the New Project dialog, navigate to C# | iOS | iPhone | Single View Application, enter NationalParks.iOS in the Name field, and click on OK. Add the MvvmCross starter kit package to the new project by selecting NationalParks.iOS and navigating to Project | Add Packages from the main menu. A number of things were added to NationalParks.iOS as a result of adding the package. They are as follows: packages.config: This file contains a list of libraries associated with the MvvmCross starter kit package. These entries are links to an actual library in the Packages folder of the overall solution, which contains the actual downloaded libraries. FirstView: This class is placed in the Views folder, which corresponds to the FirstViewModel instance created in NationalParks.Core. Setup: This class inherits from MvxTouchSetup. This class is responsible for creating an instance of the App class from the core project, which in turn displays the first ViewModel via a call to RegisterAppStart(). AppDelegate.cs.txt: This class contains the sample startup code, which should be placed in the actual AppDelete.cs file. Implementing the iOS user interface We are now ready to create the user interface for the iOS app. The good news is that we already have all the ViewModels implemented, so we can simply reuse them. The bad news is that we cannot easily reuse the storyboards from our previous work; MvvmCross apps generally use XIB files. One of the reasons for this is that storyboards are intended to provide navigation capabilities and an MvvmCross app delegates that responsibility to ViewModel and presenter. It is possible to use storyboards in combination with a custom presenter, but the remainder of this article will focus on using XIB files, as this is the more common use. The screen layouts can be used as depicted in the following screenshot: We are now ready to get started. Implementing the master view The first view we will work on is the master view. To implement the master view, complete the following steps: Create a new ViewController class named MasterView by right-clicking on the Views folder of NationalParks.iOS and navigating to Add | New File | iOS | iPhone View Controller. Open MasterView.xib and arrange controls as seen in the screen layouts. Add outlets for each of the edit controls. Open MasterView.cs and add the following boilerplate logic to deal with constraints on iOS 7, as follows: // ios7 layoutif (RespondsToSelector(new   Selector("edgesForExtendedLayout")))   EdgesForExtendedLayout = UIRectEdge.None; Within the ViewDidLoad() method, add logic to create MvxStandardTableViewSource for parksTableView: MvxStandardTableViewSource _source;. . ._source = new MvxStandardTableViewSource(   parksTableView,   UITableViewCellStyle.Subtitle,   new NSString("cell"),   "TitleText Name; DetailText Description",     0);parksTableView.Source = _source; Note that the example uses the Subtitle cell style and binds the national park name and description to the title and subtitle. Add the binding logic to the ViewDidShow() method. In the previous step, we provided specifications for properties of UITableViewCell to properties in the binding context. In this step, we need to set the binding context for the Parks property on MasterModelView: var set = this.CreateBindingSet<MasterView,   MasterViewModel>();set.Bind (_source).To (vm => vm.Parks);set.Apply(); Compile and run the app. All the parks in NationalParks.json should be displayed. Implementing the detail view Now, implement the detail view using the following steps: Create a new ViewController instance named DetailView. Open DetailView.xib and arrange controls as shown in the following code. Add outlets for each of the edit controls. Open DetailView.cs and add the binding logic to the ViewDidShow() method: this.CreateBinding (this.nameLabel).   To ((DetailViewModel vm) => vm.Park.Name).Apply ();this.CreateBinding (this.descriptionLabel).   To ((DetailViewModel vm) => vm.Park.Description).       Apply ();this.CreateBinding (this.stateLabel).   To ((DetailViewModel vm) => vm.Park.State).Apply ();this.CreateBinding (this.countryLabel).   To ((DetailViewModel vm) => vm.Park.Country).       Apply ();this.CreateBinding (this.latLabel).   To ((DetailViewModel vm) => vm.Park.Latitude).        Apply ();this.CreateBinding (this.lonLabel).   To ((DetailViewModel vm) => vm.Park.Longitude).       Apply (); Adding navigation Add navigation from the master view so that when a park is selected, the detail view is displayed, showing the park. To add navigation, complete the following steps: Open MasterView.cs, create an event handler named ParkSelected, and assign it to the SelectedItemChanged event on MvxStandardTableViewSource, which was created in the ViewDidLoad() method: . . .   _source.SelectedItemChanged += ParkSelected;. . .protected void ParkSelected(object sender, EventArgs e){   . . .} Within the event handler, invoke the ParkSelected command on MasterViewModel, passing in the selected park: ((MasterViewModel)ViewModel).ParkSelected.Execute (       (NationalPark)_source.SelectedItem); Compile and run NationalParks.iOS. Selecting a park in the list view should now navigate you to the detail view, displaying the selected park. Implementing the edit view We now need to implement the last of the Views for the iOS app, which is the edit view. To implement the edit view, complete the following steps: Create a new ViewController instance named EditView. Open EditView.xib and arrange controls as in the layout screenshots. Add outlets for each of the edit controls. Open EditView.cs and add the data binding logic to the ViewDidShow() method. You should use the same approach to data binding as the approach used for the details view. Add an event handler named DoneClicked, and within the event handler, invoke the Done command on EditViewModel: protected void DoneClicked (object sender, EventArgs e){  ((EditViewModel)ViewModel).Done.Execute();} In ViewDidLoad(), add UIBarButtonItem to NavigationItem for EditView, and assign the DoneClicked event handler to it, as follows: NavigationItem.SetRightBarButtonItem(   new UIBarButtonItem(UIBarButtonSystemItem.Done,       DoneClicked), true); Adding navigation Add navigation to two places: when New (+) is clicked from the master view and when Edit is clicked on in the detail view. Let's start with the master view. To add navigation to the master view, perform the following steps: Open MasterView.cs and add an event handler named NewParkClicked. In the event handler, invoke the NewParkClicked command on MasterViewModel: protected void NewParkClicked(object sender,       EventArgs e){   ((MasterViewModel)ViewModel).           NewParkClicked.Execute ();} In ViewDidLoad(), add UIBarButtonItem to NavigationItem for MasterView and assign the NewParkClicked event handler to it: NavigationItem.SetRightBarButtonItem(   new UIBarButtonItem(UIBarButtonSystemItem.Add,       NewParkClicked), true); To add navigation to the details view, perform the following steps: Open DetailView.cs and add an event handler named EditParkClicked. In the event handler, invoke the EditParkClicked command on DetailViewModel: protected void EditParkClicked (object sender,   EventArgs e){   ((DetailViewModel)ViewModel).EditPark.Execute ();} In ViewDidLoad(), add UIBarButtonItem to NavigationItem for MasterView, and assign the EditParkClicked event handler to it: NavigationItem.SetRightBarButtonItem(   new UIBarButtonItem(UIBarButtonSystemItem.Edit,       EditParkClicked), true); Refreshing the master view list One last detail that needs to be taken care of is to refresh the UITableView control on MasterView when items have been changed on EditView. To refresh the master view list, perform the following steps: Open MasterView.cs and call ReloadData() on parksTableView within the ViewDidAppear() method of MasterView: public override void ViewDidAppear (bool animated){   base.ViewDidAppear (animated);   parksTableView.ReloadData();} Compile and run NationalParks.iOS. You should now have a fully functional app that has the ability to create new parks and edit existing parks. Changes made to EditView should automatically be reflected in MasterView and DetailVIew. Considering the pros and cons After completing our work, we now have the basis to make some fundamental observations. Let's start with the pros: MvvmCross definitely increases the amount of code that can be reused across platforms. The ViewModels house the data required by the View, the logic required to obtain and transform the data in preparation for viewing, and the logic triggered by user interactions in the form of commands. In our sample app, the ViewModels were somewhat simple; however, the more complex the app, the more reuse will likely be gained. As MvvmCross relies on the use of each platform's native UI frameworks, each app has a native look and feel and we have a natural layer that implements platform-specific logic when required. The data binding capabilities of MvvmCross also eliminate a great deal of tedious code that would otherwise have to be written. All of these positives are not necessarily free; let's look at some cons: The first con is complexity; you have to learn another framework on top of Xamarin, Android, and iOS. In some ways, MvvmCross forces you to align the way your apps work across platforms to achieve the most reuse. As the presentation logic is contained in the ViewModels, the views are coerced into aligning with them. The more your UI deviates across platforms; the less likely it will be that you can actually reuse ViewModels. With these things in mind, I would definitely consider using MvvmCross for a cross-platform mobile project. Yes, you need to learn an addition framework and yes, you will likely have to align the way some of the apps are laid out, but I think MvvmCross provides enough value and flexibility to make these issues workable. I'm a big fan of reuse and MvvmCross definitely pushes reuse to the next level. Summary In this article, we reviewed the high-level concepts of MvvmCross and worked through a practical exercise in order to convert the national parks apps to use the MvvmCross framework and the increase code reuse. In the next article, we will follow a similar approach to exploring the Xamarin.Forms framework in order to evaluate how its use can affect code reuse. Resources for Article: Further resources on this subject: XamChat – a Cross-platform App [Article] Configuring Your Operating System [Article] Updating data in the background [Article]

In this article by Dan Toomey, author of the book R for Data Science, we will learn about the supervised learning, which involves the use of a target variable and a number of predictor variables that are put into a model to enable the system to predict the target. This is also known as predictive modeling. (For more resources related to this topic, see here.) As mentioned, in supervised learning we have a target variable and a number of possible predictor variables. The objective is to associate the predictor variables in such a way so as to accurately predict the target variable. We are using some portion of observed data to learn how our model behaves and then testing that model on the remaining observations for accuracy. We will go over the following supervised learning techniques: Decision trees Regression Neural networks Instance based learning (k-NN) Ensemble learning Support vector machines Bayesian learning Bayesian inference Random forests Decision tree For decision tree machine learning, we develop a logic tree that can be used to predict our target value based on a number of predictor variables. The tree has logical points, such as if the month is December, follow the tree logic to the left; otherwise, follow the tree logic to the right. The last leaf of the tree has a predicted value. For this example, we will use the weather data in the rattle package. We will develop a decision tree to be used to determine whether it will rain tomorrow or not based on several variables. Let's load the rattle package as follows: > library(rattle) We can see a summary of the weather data. This shows that we have some real data over a year from Australia: > summary(weather)      Date                     Location     MinTemp     Min.   :2007-11-01   Canberra     :366   Min.   :-5.300 1st Qu.:2008-01-31   Adelaide     : 0   1st Qu.: 2.300 Median :2008-05-01   Albany       : 0   Median : 7.450 Mean   :2008-05-01   Albury       : 0   Mean   : 7.266 3rd Qu.:2008-07-31   AliceSprings : 0   3rd Qu.:12.500 Max.   :2008-10-31   BadgerysCreek: 0   Max.   :20.900                      (Other)     : 0                      MaxTemp         Rainfall       Evaporation       Sunshine     Min.   : 7.60   Min.   : 0.000   Min.  : 0.200   Min.   : 0.000 1st Qu.:15.03   1st Qu.: 0.000   1st Qu.: 2.200   1st Qu.: 5.950 Median :19.65   Median : 0.000   Median : 4.200   Median : 8.600 Mean   :20.55   Mean   : 1.428   Mean   : 4.522   Mean   : 7.909 3rd Qu.:25.50   3rd Qu.: 0.200   3rd Qu.: 6.400   3rd Qu.:10.500 Max.   :35.80   Max.   :39.800   Max.   :13.800   Max.   :13.600                                                    NA's   :3       WindGustDir   WindGustSpeed   WindDir9am   WindDir3pm NW     : 73   Min.   :13.00   SE     : 47   WNW   : 61 NNW   : 44   1st Qu.:31.00   SSE   : 40   NW     : 61 E     : 37   Median :39.00   NNW   : 36   NNW   : 47 WNW   : 35   Mean   :39.84   N     : 31   N     : 30 ENE   : 30   3rd Qu.:46.00   NW     : 30   ESE   : 27 (Other):144   Max.   :98.00   (Other):151   (Other):139 NA's   : 3   NA's   :2       NA's   : 31   NA's   : 1 WindSpeed9am     WindSpeed3pm   Humidity9am     Humidity3pm   Min.   : 0.000   Min.   : 0.00   Min.   :36.00   Min.   :13.00 1st Qu.: 6.000   1st Qu.:11.00   1st Qu.:64.00   1st Qu.:32.25 Median : 7.000   Median :17.00   Median :72.00   Median :43.00 Mean   : 9.652   Mean   :17.99   Mean   :72.04   Mean   :44.52 3rd Qu.:13.000   3rd Qu.:24.00   3rd Qu.:81.00   3rd Qu.:55.00 Max.   :41.000   Max.   :52.00   Max.   :99.00   Max.   :96.00 NA's   :7                                                       Pressure9am     Pressure3pm       Cloud9am       Cloud3pm   Min.   : 996.5   Min.   : 996.8   Min.   :0.000   Min.   :0.000 1st Qu.:1015.4   1st Qu.:1012.8   1st Qu.:1.000   1st Qu.:1.000 Median :1020.1   Median :1017.4   Median :3.500   Median :4.000 Mean   :1019.7   Mean   :1016.8   Mean   :3.891   Mean   :4.025 3rd Qu.:1024.5   3rd Qu.:1021.5   3rd Qu.:7.000   3rd Qu.:7.000 Max.   :1035.7   Max.   :1033.2   Max.   :8.000   Max.   :8.000 Temp9am         Temp3pm         RainToday RISK_MM Min.   : 0.100   Min.   : 5.10   No :300   Min.   : 0.000 1st Qu.: 7.625   1st Qu.:14.15   Yes: 66   1st Qu.: 0.000 Median :12.550   Median :18.55             Median : 0.000 Mean   :12.358   Mean   :19.23             Mean   : 1.428 3rd Qu.:17.000   3rd Qu.:24.00             3rd Qu.: 0.200 Max.   :24.700   Max.   :34.50           Max.   :39.800                                                            RainTomorrow No :300     Yes: 66       We will be using the rpart function to develop a decision tree. The rpart function looks like this: rpart(formula, data, weights, subset, na.action = na.rpart, method, model = FALSE, x = FALSE, y = TRUE, parms, control, cost, ...) The various parameters of the rpart function are described in the following table: Parameter Description formula This is the formula used for the prediction. data This is the data matrix. weights These are the optional weights to be applied. subset This is the optional subset of rows of data to be used. na.action This specifies the action to be taken when y, the target value, is missing. method This is the method to be used to interpret the data. It should be one of these: anova, poisson, class, or exp. If not specified, the algorithm decides based on the layout of the data. … These are the additional parameters to be used to control the behavior of the algorithm.  Let's create a subset as follows: > weather2 <- subset(weather,select=-c(RISK_MM)) > install.packages("rpart") >library(rpart) > model <- rpart(formula=RainTomorrow ~ .,data=weather2, method="class") > summary(model) Call: rpart(formula = RainTomorrow ~ ., data = weather2, method = "class") n= 366   CPn split       rel error     xerror   xstd 1 0.19696970     0 1.0000000 1.0000000 0.1114418 2 0.09090909      1 0.8030303 0.9696970 0.1101055 3 0.01515152     2 0.7121212 1.0151515 0.1120956 4 0.01000000     7 0.6363636 0.9090909 0.1073129   Variable importance Humidity3pm WindGustSpeed     Sunshine WindSpeed3pm       Temp3pm            24           14          12             8             6 Pressure3pm       MaxTemp       MinTemp   Pressure9am       Temp9am            6             5             4             4             4 Evaporation         Date   Humidity9am     Cloud3pm     Cloud9am             3             3             2             2             1      Rainfall            1 Node number 1: 366 observations,   complexity param=0.1969697 predicted class=No   expected loss=0.1803279 P(node) =1    class counts:   300   66    probabilities: 0.820 0.180 left son=2 (339 obs) right son=3 (27 obs) Primary splits:    Humidity3pm < 71.5   to the left, improve=18.31013, (0 missing)    Pressure3pm < 1011.9 to the right, improve=17.35280, (0 missing)    Cloud3pm   < 6.5     to the left, improve=16.14203, (0 missing)    Sunshine   < 6.45   to the right, improve=15.36364, (3 missing)    Pressure9am < 1016.35 to the right, improve=12.69048, (0 missing) Surrogate splits:    Sunshine < 0.45   to the right, agree=0.945, adj=0.259, (0 split) (many more)… As you can tell, the model is complicated. The summary shows the progression of the model development using more and more of the data to fine-tune the tree. We will be using the rpart.plot package to display the decision tree in a readable manner as follows: > library(rpart.plot) > fancyRpartPlot(model,main="Rain Tomorrow",sub="Chapter 12") This is the output of the fancyRpartPlot function Now, we can follow the logic of the decision tree easily. For example, if the humidity is over 72, we are predicting it will rain. Regression We can use a regression to predict our target value by producing a regression model from our predictor variables. We will be using the forest fire data from http://archive.ics.uci.edu. We will load the data and get the following summary: > forestfires <- read.csv("http://archive.ics.uci.edu/ml/machine-learning-databases/forest-fires/forestfires.csv") > summary(forestfires)        X               Y           month     day         FFMC     Min.   :1.000   Min.   :2.0   aug   :184   fri:85 Min.   :18.70 1st Qu.:3.000   1st Qu.:4.0   sep   :172   mon:74   1st Qu.:90.20 Median :4.000   Median :4.0   mar   : 54   sat:84   Median :91.60 Mean   :4.669   Mean   :4.3   jul   : 32   sun:95   Mean   :90.64 3rd Qu.:7.000   3rd Qu.:5.0  feb   : 20   thu:61   3rd Qu.:92.90 Max.   :9.000   Max.   :9.0   jun   : 17   tue:64   Max.   :96.20                                (Other): 38   wed:54                      DMC             DC             ISI             temp     Min.   : 1.1   Min.   : 7.9   Min.   : 0.000   Min.   : 2.20 1st Qu.: 68.6   1st Qu.:437.7   1st Qu.: 6.500   1st Qu.:15.50 Median :108.3   Median :664.2   Median : 8.400   Median :19.30 Mean   :110.9   Mean   :547.9   Mean   : 9.022   Mean   :18.89 3rd Qu.:142.4   3rd Qu.:713.9   3rd Qu.:10.800   3rd Qu.:22.80 Max.   :291.3   Max.   :860.6   Max.   :56.100   Max.   :33.30                                                                         RH             wind           rain             area       Min.   : 15.00   Min.   :0.400   Min.   :0.00000   Min.   :   0.00 1st Qu.: 33.00   1st Qu.:2.700   1st Qu.:0.00000   1st Qu.:   0.00 Median : 42.00   Median :4.000   Median :0.00000   Median :   0.52 Mean   : 44.29   Mean   :4.018   Mean   :0.02166   Mean   : 12.85 3rd Qu.: 53.00   3rd Qu.:4.900   3rd Qu.:0.00000   3rd Qu.:   6.57 Max.   :100.00   Max.   :9.400   Max.   :6.40000   Max.   :1090.84 I will just use the month, temperature, wind, and rain data to come up with a model of the area (size) of the fires using the lm function. The lm function looks like this: lm(formula, data, subset, weights, na.action,    method = "qr", model = TRUE, x = FALSE, y = FALSE, qr = TRUE,    singular.ok = TRUE, contrasts = NULL, offset, ...) The various parameters of the lm function are described in the following table: Parameter Description formula This is the formula to be used for the model data This is the dataset subset This is the subset of dataset to be used weights These are the weights to apply to factors … These are the additional parameters to be added to the function Let's load the data as follows: > model <- lm(formula = area ~ month + temp + wind + rain, data=forestfires) Looking at the generated model, we see the following output: > summary(model) Call: lm(formula = area ~ month + temp + wind + rain, data = forestfires) Residuals:    Min     1Q Median     3Q     Max -33.20 -14.93   -9.10   -1.66 1063.59 Coefficients:            Estimate Std. Error t value Pr(>|t|) (Intercept) -17.390     24.532 -0.709   0.4787 monthaug     -10.342     22.761 -0.454   0.6498 monthdec     11.534     30.896   0.373   0.7091 monthfeb       2.607     25.796   0.101   0.9196 monthjan       5.988     50.493   0.119   0.9056 monthjul     -8.822    25.068 -0.352   0.7251 monthjun     -15.469     26.974 -0.573   0.5666 monthmar     -6.630     23.057 -0.288   0.7738 monthmay       6.603     50.053   0.132   0.8951 monthnov     -8.244     67.451 -0.122   0.9028 monthoct     -8.268    27.237 -0.304   0.7616 monthsep     -1.070     22.488 -0.048   0.9621 temp           1.569     0.673   2.332   0.0201 * wind           1.581     1.711   0.924   0.3557 rain         -3.179     9.595 -0.331   0.7406 --- Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1   Residual standard error: 63.99 on 502 degrees of freedom Multiple R-squared: 0.01692, Adjusted R-squared: -0.0105 F-statistic: 0.617 on 14 and 502 DF, p-value: 0.8518 Surprisingly, the month has a significant effect on the size of the fires. I would have guessed that whether or not the fires occurred in August or similar months would have effected any discernable difference. Also, the temperature has such a minimal effect. Further, the model is using the month data as categorical. If we redevelop the model (without temperature), we have a better fit (notice the multiple R-squared value drops to 0.006 from 0.01), as shown here: > model <- lm(formula = area ~ month + wind + rain, data=forestfires) > summary(model)   Call: lm(formula = area ~ month + wind + rain, data = forestfires)   Residuals:    Min     1Q Median     3Q     Max -22.17 -14.39 -10.46   -3.87 1072.43   Coefficients:           Estimate Std. Error t value Pr(>|t|) (Intercept)   4.0126   22.8496   0.176   0.861 monthaug     4.3132   21.9724   0.196   0.844 monthdec     1.3259   30.7188   0.043   0.966 monthfeb     -1.6631   25.8441 -0.064   0.949 monthjan     -6.1034   50.4475 -0.121   0.904 monthjul     6.4648   24.3021   0.266   0.790 monthjun     -2.4944   26.5099 -0.094   0.925 monthmar     -4.8431   23.1458 -0.209   0.834 monthmay     10.5754   50.2441   0.210   0.833 monthnov     -8.7169   67.7479 -0.129   0.898 monthoct     -0.9917   27.1767 -0.036   0.971 monthsep     10.2110   22.0579   0.463   0.644 wind         1.0454     1.7026   0.614   0.540 rain         -1.8504     9.6207 -0.192   0.848   Residual standard error: 64.27 on 503 degrees of freedom Multiple R-squared: 0.006269, Adjusted R-squared: -0.01941 F-statistic: 0.2441 on 13 and 503 DF, p-value: 0.9971 From the results, we can see R-squared of close to 0 and p-value almost 1; this is a very good fit. If you plot the model, you will get a series of graphs. The plot of the residuals versus fitted values is the most revealing, as shown in the following graph: > plot(model) You can see from the graph that the regression model is very accurate: Neural network In a neural network, it is assumed that there is a complex relationship between the predictor variables and the target variable. The network allows the expression of each of these relationships. For this model, we will use the liver disorder data from http://archive.ics.uci.edu. The data has a few hundred observations from patients with liver disorders. The variables are various measures of blood for each patient as shown here: > bupa <- read.csv("http://archive.ics.uci.edu/ml/machine-learning-databases/liver-disorders/bupa.data") > colnames(bupa) <- c("mcv","alkphos","alamine","aspartate","glutamyl","drinks","selector") > summary(bupa)      mcv           alkphos         alamine     Min.   : 65.00   Min.   : 23.00   Min.   : 4.00 1st Qu.: 87.00   1st Qu.: 57.00   1st Qu.: 19.00 Median : 90.00   Median : 67.00   Median : 26.00 Mean   : 90.17   Mean   : 69.81   Mean   : 30.36 3rd Qu.: 93.00   3rd Qu.: 80.00   3rd Qu.: 34.00 Max.   :103.00   Max.   :138.00   Max.   :155.00    aspartate       glutamyl         drinks     Min.   : 5.00   Min.   : 5.00   Min.  : 0.000 1st Qu.:19.00   1st Qu.: 15.00   1st Qu.: 0.500 Median :23.00   Median : 24.50   Median : 3.000 Mean   :24.64   Mean   : 38.31   Mean   : 3.465 3rd Qu.:27.00   3rd Qu.: 46.25   3rd Qu.: 6.000 Max.   :82.00   Max.   :297.00   Max. :20.000    selector   Min.   :1.000 1st Qu.:1.000 Median :2.000 Mean   :1.581 3rd Qu.:2.000 Max.   :2.000 We generate a neural network using the neuralnet function. The neuralnet function looks like this: neuralnet(formula, data, hidden = 1, threshold = 0.01,                stepmax = 1e+05, rep = 1, startweights = NULL,          learningrate.limit = NULL,          learningrate.factor = list(minus = 0.5, plus = 1.2),          learningrate=NULL, lifesign = "none",          lifesign.step = 1000, algorithm = "rprop+",          err.fct = "sse", act.fct = "logistic",          linear.output = TRUE, exclude = NULL,          constant.weights = NULL, likelihood = FALSE) The various parameters of the neuralnet function are described in the following table: Parameter Description formula This is the formula to converge. data This is the data matrix of predictor values. hidden This is the number of hidden neurons in each layer. stepmax This is the maximum number of steps in each repetition. Default is 1+e5. rep This is the number of repetitions. Let's generate the neural network as follows: > nn <- neuralnet(selector~mcv+alkphos+alamine+aspartate+glutamyl+drinks, data=bupa, linear.output=FALSE, hidden=2) We can see how the model was developed via the result.matrix variable in the following output: > nn$result.matrix                                      1 error                 100.005904355153 reached.threshold       0.005904330743 steps                 43.000000000000 Intercept.to.1layhid1   0.880621509705 mcv.to.1layhid1       -0.496298308044 alkphos.to.1layhid1     2.294158313786 alamine.to.1layhid1     1.593035613921 aspartate.to.1layhid1 -0.407602506759 glutamyl.to.1layhid1   -0.257862634340 drinks.to.1layhid1     -0.421390527261 Intercept.to.1layhid2   0.806928998059 mcv.to.1layhid2       -0.531926150470 alkphos.to.1layhid2     0.554627946150 alamine.to.1layhid2     1.589755874579 aspartate.to.1layhid2 -0.182482440722 glutamyl.to.1layhid2   1.806513419058 drinks.to.1layhid2     0.215346602241 Intercept.to.selector   4.485455617018 1layhid.1.to.selector   3.328527160621 1layhid.2.to.selector   2.616395644587 The process took 43 steps to come up with the neural network once the threshold was under 0.01 (0.005 in this case). You can see the relationships between the predictor values. Looking at the network developed, we can see the hidden layers of relationship among the predictor variables. For example, sometimes mcv combines at one ratio and on other times at another ratio, depending on its value. Let's load the neural network as follows: > plot(nn) Instance-based learning R programming has a nearest neighbor algorithm (k-NN). The k-NN algorithm takes the predictor values and organizes them so that a new observation is applied to the organization developed and the algorithm selects the result (prediction) that is most applicable based on nearness of the predictor values in the new observation. The nearest neighbor function is knn. The knn function call looks like this: knn(train, test, cl, k = 1, l = 0, prob = FALSE, use.all = TRUE) The various parameters of the knn function are described in the following table: Parameter Description train This is the training data. test This is the test data. cl This is the factor of true classifications. k This is the Number of neighbors to consider. l This is the minimum vote for a decision. prob This is a Boolean flag to return proportion of winning votes. use.all This is a Boolean variable for tie handling. TRUE means use all votes of max distance I am using the auto MPG dataset in the example of using knn. First, we load the dataset : > data <- read.table("http://archive.ics.uci.edu/ml/machine-learning-databases/auto-mpg/auto-mpg.data", na.string="?") > colnames(data) <- c("mpg","cylinders","displacement","horsepower","weight","acceleration","model.year","origin","car.name") > summary(data)      mpg         cylinders     displacement     horsepower Min.   : 9.00  Min.   :3.000   Min.   : 68.0   150   : 22 1st Qu.:17.50   1st Qu.:4.000   1st Qu.:104.2   90     : 20 Median :23.00   Median :4.000   Median :148.5   88     : 19 Mean   :23.51   Mean   :5.455   Mean   :193.4   110   : 18 3rd Qu.:29.00   3rd Qu.:8.000   3rd Qu.:262.0   100   : 17 Max.   :46.60   Max.   :8.000   Max.   :455.0   75     : 14                                                  (Other):288      weight     acceleration     model.year       origin     Min.   :1613   Min. : 8.00   Min.   :70.00   Min.   :1.000 1st Qu.:2224   1st Qu.:13.82   1st Qu.:73.00   1st Qu.:1.000 Median :2804   Median :15.50   Median :76.00   Median :1.000 Mean   :2970   Mean   :15.57   Mean   :76.01   Mean   :1.573 3rd Qu.:3608   3rd Qu.:17.18   3rd Qu.:79.00   3rd Qu.:2.000 Max.   :5140   Max.   :24.80   Max.   :82.00   Max.   :3.000                                                                           car.name ford pinto   : 6 amc matador   : 5 ford maverick : 5 toyota corolla: 5 amc gremlin   : 4 amc hornet   : 4 (Other)       :369   There are close to 400 observations in the dataset. We need to split the data into a training set and a test set. We will use 75 percent for training. We use the createDataPartition function in the caret package to select the training rows. Then, we create a test dataset and a training dataset using the partitions as follows: > library(caret) > training <- createDataPartition(data$mpg, p=0.75, list=FALSE) > trainingData <- data[training,] > testData <- data[-training,] > model <- knn(train=trainingData, test=testData, cl=trainingData$mpg) NAs introduced by coercion The error message means that some numbers in the dataset have a bad format. The bad numbers were automatically converted to NA values. Then the inclusion of the NA values caused the function to fail, as NA values are not expected in this function call. First, there are some missing items in the dataset loaded. We need to eliminate those NA values as follows: > completedata <- data[complete.cases(data),] After looking over the data several times, I guessed that the car name fields were being parsed as numerical data when there was a number in the name, such as Buick Skylark 320. I removed the car name column from the test and we end up with the following valid results; > drops <- c("car.name") > completeData2 <- completedata[,!(names(completedata) %in% drops)] > training <- createDataPartition(completeData2$mpg, p=0.75, list=FALSE) > trainingData <- completeData2[training,] > testData <- completeData2[-training,] > model <- knn(train=trainingData, test=testData, cl=trainingData$mpg) We can see the results of the model by plotting using the following command. However, the graph doesn't give us much information to work on. > plot(model) We can use a different kknn function to compare our model with the test data. I like this version a little better as you can plainly specify the formula for the model. Let's use the kknn function as follows: > library(kknn) > model <- kknn(formula = formula(mpg~.), train = trainingData, test = testData, k = 3, distance = 1) > fit <- fitted(model) > plot(testData$mpg, fit) > abline(a=0, b=1, col=3) I added a simple slope to highlight how well the model fits the training data. It looks like as we progress to higher MPG values, our model has a higher degree of variance. I think that means we are missing predictor variables, especially for the later model, high MPG series of cars. That would make sense as government mandate and consumer demand for high efficiency vehicles changed the mpg for vehicles. Here is the graph generated by the previous code: Ensemble learning Ensemble learning is the process of using multiple learning methods to obtain better predictions. For example, we could use a regression and k-NN, combine the results, and end up with a better prediction. We could average the results of both or provide heavier weight towards one or another of the algorithms, whichever appears to be a better predictor. Support vector machines We covered support vector machines (SVM), but I will run through an example here. As a reminder, SVM is concerned with binary data. We will use the spam dataset from Hewlett Packard (part of the kernlab package). First, let's load the data as follows: > library(kernlab) > data("spam") > summary(spam)      make           address           all             num3d         Min.   :0.0000   Min.   : 0.000   Min.   :0.0000   Min.   : 0.00000 1st Qu.:0.0000   1st Qu.: 0.000   1st Qu.:0.0000   1st Qu.: 0.00000 Median :0.0000   Median : 0.000   Median :0.0000   Median : 0.00000 Mean   :0.1046   Mean   : 0.213   Mean   :0.2807   Mean   : 0.06542 3rd Qu.:0.0000   3rd Qu.: 0.000   3rd Qu.:0.4200   3rd Qu.: 0.00000 Max.   :4.5400   Max.   :14.280   Max.   :5.1000   Max.   :42.81000 … There are 58 variables with close to 5000 observations, as shown here: > table(spam$type) nonspam   spam    2788   1813 Now, we break up the data into a training set and a test set as follows: > index <- 1:nrow(spam) > testindex <- sample(index, trunc(length(index)/3)) > testset <- spam[testindex,] > trainingset <- spam[-testindex,] Now, we can produce our SVM model using the svm function. The svm function looks like this: svm(formula, data = NULL, ..., subset, na.action =na.omit, scale = TRUE) The various parameters of the svm function are described in the following table: Parameter Description formula This is the formula model data This is the dataset subset This is the subset of the dataset to be used na.action This contains what action to take with NA values scale This determines whether to scale the data Let's use the svm function to produce a SVM model as follows: > library(e1071) > model <- svm(type ~ ., data = trainingset, method = "C-classification", kernel = "radial", cost = 10, gamma = 0.1) > summary(model) Call: svm(formula = type ~ ., data = trainingset, method = "C-classification",    kernel = "radial", cost = 10, gamma = 0.1) Parameters:    SVM-Type: C-classification SVM-Kernel: radial        cost: 10      gamma: 0.1 Number of Support Vectors: 1555 ( 645 910 ) Number of Classes: 2 Levels: nonspam spam We can test the model against our test dataset and look at the results as follows: > pred <- predict(model, testset) > table(pred, testset$type) pred     nonspam spam nonspam     891 104 spam         38 500 Note, the e1071 package is not compatible with the current version of R. Given its usefulness I would expect the package to be updated to support the user base. So, using SVM, we have a 90 percent ((891+500) / (891+104+38+500)) accuracy rate of prediction. Bayesian learning With Bayesian learning, we have an initial premise in a model that is adjusted with new information. We can use the MCMCregress method in the MCMCpack package to use Bayesian regression on learning data and apply the model against test data. Let's load the MCMCpack package as follows: > install.packages("MCMCpack") > library(MCMCpack) We are going to be using the transplant data on transplants available at http://lib.stat.cmu.edu/datasets/stanford. (The dataset on the site is part of the web page, so I copied into a local CSV file.) The data shows expected transplant success factor, the actual transplant success factor, and the number of transplants over a time period. So, there is a good progression over time as to the success of the program. We can read the dataset as follows: > transplants <- read.csv("transplant.csv") > summary(transplants)    expected         actual       transplants   Min.   : 0.057   Min.   : 0.000   Min.   : 1.00 1st Qu.: 0.722   1st Qu.: 0.500   1st Qu.: 9.00 Median : 1.654   Median : 2.000   Median : 18.00 Mean   : 2.379   Mean   : 2.382   Mean   : 27.83 3rd Qu.: 3.402   3rd Qu.: 3.000   3rd Qu.: 40.00 Max.   :12.131   Max.   :18.000   Max.   :152.00 We use Bayesian regression against the data— note that we are modifying the model as we progress with new information using the MCMCregress function. The MCMCregress function looks like this: MCMCregress(formula, data = NULL, burnin = 1000, mcmc = 10000,    thin = 1, verbose = 0, seed = NA, beta.start = NA,    b0 = 0, B0 = 0, c0 = 0.001, d0 = 0.001, sigma.mu = NA, sigma.var = NA,    marginal.likelihood = c("none", "Laplace", "Chib95"), ...) The various parameters of the MCMCregress function are described in the following table: Parameter Description formula This is the formula of model data This is the dataset to be used for model … These are the additional parameters for the function Let's use the Bayesian regression against the data as follows: > model <- MCMCregress(expected ~ actual + transplants, data=transplants) > summary(model) Iterations = 1001:11000 Thinning interval = 1 Number of chains = 1 Sample size per chain = 10000 1. Empirical mean and standard deviation for each variable,    plus standard error of the mean:                Mean     SD Naive SE Time-series SE (Intercept) 0.00484 0.08394 0.0008394     0.0008388 actual     0.03413 0.03214 0.0003214     0.0003214 transplants 0.08238 0.00336 0.0000336     0.0000336 sigma2     0.44583 0.05698 0.0005698     0.0005857 2. Quantiles for each variable:                2.5%     25%     50%     75%   97.5% (Intercept) -0.15666 -0.05216 0.004786 0.06092 0.16939 actual     -0.02841 0.01257 0.034432 0.05541 0.09706 transplants 0.07574 0.08012 0.082393 0.08464 0.08890 sigma2       0.34777 0.40543 0.441132 0.48005 0.57228 The plot of the data shows the range of results, as shown in the following graph. Look at this in contrast to a simple regression with one result. > plot(model) Random forests Random forests is an algorithm that constructs a multitude of decision trees for the model of the data and selects the best of the lot as the final result. We can use the randomForest function in the kernlab package for this function. The randomForest function looks like this: randomForest(formula, data=NULL, ..., subset, na.action=na.fail) The various parameters of the randomForest function are described in the following table: Parameter Description formula This is the formula of model data This is the dataset to be used subset This is the subset of the dataset to be used na.action This is the action to take with NA values For an example of random forest, we will use the spam data, as in the section Support vector machines. First, let's load the package and library as follows: > install.packages("randomForest") > library(randomForest) Now, we will generate the model with the following command (this may take a while): > fit <- randomForest(type ~ ., data=spam) Let's look at the results to see how it went: > fit Call: randomForest(formula = type ~ ., data = spam)                Type of random forest: classification                      Number of trees: 500 No. of variables tried at each split: 7        OOB estimate of error rate: 4.48% Confusion matrix:         nonspam spam class.error nonspam   2713   75 0.02690100 spam       131 1682 0.07225593 We can look at the relative importance of the data variables in the final model, as shown here: > head(importance(fit))        MeanDecreaseGini make           7.967392 address       12.654775 all           25.116662 num3d           1.729008 our           67.365754 over           17.579765 Ordering the data shows a couple of the factors to be critical to the determination. For example, the presence of the exclamation character in the e-mail is shown as a dominant indicator of spam mail: charExclamation   256.584207 charDollar       200.3655348 remove           168.7962949 free              142.8084662 capitalAve       137.1152451 capitalLong       120.1520829 your             116.6134519 Unsupervised learning With unsupervised learning, we do not have a target variable. We have a number of predictor variables that we look into to determine if there is a pattern. We will go over the following unsupervised learning techniques: Cluster analysis Density estimation Expectation-maximization algorithm Hidden Markov models Blind signal separation Cluster analysis Cluster analysis is the process of organizing data into groups (clusters) that are similar to each other. For our example, we will use the wheat seed data available at http://www.uci.edu, as shown here: > wheat <- read.csv("http://archive.ics.uci.edu/ml/machine-learning-databases/00236/seeds_dataset.txt", sep="t") Let's look at the raw data: > head(wheat) X15.26 X14.84 X0.871 X5.763 X3.312 X2.221 X5.22 X1 1 14.88 14.57 0.8811 5.554 3.333 1.018 4.956 1 2 14.29 14.09 0.9050 5.291 3.337 2.699 4.825 1 3 13.84 13.94 0.8955 5.324 3.379 2.259 4.805 1 4 16.14 14.99 0.9034 5.658 3.562 1.355 5.175 1 5 14.38 14.21 0.8951 5.386 3.312 2.462 4.956 1 6 14.69 14.49 0.8799 5.563 3.259 3.586 5.219 1 We need to apply column names so we can see the data better: > colnames(wheat) <- c("area", "perimeter", "compactness", "length", "width", "asymmetry", "groove", "undefined") > head(wheat)    area perimeter compactness length width asymmetry groove undefined 1 14.88     14.57     0.8811 5.554 3.333     1.018 4.956         1 2 14.29     14.09     0.9050 5.291 3.337     2.699 4.825         1 3 13.84     13.94     0.8955 5.324 3.379     2.259 4.805         1 4 16.14     14.99     0.9034 5.658 3.562     1.355 5.175         1 5 14.38     14.21     0.8951 5.386 3.312     2.462 4.956         1 6 14.69     14.49     0.8799 5.563 3.259     3.586 5.219         1 The last column is not defined in the data description, so I am removing it: > wheat <- subset(wheat, select = -c(undefined) ) > head(wheat)    area perimeter compactness length width asymmetry groove 1 14.88     14.57     0.8811 5.554 3.333     1.018 4.956 2 14.29     14.09     0.9050 5.291 3.337     2.699 4.825 3 13.84     13.94     0.8955 5.324 3.379     2.259 4.805 4 16.14     14.99     0.9034 5.658 3.562     1.355 5.175 5 14.38     14.21     0.8951 5.386 3.312     2.462 4.956 6 14.69    14.49     0.8799 5.563 3.259     3.586 5.219 Now, we can finally produce the cluster using the kmeans function. The kmeans function looks like this: kmeans(x, centers, iter.max = 10, nstart = 1,        algorithm = c("Hartigan-Wong", "Lloyd", "Forgy",                      "MacQueen"), trace=FALSE) The various parameters of the kmeans function are described in the following table: Parameter Description x This is the dataset centers This is the number of centers to coerce data towards … These are the additional parameters of the function Let's produce the cluster using the kmeans function: > fit <- kmeans(wheat, 5) Error in do_one(nmeth) : NA/NaN/Inf in foreign function call (arg 1) Unfortunately, there are some rows with missing data, so let's fix this using the following command: > wheat <- wheat[complete.cases(wheat),] Let's look at the data to get some idea of the factors using the following command: > plot(wheat) If we try looking at five clusters, we end up with a fairly good set of clusters with an 85 percent fit, as shown here: > fit <- kmeans(wheat, 5) > fit K-means clustering with 5 clusters of sizes 29, 33, 56, 69, 15 Cluster means:      area perimeter compactness   length   width asymmetry   groove 1 16.45345 15.35310   0.8768000 5.882655 3.462517 3.913207 5.707655 2 18.95455 16.38879   0.8868000 6.247485 3.744697 2.723545 6.119455 3 14.10536 14.20143   0.8777750 5.480214 3.210554 2.368075 5.070000 4 11.94870 13.27000   0.8516652 5.229304 2.870101 4.910145 5.093333 5 19.58333 16.64600   0.8877267 6.315867 3.835067 5.081533 6.144400 Clustering vector: ... Within cluster sum of squares by cluster: [1] 48.36785 30.16164 121.63840 160.96148 25.81297 (between_SS / total_SS = 85.4 %) If we push to 10 clusters, the performance increases to 92 percent. Density estimation Density estimation is used to provide an estimate of the probability density function of a random variable. For this example, we will use sunspot data from Vincent arlbuck site. Not clear if sunspots are truly random. Let's load our data as follows: > sunspots <- read.csv("http://vincentarelbundock.github.io/Rdatasets/csv/datasets/sunspot.month.csv") > summary(sunspots)        X             time     sunspot.month   Min.   :   1   Min.   :1749   Min.   : 0.00 1st Qu.: 795   1st Qu.:1815   1st Qu.: 15.70 Median :1589   Median :1881   Median : 42.00 Mean   :1589   Mean   :1881   Mean   : 51.96 3rd Qu.:2383   3rd Qu.:1948   3rd Qu.: 76.40 Max.   :3177   Max.   :2014   Max.   :253.80 > head(sunspots) X     time sunspot.month 1 1 1749.000         58.0 2 2 1749.083         62.6 3 3 1749.167         70.0 4 4 1749.250         55.7 5 5 1749.333         85.0 6 6 1749.417        83.5 We will now estimate the density using the following command: > d <- density(sunspots$sunspot.month) > d Call: density.default(x = sunspots$sunspot.month) Data: sunspots$sunspot.month (3177 obs.); Bandwidth 'bw' = 7.916        x               y           Min.   :-23.75   Min.   :1.810e-07 1st Qu.: 51.58   1st Qu.:1.586e-04 Median :126.90   Median :1.635e-03 Mean   :126.90   Mean   :3.316e-03 3rd Qu.:202.22   3rd Qu.:5.714e-03 Max.   :277.55   Max.   :1.248e-02 A plot is very useful for this function, so let's generate one using the following command: > plot(d) It is interesting to see such a wide variation; maybe the data is pretty random after all. We can use the density to estimate additional periods as follows: > N<-1000 > sunspots.new <- rnorm(N, sample(sunspots$sunspot.month, size=N, replace=TRUE)) > lines(density(sunspots.new), col="blue") It looks like our density estimate is very accurate. Expectation-maximization Expectation-maximization (EM) is an unsupervised clustering approach that adjusts the data for optimal values. When using EM, we have to have some preconception of the shape of the data/model that will be targeted. This example reiterates the example on the Wikipedia page, with comments. The example tries to model the iris species from the other data points. Let's load the data as shown here: > iris <- read.csv("https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data") > colnames(iris) <- c("SepalLength","SepalWidth","PetalLength","PetalWidth","Species") > modelName = "EEE" Each observation has sepal length, width, petal length, width, and species, as shown here: > head(iris) SepalLength SepalWidth PetalLength PetalWidth     Species 1         5.1       3.5         1.4       0.2 Iris-setosa 2         4.9       3.0         1.4       0.2 Iris-setosa 3         4.7       3.2         1.3       0.2 Iris-setosa 4         4.6       3.1         1.5       0.2 Iris-setosa 5         5.0       3.6         1.4       0.2 Iris-setosa 6         5.4       3.9         1.7       0.4 Iris-setosa We are estimating the species from the other points, so let's separate the data as follows: > data = iris[,-5] > z = unmap(iris[,5]) Let's set up our mstep for EM, given the data, categorical data (z) relating to each data point, and our model type name: > msEst <- mstep(modelName, data, z) We use the parameters defined in the mstep to produce our model, as shown here: > em(modelName, data, msEst$parameters) $z                [,1]         [,2]         [,3] [1,] 1.000000e+00 4.304299e-22 1.699870e-42 … [150,] 8.611281e-34 9.361398e-03 9.906386e-01 $parameters$pro [1] 0.3333333 0.3294048 0.3372619 $parameters$mean              [,1]     [,2]     [,3] SepalLength 5.006 5.941844 6.574697 SepalWidth 3.418 2.761270 2.980150 PetalLength 1.464 4.257977 5.538926 PetalWidth 0.244 1.319109 2.024576 $parameters$variance$d [1] 4 $parameters$variance$G [1] 3 $parameters$variance$sigma , , 1            SepalLength SepalWidth PetalLength PetalWidth SepalLength 0.26381739 0.09030470 0.16940062 0.03937152 SepalWidth   0.09030470 0.11251902 0.05133876 0.03082280 PetalLength 0.16940062 0.05133876 0.18624355 0.04183377 PetalWidth   0.03937152 0.03082280 0.04183377 0.03990165 , , 2 , , 3 … (there was little difference in the 3 sigma values) Covariance $parameters$variance$Sigma            SepalLength SepalWidth PetalLength PetalWidth SepalLength 0.26381739 0.09030470 0.16940062 0.03937152 SepalWidth   0.09030470 0.11251902 0.05133876 0.03082280 PetalLength 0.16940062 0.05133876 0.18624355 0.04183377 PetalWidth   0.03937152 0.03082280 0.04183377 0.03990165 $parameters$variance$cholSigma             SepalLength SepalWidth PetalLength PetalWidth SepalLength -0.5136316 -0.1758161 -0.32980960 -0.07665323 SepalWidth   0.0000000 0.2856706 -0.02326832 0.06072001 PetalLength   0.0000000 0.0000000 -0.27735855 -0.06477412 PetalWidth   0.0000000 0.0000000 0.00000000 0.16168899 attr(,"info") iterations       error 4.000000e+00 1.525131e-06 There is quite a lot of output from the em function. The highlights for me were the three sigma ranges were the same and the error from the function was very small. So, I think we have a very good estimation of species using just the four data points. Hidden Markov models The hidden Markov models (HMM) is the idea of observing data assuming it has been produced by a Markov model. The problem is to discover what that model is. I am using the Python example on Wikipedia for HMM. For an HMM, we need states (assumed to be hidden from observer), symbols, transition matrix between states, emission (output) states, and probabilities for all. The Python information presented is as follows: states = ('Rainy', 'Sunny') observations = ('walk', 'shop', 'clean') start_probability = {'Rainy': 0.6, 'Sunny': 0.4} transition_probability = {    'Rainy' : {'Rainy': 0.7, 'Sunny': 0.3},    'Sunny' : {'Rainy': 0.4, 'Sunny': 0.6},    } emission_probability = {    'Rainy' : {'walk': 0.1, 'shop': 0.4, 'clean': 0.5},    'Sunny' : {'walk': 0.6, 'shop': 0.3, 'clean': 0.1},    } trans <- matrix(c('Rainy', : {'Rainy': 0.7, 'Sunny': 0.3},    'Sunny' : {'Rainy': 0.4, 'Sunny': 0.6},    } We convert these to use in R for the initHmm function by using the following command: > hmm <- initHMM(c("Rainy","Sunny"), c('walk', 'shop', 'clean'), c(.6,.4), matrix(c(.7,.3,.4,.6),2), matrix(c(.1,.4,.5,.6,.3,.1),3)) > hmm $States [1] "Rainy" "Sunny" $Symbols [1] "walk" "shop" "clean" $startProbs Rainy Sunny 0.6   0.4 $transProbs        to from   Rainy Sunny Rainy   0.7   0.4 Sunny   0.3   0.6 $emissionProbs        symbols states walk shop clean Rainy 0.1 0.5   0.3 Sunny 0.4 0.6   0.1 The model is really a placeholder for all of the setup information needed for HMM. We can then use the model to predict based on observations, as follows: > future <- forward(hmm, c("walk","shop","clean")) > future        index states         1         2         3 Rainy -2.813411 -3.101093 -4.139551 Sunny -1.832581 -2.631089 -5.096193 The result is a matrix of probabilities. For example, it is more likely to be Sunny when we observe walk. Blind signal separation Blind signal separation is the process of identifying sources of signals from a mixed signal. Primary component analysis is one method of doing this. An example is a cocktail party where you are trying to listen to one speaker. For this example, I am using the decathlon dataset in the FactoMineR package, as shown here: > library(FactoMineR) > data(decathlon) Let's look at the data to get some idea of what is available: > summary(decathlon) 100m           Long.jump     Shot.put       High.jump Min.   :10.44   Min.   :6.61   Min.   :12.68   Min.   :1.850 1st Qu.:10.85   1st Qu.:7.03   1st Qu.:13.88   1st Qu.:1.920 Median :10.98   Median :7.30   Median :14.57   Median :1.950 Mean   :11.00   Mean   :7.26   Mean   :14.48   Mean   :1.977 3rd Qu.:11.14   3rd Qu.:7.48   3rd Qu.:14.97   3rd Qu.:2.040 Max.   :11.64   Max.   :7.96   Max.   :16.36   Max.   :2.150 400m           110m.hurdle       Discus       Pole.vault   Min.   :46.81   Min.   :13.97   Min.   :37.92   Min.   :4.200 1st Qu.:48.93   1st Qu.:14.21   1st Qu.:41.90   1st Qu.:4.500 Median :49.40   Median :14.48   Median :44.41   Median :4.800 Mean   :49.62   Mean   :14.61 Mean   :44.33   Mean   :4.762 3rd Qu.:50.30   3rd Qu.:14.98   3rd Qu.:46.07   3rd Qu.:4.920 Max.   :53.20   Max.   :15.67   Max.   :51.65   Max.   :5.400 Javeline       1500m           Rank           Points   Min.   :50.31   Min.   :262.1   Min.   : 1.00   Min.   :7313 1st Qu.:55.27   1st Qu.:271.0   1st Qu.: 6.00   1st Qu.:7802 Median :58.36   Median :278.1   Median :11.00   Median :8021 Mean   :58.32   Mean   :279.0   Mean   :12.12   Mean   :8005 3rd Qu.:60.89   3rd Qu.:285.1   3rd Qu.:18.00   3rd Qu.:8122 Max.   :70.52   Max.   :317.0   Max.   :28.00   Max.   :8893    Competition Decastar:13 OlympicG:28 The output looks like performance data from a series of events at a track meet: > head(decathlon)        100m   Long.jump Shot.put High.jump 400m 110m.hurdle Discus SEBRLE 11.04     7.58   14.83     2.07 49.81       14.69 43.75 CLAY   10.76     7.40   14.26     1.86 49.37       14.05 50.72 KARPOV 11.02     7.30   14.77     2.04 48.37       14.09 48.95 BERNARD 11.02     7.23   14.25     1.92 48.93       14.99 40.87 YURKOV 11.34     7.09   15.19     2.10 50.42       15.31 46.26 WARNERS 11.11     7.60   14.31     1.98 48.68       14.23 41.10        Pole.vault Javeline 1500m Rank Points Competition SEBRLE       5.02   63.19 291.7   1   8217   Decastar CLAY         4.92   60.15 301.5   2   8122   Decastar KARPOV       4.92   50.31 300.2   3   8099   Decastar BERNARD       5.32   62.77 280.1   4   8067   Decastar YURKOV       4.72   63.44 276.4   5   8036   Decastar WARNERS       4.92   51.77 278.1   6   8030   Decastar Further, this is performance of specific individuals in track meets. We run the PCA function by passing the dataset to use, whether to scale the data or not, and the type of graphs: > res.pca = PCA(decathlon[,1:10], scale.unit=TRUE, ncp=5, graph=T) This produces two graphs: Individual factors map Variables factor map The individual factors map lays out the performance of the individuals. For example, we see Karpov who is high in both dimensions versus Bourginon who is performing badly (on the left in the following chart): The variables factor map shows the correlation of performance between events. For example, doing well in the 400 meters run is negatively correlated with the performance in the long jump; if you did well in one, you likely did well in the other as well. Here is the variables factor map of our data: Questions Factual Which supervised learning technique(s) do you lean towards as your "go to" solution? Why are the density plots for Bayesian results off-center? When, how, and why? How would you decide on the number of clusters to use? Find a good rule of thumb to decide the number of hidden layers in a neural net. Challenges Investigate other blind signal separation techniques, such as ICA. Use other methods, such as poisson, in the rpart function (especially if you have a natural occurring dataset). Summary In this article, we looked into various methods of machine learning, including both supervised and unsupervised learning. With supervised learning, we have a target variable we are trying to estimate. With unsupervised, we only have a possible set of predictor variables and are looking for patterns. In supervised learning, we looked into using a number of methods, including decision trees, regression, neural networks, support vector machines, and Bayesian learning. In unsupervised learning, we used cluster analysis, density estimation, hidden Markov models, and blind signal separation. Resources for Article: Further resources on this subject: Machine Learning in Bioinformatics [article] Data visualization [article] Introduction to S4 Classes [article]

In this article by Curtis Bennett and Dan Violet Sagmiller, authors of the book Unity AI Programming Essentials, we will learn about navigation meshes in Unity. Navigation mesh generation controls how AI characters are able to travel around a game level and is one of the most important topics in game AI. In this article, we will provide an overview of navigation meshes and look at the algorithm for generating them. Then, we'll look at different options of customizing our navigation meshes better. To do this, we will be using RAIN 2.1.5, a popular AI plugin for Unity by Rival Theory, available for free at http://rivaltheory.com/rain/download/. In this article, you will learn about: How navigation mesh generation works and the algorithm behind it Advanced options for customizing navigation meshes Creating advanced navigation meshes with RAIN (For more resources related to this topic, see here.) An overview of a navigation mesh To use navigation meshes, also referred to as NavMeshes, effectively the first things we need to know are what exactly navigation meshes are and how they are created. A navigation mesh is a definition of the area an AI character could travel to in a level. It is a mesh, but it is not intended to be rendered or seen by the player, instead it is used by the AI system. A NavMesh usually does not cover all the area in a level (if it did we wouldn't need one) since it's just the area a character can walk. The mesh is also almost always a simplified version of the geometry. For instance, you could have a cave floor in a game with thousands of polygons along the bottom showing different details in the rock, but for the navigation mesh the areas would just be a handful of very large polys giving a simplified view of the level. The purpose of navigation mesh is to provide this simplified representation to the rest of the AI system a way to find a path between two points on a level for a character. This is its purpose; let's discuss how they are created. It used to be a common practice in the games industry to create navigation meshes manually. A designer or artist would take the completed level geometry and create one using standard polygon mesh modelling tools and save it out. As you might imagine, this allowed for nice, custom, efficient meshes, but was also a big time sink, since every time the level changed the navigation mesh would need to be manually edited and updated. In recent years, there has been more research in automatic navigation mesh generation. There are many approaches to automatic navigation mesh generation, but the most popular is Recast, originally developed and designed by Mikko Monomen. Recast takes in level geometry and a set of parameters defining the character, such as the size of the character and how big of steps it can take, and then does a multipass approach to filter and create the final NavMesh. The most important phase of this is voxelizing the level based on an inputted cell size. This means the level geometry is divided into voxels (cubes) creating a version of the level geometry where everything is partitioned into different boxes called cells. Then the geometry in each of these cells is analyzed and simplified based on its intersection with the sides of the boxes and is culled based on things such as the slope of the geometry or how big a step height is between geometry. This simplified geometry is then merged and triangulated to make a final navigation mesh that can be used by the AI system. The source code and more information on the original C++ implementation of Recast is available at https://github.com/memononen/recastnavigation. Advanced NavMesh parameters Now that we understand how navigation mesh generations works, let's look at the different parameters you can set to generate them in more detail. We'll look at how to do these with RAIN: Open Unity and create a new scene and a floor and some blocks for walls. Download RAIN from http://rivaltheory.com/rain/download/ and import it into your scene. Then go to RAIN | Create Navigation Mesh. Also right-click on the RAIN menu and choose Show Advanced Settings. The setup should look something like the following screenshot: Now let's look at some of the important parameters: Size: This is the overall size of the navigation mesh. You'll want the navigation mesh to cover your entire level and use this parameter instead of trying to scale up the navigation mesh through the Scale transform in the Inspector window. For our demo here, set the Size parameter to 20. Walkable Radius: This is an important parameter to define the character size of the mesh. Remember, each mesh will be matched to the size of a particular character, and this is the radius of the character. You can visualize the radius for a character by adding a Unity Sphere Collider script to your object (by going to Component | Physics | Sphere Collider) and adjusting the radius of the collider. Cell Size: This is also a very important parameter. During the voxel step of the Recast algorithm, this sets the size of the cubes to inspect the geometry. The smaller the size, the more detailed and finer mesh, but longer the processing time for Recast. A large cell size makes computation fast but loses detail. For example, here is a NavMesh from our demo with a cell size of 0.01: You can see the finer detail here. Here is the navigation mesh generated with a cell size of 0.1: Note the difference between the two screenshots. In the former, walking through the two walls lower down in our picture is possible, but in the latter with a larger cell size, there is no path even though the character radius is the same. Problems like this become greater with larger cell sizes. The following is a navigation mesh with a cell size of 1: As you can see, the detail becomes jumbled and the mesh itself becomes unusable. With such differing results, the big question is how large should a cell size be for a level? The answer is that it depends on the required result. However, one important consideration is that as the processing time to generate one is done during development and not at runtime even if it takes several minutes to generate a good mesh, it can be worth it to get a good result in the game. Setting a small cell size on a large level can cause mesh processing to take a significant amount of time and consume a lot of memory. It is a good practice to save the scene before attempting to generate a complex navigation mesh. The Size, Walkable Radius, and Cell Size parameters are the most important parameters when generating the navigation mesh, but there are more that are used to customize the mesh further: Max Slope: This is the largest slope that a character can walk on. This is how much a piece of geometry that is tilted can still be walked on. If you take the wall and rotate it, you can see it is walkable: The preceding is a screenshot of a walkable object with slope. Step Height: This is how high a character can step from one object to another. For example, if you have steps between two blocks, as shown in the following screenshot, this would define how far in height the blocks can be apart and whether the area is still considered walkable: This is a screenshot of the navigation mesh with step height set to connect adjacent blocks. Walkable Height: This is the vertical height that is needed for the character to walk. For example, in the previous illustration, the second block is not walkable underneath because of the walkable height. If you raise it to a least one unit off the ground and set the walkable height to 1, the area underneath would become walkable:   You can see a screenshot of the navigation mesh with walkable height set to allow going under the higher block. These are the most important parameters. There are some other parameters related to the visualization and to cull objects. We will look at culling more in the next section. Culling areas Being able to set up areas as walkable or not is an important part of creating a level. To demo this, let's divide the level into two parts and create a bridge between the two. Take our demo and duplicate the floor and pull it down. Then transform one of the walls to a bridge. Then, add two other pieces of geometry to mark areas that are dangerous to walk on, like lava. Here is an example setup: This is a basic scene with a bridge to cross. If you recreate the navigation mesh now, all of the geometry will be covered and the bridge won't be recognized. To fix this, you can create a new tag called Lava and tag the geometry under the bridge with it. Then, in the navigation meshes' RAIN component, add Lava to the unwalkable tags. If you then regenerate the mesh, only the bridge is walkable. This is a screenshot of a navigation mesh areas under bridge culled: Using layers and the walkable tag you can customize navigation meshes. Summary Navigation meshes are an important part of game AI. In this article, we looked at the different parameters to customize navigation meshes. We looked at things such as setting the character size and walkable slopes and discussed the importance of the cell size parameter. We then saw how to customize our mesh by tagging different areas as not walkable. This should be a good start for designing navigation meshes for your games. Resources for Article: Further resources on this subject: Components in Unity [article] Enemy and Friendly AIs [article] Introduction to AI [article]

In this article by C. Y. Kan, author of the book Cassandra Data Modeling and Analysis, we will see the techniques of how to evolve an existing Cassandra data model in detail. Meanwhile, the techniques of modeling by query will be demonstrated as well. (For more resources related to this topic, see here.) The Stock Screener Application is good enough to retrieve and analyze a single stock at one time. However, scanning just a single stock looks very limited in practical use. A slight improvement can be made here; it can handle a bunch of stocks instead of one. This bunch of stocks will be stored as Watch List in the Cassandra database. Accordingly, the Stock Screener Application will be modified to analyze the stocks in the Watch List, and therefore it will produce alerts for each of the stocks being watched based on the same screening rule. For the produced alerts, saving them in Cassandra will be beneficial for backtesting trading strategies and continuous improvement of the Stock Screener Application. They can be reviewed from time to time without having to review them on the fly. Backtesting is a jargon used to refer to testing a trading strategy, investment strategy, or a predictive model using existing historical data. It is also a special type of cross-validation applied to time series data. In addition, when the number of the stocks in the Watch List grows to a few hundred, it will be difficult for a user of the Stock Screener Application to recall what the stocks are by simply referring to their stock codes. Hence, it would be nice to have the name of the stocks added to the produced alerts to make them more descriptive and user-friendly. Finally, we might have an interest in finding out how many alerts were generated on a particular stock over a specified period of time and how many alerts were generated on a particular date. We will use CQL to write queries to answer these two questions. By doing so, the modeling by query technique can be demonstrated. The enhancement approach The enhancement approach consists of four change requests in total. First, we will conduct changes in the data model and then the code will be enhanced to provide the new features. Afterwards, we will test run the enhanced Stock Screener Application again. The parts of the Stock Screener Application that require modifications are highlighted in the following figure. It is remarkable that two new components are added to the Stock Screener Application. The first component, Watch List, governs Data Mapper and Archiver to collect stock quote data of those stocks in the Watch List from Yahoo! Finance. The second component is Query. It provides two Queries on Alert List for backtesting purposes: Watch List Watch List is a very simple table that merely stores the stock code of its constituents. It is rather intuitive for a relational database developer to define the stock code as the primary key, isn't it? Nevertheless, remember that in Cassandra, the primary key is used to determine the node that stores the row. As Watch List is expected to not be a very long list, it would be more appropriate to put all of its rows on the same node for faster retrieval. But how can we do that? We can create an additional column, say watch_list_code, for this particular purpose. The new table is called watchlist and will be created in the packtcdma keyspace. The CQL statement is shown in chapter06_001.py: # -*- coding: utf-8 -*- # program: chapter06_001.py ## import Cassandra driver library from cassandra.cluster import Cluster ## function to create watchlist def create_watchlist(ss):    ## create watchlist table if not exists    ss.execute('CREATE TABLE IF NOT EXISTS watchlist (' +                'watch_list_code varchar,' +                'symbol varchar,' +                'PRIMARY KEY (watch_list_code, symbol))')       ## insert AAPL, AMZN, and GS into watchlist    ss.execute("INSERT INTO watchlist (watch_list_code, " +                "symbol) VALUES ('WS01', 'AAPL')")    ss.execute("INSERT INTO watchlist (watch_list_code, " +                "symbol) VALUES ('WS01', 'AMZN')")    ss.execute("INSERT INTO watchlist (watch_list_code, " +                "symbol) VALUES ('WS01', 'GS')") ## create Cassandra instance cluster = Cluster() ## establish Cassandra connection, using local default session = cluster.connect() ## use packtcdma keyspace session.set_keyspace('packtcdma') ## create watchlist table create_watchlist(session) ## close Cassandra connection cluster.shutdown() The create_watchlist function creates the table. Note that the watchlist table has a compound primary key made of watch_list_code and symbol. A Watch List called WS01 is also created, which contains three stocks, AAPL, AMZN, and GS. Alert List It is produced by a Python program and enumerates the date when the close price was above its 10-day SMA, that is, the signal and the close price at that time. Note that there were no stock code and stock name. We will create a table called alertlist to store the alerts with the code and name of the stock. The inclusion of the stock name is to meet the requirement of making the Stock Screener Application more user-friendly. Also, remember that joins are not allowed and denormalization is really the best practice in Cassandra. This means that we do not mind repeatedly storing (duplicating) the stock name in the tables that will be queried. A rule of thumb is one table for one query; as simple as that. The alertlist table is created by the CQL statement, as shown in chapter06_002.py: # -*- coding: utf-8 -*- # program: chapter06_002.py ## import Cassandra driver library from cassandra.cluster import Cluster ## function to create alertlist def create_alertlist(ss):    ## execute CQL statement to create alertlist table if not exists    ss.execute('CREATE TABLE IF NOT EXISTS alertlist (' +                'symbol varchar,' +                'price_time timestamp,' +                'stock_name varchar,' +                'signal_price float,' +                'PRIMARY KEY (symbol, price_time))') ## create Cassandra instance cluster = Cluster() ## establish Cassandra connection, using local default session = cluster.connect() ## use packtcdma keyspace session.set_keyspace('packtcdma') ## create alertlist table create_alertlist(session) ## close Cassandra connection cluster.shutdown() The primary key is also a compound primary key that consists of symbol and price_time. Adding the descriptive stock name Until now, the packtcdma keyspace has three tables, which are alertlist, quote, and watchlist. To add the descriptive stock name, one can think of only adding a column of stock name to alertlist only. As seen in the previous section, this has been done. So, do we need to add a column for quote and watchlist? It is, in fact, a design decision that depends on whether these two tables will be serving user queries. What a user query means is that the table will be used to retrieve rows for a query raised by a user. If a user wants to know the close price of Apple Inc. on June 30, 2014, it is a user query. On the other hand, if the Stock Screener Application uses a query to retrieve rows for its internal processing, it is not a user query. Therefore, if we want quote and watchlist to return rows for user queries, they need the stock name column; otherwise, they do not need it. The watchlist table is only for internal use by the current design, and so it need not have the stock name column. Of course, if in future, the Stock Screener Application allows a user to maintain Watch List, the stock name should also be added to the watchlist table. However, for quote, it is a bit tricky. As the stock name should be retrieved from the Data Feed Provider, which is Yahoo! Finance in our case, the most suitable time to get it is when the corresponding stock quote data is retrieved. Hence, a new column called stock_name is added to quote, as shown in chapter06_003.py: # -*- coding: utf-8 -*- # program: chapter06_003.py ## import Cassandra driver library from cassandra.cluster import Cluster ## function to add stock_name column def add_stockname_to_quote(ss):    ## add stock_name to quote    ss.execute('ALTER TABLE quote ' +                'ADD stock_name varchar') ## create Cassandra instance cluster = Cluster() ## establish Cassandra connection, using local default session = cluster.connect() ## use packtcdma keyspace session.set_keyspace('packtcdma') ## add stock_name column add_stockname_to_quote(session) ## close Cassandra connection cluster.shutdown() It is quite self-explanatory. Here, we use the ALTER TABLE statement to add the stock_name column of the varchar data type to quote. Queries on alerts As mentioned previously, we are interested in two questions: How many alerts were generated on a stock over a specified period of time? How many alerts were generated on a particular date? For the first question, alertlist is sufficient to provide an answer. However, alertlist cannot answer the second question because its primary key is composed of symbol and price_time. We need to create another table specifically for that question. This is an example of modeling by query. Basically, the structure of the new table for the second question should resemble the structure of alertlist. We give that table a name, alert_by_date, and create it as shown in chapter06_004.py: # -*- coding: utf-8 -*- # program: chapter06_004.py ## import Cassandra driver library from cassandra.cluster import Cluster ## function to create alert_by_date table def create_alertbydate(ss):    ## create alert_by_date table if not exists    ss.execute('CREATE TABLE IF NOT EXISTS alert_by_date (' +               'symbol varchar,' +                'price_time timestamp,' +                'stock_name varchar,' +                'signal_price float,' +                'PRIMARY KEY (price_time, symbol))') ## create Cassandra instance cluster = Cluster() ## establish Cassandra connection, using local default session = cluster.connect() ## use packtcdma keyspace session.set_keyspace('packtcdma') ## create alert_by_date table create_alertbydate(session) ## close Cassandra connection cluster.shutdown() When compared to alertlist in chapter06_002.py, alert_by_date only swaps the order of the columns in the compound primary key. One might think that a secondary index can be created on alertlist to achieve the same effect. Nonetheless, in Cassandra, a secondary index cannot be created on columns that are already engaged in the primary key. Always be aware of this constraint. We now finish the modifications on the data model. It is time for us to enhance the application logic in the next section. Summary This article extends the Stock Screener Application by a number of enhancements. We made changes to the data model to demonstrate the modeling by query techniques and how denormalization can help us achieve a high-performance application. Resources for Article: Further resources on this subject: An overview of architecture and modeling in Cassandra [Article] About Cassandra [Article] Basic Concepts and Architecture of Cassandra [Article]

In this article by Jarosław Krochmalski, author of the book IntelliJ IDEA Essentials, we will talk about working with VCS systems such as Git and Subversion. While working on the code, one of the most important aspects is version control. A Version Control System (VCS) (also known as a Revision Control System) is a repository of source code files with monitored access. Every change made to the source is tracked, along with who made the change, why they made it, and comments about problems fixed or enhancements introduced by the change. It doesn't matter if you work alone or in a team, having the tool to efficiently work with different versions of the code is crucial. Software development is usually carried out by teams, either distributed or colocated. The version control system lets developers work on a copy of the source code and then release their changes back to the common codebase when ready. Other developers work on their own copies of the same code at the same time, unaffected by each other's changes until they choose to merge or commit their changes back to the project. Currently, probably the most popular version control system is Git. After reading this article, you will be able to set up the version control mechanism of your choice, get files from the repository, commit your work, and browse the changes. Let's start with the version control setup. (For more resources related to this topic, see here.) Enabling version control At the IDE level, version control integration is provided through a set of plugins. IntelliJ IDEA comes bundled with a number of plugins to integrate with the most popular version control systems. They include Git, CVS, Subversion, and Mercurial. The Ultimate edition additionally contains Clearcase, Visual SourceSafe, and Perforce plugins. You will need to enable them in the Plugins section of the Settings dialog box. If you find the VCS feature is not enough and you are using some other VCS, try to find it in the Browse Repositories dialog box by choosing VCS Integration from the Category drop-down menu, as shown here: The list of plugins here contains not only integration plugins, but also some useful add-ons for the installed integrations. For example, the SVN Bar plugin will create a quick access toolbar with buttons specific for Subversion (SVN) actions. Feel free to browse the list of plugins here and read the descriptions; you might find some valuable extensions. The basic principles of working with the version control systems in IntelliJ IDEA are rather similar. We will focus on the Git and Subversion integration. This article should give you an overview of how to deal with the setup and version control commands in IntelliJ IDEA in general. If you have the necessary plugins enabled in the Settings dialog box, you can start working with the version control. We will begin with fetching the project out of the version control. Doing this will set up the version control automatically so that further steps will not be required unless you decide not to use the default workflow. Later, we will cover setting the VCS integration manually, so you will be able to tweak IntelliJ's behavior then. Checking out the project from the repository To be able to work on the files, first you need to get them from the repository. To get the files from the remote Git repository, you need to use the clone command available in the VCS menu, under the Checkout from Version Control option, as shown here: In the Clone Repository dialog box, provide necessary options, such as the remote repository URL, parent directory, and the directory name to clone into, as shown in the following screenshot: After successful cloning, IntelliJ IDEA will suggest creating a project based on the cloned sources. If you don't have the remote repository for your project, you can work with the offline local Git repository. To create a local Git repository, select Create Git repository from the VCS menu, as shown in the following screenshot: This option will execute the git init command in the directory of your choice; it will most probably be the root directory of your project. For the time being, the Git plugin does not allow you to set up remote repositories. You will probably need to set up the remote host for your newly created Git repository before you can actually fetch and push changes. If you are using GitHub for your projects, the great GitHub integration plugin gives you the option to share the project on GitHub. This will create the remote repository automatically. Later, when you want to get the files from the remote repository, just use the Git Pull command. This will basically retrieve changes (fetch) and apply them to the local branch (merge). To obtain a local working copy of a subversion repository, choose Checkout from Version Control and then Subversion from the VCS menu. In the SVN Checkout Options dialog box, you will be able to specify Subversion-specific settings, such as a revision that needs to be checked (HEAD, for example). Again, IntelliJ IDEA will ask if you want to create the project from checked out sources. If you accept the suggestion to create a new project, New Project from Existing Code Wizard will start. Fetching the project out of the repository will create some default VCS configuration in IntelliJ IDEA. It is usually sufficient, but if needed, the configuration can be changed. Let's discuss how to change the configuration in the next section. Configuring version control The VCS configuration in IntelliJ IDEA can be changed at the project level. Head to the Version Control section in the Settings dialog box, as shown here: The Version Control section contains options that are common for all version control systems and also specific options for the different VCS systems (enabled by installing the corresponding plugins). IntelliJ IDEA uses a directory-based model for version control. The versioning mechanism is assigned to a specific directory that can either be a part of a project or can be just related to the project. This directory is not required to be located under the project root. Multiple directories can have different version control systems linked. To add a directory into the version control integration, use the Alt + Insert keyboard shortcut or click on the green plus button; the Add VCS Directory Mapping dialog box will appear. You have the option to put all the project contents, starting from its base directory to the version control or limit the version control only to specific directories. Select the VCS system you need from the VCS drop-down menu, as shown in the following screenshot: By default, IntelliJ IDEA will mark the changed files with a color in the Project tool window, as shown here: If you select the Show directories with changed descendants option, IntelliJ IDEA will additionally mark the directories containing the changed files with a color, giving you the possibility to quickly notice the changes without expanding the project tree, as shown in the following screenshot: The Show changed in last <number> days option will highlight the files changed recently during the debugging process and when displaying stacktraces. Displaying the changed files in color can be very useful. If you see the colored file in the stacktrace, maybe the last change to the file is causing a problem. The subsequent panes contain general version control settings, which apply to all version control systems integrated with the IDE. They include specifying actions that require confirmation, background operations set up, the ignored files list, and issuing of navigation configuration. In the Confirmation section, you specify what version control actions will need your confirmation. The Background section will tell IntelliJ IDEA what operation it should perform in the background, as shown in the following screenshot: If you choose to perform the operation in the background, IntelliJ IDEA will not display any modal windows during and after the operation. The progress and result will be presented in the status bar of the IDE and in the corresponding tool windows. For example, after the successful execution of the Git pull command, IntelliJ IDEA will present the Update Project Info tool window with the files changed and the Event Log tool window with the status of the operation, as shown in the following screenshot: In the Ignored Files section, you can specify a list of files and directories that you do not want to put under version control, as shown in the following screenshot: To add a file or directory, use the Alt + Insert keyboard shortcut or hit the green plus (+) icon. The Ignore Unversioned Files dialog box will pop up as shown here: You can now specify a single file or the directory you want to ignore. There is also the possibility to construct the filename pattern for files to be ignored. Backup and logfiles are good candidates to be specified here, for example. Most of the version control systems support the file with a list of file patterns to ignore. For Git, this will be the .gitignore file. IntelliJ IDEA will analyze such files during the project checkout from the existing repository and will fill the Ignored files list automatically. In the Issue Navigation section, you can create a list of patterns to issue navigation. IntelliJ IDEA will try to use these patterns to create links from the commit messages. These links will then be displayed in the Changes and Version Control tool windows. Clicking on the link will open the browser and take you to the issue tracker of your choice. IntelliJ IDEA comes with predefined patterns for the most popular issue trackers: JIRA and YouTrack. To create a link to JIRA, click on the first button and provide the URL for your JIRA instance, as shown in the following screenshot: To create a link to the YouTrack instance, click on the OK button and provide the URL to the YouTrack instance. If you do not use JIRA or YouTrack, you can also specify a generic pattern. Press the Alt + Insert keyboard shortcut to add a new pattern. In the IssueID field, enter the regular expression that IntelliJ IDEA will use to extract a part of the link. In the Issue Link field, provide the link expression that IntelliJ IDEA will use to replace a issue number within. Use the Example section to check if the resulting link is correct, as shown in the following screenshot: The next sections in the Version Control preferences list contain options specific to the version control system you are using. For example, the Git-specific options can be configured in the Git section, as shown here: You can specify the Git command executable here or select the associated SSH executable that will be used to perform the network Git operations such as pull and push. The Auto-update if push of the current branch was rejected option is quite useful—IntelliJ IDEA will execute the pull command first if the push command fails because of the changes in the repository revision. This saves some time.We should now have version control integration up and running. Let's use it. Working with version control Before we start working with version control, we need to know about the concept of the changelist in IntelliJ IDEA. Let's focus on this now. Changelists When it comes to newly created or modified files, IntelliJ IDEA introduces the concept of a changelist. A changelist is a set of file modifications that represents a logical change in the source. Any modified file will go to the Default changelist. You can create new changelists if you like. The changes contained in a specific changelist are not stored in the repository until committed. Only the active changelist contains the files that are going to be committed. If you modify the file that is contained in the non-active change list, there is a risk that it will not be committed. This takes us to the last section of the common VCS settings at Settings | Version Control | Changelist conflicts. In this section, you can configure the protection of files that are present in the changelist that is not currently active. In other words, you define how IntelliJ IDEA should behave when you modify the file that is not in the active changelist. The protection is turned on by default (Enable changelist conflict tracking is checked). If the Resolve Changelist Conflict checkbox is marked, the IDE will display the Resolve Changelist Conflict dialog box when you try to modify such a file. The possible options are to either shelve the changes (we will talk about the concept of shelving in a while), move a file to the active changelist, switch changelists to make the current changelist active, or ignore the conflict. If Highlight files with conflicts is checked and if you try to modify a file from the non-active change list, a warning will pop up in the editor, as shown in the following screenshot: Again, you will have the possibility to move the changes to another change list, switch the active change list, or ignore the conflict. If you select Ignore, the change will be listed in the Files with ignored conflicts list, as shown in the following screenshot: The list of all changelists in the project is listed in the Commit Changes dialog box (we will cover committing files in a while) and in the first tab of the Changes tool window, as shown here: You can create a new changelist by using the Alt + Insert keyboard shortcut. The active list will have its name highlighted in bold. The last list is special; it contains the list of unversioned files. You can drag-and-drop files between the changelists (with the exception of unversioned files). Now that we know what a changelist is, let's add some files to the repository now. Adding files to version control You will probably want newly created files to be placed in version control. If you create a file in a directory already associated with the version control system, IntelliJ IDEA will add the file to the active changelist automatically, unless you configured this differently in the Confirmation section of the Version Control pane in the Settings dialog box. If you decided to have Show options before adding to version control checked, IntelliJ IDEA will ask if you want to add the file to the VCS, as shown here: If you decide to check the Remember, don't ask again checkbox, IntelliJ IDEA will throw the future new files into version control silently. You can also add new files to the version control explicitly. Click on the file or directory you want to add in the Project tool window and choose the corresponding VCS command; for example: Alternatively, you can open the Changes tool window, and browse Unversioned Files, where you can right-click on the file you want to add and select Add to VCS from the context menu, as shown in the following screenshot: If there are many unversioned files, IntelliJ IDEA will render a link that allows you to browse the files in a separate dialog box, as shown in the following screenshot: In the Unversioned Files dialog box, right-click on the file you want to add and select Add to VCS from the context menu, as shown in the following screenshot: From now on, the file will be ready to commit to the repository. If you've accidently added some files to version control and want to change them to unversioned, you can always revert the file so that it is no longer marked as part of the versioned files. Summary After reading this article, you know how to set up version control, get the project from the repository, commit your work, and get the changes made by other members of your team. Version control in IntelliJ IDEA is tightly integrated into the IDE. All the versioning activities can be executed from the IDE itself—you will not need to use an external tool for this. I believe it will shortly become natural for you to use the provided functionalities. Not being distracted by the use of external tools will result in higher effectiveness. Resources for Article: Further resources on this subject: Improving Your Development Speed [Article] Ridge Regression [Article] Function passing [Article]

In this article, by Richard Grimmett, author of Mastering BeagleBone Robotics, you'll build a quadruped, that is, a robot with four legs. You'll be using 12 servos so that each leg has three points that can move, or three Degrees of Freedom (DOF). In this project, you'll control 12 servos at the same time, so it will make more sense to use an external servo controller that can supply the control signals and supply voltages for all the 12 servos. (For more resources related to this topic, see here.) Since servos are the main component of this project, it is perhaps useful to go through a tutorial on servos and how to control them. Working of servomotors Servomotors are somewhat similar to DC motors; however, there is an important difference. While DC motors are generally designed to move in a continuous way—rotating 360 degrees at a given speed—servos are generally designed to move within a limited set of angles. In other words, in the case of a DC motor, you would generally want your motors to spin with continuous rotation speed that you control. But in the case of a servomotor, you would want your motor to move to a specific position that you control. This is done by sending a Pulse-Width-Modulated (PWM) signal to the control connector of the servo. PWM simply means that you are going to change the length of each pulse of electrical energy in order to control something. In this case, the length of this pulse will control the angle of the servo, as shown in the following diagram: These pulses are sent out with a repetition rate of 60 Hz. You can position the servo to any angle by setting the correct control pulse. Building the quadruped platform You'll first need some parts so that you can build your quadruped robot. There are several kit possibilities out there, including the one available at www.trossenrobotics.com/p/PhantomX-AX-12-Quadruped.aspx. However, such kits can be expensive, so for this example, you'll create your own kit using a set of Lynxmotion parts. These are available from several online retailers such as robotshop.com. To build this quadruped, you'll need four legs, each leg requires two Lynxmotion parts. Here are the parts with their Robotshop part numbers: Quantity Description 1 Lynxmotion symmetric quadruped body kit: Mini QBK-02 2 Lynxmotion 3'' aluminum femur pair 2 Lynxmotion Robot Leg "A" pair (No servo) RL-01 4 Lynxmotion aluminum multi-purpose servo bracket Two Pack ASB-04 2 Ball bearing with flange: 3mm ID (pair) Product code: RB-Lyn-317 This last part a bearing you'll need to connect the leg to the body. You'll also need 12 servos of a standard size. There are several possible choices, but I personally like the Hitec servos. They are very inexpensive and you can get them from most hobby shops and online electronic retailers. Now, for a moment on the model of servo: Servos come in different model numbers, primarily based on the amount of torque they can generate. Torque is the force that the servo can exert to move the part connected to it. In this case, your servos will need to lift and move the weight associated with your quadruped, so you'll need a servo with enough torque to do this. I suggest that you use eight Hitec model HS-485HB servos. You'll use these for the servos attached to the end of the leg and for the body. Then you'll use four Hitec model HS-645MG servos for the middle of the leg; this is the servo that will require the highest amount of torque. You can use the 12 Hitec model HS-645MG servos instead, but they are more expensive than the HS-485 servos, so using two different servos will be less expensive. Here are the steps required to assemble the quadruped: Put the two parts of the lower right leg together and insert the servo with the servo mounting screws. It should look like this: Now connect this assembly to the interconnect part, like this: Complete the leg by connecting two of the servo brackets together at right angles, mounting the HS-645MG on one of the brackets, and then connecting this servo to the interconnect piece, like this: Put another right leg together. Now put two left legs together following the same preceding steps, but in left leg configuration. They look like this: The next step is to build the body kit. There are some instructions given at www.lynxmotion.com/images/html/sq3u-assembly.htm, but it should be like the following image: Then connect each leg to the body kit. First connect the empty servo bracket to the body using the bearing, as shown in the following image: Now connect the other servo to the empty servo bracket and the body, like this: After performing all the preceding steps, your quadruped should now look like this: Now that you have the basic hardware assembled, you can turn your attention to the electronics. Using a servo controller to control the servos To make your quadruped walk, you will first need to connect the servomotor controller to the servos. The servo controller you are going to use for this project is a simple servomotor controller utilizing USB from Pololu (Pololu item number 1354, available at pololu.com) that can control 18 servomotors. Here is an image of the unit: Make sure that you order the assembled version. This piece of hardware will turn USB commands from the BeagleBone Black into signals that control your servomotors. Pololu makes a number of different versions of this controller, each able to control a certain number of servos. In this case, you may want to choose the 18-servo version, so you can control all 12 servos with one controller and also add an additional servo to control the direction of a camera or sensor. You could also choose the 12-servo version. One advantage of the 18-servo controller is the ease of connecting power to the unit via screw-type connectors. There are two connections you'll need to make to the servo controller to get started: the first is to the servomotors and the second is to a battery. First, connect the servos to the controller. In order to be consistent, let's connect your 12 servos to the connections marked 0 through 11 on the controller using the configuration shown in the following table: Servo Connector Servo 0 Right front lower leg 1 Right front middle leg 2 Right front upper leg 3 Right rear lower leg 4 Right rear middle leg 5 Right rear upper leg 6 Left front lower leg 7 Left front middle leg 8 Left front upper leg 9 Left rear lower leg 10 Left rear middle leg 11 Left rear upper leg Here is an image of the back of the controller. This will tell us where to connect our servos: Now you need to connect the servomotor controller to your battery. For this project, you can use a 2S RC LiPo battery. The 2S means that the battery will have two cells, with an output voltage of 7.2 volts. It will supply the voltage and current needed by your servos, which can be of the order of 2 amperes. Here is an image of the battery: This battery will come with two connectors: one with large gauge wires for normal usage and a smaller connector used to connect to the battery recharger. You'll want to build connectors that can connect to the screw-type connectors of the servo controller. I purchased some XT60 connector pairs, soldered some wires to the mating connector of the battery, and screwed these into the servo controller. Your system is now functional. Now you'll connect the motor controller to your personal computer to check if you can communicate with it. To do this, connect a mini USB cable between the servo controller and your personal computer. Communicating with the servo controller via a PC Now that the hardware is connected, you can use some software provided by Pololu to control the servos. Let's do this using your personal computer. First download the Pololu software from www.pololu.com/docs/0J40/3.a and install it according to the instructions on the website. Once it is installed, run the software, and you should see something like the following screenshot: You will first need to change the configuration of the serial settings, so select the Serial Settings tab and you should see this: Make sure that USB Chained is selected; this will allow you to communicate with and control the motor controller over USB. Now go back to the main screen by selecting the Status tab, and now you can actually turn on the 12 servos. The screen should look like this screenshot: Now you can use the sliders to actually control the servos. Check that servo 0 moves the right front lower servo, servo 1 moves the right front middle servo, servo 2 moves the right front upper servo, and so on. You can also use this to center the servos. Set all the servos such that the slider is in the middle. Now unscrew the servo horn on each servo until the servos are centered at this location. At the zero degree location of all servos, your quadruped should look like this: Your quadruped is now ready to actually do something. Now you'll need to send the servos the electronic signals they need to move your quadruped. Connecting the servo controller to the BeagleBone Black You've checked the servomotor controller and the servos. You'll now connect the motor controller to the BeagleBone Black and make sure you can control the servos from it. Remove the USB cable from the PC and connect it to the BeagleBone Black. The entire system will look like this: Let's now talk to the motor controller by downloading the Linux code from Pololu at www.pololu.com/docs/0J40/3.b. Perhaps the best way is to log in to your BeagleBone Black using PuTTY, then type wget http://www.pololu.com/file/download/maestro-linux-100507.tar.gz?file_id=0J315. Then move the file by typing mv maestro-linux-100507.tar.gz?file_id=0J315 maestro-linux-100507.tar.gz. Unpack the file by typing tar –xzfv maestro_linux_011507.tar.gz. This will create a directory called maestro_linux. Go to that directory by typing cd maestro_linux and then type ls. You should see something like this: The README.txt document will give you explicit instructions on how to install the software. Unfortunately, you can't run MaestroControlCenter on your BeagleBone Black. Your version of Windows doesn't support the graphics, but you can control your servos using the UscCmd command-line application to ensure that they are connected and working correctly. First, type ./UscCmd --list and you should see something like the following screenshot: The unit sees our servo controller. By just typing ./UscCmd, you can see all the commands that you can send to your controller, as shown in the following screenshot: Note that although you can send a servo a specific target angle, the target is not in angle values, so it makes it a bit difficult to know where you are sending your servo. Try typing ./UscCmd --servo 0, 10. The servo will move to its maximum angle position. Type ./UscCmd – servo 0, 0 and it will prevent the servo from trying to move. In the next section, you'll write some Python code that will translate your angles to the commands that the servo controller will want to receive to move it to specific angle locations. If you didn't run the Windows version of Maestro Controller and set the Serial Settings to USB Chained, your motor controller might not respond. Rerun the Maestro Controller code and set the Serial Settings to USB Chained. Creating a program on Linux to control your quadruped You now know that you can talk to your servomotor controller, and move your servos. In this section, you'll create a Python program that will let you talk to your servos to move them to specific angles. Let's start with a simple program that will make your legged mobile robot's servos go to 90 degrees (the middle of the 0 to 180 degrees you can set). This particular controller uses bytes of information, so the code will translate the input of the channel and angle to numbers that the controller can understand. For more details, see http://www.pololu.com/docs/0J40. Here is the code to move all the connected servos to the 90 degree point: Here is an explanation of the code: #! /usr/bin/python: This first line allows you to make this Python file executable from the command line. import serial: This line imports the serial library. You need the serial library to talk to your unit via USB. def setAngle(ser, channel, angle): This function converts your desired setting of servo and angle into the serial command that the servomotor controller needs. ser = serial.Serial("/dev/ttyACM0", 9600): This opens the serial port connection to your servo controller. for i in range(0, 15): This is for all 16 servo possibilities. setAngle(ser, i, 90): This allows you to set each servo to the middle (home) position. The default would be to set each servo to 90 degrees. If the legs of your robot aren't in their middle position, you can adjust them by adjusting the position of the servo horns on each servo. To access the serial port, you'll need to make sure that you have the Python serial library. If you don't, then type sudo apt-get install python-serial. After you have installed the serial library, you can run your program by typing sudo python quad.py. Once you have the basic home position set, you can now ask your robot to do some things. Let's start by making your quadruped wave an arm. Here is the Python code that waves the arm: In this case, you are using the setAngle command to set your servos to manipulate your robot's front-right arm. The middle servo raises the arm, and the lower servo then goes back and forth between angle 100 and 130. One of the most basic actions you'll want your robot to do is to walk forward. Here is an example of how to manipulate the legs to make this happen: This program lifts and moves each leg forward one at a time, and then moves all the legs to home position, which moves the robot forward. Not the most elegant motion, but it does work. There are more sophisticated algorithms to make your quadruped walk as shown at http://letsmakerobots.com/node/35354 and https://www.youtube.com/watch?v=jWP3RnYa_tw. Once you have the program working, you'll want to package all of your hardware onto the mobile robot. You can make your robot do many amazing things: walk forward, walk backward, dance, turn around—any kinds of movements are possible. The best way to learn is to try new and different positions with the servos. Issuing voice commands to your quadruped You should now have a mobile platform that you can program to move in any number of ways. Unfortunately, you still have your LAN cable connected, so the platform isn't completely mobile. And once you have begun the program, you can't alter the behavior of your program. You'll need to modify your voice recognition program so that it can run your Python program when it gets a voice command. You have to make a simple modification to the continuous.c program in /home/ubuntu/pocketsphinx-0.8/src/programs. To do this, type cd /home/ubuntu/ pocketsphinx-0.8/src/programs, and then type emacs continuous.c. The changes will occur in the same section as your other voice commands and will look like this: The additions are pretty straightforward. Let's walk through them: else if (strcmp(word, "FORWARD") == 0): This checks the word as recognized by your voice command program. If it corresponds with the word FORWARD, it will execute everything inside the if statement. We use { } to tell the system which commands go with this else if clause. system("/home/ubuntu/maestro_linux/robot.py"): This is the program we will execute. In this case, our mobile platform will do whatever the robot.py program tells it to do. After doing this, you will need to recompile the program, so type make and the pocketSphinx_continuous executable will be created. Run the program by typing ./pocketSphinx_continuous. Don't forget the ./ at the start of this command or you'll run a different version of the program. When the program is running, you can disconnect the LAN cable, and the mobile platform will now take the forward voice command and execute your program. Summary You now have a robot than can walk! You can also add other sensors, like the ones you discovered for your tracked robot, sensors that can watch for barriers, or even a webcam. Resources for Article: Further resources on this subject: Protecting GPG Keys in BeagleBone [Article] Home Security by BeagleBone [Article] Introducing BeagleBoard [Article]

In this article, by Tim Rogers author of Twilio Best Practices, we'll see how to keep our Twilio account and applications (and ultimately credit) secure by: Enabling two-factor authentication on our Twilio account Verifying that requests to our application are really coming from Twilio Setting up a circuit breaker for our account and any subaccounts (For more resources related to this topic, see here.) Enabling two-factor authentication Twilio offers two-factor authentication functionality that we can enable on our account. This will give you much greater security if someone tries to break into your account following the something you know and something you have model. Apart from your password, Twilio will send you an SMS or call you when a login attempt is made, requiring you to enter a one-time password. Not only will this largely prevent malicious access to your account, but you'll also know that someone is attempting to access your account, and what's more, that they have your password. It's worth noting that, unsurprisingly, you can quite easily roll two-factor authentication functionality for your own application using Twilio's call and SMS functionality. Check out https://www.twilio.com/docs/howto/two-factor-authentication for help with getting started. There are two steps to enable two-factor authentication: First, you'll need to add a phone number. You can do this from your Twilio dashboard by clicking on the dropdown in the top-right corner, and then clicking on the first entry with your name and e-mail address. Next, click on Add phone number, and then enter your phone number. Twilio will send you an SMS (or alternatively, call you if you'd like), thereby ensuring that you own the phone number that you've provided. Once you've added and verified your phone number on your user profile, you'll need to set up two-factor authentication on your account(s). To get to the right place, click on Account Settings in the dropdown. If your login has been given access to another user's Twilio account, (that is, the account you access from the Switch Accounts menu option) an administrator on that account will need to repeat this process. From this page, you'll be able to choose between two different two-factor options (you can also disable the feature here): Once Per Computer: This will effectively make the device you're using trusted, which means that subsequent logins for the next 30 days won't require you to use your phone. Every log-in: Every time you try to log in to Twilio, you'll have to provide a one-time password from your phone. Once you're done, click on the Save Settings button at the bottom of your page to set up the two-factor authentication. There are a couple of other features you might want to check out on the Account Settings page. You can reset your API credentials if you accidentally reveal them and you can disable parts of Twilio's Request Inspector which might potentially store sensitive information from your application and require passwords in order to access recordings and media you send in MMS messages. Verifying that requests are from Twilio If parties other than Twilio are able to make requests to your application, they can potentially change and corrupt data or access sensitive information. Without authentication measures, if an attacker was able to guess the URLs of the endpoints on your application that Twilio hits with its webhooks, they could wreak havoc. For instance, they could spoof fake SMS messages so that they appear to come from users or they could access the private phones numbers of users they should only be able to call through a public line you provide. There are two routes you can take to prevent this, ensuring with a reasonable degree of certainty that a request genuinely comes from Twilio: Set up HTTP Basic Authentication Verify the signature of requests to ensure they're signed by Twilio HTTP Basic Authentication HTTP Basic Authentication simply allows you to require a username and password to access your web server's resources. If you're working with PHP, you'll want to set this up on the web server level. This is possible in most servers, including: Apache () Nginx () IIS () If you're not using one of these, you can be virtually certain anyway that this option will be available to you; simply have a look at its documentation or search the web. Alternatively, you can implement Basic Authentication in your PHP code using code along these lines. We'll store the username and password in environment variables for security: <?php if (!isset($_SERVER['PHP_AUTH_USER'])) { // The user didn't even try to authenticate, so sent 401 Unauthorized header('WWW-Authenticate: Basic realm="Twilio only!"'); header('HTTP/1.0 401 Unauthorized'); exit; } elseif ($_SERVER['PHP_AUTH_USER'] == $_ENV["TWILIO_USERNAME"] && $_SERVER['PHP_AUTH_PW'] == $_ENV["TWILIO_PASSWORD"]) { // The user authenticated successfully, so perform actions and output TwiML } else { // The user tried to authenticate, but didn't have the right credentials header('WWW-Authenticate: Basic realm="Twilio only!"'); header('HTTP/1.0 401 Unauthorized'); exit; } ?> Let's go through this bit by bit: If $_SERVER['PHP_AUTH_USER'] isn't set, then no username and password has been provided, so we respond with a 401 Unauthorized error (that is, a header request, and the user provides a username and password, as well as the WWW-Authenticate header), which will make browsers display Twilio only! in the login dialog. If the provided username and password do match what is stored in the TWILIO_USERNAME and TWILIO_PASSWORD environment variables respectively, then we perform actions that the request requires and respond with TwiML. If a username and password was provided, but didn't match those we expected, then we send our 401 error and associated headers again. When we're providing a URL to Twilio (for instance, when initiating a call via the REST API, or setting it for incoming calls or SMS messages from our Dashboard), we can set the username and password in this format: Verifying the signature Alternatively, instead of using a username and password, we can verify the cryptographic signature Twilio generates with its requests based upon our auth token which is sent in the X-Twilio-Signature header. The scheme for doing this is somewhat complicated (you can find it in full at https://www.twilio.com/docs/security#validating-requests) but fortunately, Twilio provides validation functionality in their API libraries alongside code samples. For this method of verification to be available, you'll need to serve your application over HTTPS with Transport Layer Security (TLS) enabled. In fact, you should always do this with your Twilio application, as a good security practice. Following the SSLv3 vulnerability discovered in October, 2014 known as POODLE, you'll want to double-check the security of any SSL configuration. See https://www.digitalocean.com/community/tutorials/how-to-protect-your-server-against-the-poodle-sslv3-vulnerability for details. In PHP, we'd execute the following: <?php // Load auth token from the TWILIO_AUTH_TOKEN environment variable $authToken = $_ENV['TWILIO_AUTH_TOKEN']; // You'll need to make sure the Twilio library is included, either by requiring // it manually or loading Composer's autoload.php $validator = new Services_Twilio_RequestValidator($authToken); $url = $_SERVER["SCRIPT_URI"]; $vars = $_GET; $signature = $_SERVER["HTTP_X_TWILIO_SIGNATURE"]; if ($validator->validate($signature, $url, $vars)) { // This request definitely came from Twilio, so continue onwards... } else { // Watch out - this is not a real request from Twilio. header('HTTP/1.0 401 Unauthorized'); } ?> Here, we instantiate a Services_Twilio_RequestValidator object from the API library with our auth token before passing in the requested URL, the request body ($_GET in this case, but for a POST request, this would be $_POST), and the signature. We then call the validator's validate method with these pieces of data, allowing it to generate the signature itself, and comparing it against what we received in the X-Twilio-Signature header. If it matches, the request is genuine, but if not, the request is spoofed and is not from Twilio. Building a circuit breaker Using Twilio's Usage Triggers allows us to build a circuit breaker. In short, this will let us know when one of our subaccounts passes certain amounts of usage, which will help us detect possible abuse of our account, as well as mistakes in our code. It can even help detect abuse if we were running a multitenant app (that is, offering Twilio-based services to our users). When our specified usage threshold is surpassed, Twilio will send a webhook to a URL of our choice. From this URL, we can perform a range of actions, whether that is sending ourselves an e-mail or even suspending the account in question. Here, we'll just run through a quick example of suspending an account if it spends more than $50 in one day. We'll set up our Usage Trigger using the Twilio dashboard. To do this, first log in, and then switch to the appropriate subaccount you'd like to set up the trigger for by clicking on your name in the top-right corner. Next, click on Subaccounts, and then click on the desired account. Next, click on Usage in the navigation bar, then click on Triggers underneath, and then click on the Create Usage Trigger button. Fill out the fields as shown in the following image. First, you'll need to click on the Trigger a webhook link on the right-hand side of the page where Send an email appears in the screenshot to set up a webhook, replacing the URL with one that would be accessible from a domain of your own, of course. We might also want to automate this process of setting up a usage trigger using the REST API. For example, we might want to automatically suspend a subaccount we've just created for a customer if their usage goes beyond reasonable limits. In order to do so, we'd do the same thing we did previously in PHP like this: <?php $accountSid = $_ENV['TWILIO_ACCOUNT_SID']; $authToken = $_ENV['TWILIO_AUTH_TOKEN']; $subaccountSid = '<the SID of the subaccount>'; // You'll need to make sure the Twilio library is included, either by requiring // it manually or loading Composer's autoload.php $client = new Services_Twilio($accountSid, $authToken); $account = $client->accounts->get($subaccountSid); $account->usage_triggers->create( 'totalprice', '+50', 'https://twiliobestpractices.com/path/to/trigger_action.php', array(    'Recurring' => 'daily',    'TriggerBy' => 'price',    'FriendlyName' => 'Suspend if uses more than $50 per day' ) ); ?> Both options do exactly the same thing. They create a trigger that will send a webhook to when more than $50 is spent by our subaccount on any one day. It's completely up to us what we do from the endpoint that receives the webhook. Anything we can program is possible, from suspending the account to calling an engineer to look into it. Here's some example code to go with the usage trigger we just set up that will automatically suspend the account once it goes over $50 spend in a day. In this code sample, we also verify the authenticity of the request, as we saw previously, making sure that this is a genuine Usage Trigger webhook from Twilio: <?php // Before starting, you'll need to require the Twilio PHP library // We'll load the SID of the subaccount that the trigger relates to and its // auth token from environment variables, but in reality, you're likely to be // loading them from a database of your users' details based on a passed-in ID, // or something along those lines. $subaccountSid = $_ENV['TWILIO_SUBACCOUNT_SID']; $subaccountAuthToken = $_ENV['TWILIO_SUBACCOUNT_AUTH_TOKEN']; $url = $_SERVER['SCRIPT_URI']; $signature = $_SERVER['HTTP_X_TWILIO_SIGNATURE']; $validator = new Services_Twilio_RequestValidator($subaccountAuthToken); if ($validator->validate($signature, $url, $_POST)) {    $client = new Services_Twilio($subaccountSid, $subaccountAuthToken);    $client->account->update(array('Status' => 'suspended')); } else {    header('HTTP/1.0 401 Unauthorized'); } ?> We've shown you examples of doing all of this using the PHP library, but you can do it with any of Twilio's libraries . You can also do this directly with the REST API using a tool such as Postman. Summary In this article, we saw three helpful tips to keep your Twilio account and application secure: First, we enabled Two-Factor Authentication to keep our account secure, even if someone finds out our password. Next, we learned how to make sure that the requests your app receives genuinely come from Twilio, by either using the HTTP Basic Authentication or by verifying the cryptographic request signature. Finally, we set up alerts to inform us and take appropriate action when certain usage thresholds are reached using Twilio's Usage Triggers, helping protect your app from abuse and coding errors. Resources for Article: Further resources on this subject: Make phone calls, send SMS from your website using Twilio [article] Execution of Test Plans [article] Configuring Your Operating System [article]

In this article by Kyle D'Aoust, author of the book Unity Game Development Scripting, we will see how to create enemy and friendly AIs. (For more resources related to this topic, see here.) Artificial Intelligence, also known as AI, is something that you'll see in every video game that you play. First-person shooter, real-time strategy, simulation, role playing games, sports, puzzles, and so on, all have various forms of AI in both large and small systems. In this article, we'll be going over several topics that involve creating AI, including techniques, actions, pathfinding, animations, and the AI manager. Then, finally, we'll put it all together to create an AI package of our own. In this article, you will learn: What a finite state machine is What a behavior tree is How to combine two AI techniques for complex AI How to deal with internal and external actions How to handle outside actions that affect the AI How to play character animations What is pathfinding? How to use a waypoint system How to use Unity's NavMesh pathfinding system How to combine waypoints and NavMesh for complete pathfinding AI techniques There are two very common techniques used to create AI: the finite state machine and the behavior tree. Depending on the game that you are making and the complexity of the AI that you want, the technique you use will vary. In this article, we'll utilize both the techniques in our AI script to maximize the potential of our AI. Finite state machines Finite state machines are one of the most common AI systems used throughout computer programming. To define the term itself, a finite state machine breaks down to a system, which controls an object that has a limited number of states to exist in. Some real-world examples of a finite state machine are traffic lights, television, and a computer. Let's look at an example of a computer finite state machine to get a better understanding. A computer can be in various states. To keep it simple, we will list three main states. These states are On, Off, and Active. The Off state is when the computer does not have power running it, the On state is when the computer does have power running it, and the Active state is when someone is using the computer. Let's take a further look into our computer finite state machine and explore the functions of each of its states: State Functions On Can be used by anyone Can turn off the computer Off Can turn on the computer Computer parts can be operated on Active Can access the Internet and various programs Can communicate with other devices Can turn off the computer Each state has its own functions. Some of the functions of each state affect each other, while some do not. The functions that do affect each other are the functions that control what state the finite state machine is in. If you press the power button on your computer, it will turn on and change the state of your computer to On. While the state of your computer is On, you can use the Internet and possibly some other programs, or communicate to other devices such as a router or printer. Doing so will change the state of your computer to Active. When you are using the computer, you can also turn off the computer by its software or by pressing the power button, therefore changing the state to Off. In video games, you can use a finite state machine to create AI with a simple logic. You can also combine finite state machines with other types of AI systems to create a unique and perhaps more complex AI system. In this article, we will be using finite state machines as well as what is known as a behavior tree. The behavior tree form of the AI system A behavior tree is another kind of AI system that works in a very similar way to finite state machines. Actually, behavior trees are made up of finite state machines that work in a hierarchical system. This system of hierarchy gives us great control over an individual, and perhaps many finite state systems within the behavior tree, allowing us to have a complex AI system. Taking a look back at the table explaining a finite state machine, a behavior tree works the same way. Instead of states, you have behaviors, and in place of the state functions, you have various finite state machines that determine what is done while the AI is in a specific behavior. Let's take a look at the behavior tree that we will be using in this article to create our AI: On the left-hand side, we have four behaviors: Idle, Guard, Combat, and Flee. To the right are the finite state machines that make up each of the behaviors. Idle and Flee only have one finite state machine, while Guard and Combat have multiple. Within the Combat behavior, two of its finite state machines even have a couple of their own finite state machines. As you can see, this hierarchy-based system of finite state machines allows us to use a basic form of logic to create an even more complex AI system. At the same time, we are also getting a lot of control by separating our AI into various behaviors. Each behavior will run its own silo of code, oblivious to the other behaviors. The only time we want a behavior to notice another behavior is either when an internal or external action occurs that forces the behavior of our AI to change. Combining the techniques In this article, we will take both of the AI techniques and combine them to create a great AI package. Our behavior tree will utilize finite state machines to run the individual behaviors, creating a unique and complex AI system. This AI package can be used for an enemy AI as well as a friendly AI. Let's start scripting! Now, let's begin scripting our AI! To start off, create a new C# file and name it AI_Agent. Upon opening it, delete any functions within the main class, leaving it empty. Just after the using statements, add this enum to the script: public enum Behaviors {Idle, Guard, Combat, Flee}; This enum will be used throughout our script to determine what behavior our AI is in. Now let's add it to our class. It is time to declare our first variable: public Behaviors aiBehaviors = Behaviors.Idle; This variable, aiBehaviors, will be the deciding factor of what our AI does. Its main purpose is to have its value checked and changed when needed. Let's create our first function, which will utilize one of this variable's purposes: void RunBehaviors(){switch(aiBehaviors){case Behaviors.Idle:   RunIdleNode();   break;case Behaviors.Guard:   RunGuardNode();   break;case Behaviors.Combat:   RunCombatNode();   break;case Behaviors.Flee:   RunFleeNode();   break;}} What this function will do is check the value of our aiBehaviors variable in a switch statement. Depending on what the value is, it will then call a function to be used within that behavior. This function is actually going to be a finite state machine, which will decide what that behavior does at that point. Now, let's add another function to our script, which will allow us to change the behavior of our AI: void ChangeBehavior(Behaviors newBehavior){aiBehaviors = newBehavior; RunBehaviors();} As you can see, this function works very similarly to the RunBehaviors function. When this function is called, it will take a new behaviors variable and assign its value to aiBehaviors. By doing this, we changed the behavior of our AI. Now let's add the final step to running our behaviors; for now, they will be empty functions that act as placeholders for our internal and external actions. Add these functions to the script: void RunIdleNode(){ } void RunGuardNode(){ }void RunCombatNode(){ }void RunFleeNode(){ } Each of these functions will run the finite state machines that make up the behaviors. These functions are essentially a middleman between the behavior and the behavior's action. Using these functions is the beginning of having more control over our behaviors, something that can't be done with a simple finite state machine. Internal and external actions The actions of a finite state machine can be broken up into internal and external actions. Separating the actions into the two categories makes it easier to define what our AI does in any given situation. The separation is helpful in the planning phase of creating AI, but it can also help in the scripting part as well, since you will know what will and will not be called by other classes and GameObjects. Another way this separation is beneficial is that it eases the work of multiple programmers working on the same AI; each programmer could work on separate parts of the AI without as many conflicts. External actions External actions are functions and activities that are activated when objects outside of the AI object act upon the AI object. Some examples of external actions include being hit by a player, having a spell being cast upon the player, falling from heights, losing the game by an external condition, communicating with external objects, and so on. The external actions that we will be using for our AI are: Changing its health Raising a stat Lowering a stat Killing the AI Internal actions Internal actions are the functions and activities that the AI runs within itself. Examples of these are patrolling a set path, attacking a player, running away from the player, using items, and so on. These are all actions that the AI will choose to do depending on a number of conditions. The internal actions that we will be using for our AI are: Patrolling a path Attacking a player Fleeing from a player Searching for a player Scripting the actions It's time to add some internal and external actions to the script. First, be sure to add the using statement to the top of your script with the other using statements: using System.Collections.Generic; Now, let's add some variables that will allow us to use the actions: public bool isSuspicious = false;public bool isInRange = false;public bool FightsRanged = false;public List<KeyValuePair<string, int>> Stats = new List<KeyValuePair<string, int>>();public GameObject Projectile; The first three of our new variables are conditions to be used in finite state machines to determine what function should be called. Next, we have a list of the KeyValuePair variables, which will hold the stats of our AI GameObject. The last variable is a GameObject, which is what we will use as a projectile for ranged attacks. Remember the empty middleman functions that we previously created? Now with these new variables, we will be adding some code to each of them. Add this code so that the empty functions are now filled: void RunIdleNode(){Idle();} void RunGuardNode(){Guard();}void RunCombatNode(){if(FightsRanged)   RangedAttack();else   MeleeAttack();}void RunFleeNode(){Flee();} Two of the three boolean variables we just created are being used as conditionals to call different functions, effectively creating finite state machines. Next, we will be adding the rest of our actions; these are what is being called by the middleman functions. Some of these functions will be empty placeholders, but will be filled later on in the article: void Idle(){} void Guard(){if(isSuspicious){   SearchForTarget();}else{   Patrol();}}void Combat(){if(isInRange){   if(FightsRanged)   {     RangedAttack();   }   else   {     MeleeAttack();   }}else{   SearchForTarget();}}void Flee(){} void SearchForTarget(){} void Patrol(){} void RangedAttack(){GameObject newProjectile;newProjectile = Instantiate(Projectile, transform.position, Quaternion.identity) as GameObject;} void MeleeAttack(){} In the Guard function, we check to see whether the AI notices the player or not. If it does, then it will proceed to search for the player; if not, then it will continue to patrol along its path. In the Combat function, we first check to see whether the player is within the attacking range; if not, then the AI searches again. If the player is within the attacking range, we check to see whether the AI prefers attacking up close or far away. For ranged attacks, we first create a new, temporary GameObject variable. Then, we set it to an instantiated clone of our Projectile GameObject. From here, the projectile will run its own scripts to determine what it does. This is how we allow our AI to attack the player from a distance. To finish off our actions, we have two more functions to add. The first one will be to change the health of the AI, which is as follows: void ChangeHealth(int Amount){if(Amount < 0){   if(!isSuspicious)   {     isSuspicious = true;     ChangeBehavior(Behaviors.Guard);   }}for(int i = 0; i < Stats.Capacity; i++){   if(Stats[i].Key == "Health")   {     int tempValue = Stats[i].Value;     Stats[i] = new KeyValuePair<string, int>(Stats[i].Key, tempValue += Amount);     if(Stats[i].Value <= 0)     {       Destroy(gameObject);     }     else if(Stats[i].Value < 25)     {       isSuspicious = false;       ChangeBehavior(Behaviors.Flee);     }     break;   }}} This function takes an int variable, which is the amount by which we want to change the health of the player. The first thing we do is check to see if the amount is negative; if it is, then we make our AI suspicious and change the behavior accordingly. Next, we search for the health stat in our list and set its value to a new value that is affected by the Amount variable. We then check if the AI's health is below zero to kill it; if not, then we also check if its health is below 25. If the health is that low, we make our AI flee from the player. To finish off our actions, we have one last function to add. It will allow us to affect a specific stat of the AI. These modifications will either add to or subtract from a stat. The modifications can be permanent or restored anytime. For the following instance, the modifications will be permanent: void ModifyStat(string Stat, int Amount){for(int i = 0; i < Stats.Capacity; i++){   if(Stats[i].Key == Stat)   {     int tempValue = Stats[i].Value;     Stats[i] = new KeyValuePair<string, int>(Stats[i].Key, tempValue += Amount);     break;   }}if(Amount < 0){   if(!isSuspicious)   {     isSuspicious = true;     ChangeBehavior(Behaviors.Guard);   }}} This function takes a string and an integer. The string is used to search for the specific stat that we want to affect and the integer is how much we want to affect that stat by. It works in a very similar way to how the ChangeHealth function works, except that we first search for a specific stat. We also check to see if the amount is negative. This time, if it is negative, we change our AI behavior to Guard. This seems to be an appropriate response for the AI after being hit by something that negated one of its stats! Pathfinding Pathfinding is how the AI will maneuver around the level. For our AI package, we will be using two different kinds of pathfinding, NavMesh and waypoints. The waypoint system is a common approach to create paths for AI to move around the game level. To allow our AI to move through our level in an intelligent manner, we will use Unity's NavMesh component. Creating paths using the waypoint system Using waypoints to create paths is a common practice in game design, and it's simple too. To sum it up, you place objects or set locations around the game world; these are your waypoints. In the code, you will place all of your waypoints that you created in a container of some kind, such as a list or an array. Then, starting at the first waypoint, you tell the AI to move to the next waypoint. Once that waypoint has been reached, you send the AI off to another one, ultimately creating a system that iterates through all of the waypoints, allowing the AI to move around the game world through the set paths. Although using the waypoint system will grant our AI movement in the world, at this point, it doesn't know how to avoid obstacles that it may come across. That is when you need to implement some sort of mesh navigation system so that the AI won't get stuck anywhere. Unity's NavMesh system The next step in creating AI pathfinding is to create a way for our AI to navigate through the game world intelligently, meaning that it does not get stuck anywhere. In just about every game out there that has a 3D-based AI, the world it inhabits has all sorts of obstacles. These obstacles could be plants, stairs, ramps, boxes, holes, and so on. To get our AI to avoid these obstacles, we will use Unity's NavMesh system, which is built into Unity itself. Setting up the environment Before we can start creating our pathfinding system, we need to create a level for our AI to move around in. To do this, I am just using Unity primitive models such as cubes and capsules. For the floor, create a cube, stretch it out, and squish it to make a rectangle. From there, clone it several times so that you have a large floor made up of cubes. Next, delete a bunch of the cubes and move some others around. This will create holes in our floor, which will be used and tested when we implement the NavMesh system. To make the floor easy to see, I've created a material in green and assigned it to the floor cubes. After this, create a few more cubes, make one really long and one shorter than the previous one but thicker, and the last one will be used as a ramp. I've created an intersection of the really long cube and the thick cube. Then, place the ramp towards the end of the thick cube, giving access to the top of the cubes. Our final step in creating our test environment is to add a few waypoints for our AI. For testing purposes, create five waypoints in this manner. Place one in each corner of the level and one in the middle. For the actual waypoints, use the capsule primitive. For each waypoint, add a rigid body component. Name the waypoints as Waypoint1, Waypoint2, Waypoint3, and so on. The name is not all that important for our code; it just makes it easier to distinguish between waypoints in the inspector. Here's what I made for my level:  Creating the NavMesh Now, we will create the navigation mesh for our scene. The first thing we will do is select all of the floor cubes. In the menu tab in Unity, click on the Window option, and then click on the Navigation option at the bottom of the dropdown; this will open up the Navigation window. This is what you should be seeing right now:  By default, the OffMeshLink Generation option is not checked; be sure to check it. What this does is create links at the edges of the mesh allowing it to communicate with any other OffMeshLink nearby, creating a singular mesh. This is a handy tool since game levels typically use more than one mesh as a floor. The Scene filter will just show specific objects within the hierarchy view list. Selecting all the objects will show all of your GameObjects. Selecting mesh renderers will only show GameObjects that have the mesh renderer component. Then, finally, if you select terrains, only terrains will be shown in the Hierarchy view list. The Navigation Layer dropdown will allow you to set the area as either walkable, not walkable, or jump accessible. Walkable areas typically refer to floors, ramps, and so on. Non-walkable areas refer to walls, rocks, and other various obstacles. Next, click on the Bake tab next to the Object tab. You should see information that looks like this: For this article, I am leaving all the values at their defaults. The Radius property is used to determine how close to the walls the navigation mesh will exist. Height determines how much vertical space is needed for the AI agent to be able to walk on the navigation mesh. Max Slope is the maximum angle that the AI is allowed to travel on for ramps, hills, and so on. The Step Height property is used to determine how high the AI can step up onto surfaces higher than the ground level. For Generated Off Mesh Links, the properties are very similar to each other. The Drop Height value is the maximum amount of space the AI can intelligently drop down to another part of the navigation mesh. Jump Distance is the opposite of Height; it determines how high the AI can jump up to another part of the navigation mesh. The Advanced options are to be used when you have a better understanding of the NavMesh component and want a little more out of it. Here, you can further tweak the accuracy of the NavMesh as well as create Height Mesh to coincide with the navigation mesh. Now that you know all the basics of the Unity NavMesh, let's go ahead and create our navigation mesh. At the bottom-right corner of the Navigation tab in the Inspector window, you should see two buttons: one that says Clear and the other that says Bake. Click on the Bake button now to create your new navigation mesh. Select the ramp and the thick cube that we created earlier. In the Navigation window, make sure that the OffMeshLink Generation option is not checked, and that Navigation Layer is set to Default. If the ramp and the thick cube are not selected, reselect the floor cubes so that you have the floors, ramp, and thick wall selected. Bake the navigation mesh again to create a new one. This is what my scene looks like now with the navigation mesh: You should be able to see the newly generated navigation mesh overlaying the underlying mesh. This is what was created using the default Bake properties. Changing the Bake properties will give you different results, which will come down to what kind of navigation mesh you want the AI to use. Now that we have a navigation mesh, let's create the code for our AI to utilize. First, we will code the waypoint system, and then we will code what is needed for the NavMesh system. Adding our variables To start our navigation system, we will need to add a few variables first. Place these with the rest of our variables: public Transform[] Waypoints;public int curWaypoint = 0;bool ReversePath = false;NavMeshAgent navAgent;Vector3 Destination;float Distance; The first variable is an array of Transforms; this is what we will use to hold our waypoints. Next, we have an integer that is used to iterate through our Transform array. We have a bool variable, which will decide how we should navigate through the waypoints. The next three variables are more oriented towards our navigation mesh that we created earlier. The NavMeshAgent object is what we will reference when we want to interact with the navigation mesh. The destination will be the location that we want the AI to move towards. The distance is what we will use to check how far away we are from that location. Scripting the navigation functions Previously, we created many empty functions; some of these are dependent on pathfinding. Let's start with the Flee function. Add this code to replace the empty function: void Flee(){for(int fleePoint = 0; fleePoint < Waypoints.Length; fleePoint++){   Distance = Vector3.Distance(gameObject.transform.position, Waypoints[fleePoint].position);   if(Distance > 10.00f)   {     Destination = Waypoints[curWaypoint].position;     navAgent.SetDestination(Destination);     break;   }   else if(Distance < 2.00f)   {     ChangeBehavior(Behaviors.Idle);   }}} What this for loop does is pick a waypoint that has Distance of more than 10. If it does, then we set the Destination value to the current waypoint and move the AI accordingly. If the distance from the current waypoint is less than 2, we change the behavior to Idle. The next function that we will adjust is the SearchForTarget function. Add the following code to it, replacing its previous emptiness: void SearchForTarget(){Destination = GameObject.FindGameObjectWithTag("Player").transform.position;navAgent.SetDestination(Destination);Distance = Vector3.Distance(gameObject.transform.position, Destination);if(Distance < 10)   ChangeBehavior(Behaviors.Combat);} This function will now be able to search for a target, the Player target to be more specific. We set Destination to the player's current position, and then move the AI towards the player. When Distance is less than 10, we set the AI behavior to Combat. Now that our AI can run from the player as well as chase them down, let's utilize the waypoints and create paths for the AI. Add this code to the empty Patrol function: void Patrol(){Distance = Vector3.Distance(gameObject.transform.position, Waypoints[curWaypoint].position);if(Distance > 2.00f){   Destination = Waypoints[curWaypoint].position;   navAgent.SetDestination(Destination);}else{   if(ReversePath)   {     if(curWaypoint <= 0)     {       ReversePath = false;     }     else     {       curWaypoint--;       Destination = Waypoints[curWaypoint].position;     }   }   else   {     if(curWaypoint >= Waypoints.Length - 1)     {       ReversePath = true;     }     else     {       curWaypoint++;       Destination = Waypoints[curWaypoint].position;     }   }}} What Patrol will now do is check the Distance variable. If it is far from the current waypoint, we set that waypoint as the new destination of our AI. If the current waypoint is close to the AI, we check the ReversePath Boolean variable. When ReversePath is true, we tell the AI to go to the previous waypoint, going through the path in the reverse order. When ReversePath is false, the AI will go on to the next waypoint in the list of waypoints. With all of this completed, you now have an AI with pathfinding abilities. The AI can also patrol a path set by waypoints and reverse the path when the end has been reached. We have also added abilities for the AI to search for the player as well as flee from the player. Character animations Animations are what bring the characters to life visually in the game. From basic animations to super realistic movements, all the animations are important and really represent what scripters do to the player. Before we add animations to our AI, we first need to get a model mesh for it! Importing the model mesh For this article, I am using a model mesh that I got from the Unity Asset Store. To use the same model mesh that I am using, go to the Unity Asset Store and search for Skeletons Pack. It is a package of four skeleton model meshes that are fully textured, propped, and animated. The asset itself is free and great to use. When you import the package into Unity, it will come with all four models as well as their textures, and an example scene named ShowCase. Open that scene and you should see the four skeletons. If you run the scene, you will see all the skeletons playing their idle animations. Choose the skeleton you want to use for your AI; I chose skeletonDark for mine. Click on the drop-down list of your skeleton in the Hierarchy window, and then on the Bip01 drop-down list. Then, select the magicParticle object. For our AI, we will not need it, so delete it from the Hierarchy window. Create a new prefab in the Project window and name it Skeleton. Now select the skeleton that you want to use from the Hierarchy window and drag it onto the newly created prefab. This will now be the model that you will use for this article. In your AI test scene, drag and drop Skeleton Prefab onto the scene. I have placed mine towards the center of the level, near the waypoint in the middle. In the Inspector window, you will be able to see the Animation component full of animations for the model. Now, we will need to add a few components to our skeleton. Go to the Components menu on the top of the Unity window, select Navigation, and then select NavMesh Agent. Doing this will allow the skeleton to utilize the NavMesh we created earlier. Next, go into the Components menu again and click on Capsule Collider as well as Rigidbody. Your Inspector window should now look like this after adding the components: Your model now has all the necessary components needed to work with our AI script. Scripting the animations To script our animations, we will take a simple approach to it. There won't be a lot of code to deal with, but we will spread it out in various areas of our script where we need to play the animations. In the Idle function, add this line of code: animation.Play("idle"); This simple line of code will play the idle animation. We use animation to access the model's animation component, and then use the Play function of that component to play the animation. The Play function can take the name of the animation to call the correct animation to be played; for this one, we call the idle animation. In the SearchForTarget function, add this line of code to the script: animation.Play("run"); We access the same function of the animation component and call the run animation to play here. Add the same line of code to the Patrol function as well, since we will want to use that animation for that function too. In the RangedAttack and MeleeAttack functions, add this code: animation.Play("attack"); Here, we call the attack animation. If we had a separate animation for ranged attacks, we would use that instead, but since we don't, we will utilize the same animation for both attack types. With this, we finished coding the animations into our AI. It will now play those animations when they are called during gameplay. Putting it all together To wrap up our AI package, we will now finish up the script and add it to the skeleton. Final coding touches At the beginning of our AI script, we created some variables that we have yet to properly assign. We will do that in the Start function. We will also add the Update function to run our AI code. Add these functions to the bottom of the class: void Start(){navAgent = GetComponent<NavMeshAgent>(); Stats.Add(new KeyValuePair<string, int>("Health", 100));Stats.Add(new KeyValuePair<string, int>("Speed", 10));Stats.Add(new KeyValuePair<string, int>("Damage", 25));Stats.Add(new KeyValuePair<string, int>("Agility", 25));Stats.Add(new KeyValuePair<string, int>("Accuracy", 60));} void Update (){RunBehaviors();} In the Start function, we first assign the navAgent variable by getting the NavMeshAgent component from the GameObject. Next, we add new KeyValuePair variables to the Stats array. The Stats array is now filled with a few stats that we created. The Update function calls the RunBehaviors function. This is what will keep the AI running; it will run the correct behavior as long as the AI is active. Filling out the inspector To complete the AI package, we will need to add the script to the skeleton, so drag the script onto the skeleton in the Hierarchy window. In the Size property of the waypoints array, type the number 5 and open up the drop-down list. Starting with Element 0, drag each of the waypoints into the empty slots. For the projectile, create a sphere GameObject and make it a prefab. Now, drag it onto the empty slot next to Projectile. Finally, set the AI Behaviors to Guard. This will make it so that when you start the scene, your AI will be patrolling. The Inspector window of the skeleton should look something like this: Your AI is now ready for gameplay! To make sure everything works, we will need to do some playtesting. Playtesting A great way to playtest the AI is to play the scene in every behavior. Start off with Guard, then run it in Idle, Combat, and Flee. For different outputs, try adjusting some of the variables in the NavMesh Agent component, such as Speed, Angular Speed, and Stopping Distance. Try mixing your waypoints around so the path is different. Summary In this article, you learned how to create an AI package. We explored a couple of techniques to handle AI, such as finite state machines and behavior trees. Then, we dived into AI actions, both internal and external. From there, we figured out how to implement pathfinding with both a waypoint system and Unity's NavMesh system. Finally, we topped the AI package off with animations and put everything together, creating our finalized AI. Resources for Article: Further resources on this subject: Getting Started – An Introduction to GML [article] Animations in Cocos2d-x [article] Components in Unity [article]

In this article, by James Miller, author of the book Mastering Splunk, we will discuss Splunk lookups and workflows. The topics that will be covered in this article are as follows: The value of a lookup Design lookups File lookups Script lookups (For more resources related to this topic, see here.) Lookups Machines constantly generate data, usually in a raw form that is most efficient for processing by machines, but not easily understood by "human" data consumers. Splunk has the ability to identify unique identifiers and/or result or status codes within the data. This gives you the ability to enhance the readability of the data by adding descriptions or names as new search result fields. These fields contain information from an external source such as a static table (a CSV file) or the dynamic result of a Python command or a Python-based script. Splunk's lookups can use information within returned events or time information to determine how to add other fields from your previously defined external data sources. To illustrate, here is an example of a Splunk static lookup that: Uses the Business Unit value in an event Matches this value with the organization's business unit name in a CSV file Adds the definition to the event (as the Business Unit Name field) So, if you have an event where the Business Unit value is equal to 999999, the lookup will add the Business Unit Name value as Corporate Office to that event. More sophisticated lookups can: Populate a static lookup table from the results of a report. Use a Python script (rather than a lookup table) to define a field. For example, a lookup can use a script to return a server name when given an IP address. Perform a time-based lookup if your lookup table includes a field value that represents time. Let's take a look at an example of a search pipeline that creates a table based on IBM Cognos TM1 file extractions: sourcetype=csv 2014 "Current Forecast" "Direct" "513500" |rename May as "Month" Actual as "Version" "FY 2012" as Year 650693NLR001 as "Business Unit" 100000 as "FCST" "09997_Eliminations Co 2" as "Account" "451200" as "Activity" | eval RFCST= round(FCST) |Table Month, "Business Unit", RFCST The following table shows the results generated:   Now, add the lookup command to our search pipeline to have Splunk convert Business Unit into Business Unit Name: sourcetype=csv 2014 "Current Forecast" "Direct" "513500" |rename May as "Month" Actual as "Version" "FY 2012" as Year 650693NLR001 as "Business Unit" 100000 as "FCST" "09997_Eliminations Co 2" as "Account" "451200"as "Activity" | eval RFCST= round(FCST) |lookup BUtoBUName BU as "Business Unit" OUTPUT BUName as "Business Unit Name" | Table Month, "Business Unit", "Business Unit Name", RFCST The lookup command in our Splunk search pipeline will now add Business Unit Name in the results table:   Configuring a simple field lookup In this section, we will configure a simple Splunk lookup. Defining lookups in Splunk Web You can set up a lookup using the Lookups page (in Splunk Web) or by configuring stanzas in the props.conf and transforms.conf files. Let's take the easier approach first and use the Splunk Web interface. Before we begin, we need to establish our lookup table that will be in the form of an industry standard comma separated file (CSV). Our example is one that converts business unit codes to a more user-friendly business unit name. For example, we have the following information: Business unit code Business unit name 999999 Corporate office VA0133SPS001 South-western VA0133NLR001 North-east 685470NLR001 Mid-west In the events data, only business unit codes are included. In an effort to make our Splunk search results more readable, we want to add the business unit name to our results table. To do this, we've converted our information (shown in the preceding table) to a CSV file (named BUtoBUName.csv):   For this example, we've kept our lookup table simple, but lookup tables (files) can be as complex as you need them to be. They can have numerous fields (columns) in them. A Splunk lookup table has a few requirements, as follows: A table must contain a minimum of two columns Each of the columns in the table can have duplicate values You should use (plain) ASCII text and not non-UTF-8 characters Now, from Splunk Web, we can click on Settings and then select Lookups:   From the Lookups page, we can select Lookup table files:   From the Lookup table files page, we can add our new lookup file (BUtoBUName.csv):   By clicking on the New button, we see the Add new page where we can set up our file by doing the following: Select a Destination app (this is a drop-down list and you should select Search). Enter (or browse to) our file under Upload a lookup file. Provide a Destination filename. Then, we click on Save:   Once you click on Save, you should receive the Successfully saved "BUtoBUName" in search" message:   In the previous screenshot, the lookup file is saved by default as private. You will need to adjust permissions to allow other Splunk users to use it. Going back to the Lookups page, we can select Lookup definitions to see the Lookup definitions page:   In the Lookup definitions page, we can click on New to visit the Add new page (shown in the following screenshot) and set up our definition as follows: Destination app: The lookup will be part of the Splunk search app Name: Our file is BUtoBUName Type: Here, we will select File-based Lookup file: The filename is ButoBUName.csv, which we uploaded without the .csv suffix Again, we should see the Successfully saved "BUtoBUName" in search message:   Now, our lookup is ready to be used: Automatic lookups Rather than having to code for a lookup in each of your Splunk searches, you have the ability to configure automatic lookups for a particular source type. To do this from Splunk Web, we can click on Settings and then select Lookups:   From the Lookups page, click on Automatic lookups:   In the Automatic lookups page, click on New:   In the Add New page, we will fill in the required information to set up our lookup: Destination app: For this field, some options are framework, launcher, learned, search, and splunk_datapreview (for our example, select search). Name: This provide a user-friendly name that describes this automatic lookup. Lookup table: This is the name of the lookup table you defined with a CSV file (discussed earlier in this article). Apply to: This is the type that you want this automatic lookup to apply to. The options are sourcetype, source, or host (I've picked sourcetype). Named: This is the name of the type you picked under Apply to. I want my automatic search to apply for all searches with the sourcetype of csv. Lookup input fields: This is simple in my example. In my lookup table, the field to be searched on will be BU and the = field value will be the field in the event results that I am converting; in my case, it was the field 650693NLR001. Lookup output fields: This will be the field in the lookup table that I am using to convert to, which in my example is BUName and I want to call it Business Unit Name, so this becomes the = field value. Overwrite field values: This is a checkbox where you can tell Splunk to overwrite existing values in your output fields—I checked it. The Add new page The Splunk Add new page (shown in the following screenshot) is where you enter the lookup information (detailed in the previous section):   Once you have entered your automatic lookup information, you can click on Save and you will receive the Successfully saved "Business Unit to Business Unit Name" in search message:   Now, we can use the lookup in a search. For example, you can run a search with sourcetype=csv, as follows: sourcetype=csv 2014 "Current Forecast" "Direct" "513500" |rename May as "Month" Actual as "Version" "FY 2012" as Year 650693NLR001 as "Business Unit" 100000 as "FCST" "09997_Eliminations Co 2"as "Account" "451200" as "Activity" | eval RFCST= round(FCST) |Table "Business Unit", "Business Unit Name", Month, RFCST Notice in the following screenshot that Business Unit Name is converted to the user-friendly values from our lookup table, and we didn't have to add the lookup command to our search pipeline:   Configuration files In addition to using the Splunk web interface, you can define and configure lookups using the following files: props.conf transforms.conf To set up a lookup with these files (rather than using Splunk web), we can perform the following steps: Edit transforms.conf to define the lookup table. The first step is to edit the transforms.conf configuration file to add the new lookup reference. Although the file exists in the Splunk default folder ($SPLUNK_HOME/etc/system/default), you should edit the file in $SPLUNK_HOME/etc/system/local/ or $SPLUNK_HOME/etc/apps/<app_name>/local/ (if the file doesn't exist here, create it). Whenever you edit a Splunk .conf file, always edit a local version, keeping the original (system directory version) intact. In the current version of Splunk, there are two types of lookup tables: static and external. Static lookups use CSV files, and external (which are dynamic) lookups use Python scripting. You have to decide if your lookup will be static (in a file) or dynamic (use script commands). If you are using a file, you'll use filename; if you are going to use a script, you use external_cmd (both will be set in the transforms.conf file). You can also limit the number of matching entries to apply to an event by setting the max_matches option (this tells Splunk to use the first <integer> (in file order) number of entries). I've decided to leave the default for max_matches, so my transforms.conf file looks like the following: [butobugroup]filename = butobugroup.csv This step is optional. Edit props.conf to apply your lookup table automatically. For both static and external lookups, you stipulate the fields you want to match in the configuration file and the output from the lookup table that you defined in your transforms.conf file. It is okay to have multiple field lookups defined in one source lookup definition, but each lookup should have its own unique lookup name; for example, if you have multiple tables, you can name them LOOKUP-table01, LOOKUP-table02, and so on, or something perhaps more easily understood. If you add a lookup to your props.conf file, this lookup is automatically applied to all events from searches that have matching source types (again, as mentioned earlier; if your automatic lookup is very slow, it will also impact the speed of your searches). Restart Splunk to see your changes. Implementing a lookup using configuration files – an example To illustrate the use of configuration files in order to implement an automatic lookup, let's use a simple example. Once again, we want to convert a field from a unique identification code for an organization's business unit to a more user friendly descriptive name called BU Group. What we will do is match the field bu in a lookup table butobugroup.csv with a field in our events. Then, add the bugroup (description) to the returned events. The following shows the contents of the butobugroup.csv file: bu, bugroup 999999, leadership-groupVA0133SPS001, executive-group650914FAC002, technology-group You can put this file into $SPLUNK_HOME/etc/apps/<app_name>/lookups/ and carry out the following steps: Put the butobugroup.csv file into $SPLUNK_HOME/etc/apps/search/lookups/, since we are using the search app. As we mentioned earlier, we edit the transforms.conf file located at either $SPLUNK_HOME/etc/system/local/ or $SPLUNK_HOME/etc/apps/<app_name>/local/. We add the following two lines: [butobugroup]filename = butobugroup.csv Next, as mentioned earlier in this article, we edit the props.conf file located at either $SPLUNK_HOME/etc/system/local/ or $SPLUNK_HOME/etc/apps/<app_name>/local/. Here, we add the following two lines: [csv]LOOKUP-check = butobugroup bu AS 650693NLR001 OUTPUT bugroup Restart the Splunk server. You can (assuming you are logged in as an admin or have admin privileges) restart the Splunk server through the web interface by going to Settings, then select System and finally Server controls. Now, you can run a search for sourcetype=csv (as shown here): sourcetype=csv 2014 "Current Forecast" "Direct" "513500" |rename May as "Month" ,650693NLR001 as "Business Unit" 100000 as "FCST"| eval RFCST= round(FCST) |Table "Business Unit", "Business Unit Name", bugroup, Month, RFCST You will see that the field bugroup can be returned as part of your event results:   Populating lookup tables Of course, you can create CSV files from external systems (or, perhaps even manually?), but from time to time, you might have the opportunity to create lookup CSV files (tables) from event data using Splunk. A handy command to accomplish this is outputcsv (which is covered in detail later in this article). The following is a simple example of creating a CSV file from Splunk event data that can be used for a lookup table: sourcetype=csv "Current Forecast" "Direct" | rename 650693NLR001 as "Business Unit" | Table "Business Unit", "Business Unit Name", bugroup | outputcsv splunk_master The results are shown in the following screeshot:   Of course, the output table isn't quite usable, since the results have duplicates. Therefore, we can rewrite the Splunk search pipeline introducing the dedup command (as shown here): sourcetype=csv   "Current Forecast" "Direct"   | rename 650693NLR001 as "Business Unit" | dedup "Business Unit" | Table "Business Unit", "Business Unit Name", bugroup | outputcsv splunk_master Then, we can examine the results (now with more desirable results):   Handling duplicates with dedup This command allows us to set the number of duplicate events to be kept based on the values of a field (in other words, we can use this command to drop duplicates from our event results for a selected field). The event returned for the dedup field will be the first event found (if you provide a number directly after the dedup command, it will be interpreted as the number of duplicate events to keep; if you don't specify a number, dedup keeps only the first occurring event and removes all consecutive duplicates). The dedup command also lets you sort by field or list of fields. This will remove all the duplicates and then sort the results based on the specified sort-by field. Adding a sort in conjunction with the dedup command can affect the performance as Splunk performs the dedup operation and then sorts the results as a final step. Here is a search command using dedup: sourcetype=csv   "Current Forecast" "Direct"   | rename 650693NLR001 as "Business Unit" | dedup "Business Unit" sortby bugroup | Table "Business Unit", "Business Unit Name", bugroup | outputcsv splunk_master The result of the preceding command is shown in the following screenshot:   Now, we have our CSV lookup file (outputcsv splunk_master) generated and ready to be used:   Look for your generated output file in $SPLUNK_HOME/var/run/splunk. Dynamic lookups With a Splunk static lookup, your search reads through a file (a table) that was created or updated prior to executing the search. With dynamic lookups, the file is created at the time the search executes. This is possible because Splunk has the ability to execute an external command or script as part of your Splunk search. At the time of writing this book, Splunk only directly supports Python scripts for external lookups. If you are not familiar with Python, its implementation began in 1989 and is a widely used general-purpose, high-level programming language, which is often used as a scripting language (but is also used in a wide range of non-scripting contexts). Keep in mind that any external resources (such as a file) or scripts that you want to use with your lookup will need to be copied to a location where Splunk can find it. These locations are: $SPLUNK_HOME/etc/apps/<app_name>/bin $SPLUNK_HOME/etc/searchscripts The following sections describe the process of using the dynamic lookup example script that ships with Splunk (external_lookup.py). Using Splunk Web Just like with static lookups, Splunk makes it easy to define a dynamic or external lookup using the Splunk web interface. First, click on Settings and then select Lookups:   On the Lookups page, we can select Lookup table files to define a CSV file that contains the input file for our Python script. In the Add new page, we enter the following information: Destination app: For this field, select Search Upload a lookup file: Here, you can browse to the filename (my filename is dnsLookup.csv) Destination filename: Here, enter dnslookup The Add new page is shown in the following screenshot:   Now, click on Save. The lookup file (shown in the following screenshot) is a text CSV file that needs to (at a minimum) contain the two field names that the Python (py) script accepts as arguments, in this case, host and ip. As mentioned earlier, this file needs to be copied to $SPLUNK_HOME/etc/apps/<app_name>/bin.   Next, from the Lookups page, select Lookup definitions and then click on New. This is where you define your external lookup. Enter the following information: Type: For this, select External (as this lookup will run an external script) Command: For this, enter external_lookup.py host ip (this is the name of the py script and its two arguments) Supported fields: For this, enter host, ip (this indicates the two script input field names) The following screenshot describes a new lookup definition:   Now, click on Save. Using configuration files instead of Splunk Web Again, just like with static lookups in Splunk, dynamic lookups can also be configured in the Splunk transforms.conf file: [myLookup]external_cmd = external_lookup.py host ipexternal_type = pythonfields_list = host, ipmax_matches = 200 Let's learn more about the terms here: [myLookup]: This is the report stanza. external_cmd: This is the actual runtime command definition. Here, it executes the Python (py) script external_lookup, which requires two arguments (or parameters), host and ip. external_type (optional): This indicates that this is a Python script. Although this is an optional entry in the transform.conf file, it's a good habit to include this for readability and support. fields_list: This lists all the fields supported by the external command or script, delimited by a comma and space. The next step is to modify the props.conf file, as follows: [mylookup]LOOKUP-rdns = dnslookup host ip OUTPUT ip After updating the Splunk configuration files, you will need to restart Splunk. External lookups The external lookup example given uses a Python (py) script named external_lookup.py, which is a DNS lookup script that can return an IP address for a given host name or a host name for a provided IP address. Explanation The lookup table field in this example is named ip, so Splunk will mine all of the IP addresses found in the indexed logs' events and add the values of ip from the lookup table into the ip field in the search events. We can notice the following: If you look at the py script, you will notice that the example uses an MS Windows supported socket.gethostbyname_ex(host) function The host field has the same name in the lookup table and the events, so you don't need to do anything else Consider the following search command: sourcetype=tm1* | lookup dnslookup host | table host, ip When you run this command, Splunk uses the lookup table to pass the values for the host field as a CSV file (the text CSV file we looked at earlier) into the external command script. The py script then outputs the results (with both the host and ip fields populated) and returns it to Splunk, which populates the ip field in a result table:   Output of the py script with both the host and ip fields populated Time-based lookups If your lookup table has a field value that represents time, you can use the time field to set up a Splunk fields lookup. As mentioned earlier, the Splunk transforms.conf file can be modified to add a lookup stanza. For example, the following screenshot shows a file named MasteringDCHP.csv:   You can add the following code to the transforms.conf file: [MasteringDCHP]filename = MasteringDCHP.csvtime_field = TimeStamptime_format = %d/%m/%y %H:%M:%S $pmax_offset_secs = <integer>min_offset_secs = <integer> The file parameters are defined as follows: [MasteringDCHP]: This is the report stanza filename: This is the name of the CSV file to be used as the lookup table time_field: This is the field in the file that contains the time information and is to be used as the timestamp time_format: This indicates what format the time field is in max_offset_secs and min_offset_secs: This indicates min/max amount of offset time for an event to occur after a lookup entry Be careful with the preceding values; the offset relates to the timestamp in your lookup (CSV) file. Setting a tight (small) offset range might reduce the effectiveness of your lookup results! The last step will be to restart Splunk. An easier way to create a time-based lookup Again, it's a lot easier to use the Splunk Web interface to set up our lookup. Here is the step-by-step process: From Settings, select Lookups, and then Lookup table files: In the Lookup table files page, click on New, configure our lookup file, and then click on Save: You should receive the Successfully saved "MasterDHCP" in search message: Next, select Lookup definitions and from this page, click on New: In the Add new page, we define our lookup table with the following information: Destination app: For this, select search from the drop-down list Name: For this, enter MasterDHCP (this is the name you'll use in your lookup) Type: For this, select File-based (as this lookup table definition is a CSV file) Lookup file: For this, select the name of the file to be used from the drop-down list (ours is MasteringDCHP) Configure time-based lookup: Check this checkbox Name of time field: For this, enter TimeStamp (this is the field name in our file that contains the time information) Time format: For this, enter the string to describe to Splunk the format of our time field (our field uses this format: %d%m%y %H%M%S) You can leave the rest blank and click on Save. You should receive the Successfully saved "MasterDHCP" in search message: Now, we are ready to try our search: sourcetype=dh* | Lookup MasterDHCP IP as "IP" | table DHCPTimeStamp, IP, UserId | sort UserId The following screenshot shows the output:   Seeing double? Lookup table definitions are indicated with the attribute LOOKUP-<class> in the Splunk configuration file, props.conf, or in the web interface under Settings | Lookups | Lookup definitions. If you use the Splunk Web interface (which we've demonstrated throughout this article) to set up or define your lookup table definitions, Splunk will prevent you from creating duplicate table names, as shown in the following screenshot:   However, if you define your lookups using the configuration settings, it is important to try and keep your table definition names unique. If you do give the same name to multiple lookups, the following rules apply: If you have defined lookups with the same stanza (that is, using the same host, source, or source type), the first defined lookup in the configuration file wins and overrides all others. If lookups have different stanzas but overlapping events, the following logic is used by Splunk: Events that match the host get the host lookup Events that match the sourcetype get the sourcetype lookup Events that match both only get the host lookup It is a proven practice recommendation to make sure that all of your lookup stanzas have unique names. Command roundup This section lists several important Splunk commands you will use when working with lookups. The lookup command The Splunk lookup command is used to manually invoke field lookups using a Splunk lookup table that is previously defined. You can use Splunk Web (or the transforms.conf file) to define your lookups. If you do not specify OUTPUT or OUTPUTNEW, all fields in the lookup table (excluding the lookup match field) will be used by Splunk as output fields. Conversely, if OUTPUT is specified, the output lookup fields will overwrite existing fields and if OUTPUTNEW is specified, the lookup will not be performed for events in which the output fields already exist. For example, if you have a lookup table specified as iptousername with (at least) two fields, IP and UserId, for each event, Splunk will look up the value of the field IP in the table and for any entries that match, the value of the UserId field in the lookup table will be written to the field user_name in the event. The query is as follows: ... Lookup iptousernameIP as "IP" output UserId as user_name Always strive to perform lookups after any reporting commands in your search pipeline, so that the lookup only needs to match the results of the reporting command and not every individual event. The inputlookup and outputlookup commands The inputlookup command allows you to load search results from a specified static lookup table. It reads in a specified CSV filename (or a table name as specified by the stanza name in transforms.conf). If the append=t (that is, true) command is added, the data from the lookup file is appended to the current set of results (instead of replacing it). The outputlookup command then lets us write the results' events to a specified static lookup table (as long as this output lookup table is defined). So, here is an example of reading in the MasterDHCP lookup table (as specified in transforms.conf) and writing these event results to the lookup table definition NewMasterDHCP: | inputlookup MasterDHCP | outputlookup NewMasterDHCP After running the preceding command, we can see the following output:   Note that we can add the append=t command to the search in the following fashion: | inputlookup MasterDHCP.csv | inputlookup NewMasterDHCP.csv append=t | The inputcsv and outputcsv commands The inputcsv command is similar to the inputlookup command; in this, it loads search results, but this command loads from a specified CSV file. The filename must refer to a relative path in $SPLUNK_HOME/var/run/splunk and if the specified file does not exist and the filename did not have an extension, then a filename with a .csv extension is assumed. The outputcsv command lets us write our result events to a CSV file. Here is an example where we read in a CSV file named splunk_master.csv, search for the text phrase FPM, and then write any matching events to a CSV file named FPMBU.csv: | inputcsv splunk_master.csv | search "Business Unit Name"="FPM" | outputcsv FPMBU.csv The following screenshot shows the results from the preceding search command:   The following screenshot shows the resulting file generated as a result of the preceding command:   Here is another example where we read in the same CSV file (splunk_master.csv) and write out only events from 51 to 500: | inputcsv splunk_master start=50 max=500 Events are numbered starting with zero as the first entry (rather than 1). Summary In this article, we defined Splunk lookups and discussed their value. We also went through the two types of lookups, static and dynamic, and saw detailed, working examples of each. Various Splunk commands typically used with the lookup functionality were also presented. Resources for Article: Further resources on this subject: Working with Apps in Splunk [article] Processing Tweets with Apache Hive [article] Indexes [article]

In this article, Marco Schwartz and Stefan Buttigieg, the authors of the Arduino Android Blueprints, we are going to use most of the concepts we have learned to control a mobile robot via an Android app. The robot will have two motors that we can control, and also an ultrasonic sensor in the front so that it can detect obstacles. The robot will also have a BLE chip so that it can receive commands from the Android app. The following will be the major takeaways: Building a mobile robot based on the Arduino platform Connecting a BLE module to the Arduino robot (For more resources related to this topic, see here.) Configuring the hardware We are first going to assemble the robot itself, and then see how to connect the Bluetooth module and the ultrasonic sensor. To give you an idea of what you should end up with, the following is a front-view image of the robot when fully assembled: The following image shows the back of the robot when fully assembled: The first step is to assemble the robot chassis. To do so, you can watch the DFRobot assembly guide at https://www.youtube.com/watch?v=tKakeyL_8Fg. Then, you need to attach the different Arduino boards and shields to the robot. Use the spacers found in the robot chassis kit to mount the Arduino Uno board first. Then put the Arduino motor shield on top of that. At this point, use the screw header terminals to connect the two DC motors to the motor shield. This is how it should look at this point: Finally, mount the prototyping shield on top of the motor shield. We are now going to connect the BLE module and the ultrasonic sensor to the Arduino prototyping shield. The following is a schematic diagram showing the connections between the Arduino Uno board (done via the prototyping shield in our case) and the components: Now perform the following steps: First, we are now going to connect the BLE module. Place the module on the prototyping shield. Connect the power supply of the module as follows: GND goes to the prototyping shield's GND pin, and VIN goes to the prototyping shield's +5V. After that, you need to connect the different wires responsible for the SPI interface: SCK to Arduino pin 13, MISO to Arduino pin 12, and MOSI to Arduino pin 11. Then connect the REQ pin to Arduino pin 10. Finally, connect the RDY pin to Arduino pin 2 and the RST pin to Arduino pin 9. For the URM37 module, connect the VCC pin of the module to Arduino +5V, GND to GND, and the PWM pin to the Arduino A3 pin. To review the pin order on the URM37 module, you can check the official DFRobot documentation at http://www.dfrobot.com/wiki/index.php?title=URM37_V3.2_Ultrasonic_Sensor_(SKU:SEN0001). The following is a close-up image of the prototyping shield with the BLE module connected: Finally, connect the 7.4 V battery to the Arduino Uno board power jack. The battery is simply placed below the Arduino Uno board. Testing the robot We are now going to write a sketch to test the different functionalities of the robot, first without using Bluetooth. As the sketch is quite long, we will look at the code piece by piece. Before you proceed, make sure that the battery is always plugged into the robot. Now perform the following steps: The sketch starts by including the aREST library that we will use to control the robot via serial commands: #includ e <aREST.h> Now we declare which pins the motors are connected to: int speed_motor1 = 6; int speed_motor2 = 5; int direction_motor1 = 7; int direction_motor2 = 4; We also declare which pin the ultrasonic sensor is connected to: int distance_sensor = A3; Then, we create an instance of the aREST library: aREST rest = aREST(); To store the distance data measured by the ultrasonic sensor, we declare a distance variable: int distance; In the setup() function of the sketch, we first initialize serial communications that we will use to communicate with the robot for this test: Serial.begin(115200); We also expose the distance variable to the REST API, so we can access it easily: rest.variable("distance",&distance); To control the robot, we are going to declare a whole set of functions that will perform the basic operations: going forward, going backward, turning on itself (left or right), and stopping. We will see the details of these functions in a moment; for now, we just need to expose them to the API: rest.function("forward",forward); rest.function("backward",backward); rest.function("left",left); rest.function("right",right); rest.function("stop",stop); We also give the robot an ID and a name: rest.set_id("001"); rest.set_name("mobile_robot"); In the loop() function of the sketch, we first measure the distance from the sensor: distance = measure_distance(distance_sensor); We then handle the requests using the aREST library: rest.handle(Serial); Now, we will look at the functions for controlling the motors. They are all based on a function to control a single motor, where we need to set the motor pins, the speed, and the direction of the motor: void send_motor_command(int speed_pin, int direction_pin, int pwm, boolean dir) { analogWrite(speed_pin, pwm); // Set PWM control, 0 for stop, and 255 for maximum speed digitalWrite(direction_pin, dir); // Dir set the rotation direction of the motor (true or false means forward or reverse) } Based on this function, we can now define the different functions to move the robot, such as forward: int forward(String command) { send_motor_command(speed_motor1,direction_motor1,100,1); send_motor_command(speed_motor2,direction_motor2,100,1); return 1; } We also define a backward function, simply inverting the direction of both motors: int backward(String command) { send_motor_command(speed_motor1,direction_motor1,100,0); send_motor_command(speed_motor2,direction_motor2,100,0); return 1; } To make the robot turn left, we simply make the motors rotate in opposite directions: int left(String command) { send_motor_command(speed_motor1,direction_motor1,75,0); send_motor_command(speed_motor2,direction_motor2,75,1); return 1; } We also have a function to stop the robot: int stop(String command) { send_motor_command(speed_motor1,direction_motor1,0,1); send_motor_command(speed_motor2,direction_motor2,0,1); return 1; } There is also a function to make the robot turn right, which is not detailed here. We are now going to test the robot. Before you do anything, ensure that the battery is always plugged into the robot. This will ensure that the motors are not trying to get power from your computer USB port, which could damage it. Also place some small support at the bottom of the robot so that the wheels don't touch the ground. This will ensure that you can test all the commands of the robot without the robot moving too far from your computer, as it is still attached via the USB cable. Now you can upload the sketch to your Arduino Uno board. Open the serial monitor and type the following: /forward This should make both the wheels of the robot turn in the same direction. You can also try the other commands to move the robot to make sure they all work properly. Then, test the ultrasonic distance sensor by typing the following: /distance You should get back the distance (in centimeters) in front of the sensor: {"distance": 24, "id": "001", "name": "mobile_robot", "connected": true} Try changing the distance by putting your hand in front of the sensor and typing the command again. Writing the Arduino sketch Now that we have made sure that the robot is working properly, we can write the final sketch that will receive the commands via Bluetooth. As the sketch shares many similarities with the test sketch, we are only going to see what is added compared to the test sketch. We first need to include more libraries: #include <SPI.h> #include "Adafruit_BLE_UART.h" #include <aREST.h> We also define which pins the BLE module is connected to: #define ADAFRUITBLE_REQ 10 #define ADAFRUITBLE_RDY 2     // This should be an interrupt pin, on Uno thats #2 or #3 #define ADAFRUITBLE_RST 9 We have to create an instance of the BLE module: Adafruit_BLE_UART BTLEserial = Adafruit_BLE_UART(ADAFRUITBLE_REQ, ADAFRUITBLE_RDY, ADAFRUITBLE_RST); In the setup() function of the sketch, we initialize the BLE chip: BTLEserial.begin(); In the loop() function, we check the status of the BLE chip and store it in a variable: BTLEserial.pollACI(); aci_evt_opcode_t status = BTLEserial.getState(); If we detect that a device is connected to the chip, we handle the incoming request with the aREST library, which will allow us to use the same commands as before to control the robot: if (status == ACI_EVT_CONNECTED) { rest.handle(BTLEserial); } You can now upload the code to your Arduino board, again by making sure that the battery is connected to the Arduino Uno board via the power jack. You can now move on to the development of the Android application to control the robot. Summary In this article, we managed to create our very own mobile robot together with a companion Android application that we can use to control our robot. We achieved this step by step by setting up an Arduino-enabled robot and coding the companion Android application. It uses the BLE software and hardware of an Android physical device running on Android 4.3 or higher. Resources for Article: Further resources on this subject: Our First Project – A Basic Thermometer [article] Hardware configuration [article] Avoiding Obstacles Using Sensors [article]