How-To Tutorials

(For more resources on Microsoft, see here.) Creating your first WCF application in Visual Studio 2008 You start creating a WCF project by creating a new project from File | New | Project.... This opens the New Project window. You can see that there are four different templates available. We will be using the WCF Service Library template. Change the default name and provide a name for the project (herein JayWcf01) and click OK. The project JayWcf01 gets created with the folder structure shown in the next image: If you were to expand References node in the above you would notice that System.ServiceModel is already referenced. If it is not, for some reason, you can bring it in by using the Add Reference... window which is displayed when you right click the project in the Solution Explorer. IService1.vb is a service interface file as shown in the next listing. This defines the service contract and the operations expected of the service. If you change the interface name "IService1" here, you must also update the reference to "IService1" in App.config. <ServiceContract()> _Public Interface IService1 <OperationContract()> _ Function GetData(ByVal value As Integer) As String <OperationContract()> _ Function GetDataUsingDataContract(ByVal composite As CompositeType) As CompositeType ' TODO: Add your service operations hereEnd Interface' Use a data contract as illustrated in the sample below to add composite types to service operations<DataContract()> _Public Class CompositeType Private boolValueField As Boolean Private stringValueField As String <DataMember()> _ Public Property BoolValue() As Boolean Get Return Me.boolValueField End Get Set(ByVal value As Boolean) Me.boolValueField = value End Set End Property <DataMember()> _ Public Property StringValue() As String Get Return Me.stringValueField End Get Set(ByVal value As String) Me.stringValueField = value End Set End PropertyEnd Class The Service Contract is a contract that will be agreed to between the Client and the Server. Both the Client and the Server should be working with the same service contract. The one shown above is in the server. Inside the service, data is handled as simple (e.g. GetData) or complex types (e.g. GetDataUsingDataContract). However outside the Service these are handled as XML Schema Definitions. WCF Data contracts provides a mapping between the data defined in the code and the XML Schema defined by W3C organization, the standards organization. The service performed when the terms of the contract are properly adhered to is in the listing of Service1.vb file shown here. ' NOTE: If you change the class name "Service1" here, you must also update the reference to "Service1" in App.config.Public Class Service1 Implements IService1 Public Function GetData(ByVal value As Integer) As _ String Implements IService1.GetData Return String.Format("You entered: {0}", value) End Function Public Function GetDataUsingDataContract(ByVal composite As CompositeType) _ As CompositeType Implements IService1.GetDataUsingDataContract If composite.BoolValue Then composite.StringValue = (composite.StringValue & "Suffix") End If Return composite End FunctionEnd Class Service1 is defining two methods of the service by way of Functions. The GetData accepts a number and returns a string. For example, if the Client enters a value 50, the Server response will be "You entered: 50". The function GetDataUsingDataContract returns a Boolean and a String with 'Suffix' appended for an input which consists of a Boolean and a string. The JayWcf01 is a completed program with a default example contract IService1 and a defined service, Service1. This program is complete in itself. It is a good practice to provide your own names for the objects. Notwithstanding the default names are accepted in this demo. In what follows we test this program as is and then slightly modify the contract and test it again. The testing in the next section will invoke an in-built client and then later on we will publish it to the localhost which is an IIS 7 web server. How to test this program The program has a valid pair of contract and service and we should be able to test this service. The Windows Communication Foundation allows Visual Studio 2008 (also Visual Studio 2010 Express) to launch a host to test the service with a client. Build the program and after it succeeds hit F5. The WcfSvcHost is spawned which stays in the taskbar as shown. You can click WcfSvcHost to display the WCF Service Host window popping-up as shown. The host gets started as shown here. The service is hosted on the developmental server. This is immediately followed by the WCF Test Client user interface popping-up as shown. In this harness you can test the service.

In this article by Yuvraj Gupta, author of the book, Kibana Essentials, explains Kibana is a tool that is part of the ELK stack, which consists of Elasticsearch, Logstash, and Kibana. It is built and developed by Elastic. Kibana is a visualization platform that is built on top of Elasticsearch and leverages the functionalities of Elasticsearch. (For more resources related to this topic, see here.) To understand Kibana better, let's check out the following diagram: This diagram shows that Logstash is used to push data directly into Elasticsearch. This data is not limited to log data, but can include any type of data. Elasticsearch stores data that comes as input from Logstash, and Kibana uses the data stored in Elasticsearch to provide visualizations. So, Logstash provides an input stream of data to Elasticsearch, from which Kibana accesses the data and uses it to create visualizations. Kibana acts as an over-the-top layer of Elasticsearch, providing beautiful visualizations for data (structured or nonstructured) stored in it. Kibana is an open source analytics product used to search, view, and analyze data. It provides various types of visualizations to visualize data in the form of tables, charts, maps, histograms, and so on. It also provides a web-based interface that can easily handle a large amount of data. It helps create dashboards that are easy to create and helps query data in real time. Dashboards are nothing but an interface for underlying JSON documents. They are used for saving, templating, and exporting. They are simple to set up and use, which helps us play with data stored in Elasticsearch in minutes without requiring any coding. Kibana is an Apache-licensed product that aims to provide a flexible interface combined with the powerful searching capabilities of Elasticsearch. It requires a web server (included in the Kibana 4 package) and any modern web browser, that is, a browser that supports industry standards and renders the web page in the same way across all browsers, to work. It connects to Elasticsearch using the REST API. It helps to visualize data in real time with the use of dashboards to provide real-time insights. As Kibana uses the functionalities of Elasticsearch, it is easier to learn Kibana by understanding the core functionalities of Elasticsearch. In this article, we are going to take a look at the following topics: The basic concepts of Elasticsearch Installation of Java Installation of Elasticsearch Installation of Kibana Importing a JSON file into Elasticsearch Understanding Elasticsearch Elasticsearch is a search server built on top of Lucene (licensed under Apache), which is completely written in Java. It supports distributed searches in a multitenant environment. It is a scalable search engine allowing high flexibility of adding machines easily. It provides a full-text search engine combined with a RESTful web interface and JSON documents. Elasticsearch harnesses the functionalities of Lucene Java Libraries, adding up by providing proper APIs, scalability, and flexibility on top of the Lucene full-text search library. All querying done using Elasticsearch, that is, searching text, matching text, creating indexes, and so on, is implemented by Apache Lucene. Without a setup of an Elastic shield or any other proxy mechanism, any user with access to Elasticsearch API can view all the data stored in the cluster. The basic concepts of Elasticsearch Let's explore some of the basic concepts of Elasticsearch: Field: This is the smallest single unit of data stored in Elasticsearch. It is similar to a column in a traditional relational database. Every document contains key-value pairs, which are referred to as fields. Values in a field can contain a single value, such as integer [27], string ["Kibana"], or multiple values, such as array [1, 2, 3, 4, 5]. The field type is responsible for specifying which type of data can be stored in a particular field, for example, integer, string, date, and so on. Document: This is the simplest unit of information stored in Elasticsearch. It is a collection of fields. It is considered similar to a row of a table in a traditional relational database. A document can contain any type of entry, such as a document for a single restaurant, another document for a single cuisine, and yet another for a single order. Documents are in JavaScript Object Notation (JSON), which is a language-independent data interchange format. JSON contains key-value pairs. Every document that is stored in Elasticsearch is indexed. Every document contains a type and an ID. An example of a document that has JSON values is as follows: { "name": "Yuvraj", "age": 22, "birthdate": "2015-07-27", "bank_balance": 10500.50, "interests": ["playing games","movies","travelling"], "movie": {"name":"Titanic","genre":"Romance","year" : 1997} } In the preceding example, we can see that the document supports JSON, having key-value pairs, which are explained as follows: The name field is of the string type The age field is of the numeric type The birthdate field is of the date type The bank_balance field is of the float type The interests field contains an array The movie field contains an object (dictionary) Type: This is similar to a table in a traditional relational database. It contains a list of fields, which is defined for every document. A type is a logical segregation of indexes, whose interpretation/semantics entirely depends on you. For example, you have data about the world and you put all your data into an index. In this index, you can define a type for continent-wise data, another type for country-wise data, and a third type for region-wise data. Types are used with a mapping API; it specifies the type of its field. An example of type mapping is as follows: { "user": { "properties": { "name": { "type": "string" }, "age": { "type": "integer" }, "birthdate": { "type": "date" }, "bank_balance": { "type": "float" }, "interests": { "type": "string" }, "movie": { "properties": { "name": { "type": "string" }, "genre": { "type": "string" }, "year": { "type": "integer" } } } } } } Now, let's take a look at the core data types specified in Elasticsearch, as follows: Type Definition string This contains text, for example, "Kibana" integer This contains a 32-bit integer, for example, 7 long This contains a 64-bit integer float IEEE float, for example, 2.7 double This is a double-precision float boolean This can be true or false date This is the UTC date/time, for example, "2015-06-30T13:10:10" geo_point This is the latitude or longitude Index: This is a collection of documents (one or more than one). It is similar to a database in the analogy with traditional relational databases. For example, you can have an index for user information, transaction information, and product type. An index has a mapping; this mapping is used to define multiple types. In other words, an index can contain single or multiple types. An index is defined by a name, which is always used whenever referring to an index to perform search, update, and delete operations for documents. You can define any number of indexes you require. Indexes also act as logical namespaces that map documents to primary shards, which contain zero or more replica shards for replicating data. With respect to traditional databases, the basic analogy is similar to the following: MySQL => Databases => Tables => Columns/Rows Elasticsearch => Indexes => Types => Documents with Fields You can store a single document or multiple documents within a type or index. As a document is within an index, it must also be assigned to a type within an index. Moreover, the maximum number of documents that you can store in a single index is 2,147,483,519 (2 billion 147 million), which is equivalent to Integer.Max_Value. ID: This is an identifier for a document. It is used to identify each document. If it is not defined, it is autogenerated for every document.The combination of index, type, and ID must be unique for each document. Mapping: Mappings are similar to schemas in a traditional relational database. Every document in an index has a type. A mapping defines the fields, the data type for each field, and how the field should be handled by Elasticsearch. By default, a mapping is automatically generated whenever a document is indexed. If the default settings are overridden, then the mapping's definition has to be provided explicitly. Node: This is a running instance of Elasticsearch. Each node is part of a cluster. On a standalone machine, each Elasticsearch server instance corresponds to a node. Multiple nodes can be started on a single standalone machine or a single cluster. The node is responsible for storing data and helps in the indexing/searching capabilities of a cluster. By default, whenever a node is started, it is identified and assigned a random Marvel Comics character name. You can change the configuration file to name nodes as per your requirement. A node also needs to be configured in order to join a cluster, which is identifiable by the cluster name. By default, all nodes join the Elasticsearch cluster; that is, if any number of nodes are started up on a network/machine, they will automatically join the Elasticsearch cluster. Cluster: This is a collection of nodes and has one or multiple nodes; they share a single cluster name. Each cluster automatically chooses a master node, which is replaced if it fails; that is, if the master node fails, another random node will be chosen as the new master node, thus providing high availability. The cluster is responsible for holding all of the data stored and provides a unified view for search capabilities across all nodes. By default, the cluster name is Elasticsearch, and it is the identifiable parameter for all nodes in a cluster. All nodes, by default, join the Elasticsearch cluster. While using a cluster in the production phase, it is advisable to change the cluster name for ease of identification, but the default name can be used for any other purpose, such as development or testing.The Elasticsearch cluster contains single or multiple indexes, which contain single or multiple types. All types contain single or multiple documents, and every document contains single or multiple fields. Sharding: This is an important concept of Elasticsearch while understanding how Elasticsearch allows scaling of nodes, when having a large amount of data termed as big data. An index can store any amount of data, but if it exceeds its disk limit, then searching would become slow and be affected. For example, the disk limit is 1 TB, and an index contains a large number of documents, which may not fit completely within 1 TB in a single node. To counter such problems, Elasticsearch provides shards. These break the index into multiple pieces. Each shard acts as an independent index that is hosted on a node within a cluster. Elasticsearch is responsible for distributing shards among nodes. There are two purposes of sharding: allowing horizontal scaling of the content volume, and improving performance by providing parallel operations across various shards that are distributed on nodes (single or multiple, depending on the number of nodes running).Elasticsearch helps move shards among multiple nodes in the event of an addition of new nodes or a node failure. There are two types of shards, as follows: Primary shard: Every document is stored within a primary index. By default, every index has five primary shards. This parameter is configurable and can be changed to define more or fewer shards as per the requirement. A primary shard has to be defined before the creation of an index. If no parameters are defined, then five primary shards will automatically be created.Whenever a document is indexed, it is usually done on a primary shard initially, followed by replicas. The number of primary shards defined in an index cannot be altered once the index is created. Replica shard: Replica shards are an important feature of Elasticsearch. They help provide high availability across nodes in the cluster. By default, every primary shard has one replica shard. However, every primary shard can have zero or more replica shards as required. In an environment where failure directly affects the enterprise, it is highly recommended to use a system that provides a failover mechanism to achieve high availability. To counter this problem, Elasticsearch provides a mechanism in which it creates single or multiple copies of indexes, and these are termed as replica shards or replicas. A replica shard is a full copy of the primary shard. Replica shards can be dynamically altered. Now, let's see the purposes of creating a replica. It provides high availability in the event of failure of a node or a primary shard. If there is a failure of a primary shard, replica shards are automatically promoted to primary shards. Increase performance by providing parallel operations on replica shards to handle search requests.A replica shard is never kept on the same node as that of the primary shard from which it was copied. Inverted index: This is also a very important concept in Elasticsearch. It is used to provide fast full-text search. Instead of searching text, it searches for an index. It creates an index that lists unique words occurring in a document, along with the document list in which each word occurs. For example, suppose we have three documents. They have a text field, and it contains the following: I am learning Kibana Kibana is an amazing product Kibana is easy to use To create an inverted index, the text field is broken into words (also known as terms), a list of unique words is created, and also a listing is done of the document in which the term occurs, as shown in this table: Term Doc 1 Doc 2 Doc 3 I X     Am X     Learning X     Kibana X X X Is   X X An   X   Amazing   X   Product   X   Easy     X To     X Use     X Now, if we search for is Kibana, Elasticsearch will use an inverted index to display the results: Term Doc 1 Doc 2 Doc 3 Is   X X Kibana X X X With inverted indexes, Elasticsearch uses the functionality of Lucene to provide fast full-text search results. An inverted index uses an index based on keywords (terms) instead of a document-based index. REST API: This stands for Representational State Transfer. It is a stateless client-server protocol that uses HTTP requests to store, view, and delete data. It supports CRUD operations (short for Create, Read, Update, and Delete) using HTTP. It is used to communicate with Elasticsearch and is implemented by all languages. It communicates with Elasticsearch over port 9200 (by default), which is accessible from any web browser. Also, Elasticsearch can be directly communicated with via the command line using the curl command. cURL is a command-line tool used to send, view, or delete data using URL syntax, as followed by the HTTP structure. A cURL request is similar to an HTTP request, which is as follows: curl -X <VERB> '<PROTOCOL>://<HOSTNAME>:<PORT>/<PATH>?<QUERY_STRING>' -d '<BODY>' The terms marked within the <> tags are variables, which are defined as follows: VERB: This is used to provide an appropriate HTTP method, such as GET (to get data), POST, PUT (to store data), or DELETE (to delete data). PROTOCOL: This is used to define whether the HTTP or HTTPS protocol is used to send requests. HOSTNAME: This is used to define the hostname of a node present in the Elasticsearch cluster. By default, the hostname of Elasticsearch is localhost. PORT: This is used to define the port on which Elasticsearch is running. By default, Elasticsearch runs on port 9200. PATH: This is used to define the index, type, and ID where the documents will be stored, searched, or deleted. It is specified as index/type/ID. QUERY_STRING: This is used to define any additional query parameter for searching data. BODY: This is used to define a JSON-encoded request within the body. In order to put data into Elasticsearch, the following curl command is used: curl -XPUT 'http://localhost:9200/testing/test/1' -d '{"name": "Kibana" }' Here, testing is the name of the index, test is the name of the type within the index, and 1 indicates the ID number. To search for the preceding stored data, the following curl command is used: curl -XGET 'http://localhost:9200/testing/_search? The preceding commands are provided just to give you an overview of the format of the curl command. Prerequisites for installing Kibana 4.1.1 The following pieces of software need to be installed before installing Kibana 4.1.1: Java 1.8u20+ Elasticsearch v1.4.4+ A modern web browser—IE 10+, Firefox, Chrome, Safari, and so on The installation process will be covered separately for Windows and Ubuntu so that both types of users are able to understand the process of installation easily. Installation of Java In this section, JDK needs to be installed so as to access Elasticsearch. Oracle Java 8 (update 20 onwards) will be installed as it is the recommended version for Elasticsearch from version 1.4.4 onwards. Installation of Java on Ubuntu 14.04 Install Java 8 using the terminal and the apt package in the following manner: Add the Oracle Java Personal Package Archive (PPA) to the apt repository list: sudo add-apt-repository -y ppa:webupd8team/java In this case, we use a third-party repository; however, the WebUpd8 team is trusted to install Java. It does not include any Java binaries. Instead, the PPA directly downloads from Oracle and installs it. As shown in the preceding screenshot, you will initially be prompted for the password for running the sudo command (only when you have not logged in as root), and on successful addition to the repository, you will receive an OK message, which means that the repository has been imported. Update the apt package database to include all the latest files under the packages: sudo apt-get update Install the latest version of Oracle Java 8: sudo apt-get -y install oracle-java8-installer Also, during the installation, you will be prompted to accept the license agreement, which pops up as follows: To check whether Java has been successfully installed, type the following command in the terminal:java –version This signifies that Java has been installed successfully. Installation of Java on Windows We can install Java on windows by going through the following steps: Download the latest version of the Java JDK from the Sun Microsystems site at http://www.oracle.com/technetwork/java/javase/downloads/index.html:                                                                                     As shown in the preceding screenshot, click on the DOWNLOAD button of JDK to download. You will be redirected to the download page. There, you have to first click on the Accept License Agreement radio button, followed by the Windows version to download the .exe file, as shown here: Double-click on the file to be installed and it will open as an installer. Click on Next, accept the license by reading it, and keep clicking on Next until it shows that JDK has been installed successfully. Now, to run Java on Windows, you need to set the path of JAVA in the environment variable settings of Windows. Firstly, open the properties of My Computer. Select the Advanced system settings and then click on the Advanced tab, wherein you have to click on the environment variables option, as shown in this screenshot: After opening Environment Variables, click on New (under the System variables) and give the variable name as JAVA_HOME and variable value as C:Program FilesJavajdk1.8.0_45 (do check in your system where jdk has been installed and provide the path corresponding to the version installed as mentioned in system directory), as shown in the following screenshot: Then, double-click on the Path variable (under the System variables) and move towards the end of textbox. Insert a semicolon if it is not already inserted, and add the location of the bin folder of JDK, like this: %JAVA_HOME%bin. Next, click on OK in all the windows opened. Do not delete anything within the path variable textbox. To check whether Java is installed or not, type the following command in Command Prompt: java –version This signifies that Java has been installed successfully. Installation of Elasticsearch In this section, Elasticsearch, which is required to access Kibana, will be installed. Elasticsearch v1.5.2 will be installed, and this section covers the installation on Ubuntu and Windows separately. Installation of Elasticsearch on Ubuntu 14.04 To install Elasticsearch on Ubuntu, perform the following steps: Download Elasticsearch v 1.5.2 as a .tar file using the following command on the terminal:  curl -L -O https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.5.2.tar.gz Curl is a package that may not be installed on Ubuntu by the user. To use curl, you need to install the curl package, which can be done using the following command: sudo apt-get -y install curl Extract the downloaded .tar file using this command: tar -xvzf elasticsearch-1.5.2.tar.gzThis will extract the files and folder into the current working directory. Navigate to the bin directory within the elasticsearch-1.5.2 directory: cd elasticsearch-1.5.2/bin Now run Elasticsearch to start the node and cluster, using the following command:./elasticsearch The preceding screenshot shows that the Elasticsearch node has been started, and it has been given a random Marvel Comics character name. If this terminal is closed, Elasticsearch will stop running as this node will shut down. However, if you have multiple Elasticsearch nodes running, then shutting down a node will not result in shutting down Elasticsearch. To verify the Elasticsearch installation, open http://localhost:9200 in your browser. Installation of Elasticsearch on Windows The installation on Windows can be done by following similar steps as in the case of Ubuntu. To use curl commands on Windows, we will be installing GIT. GIT will also be used to import a sample JSON file into Elasticsearch using elasticdump, as described in the Importing a JSON file into Elasticsearch section. Installation of GIT To run curl commands on Windows, first download and install GIT, then perform the following steps: Download the GIT ZIP package from https://git-scm.com/download/win. Double-click on the downloaded file, which will walk you through the installation process. Keep clicking on Next by not changing the default options until the Finish button is clicked on. To validate the GIT installation, right-click on any folder in which you should be able to see the options of GIT, such as GIT Bash, as shown in the following screenshot: The following are the steps required to install Elasticsearch on Windows: Open GIT Bash and enter the following command in the terminal:  curl –L –O https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.5.2.zip Extract the downloaded ZIP package by either unzipping it using WinRar, 7Zip, and so on (if you don't have any of these, download one of them) or using the following command in GIT Bash: unzip elasticsearch-1.5.2.zip This will extract the files and folder into the directory. Then click on the extracted folder and navigate through it to reach the bin folder. Click on the elasticsearch.bat file to run Elasticsearch. The preceding screenshot shows that the Elasticsearch node has been started, and it is given a random Marvel Comics character's name. Again, if this window is closed, Elasticsearch will stop running as this node will shut down. However, if you have multiple Elasticsearch nodes running, then shutting down a node will not result in shutting down Elasticsearch. To verify the Elasticsearch installation, open http://localhost:9200 in your browser. Installation of Kibana In this section, Kibana will be installed. We will install Kibana v4.1.1, and this section covers installations on Ubuntu and Windows separately. Installation of Kibana on Ubuntu 14.04 To install Kibana on Ubuntu, follow these steps: Download Kibana version 4.1.1 as a .tar file using the following command in the terminal:  curl -L -O https://download.elasticsearch.org/kibana/kibana/kibana-4.1.1-linux-x64.tar.gz Extract the downloaded .tar file using this command: tar -xvzf kibana-4.1.1-linux-x64.tar.gz The preceding command will extract the files and folder into the current working directory. Navigate to the bin directory within the kibana-4.1.1-linux-x64 directory: cd kibana-4.1.1-linux-x64/bin Now run Kibana to start the node and cluster using the following command: Make sure that Elasticsearch is running. If it is not running and you try to start Kibana, the following error will be displayed after you run the preceding command: To verify the Kibana installation, open http://localhost:5601 in your browser. Installation of Kibana on Windows To install Kibana on Windows, perform the following steps: Open GIT Bash and enter the following command in the terminal:  curl -L -O https://download.elasticsearch.org/kibana/kibana/kibana-4.1.1-windows.zip Extract the downloaded ZIP package by either unzipping it using WinRar or 7Zip (download it if you don't have it), or using the following command in GIT Bash: unzip kibana-4.1.1-windows.zip This will extract the files and folder into the directory. Then click on the extracted folder and navigate through it to get to the bin folder. Click on the kibana.bat file to run Kibana. Make sure that Elasticsearch is running. If it is not running and you try to start Kibana, the following error will be displayed after you click on the kibana.bat file: Again, to verify the Kibana installation, open http://localhost:5601 in your browser. Additional information You can change the Elasticsearch configuration for your production environment, wherein you have to change parameters such as the cluster name, node name, network address, and so on. This can be done using the information mentioned in the upcoming sections.. Changing the Elasticsearch configuration To change the Elasticsearch configuration, perform the following steps: Run the following command in the terminal to open the configuration file: sudo vi ~/elasticsearch-1.5.2/config/elasticsearch.yml Windows users can open the elasticsearch.yml file from the config folder. This will open the configuration file as follows: The cluster name can be changed, as follows: #cluster.name: elasticsearch to cluster.name: "your_cluster_name". In the preceding figure, the cluster name has been changed to test. Then, we save the file. To verify that the cluster name has been changed, run Elasticsearch as mentioned in the earlier section. Then open http://localhost:9200 in the browser to verify, as shown here: In the preceding screenshot, you can notice that cluster_name has been changed to test, as specified earlier. Changing the Kibana configuration To change the Kibana configuration, follow these steps: Run the following command in the terminal to open the configuration file: sudo vi ~/kibana-4.1.1-linux-x64/config/kibana.yml Windows users can open the kibana.yml file from the config folder In this file, you can change various parameters such as the port on which Kibana works, the host address on which Kibana works, the URL of Elasticsearch that you wish to connect to, and so on For example, the port on which Kibana works can be changed by changing the port address. As shown in the following screenshot, port: 5601 can be changed to any other port, such as port: 5604. Then we save the file. To check whether Kibana is running on port 5604, run Kibana as mentioned earlier. Then open http://localhost:5604 in the browser to verify, as follows: Importing a JSON file into Elasticsearch To import a JSON file into Elasticsearch, we will use the elasticdump package. It is a set of import and export tools used for Elasticsearch. It makes it easier to copy, move, and save indexes. To install elasticdump, we will require npm and Node.js as prerequisites. Installation of npm In this section, npm along with Node.js will be installed. This section covers the installation of npm and Node.js on Ubuntu and Windows separately. Installation of npm on Ubuntu 14.04 To install npm on Ubuntu, perform the following steps: Add the official Node.js PPA: sudo curl --silent --location https://deb.nodesource.com/setup_0.12 | sudo bash - As shown in the preceding screenshot, the command will add the official Node.js repository to the system and update the apt package database to include all the latest files under the packages. At the end of the execution of this command, we will be prompted to install Node.js and npm, as shown in the following screenshot: Install Node.js by entering this command in the terminal: sudo apt-get install --yes nodejs This will automatically install Node.js and npm as npm is bundled within Node.js. To check whether Node.js has been installed successfully, type the following command in the terminal: node –v Upon successful installation, it will display the version of Node.js. Now, to check whether npm has been installed successfully, type the following command in the terminal: npm –v Upon successful installation, it will show the version of npm. Installation of npm on Windows To install npm on Windows, follow these steps: Download the Windows Installer (.msi) file by going to https://nodejs.org/en/download/. Double-click on the downloaded file and keep clicking on Next to install the software. To validate the successful installation of Node.js, right-click and select GIT Bash. In GIT Bash, enter this: node –v Upon successful installation, you will be shown the version of Node.js. To validate the successful installation of npm, right-click and select GIT Bash. In GIT Bash, enter the following line: npm –v Upon successful installation, it will show the version of npm. Installing elasticdump In this section, elasticdump will be installed. It will be used to import a JSON file into Elasticsearch. It requires npm and Node.js installed. This section covers the installation on Ubuntu and Windows separately. Installing elasticdump on Ubuntu 14.04 Perform these steps to install elasticdump on Ubuntu: Install elasticdump by typing the following command in the terminal: sudo npm install elasticdump -g Then run elasticdump by typing this command in the terminal: elasticdump Import a sample data (JSON) file into Elasticsearch, which can be downloaded from https://github.com/guptayuvraj/Kibana_Essentials and is named tweet.json. It will be imported into Elasticsearch using the following command in the terminal: elasticdump --bulk=true --input="/home/yuvraj/Desktop/tweet.json" --output=http://localhost:9200/ Here, input provides the location of the file, as shown in the following screenshot: As you can see, data is being imported to Elasticsearch from the tweet.json file, and the dump complete message is displayed when all the records are imported to Elasticsearch successfully. Elasticsearch should be running while importing the sample file. Installing elasticdump on Windows To install elasticdump on Windows, perform the following steps: Install elasticdump by typing the following command in GIT Bash: npm install elasticdump -g                                                                                                           Then run elasticdump by typing this command in GIT Bash: elasticdump Import the sample data (JSON) file into Elasticsearch, which can be downloaded from https://github.com/guptayuvraj/Kibana_Essentials and is named tweet.json. It will be imported to Elasticsearch using the following command in GIT Bash: elasticdump --bulk=true --input="C:UsersyguptaDesktoptweet.json" --output=http://localhost:9200/ Here, input provides the location of the file. The preceding screenshot shows data being imported to Elasticsearch from the tweet.json file, and the dump complete message is displayed when all the records are imported to Elasticsearch successfully. Elasticsearch should be running while importing the sample file. To verify that the data has been imported to Elasticsearch, open http://localhost:5601 in your browser, and this is what you should see: When Kibana is opened, you have to configure an index pattern. So, if data has been imported, you can enter the index name, which is mentioned in the tweet.json file as index: tweet. After the page loads, you can see to the left under Index Patterns the name of the index that has been imported (tweet). Now mention the index name as tweet. It will then automatically detect the timestamped field and will provide you with an option to select the field. If there are multiple fields, then you can select them by clicking on Time-field name, which will provide a drop-down list of all fields available, as shown here: Finally, click on Create to create the index in Kibana. After you have clicked on Create, it will display the various fields present in this index. If you do not get the options of Time-field name and Create after entering the index name as tweet, it means that the data has not been imported into Elasticsearch. Summary In this article, you learned about Kibana, along with the basic concepts of Elasticsearch. These help in the easy understanding of Kibana. We also looked at the prerequisites for installing Kibana, followed by a detailed explanation of how to install each component individually in Ubuntu and Windows. Resources for Article: Further resources on this subject: Understanding Ranges [article] Working On Your Bot [article] Welcome to the Land of BludBorne [article]

(For more resources related to this topic, see here.) Easy audio grouping Pro Tools offers an interesting shortcut to assist the process. To create a new audio group, the simplest way is to follow these steps: Select the tracks to be grouped. Press option + shift (Mac) or Alt + Shift (Windows) and click on the track output. Select New Track and create a mono or stereo auxiliary track. The bus name and routing will be automatically assigned. We can later select or show the corresponding tracks associated with any type of input or output by right-clicking on any track input or output. This also works for hardware sends. Multiple track outputs In Pro Tools, a track can have as many outputs as you like (as long as you have available voices); it can even be routed to all available outputs simultaneously. Hold the control key (Mac) or Start key (Windows) and click on the track's output to add another output. When a track is routed to multiple outputs, a + button is displayed like this: Routing to multiple outputs; this track is assigned to Mixbus and other outputs There are also key modifiers that we can use. Key modifier Mac Windows Add the desired output to all the tracks control + option Start + Alt Add the desired output to the selected tracks control + option + shift Start + Alt + Shift Stemming with multiple outputs Unfortunately, there is no easy way of creating stems or multitrack bounces automatically with Pro Tools 10; the process remains mainly manual. Pro Tools 11 addresses some of these issues in the next section, but for now, assuming that our master fader track contains processing, our first and only stemming option is to solo the appropriate tracks and use the bounce to disk function. If we have no processing for our master output fader, we can bounce inside the session instead by assigning multiple outputs to tracks or buses by grouping them and creating auxiliary sends. Using the key modifiers defined previously, you can quickly select tracks and assign their outputs or auxiliary sends to new audio tracks for recording. To stem tracks to a new audio track without affecting their current routing, use the following methods. Using multiple outputs Select the tracks. Hold control + option + shift (Mac) or Start + Alt + Shift (Windows). Click on New track from the track output menu. Create the appropriate audio track and record. Using auxiliary sends Select the tracks. Hold option + shift (Mac) or Alt + Shift (Windows). Click on N ew track from the track output menu. Create the appropriate audio track and record. We can select the desired recording format from the session window accessible from Setup | Session . If you're recording the track at anything other than a 32-bit float, make sure your levels do not exceed 0 dBfs. We can also use complex mixing routing techniques to our advantage, as this will give us many summing points from which we can easily create the New track stems. We will discuss more on how to increase routing complexity to create more summing points inside the mix. Sound layering Using auxiliary tracks is the best way to perform parallel processing in Pro Tools, but multiple output routing can also be very useful for sound layering, keeping your track sends free for other effects. By sending the output to multiple auxiliary tracks, we can very easily add textures and effects. As an example, I like to layer my basses this way with different distortions and other parallel processing. To me, sound layering is more suited to multiple outputs because their relative level mostly stays the same. Stemming with Pro Tools 11 Pro Tools 11 brings new offline bouncing capabilities with added simultaneous bounce from internal busses or physical outputs. Inside the Bounce to disk window, click on the + button to add another source to a maximum of 16. Each source can have a different file format, from mono to multichannel audio. Because we can bounce physical outputs, Pro Tools 11 also allows for stemming tracks that are processed using master fader inserts. These new features added to offline bounces have drastically improved the Pro Tools 10 workflow, saving us a lot of time and leaving us with the following two main stemming options: While working with multiple outputs, we can source our stem from the physical outputs directly While summing inside Pro Tools, it becomes a lot more relevant to group as many tracks as we can to create available bounce sources Increasing mixing complexity One of the biggest advantages of mixing inside a DAW over traditional analogue console setups is its expanded routing capabilities, giving us far more control over the signal. In this section, I'd like to discuss how to use buses to create more complex mixes, that is, creating extra summing points for several uses and comparing the digital approach over a traditional frontend analog mixing console setup. As we saw previously, Pro Tools 10 introduced a new 64-bit floating point mixer, allowing for almost unlimited headroom and greater precision, which made digital summing even more digitally perfect. This means that it is entirely up to the engineer to create his sound textures through recording or digital processing. Pro Tools 11 took this quest for digital perfection even further with a full 64-bit path. On the other hand, analog is technically flawed and has always been; every single piece of circuitry will color the signal in various amounts, but these are those imperfections that many of us came to love and successfully or not tried to recreate in the digital domain. A digital mixer will sum signals with a far lower harmonic distortion. Applying different distortions is therefore the key to restore an analog feel to the mix. The first step toward this approach is to increase the mixer's complexity to be able to apply different types of distortions in many more places. Audio groups or buses are the most basic routing idea. As an example, working with drums, I would group all the kicks together, the snare together, the toms together, the overheads together, the rooms together, and so on. Those subbuses will then be bused to a master drum bus. This allows us to do the following: Process multiple microphones at a time Bounce stems more easily later Have greater control over my individual drum sounds To me, routing is the heart of digital mixing and making it as complex as possible begins to open many new creative doors. But we can go a step further. My drum bus will be first bused to my rhythm bus, then to my instrumental bus, and finally to my mix bus. This introduces more advanced groupings based on other criteria such as rhythm or instrumental. This might not seem very useful at first, but it will make you think about the track in different ways, giving you control at every stages of the mix. Here is an example of what a complex mix would look like: An example of a more complex routing diagram I might not use all the possibilities all the time, and this is only a small example. But this helped me very much over the years to create better depth in my mixes and also considerably speed up my workflow while creating stem mixes. I have instant access to many more summing points in my mixes, allowing me to group and differentiate even more sounds from each other. The previous diagram only shows how to increase a mix complexity by busing tracks into each other, but these summing points are also open doors to some very effective parallel processing. Default output bus options While working with multiple buses, it becomes quite hard to systematically remember to update the track output to the user's Mixbus. We can change the default track's Pro Tools's default bus output through Preferences | I/O | Output | Default Output Bus . Along with changing the default bus output, there are a few key modifiers worth remembering to assign tracks to the desired output or the bus. Key modifier Mac Windows Assign all the tracks to the same I/O option Alt Assign all the selected tracks to the same I/O option + shift Alt + Shift Assign all the tracks' I/O incrementally command + option Ctrl + Alt Assign all the selected tracks' I/O incrementally command + option + shift Ctrl + Alt + Shift Greater control over parallel processing Parallel processing is widely used for mixing. It allows adding to the signal while preserving the original content. It usually sounds a lot better to mix this way rather than process everything on groups; drums can particularly benefit from this technique. The main issue with group processing is that it tends to make everything sound very clinical and lifeless, especially if too much dynamic treatment is applied. Digital dynamic processors especially do not compare well to their analog counterpart when it comes to attenuating levels without taking life out of the recording. For these reasons, I tend to complement my mixes with a lot of parallel processing to increase harmonics, depth, loudness, and focus frequencies rather than cutting or boosting them too much. Parallel processing allows me to get a richer sound. Creating parallel processing requires extra buses. We can use the track sends, multiple outputs, or duplicate the track to achieve quick results. As an example, using track sends in pre fader mode, I can quickly create a perfect copy of my signal independent of my main mix level to process and automate creatively. Select the tracks and press command + G (Mac) or Ctrl + G (Windows) to create a new group. Inside the Create group window, disable Follow Global and select Mix or Edit/Mix and the appropriate send and settings you would like to link.In my example, I will only link Volume and Mute . Now that we have created the group, a neat thing to do would be to copy the existing track volume levels to the send. To do so, perform the following steps: Press command + option + H (Mac) or Ctrl + Alt + H (Windows). The Copy to Send window appears. Select the correct destination send and the parameter to copy. We can copy everything, even automation, to have the exact same mix across the entire song. Now with the tracks still selected, display any send window, hold option + shift (Mac) or Alt + Shift (Windows), and click on PRE to make the sends pre fader and have the auxiliary mix independent of the main mix. Having copied the current mix to the send, we can not only use it as a perfect parallel copy, but now we also have the choice to alter the balance between the current mix and the send to better suit the type of parallel processing we will be applying. Trying to improve the low end might not require too much high frequency information, so I might change the balance to pull down the appropriate tracks. On the other hand, trying to improve my high end might benefit from pulling down low frequency content. Copying mix to send is also very useful while setting up cue mixes for recording. Using the copy to send function is very rewarding while working with parallel processing, but the default mixer's send view does not look very user friendly. Pro Tools offers an alternate way of displaying the send information inside the mixer window. We can focus on a particular send to display mini fader sends and level information. To do so, let's navigate to View | Sends A-E or View | Sends F-J and select the particular send you would like to display. We can also press command + click (Mac) or Ctrl + click (Windows) from the left-hand side of the send in the mixer view. We can display a maximum of two sends at a time in Pro Tools 10 and all of them simultaneously in Pro Tools 11, giving a great matrix view for parallel processing or headphone mixes. This new functionality is accessible from View | Expanded Sends . Pro Tools 11 Expanded Sends view, showing Sends A-E We now have a nice send display with linked parameters: To change the level of all the grouped tracks, keeping their relative level with the group active, just move one of the send fader To change one send value without affecting the entire group, hold control (Mac) or Start (Windows) and move the appropriate fader Advanced side chain Side chaining is one of the most commonly used techniques for music production, sound design, and mixing. Pro Tools allow for some interesting side chaining possibilities, allowing us to not only mix many different tracks and process them, but also to play with their timing by manipulating the automatic delay compensation. For bass I will stay with my drum recording and take the classic example of compressing the bass track according to the kick. By doing so, we make the groove tighter, reinforcing the impression that the bass player is locked to the drum groove. To do so, I could use the Bomb Factory BF76 plugin on the bass with my side chain activated on bus 15 carrying the kick. Once it is set up, every time the kick hits, the bass track gain will be pulled down, enhancing the groove and also giving more space in the mix for the "kick drum," but that is not all. Since Pro Tools uses buses to create external side chains, we can send as many signals as we'd like to control the bass accordingly. As an example, I could have the kick not only during the verses but could also add some of my percussion to the side chain mix. Tweaking the side chain balance mix can give you very natural results and give the impression that the bass player is grooving more with the entire performance. When we group multiple signals together into a side chain, we can also alter their frequency spectrum or even their dynamics to create different compression behaviors. Because my side chain information is contained on a bus, I can create a master fader or auxiliary track to process the audio before it reaches my plugin. While working with recorded drums, it might be useful to gate the side chain bus a lot more aggressively, since it cannot be heard. We can also boost one of the low kick frequencies and cut the high end to avoid hats, crashes, and so on triggering unwanted compressions. To do so, we could boost the low frequencies first and then gate the result more aggressively. Summary In this article, we learned how to use internal routing and automations to increase creative output. Resources for Article: Further resources on this subject: Audio Playback [Article] Automating the Audio Parameters – How it Works [Article] Getting Started with Adobe Premiere Pro CS6 Hotshot [Article]

Please refer to an earlier article by the author to learn how to build queries visually. Creating a View from an Existing Query To create a view from a query, you must have executed the query successfully. To be more precise, the view is created from the latest successfully executed query, not necessarily from the query currently in the Query Area. To further clarify, the following three examples are cases where the view is not created from the current query: Your current query fails, and immediately after you create a view from the query. The view created is not from the failed query. If the failed query is the first query in your MySQL Query Browser session, you can’t create any view. You have just moved forward or backward the query in the Query Area without executing it, and then your current query is not the latest successfully executed. You open a saved query that you have never executed successfully in your active Resultset. Additionally, if you’re changing your Resultset, the view created is from the latest successfully executed query that uses the currently active Resultset to display its output. To make sure your view is from the query you want, select the query, confirm it as written in the Query Area, execute the query, and then, immediately create its view. You create a view from an existing query by selecting Query | Create View from Select from the Menu bar. Type in the name you want to give to the view, and then click Create View. MySQL Query Browser creates the view. When successfully created, you can see the view in the Schemata. You can modify a view by editing it: Right-click the view and select Edit View. You can edit the CREATE view statement by right-clicking it and select Edit View. The CREATE view statement opens in its Script tab. When you finish editing, you can execute the modified view. If successful, the existing view is replaced with the modified one. To replace the view you’re editing with the modified view, change the name of the view before you execute it. If you want to keep the view you’re editing, remove the DROP VIEW statement.

Core Location framework – an overview Core Location implements all the three methods of location detection: GPS, Wi-Fi, and Cell Tower Triangulation. The developer can control location detection by only specifying the accuracy needed. Core Location then decides internally on which approach to use for actual location detection. When creating an application that uses the Core Location framework, you need to first add it to your project in Xcode and include the <CoreLocation/Corelocation.h> header files in your .m or .h file. The new iOS SDK 5 and Xcode 4.2 includes a nifty location simulation and debugging capability. Now we will understand how the location debugging feature helps you test your app by using different location values when your app is actually running in the simulator or a connected device. Time for action – location debugging Open the Hello Location application and run the application. If you are running your app on the iOS Simulator, then go to the Debug | Location Menu option where you can simulate multiple location inputs for your app, as shown in the following screenshot: If you are running your app on a connected device, you need to go to the Product | Debug | Simulate Location menu option in the Xcode 4 menu bar. Try changing to different locations and testing the app by clicking the Detect Location button in the app. Location data is reported in your application via the Core Location's Delegate object, CLLocationManagerDelegate. Based on the location service type used in your app, the corresponding Core Location Delegate function has to be implemented by your application to catch the appropriate location change event. We will look at it as we inspect the different Core Location Services. What just happened? We simulated location information on our iPhone, using the new feature of "Location Simulation" in iOS 5 SDK and Xcode 4.2. This new feature helps us analyze our app's behavior in different locations. In the preceding example, we changed the location values one-by-one and clicked on the Detect Location button to echo the geo co-ordinates of our labels. iOS 5 Location simulation includes significant location updates, region monitoring, and continuous location updates via the GPX file support (a GPX file is an XML file format that contains a sequence of Geo Coordinates, typically for Tours or Navigational purposes). Core location services The Core Location framework provides the following services: Standard location Significant change Region monitoring Geocoding and reverse Geocoding – CLGeocoder (Added in iOS 5 SDK) Direction using heading information Standard location Standard Location is the latitude and longitude information retrieved from Core Location. The Core Location Manager (the CLLocationManager object in the iOS SDK) returns this information in the CLLocation object. NSString *latitudeTextData = [[NSString alloc]initWithFormat: @"%g",locMgr.location.coordinate.latitude]; Here the location object is an instance of the CLLocation object that contains the latitude and longitude variables. Standard location service with the Core Location Manager is started with the startUpdatingLocation function. You can tell the Location Manager to stop updating the location with the stopUpdatingLocation function. distanceFilter and desiredAccuracy are two properties that define how often you will receive the location updates and how much accuracy (in meters) is required by your app. With distanceFilter, you will receive location information if the device has moved distance equal to or more than the value specified in the distanceFilter property. Accuracy of the location detections can be chosen from the following desiredAccuracy values: Table 1 Use/Try to use the lowest accuracy possible (the lowest accuracy your application can work with) to avoid more battery power consumption. Significant change With the Core Location framework, you can also request for location updates having significant location value changes only. This method provides excellent power saving options, as well as the ability of the device to send location updates even when your application is not running. This method uses Cellular Radio to detect the device location. To use the significant change location service in your app, you need to use the startMonitoringSignificantLocationChanges and stopMonitoringSignificantLocationChanges functions. Core Location framework caches the location data, it is a good idea to get the timestamp on the measurement objects to make sure your application receives the correct and updated location information. Region monitoring With the Region monitoring services, you can define geographical boundary-based tracking for your apps. Consider a simple example of a Weather app that can use Region Monitoring to detect the user's location based on physical boundaries and alert them if they cross a particular boundary, for example, if a user crosses a San Francisco city boundary towards San Jose, the app can trigger a boundary alert for the user and show him the new San Jose weather information. The startMonitoringForRegion and stopMonitoringForRegion methods of the location framework are used to start and stop region monitoring in your application. Boundary entering and exiting are monitored by locationManager:didEnterRegion and locationManager:didExitRegion. Boundary crossing deteci on also requires an accuracy factor to determine the crossing factor needed to trigger the alert. This is done by the startMonitoringForRegion:desiredAccuracy method. As with the Significant Change service, Region monitoring also works even if your application is not running. The most important part is that you need to register the Regions to be monitored with the device using the monitoredRegions property. Use smart programming techniques to shut down location services when not required in order to conserve battery power. Another good idea is to turn off location if accuracy does not improve over a course of time. Geocoding and reverse Geocoding – CLGeocoder The CLGeocoder along with the CLPlacemark object provide the Geocoding and Reverse Geocoding functions in the Core Location framework. Note that these are new APIs added in the iOS 5.0 SDK. The MKReverseGeocoder from the MapKit Framework has been deprecated. The CLGeocode object now handles the same. CLGeocode features as follows: Requests are asynchronous and support only one operation per request Supports multiple languages Supports Forward and Reverse Geocoding Does not require results to be displayed on a map Worldwide coverage Geocoding is done by any of the following three methods: geocodeAddressString:completionHandler: Geocodes a simple string, for example, Mountain View, San Francisco. geocodeAddressString:inRegion:completionHandler: Geocodes a specified string using regional information. Think of this as searching for the String Market Street in region San Francisco. geocodeAddressDictionary:completionHandler: Geocodes is a speci? ed address dictionary. This is a more structured geocoding request, usually providing the Address Street, Address City, and Address State fields in the AddressBook format. The following is an example code snippet for this function: CLGeocoder *geocoder =[[CLGeocoder alloc]init]; NSDictionary *address=[NSDictionary dictionaryWithObjectsAndKeys: @"32 Lincoln RoadRoad",kABPersonAddressStreetKey, @"Birmingham",kABPersonAddressCityKey,nil ]; [geocoder geocodeAddressDictionary:address completionHandler:^(NSArray *placemarks, NSError *error) { for(CLPlacemark *placemark in placemarks) { NSLog(@"Placemark %@",placemark); } }]; Don't forget to add the AddressBook framework in build phases in Xcode and import the header files required in your Hello_LocationViewController.m file from the Hello World example. #import <AddressBook/AddressBook.h> #import <AddressBook/ABPerson.h> You can find the code at the book's page at http://www.packtpub.com/iphone-location-aware-apps-beginners-guide/book – in a project called Hello Location – Geocode. Run the application and click on the Detect Location button and observe the Debug Console in Xcode; you should see an output as follows: Placemark 32 Lincoln Road, Solihull, England, B27 6, United Kingdom @ <+52.44378245,-1.81094734> +/- 100.00m This is the result of the geocoding, along with the accuracy of 100 meters. Reverse Geocoding is handled by the reverseGeocodeLocation:completionHandler method in the CLGeocoder class. The CLPlaceMark object is returned for both Forward and Reverse Geocoding. Direction using heading Heading information in the Core Location service signifies the direction in which the device is oriented. This information is very critical for augmented reality, navigation, and gaming applications. The direction in which a device is poini ng, reported by iOS devices with a magnetometer is known as heading, while direci on in which the iOS device is moving, reported by the GPS hardware, is known as course. The CLHeading object holds the heading data reported by the Location Manager. The startUpdatingHeading method in the Location Manager is used to start the heading update process, while stopUpdatingHeading is used to stop it. The CLHeading object contains the following properties: Table 2

Supported charts Pentaho Reporting relies on JFreeChart, an open source Java chart library, for charting visualization within reports. From within Report Designer, many chart types are supported. In the chart editor, two areas of properties appear when editing a chart. The first area of properties is related to chart rendering, and the second tabbed area of properties is related to the data that populates a chart. Following is the screenshot of the chart editor within Pentaho Report Designer: All chart types receive their data from three general types of datasets. The first type is known as a Category Dataset , where the dataset series and values are grouped by categories. A series is like a sub-group. If the exact category and series appear, the chart will sum the values into a single result. The following table is a simple example of a category dataset: Category Series Sale Price Store 1 Sales Cash $14 Store 1 Sales Credit $12 Store 2 Sales Cash $100 Store 2 Sales Credit $120 Pentaho Reporting builds a Category Dataset using the CategorySetDataCollector. Also available is the PivotCategorySetCollector, which pivots the category and series data. Collector classes implement Pentaho Reporting’s Function API. The second type of dataset is known as an XY Series Dataset, which is a two dimensional group of values that may be plotted in various forms. In this dataset, the series may be used to draw different lines, and so on. Here is a simple example of an XY series dataset : Series Cost of Goods (X) Sale Price (Y) Cash 10 14 Credit 11 12 Cash 92 100 Credit 105 120 Note that X is often referred to as the domain, and Y is referred to as the range. Pentaho Reporting builds an XY Series Dataset using the XYSeriesCollector. The XYZSeriesCollector also exists for three dimensional data. The third type of dataset is known as a Time Series Dataset , which is a two dimensional group of values that are plotted based on a time and date. The Time Series Dataset is more like an XY Series than a Category Dataset, as the time scale is displayed in a linear fashion with appropriate distances between the different time references. Time Series Sale Price May 05, 2009 11:05pm Cash $14 June 07, 2009 12:42pm Credit $12 June 14, 2009 4:20pm Cash $100 June 01, 2009 1:22pm Credit $120 Pentaho Reporting builds a Time Series Dataset using the TimeSeriesCollector. Common chart rendering properties Most charts share a common set of properties. The following properties are common across most charts. Any exceptions are mentioned as part of the specific chart type. Required Property Group Property name Description name The name of the chart object within the report. This is not displayed during rendering, but must be unique in the report. A default name is generated for each chart added to the report. data-source The dataset name for the chart, which is automatically populated with the name of the dataset in the Primary DataSource panel of the chart editor. no-data-message The message to display if no data is available to render the chart. Title Property Group Property name Description chart-title The title of the chart, which is rendered in the report. chart-title-field A field representing the chart title. title-font The chart title's font family, size, and style. Options Property Group Property name Description horizontal If set to True, the chart's X and Y axis are rotated horizontally. The default value is set to False. series-colors The color in which to render each series. The default for the first three series colors are red, blue, and green. General Property Group Property name Description 3-D If set to True, renders the chart in a 3D perspective. The default value is set to False. anti-alias If set to True, renders chart fonts as anti-aliased. The default value is set to True. bg-color Sets the background around the chart to the specified color. If not set, defaults to gray. bg-image Sets the background of the chart area to the specified image. If not set, the background of the chart area defaults to white. The chart area is the area within the axes of the chart. Supported image types include PNG, JPG, and GIF file formats. show-border If set to True, displays a border around the chart. The default value is set to True. border-color Sets the border to the specified color. If not set, defaults to black. plot-border If set to False, clears the default rendering value of the chart border. plot-bg-color Sets the plot background color to the specified color. If not set, defaults to white. plot-fg-alpha Sets the alpha value of the plot foreground colors relative to the plot background. The default value is set to 1.0. plot-bg-alpha Sets the alpha value of the plot background color relative to the chart background color. The default value is set to 1.0.   Legend Property Group Property name Description show-legend If set to True, displays the legend for the chart. The default value is set to False. location The location of the legend in relation to the chart, which may be set to top, bottom, left, or right. The default location is bottom. legend-border If set to True, renders a border around the legend. The default value is set to True. legend-font The type of Java font to render the legend labels in. legend-bg-color Sets the legend background color. If not set, defaults to white. legend-font-color Sets the legend font color. If not set, defaults to black. Advanced Property Group Property name Description dependencyLevel The dependency level field informs the reporting engine what order the chart should be executed in relation to other items in the report. This is useful if you are using special functions that may need to execute prior to generating the chart. The default value is set to 0. Negative values execute before 0, and positive values execute after 0.   Common category series rendering properties The following properties appear in charts that render category information: Options Property Group Property name Description stacked If set to True, the series values will appear layered on top of one another instead of being displayed relative to one another. stacked-percent If set to True, determines the percentages of each series, and renders the bar height based on those percentages. The property stacked must be set to True for this property to have an effect. General Property Group Property name Description gridlines If set to True, displays category grid lines. This value is set to True by default. X-Axis Property Group Property name Description label-rotation If set, adjusts the inline item label rotation value. The value should be specified in degrees. If not specified, labels are rendered horizontally. You must have show-labels set to true for this value to be relevant. date-format If the item value is a date, a Java date format string may be provided to format the date appropriately. Please see Java's SimpleDateFormat JavaDoc for formatting details. numeric-format If the item value is a decimal number, a Java decimal format string may be provided to format the number appropriately. Please see Java's DecimalFormat JavaDoc for formatting details. text-format The label format used for displaying category items within the chart. This property is required if you would like to display the category item values. The following parameters may be defined in the format string to access details of the item: {0}: To access the Series Name detail of an item {1}: To access the Category detail of an item {2}: To access the Item value details of an item To display just the item value, set the format string to "{2}". x-axis-title If set, displays a label describing the category axis. show-labels If set to true, displays x-axis labels in the chart. x-axis-label-width Sets the maximum category label width ratio, which determines the maximum length each category label should render in. This might be useful if you have really long category names. x-axis-label-rotation If set, adjusts the category item label rotation value. The value should be specified in degrees. If not specified, labels are rendered horizontally. x-font The font to render the category axis title and labels in. Y-Axis Property Group Property name Description y-axis-title If set, displays a label along the value axis of the chart. label-rotation If set, determines the upward angle position of the label, where the value passed into JFreeChart is the mathematical pie over the value. Unfortunately, this property is not very flexible and you may find it difficult to use. y-tick-interval The numeric interval value to separate range ticks in the chart. y-font The font to render the range axis title in. y-sticky-0 If the range includes zero in the axis, making it sticky will force truncation of the axis to zero if set to True. The default value of this property is True. y-incl-0 If set to True, the range axis will force zero to be included in the axis. y-min The minimum value to render in the range axis. y-max The maximum value to render in the range axis. y-tick-font The font to render the range tick value in. y-tick-fmt-str The DecimalFormat string to render the numeric range tick value. enable-log-axis If set to true, displays the y-axis as a logarithmic scale. log-format If set to true, will present the logarithmic scale in a human readable view.

  Blender 2.5 Materials and Textures Cookbook Over 80 great recipes to create life-like Blender objects Master techniques to create believable natural surface materials Take your models to the next level of realism or artistic development by using the material and texture settings within Blender 2.5. Take the hassle out of material simulation by applying faster and more efficient material and texture strategies Part of Packt's Cookbook series: Each recipe is a logically organized according to the surface types with clear instructions and explanations on how these recipes can be applied across a range of materials including complex materials such as oceans, smoke, fire and explosions. Introduction Organizing your work, as you develop any project, will ensure that you achieve your task sooner and more efficiently. How to do this in Blender may not be immediately obvious. However, Blender has a raft of tools that will make your life, as a materials creator, so much easier. This article deals with the techniques that can be used to organize your textures and materials, and thus encourage some order to complex tasks. While Blender can be a very flexible 3D suite, allowing the designer more than a single approach to a simulation, it is better to develop a more ordered strategy to your material and texture creations. We will explore several recipes that attempt to show how to control material creation. However, apart from the inbuilt tools, there are several setups that will be dependent on personal preference. You are therefore encouraged to modify any of these recipes to suit your own approaches to organizing material production. Setting a default scene for materials creation It's always a good idea to set the initial state of Blender to suit your needs. For us, the primary task is to explore materials and texture creation. When you first install Blender, a default layout will be presented. From here, you can perform most tasks, such as modeling, and rendering, as you create your desired objects. We can aid the process of surface creation by improving the lighting in the default setup. Adding a second light can give better definition to objects that are rendered. Getting ready When you first download Blender, the default factory settings provide a simple cube illuminated by a single light. Even if you have already changed some of these defaults, you will be able to apply the suggested changes in this recipe on top of your personalized settings. So, you can either start with the factory settings or your own. How to do it... Start Blender, or select New from the File menu. This will ensure that any previous default settings are loaded. Move the mouse cursor into the main 3D view and press SHIFT+A to bring up the Add menu and select Lamp of type Hemi. Move, and rotate the lamp so that it will illuminate the shaded side of the default cube. Try to adjust its height and distance from the object similar to the default lamp. From the lamp menu, set the Energy value between 0.200 and 0.500. Render a quick scene and adjust as necessary. Move to the Render panel. In the Dimensions tab, select Render Presets. From the list, select HDTV 1080p. This will give a render size of 1,920 x 1,080 square pixels. However, alter the Resolution percentage slider to 25%. Just below the Aspect Ratio settings are two buttons, check Border and then ,Crop. Now, uncheck Border. This might seem strange but although the Crop checkbox is grayed out it is still set. Ensure the Anti-Aliasing tab is selected, and the figure below that is set to 8, with the anti-aliasing method set to Mitchell-Netravali. Ensure that the Full Sample is NOT, set. Under Shading, ensure Textures, ,Shadows, Ray Tracing, and Color Management are set, while Subsurface Scattering and Environment Map are not. Move down to the ,Output tab and from the list of choices select PNG. You can also change the Compression percentage to between 0% for a loss less saved image or up to 100% for full compression. I usually set to 0% to produce the clearest images. Finally, in the Performance tab, ensure that Threads is set to Autodetect. You can save these settings as the default scene by pressing CTRL+U and selecting Save User Settings. Now, whenever you restart Blender, or select a new scene, you will have a better lit setup with render settings providing an ideal environment to create materials. How it works... The recipe provides a relatively simple set of changes to the factory default scene. However, they are ideal for materials creation because they make it easier to judge the surface characteristics as you develop the material. We started by improving the default lighting by adding a second light to give a little more illumination to the shaded side of the objects you will be creating materials and textures for. Being able to set this as the default scene means we don't have to worry about special light setups every time we create a new material. It also helps with consistency because the levels of light will be very similar between every new material you create. It is not there to provide the finished lighting for every scene you create, but just to give a more even illumination when you test render materials you are developing. What we have done is produced a key light and a fill light, which is the minimum in almost any 3D lighting arrangement. The Hemi light offers a nice broad illumination but will not cast shadows. This is ideal for a fill light as it can represent bounced light off of ceilings, or walls, or even the outside world. The Point light source acting as our key light will cast shadows just as the strongest light in a natural environment would. When you have developed your materials, you can light the actual scene with a more complex or artistic lighting setup if you wish. For the majority of digital work, we need to use square pixels and a resolution that matches the size we wish to render to. Here, we have set the render size and resolution from the presets to HDTV 1080p. This produces a relatively large render area with square pixels. Square pixels are really important when developing objects or materials for digital work. It's possible to set different aspect ratios that would alter the screen and render proportions, which are of no value when creating models or developing and placing textures on them. If you eventually want to render out to these none square pixel ratios, do so only when all you're modeling and scene creation is finalized. The render panel offers several useful presets. These are based on screen resolution and pixel aspect ratio to exactly match the desired output. Be careful not to inadvertently select one of the non-square pixel ratios, like HDV 1080p. In the same step, we set the render resolution down to 25%. This will still give a render size of 480 x 270, which is OK for initial quick renders to check how a material is progressing. You can easily scale that up to 50% or 75% for more detailed renders. However, these will obviously increase the render times. If you create a border in the camera view, by pressing SHIFT+B, and dragging the orange dotted border, Blender will only render what's inside that rectangle. This is why we also checked the Crop checkbox so that Blender automatically crops the rendered image. If you perform a bordered render without the Crop set, the unrendered part is filled with black pixels. Therefore, setting the crop checkbox will ensure it will be cropped if selected. This will save valuable render time and also smaller image saves. Anti-aliasing Even if we are rendering to a large size, we should set Blender to anti-aliase the resultant render to remove the jagged edges that would otherwise appear. Here, we have set the antialiasing method to Mitchell-Netravali. This is probably the best of the available options. It will give very reasonable anti-aliasing at the relatively low setting of 8 without unreasonable render times. For final render, you might want to consider raising the level to 16. Turning off unneeded render settings Subsurface scattering and Environment map are not always required so can be turned off in the default scene. They can always be turned on for a particular material simulation that might require them. However, normally, they are not required and render times will be reduced by having them turned off. If you are working on a material that requires environment mapping or subsurface scattering, you can set these, then save your first file of the simulation. Saving a blendfile will save all additional settings as well as objects and materials. Blender offers an enormous range of output formats for your rendered still or animation masterpieces. You will not need to use them all so which should we choose as a default? PNG (Portable Network Graphics) has lossless data compression, as compared to JPG which is lossy, and therefore, the picture is degraded every time you save. However, PNG can efficiently compress images without the subsequent loss of quality. It can also handle alpha channels. It can be read by most web browsers so is suitable for the Internet. Because Blender can render an animation as a series of still images, it is ideal for producing animations as well. Several video editors, including Quicktime, Adobe Premiere, and of course Blender, can take these sequenced still PNG images and combine them into movie formats like .mov, .avi, .mp4, .mpeg, and so on. I would therefore suggest that PNG is probably the best all-round image format to set as default. If you're primary work is in either the game development, or print, fields you might want to consider using TGA (Truevision Advanced Raster Graphics Adapter), or TIFF (Tagged Image File Format). However, windows-based systems will not display thumbnails of TIFF formatted images at this time. There's more... There are other settings that you may want to consider as appropriate in a default scene. You can set locations of often used resources from the Blender User Preferences window, CTRL+ALT+U. Under the file menu of this preferences window, you can set file paths for such things as Fonts, Textures, Render Output, and so on. Many of these locations will be specific to your operating system and where you choose them to be. Blender defaults are fine, but if you want to be specific go to this window and enter your appropriate choices. To ensure they are saved as default, click the Save User Defaults button, or press CTRL+U. (Move the mouse over the image to enlarge it). Additional settings for default scene If you have a powerful enough computer system, you might want to consider setting some more advanced options to make your test renders look really special. In some ways, what you will be doing with this recipe is creating a more production-ready materials creation environment. However, each render will take longer and if it is a complex mesh object with transparency and multiple large-scale image textures, you may have to wait several minutes for renders to complete. Although this may not seem to be a significant disadvantage, the extra render time can build up as you produce multiple renders to test the look of a material simulation. The renders reproduced in this article, and online, were created with these additional settings. The majority of development work also used these additional settings. While the majority of the simulations only took a few minutes to render at maximum resolution, one or two took a little longer. If that is the case it can slow down your material development, just turn some of these additional settings off before your first save of the blendfile. Any settings will be saved with the blendfile. Getting ready As we are adding additional settings to the default scene, ensure you have either just started Blender, or selected New from the File menu. This will set Blender back to the default scene ready for you to append the additional features suggested here. How to do it... In the 3D window, ensure that the cursor is at the center. SHIFT+S, Cursor to Center. Add new object of type plane. SHIFT+A, Mesh, Plane. Scale the plane by 50 Blender units. The easiest way to achieve this is to type S, enter 50 and press the ENTER key to confirm. Grab the plane and transpose it down -1 in the Z direction. You can either select the plane, then type G, Z, -1, and ENTER to confirm. Or press N to bring up the Properties panel and alter the Z location to -1.000. That has ensured that the plane is below any object you create from the origin. New objects are created at the cursor position and are always 1 Blender unit from their own origin, which will mean they should stand on the ground plane you have just created. Move to the Materials panel and create a new material, naming it ground. Under the Diffuse tab, change its color to R, G, B 1.000 or pure white. In the Specular tab, change the type to WardIso with an Intensity of 0.250, and a Slope of 0.300. Under the Shadow tab, select Receive and Receive Transparent. That has set up the ground ready to act as a shadow-receiving backdrop to our material creations. However, we need to set up a better shadow than that in the default key light setup. Select the key light. From the Light panel, select the Shadow tab and set Ray Shadow with Sampling set to Adaptive QMC, with Samples set to 6, and Soft Size of 1.000 and Threshold 0.001. Finally, we will set up ambient occlusion to give our models a little more shape. Move to the World panel and select Ambient Occlusion. Set its Factor to 1.30 and its Mix type to Multiply. In the Gathering tab, select Approximate, with Attenuation/Falloff selected and its Strength changed to 0.900. Set Sampling/Passes to 6. Select the Render tab, then save this as the new default setting by pressing CTRL+U. How it works... The plane is there just so that any shadows will have somewhere to fall. Shadow casting is normally created as a default once an object has a material assigned. In step 19, we also said let the plane receive transparent shadows. That means that objects casting shadows with an alpha component such as windows, or transparent materials, will have accurate shadows showing that transparency. We also set up soft raytrace shadows by turning on this feature for the key light. The important setting here is the number of samples. Too low and the shadow will look fake. Too high and the render times will become rather lengthy. Setting this to 6 is a good compromise for accuracy and speed. Finally, we have set up ambient occlusion, which simulates the darkening of shadows in crevices and shaded portions of a model. The higher the Factor level, the darker the ambient occlusion will become. Essentially, the darkening is being multiplied on top of the rendered image, although it can also be set to Add. Full raytraced ambient occlusion can take some time to compute, so it is good news that Blender has an excellent Approximate method, which is very quick. The Attenuation and Passes can be tweaked to give the best balance between accuracy and render time. Too low a setting will produce a spotty darkening that is not very real. We set Passes at 6, which is another excellent compromise. Ambient occlusion is only available if you render with raytrace enabled. To save all these extra setting as the default, we only had to press CTRL+U. Now, whenever you start a new scene, these settings will be pre-set. There's more... Occasionally, you may go too far with default settings and find that render times become too long when you only want to check how a material is progressing. Another common problem is that users will sometimes inadvertently save a pre-created scene as the default. If this happens, you can always return to the factory settings by pressing CTRL+N or choosing the Load Factory Settings from the File menu. As you will be doing this regularly, the default light setup is perfectly adequate. However, for a 'hero' render, I would recommend the following settings: Lights Your key light, the one to the right of frame, should be set to Energy 0.500. Under the Light panel, set ShadowRay Shadow. Under Sampling, set to Adaptive QMC and Soft Size to 5.318, and Samples to 6. This will create a nice soft shadow, which is more realistic than the sharp ray shadows produced by the default settings. Ambient occlusion Ambient occlusion can produce a nice darkening of overall illumination around those shaded parts in our renders. It adds a decent approximation of how real light and shadow spread through an environment giving a render depth. In our example, a Blend sky has been set in the World panel, with a Horizon color R and G set to 0.80, and B set to 0.69. The Zenith color has been set to R 0.69, G 0.75, and B 0.80. Ambient Occlusion has been selected with the following settings: Factor 1.00, and Multiply. Under Gather, Raytrace is selected and Sampling is set to Adaptive QMC, with Samples 24. Threshold and Adapt To Speed are all set to defaults. Under Attenuation, Distance is set to 10.000 with Falloff and Strength set to 0.220.

[box type="note" align="" class="" width=""]This article is a book excerpt from Learning Pentaho Data Integration 8 CE - Third Edition written by María Carina Roldán.  In this book you will explore the features and capabilities of Pentaho Data Integration 8 Community Edition.[/box] In today’s tutorial, we will introduce you to Pentaho Data Integration (PDI) and learn to use it in real world scenario. Pentaho Data Integration (PDI) is an engine along with a suite of tools responsible for the processes of Extracting, Transforming, and Loading (also known as ETL processes). The Pentaho Business Intelligence Suite is a collection of software applications intended to create and deliver solutions for decision making. The main functional areas covered by the suite are: Analysis: The analysis engine serves multidimensional analysis. It's provided by the Mondrian OLAP server. Reporting: The reporting engine allows designing, creating, and distributing reports in various known formats (HTML, PDF, and so on), from different kinds of sources. In the Enterprise Edition of Pentaho, you can also generate interactive Reports. Data mining: Data mining is used for running data through algorithms in order to understand the business and do predictive analysis. Data mining is possible thanks to Weka project. Dashboards: Dashboards are used to monitor and analyze Key Performance Indicators (KPIs). CTools is a set of tools and components created to help the user to build custom dashboards on top of Pentaho. There are specific CTools for different purposes, including a Community Dashboard Editor (CDE), a very powerful charting library (CCC), and a plugin for accessing data with great flexibility (CDA), among others. While the Ctools allow to develop advanced and custom dashboards, there is a Dashboard Designer, available only in Pentaho Enterprise Edition, that allows to build dashboards in an easy way. Data integration: Data integration is used to integrate scattered information from different sources (for example, applications, databases, and files) and make the integrated information available to the final user. PDI—the tool that we will learn to use throughout the book—is the engine that provides this functionality. PDI also interacts with the rest of the tools, as, for example, reading OLAP cubes, generating Pentaho Reports, and doing data mining with R Executor Script and the CPython Script Executor. All of these tools can be used standalone but also integrated. Pentaho tightly couples data integration with analytics in a modern platform: the PDI and Business Analytics Platform. This solution offers critical services, for example: Authentication and authorization Scheduling Security Web services Scalability and failover This set of software and services forms a complete BI Suite, which makes Pentaho the world's leading open source BI option on the market. Note: You can find out more about the platform at https://community.hds.com/community/products-and-solutions/pentaho/. There is also an Enterprise Edition with additional features and support. You can find more on this at http://www.pentaho.com/. Introducing Pentaho Data Integration Most of the Pentaho engines, including the engines mentioned earlier, were created as community projects and later adopted by Pentaho. The PDI engine is not an exception; Pentaho Data Integration is the new denomination for the business intelligence tool born as Kettle. By joining forces with Pentaho, Kettle benefited from a huge developer community, as well as from a company that would support the future of the project. From that moment, the tool has grown with no pause. Every few months a new release is available, bringing to the user's improvements in performance and existing functionality, new functionality, and ease of use, along with great changes in look and feel. The following is a timeline of the major events related to PDI since its acquisition by Pentaho: June 2006: PDI 2.3 was released. Numerous developers had joined the project and there were bug fixes provided by people in various regions of the world. The version included, among other changes, enhancements for large-scale environments and multilingual capabilities. November 2007: PDI 3.0 emerged totally redesigned. Its major library changed to gain massive performance improvements. The look and feel had also changed completely. April 2009: PDI 3.2 was released with a really large amount of changes for a minor version: new functionality, visualization and performance improvements,and a huge amount of bug fixes. June 2010: PDI 4.0 was released, delivering mostly improvements with regard to enterprise features, for example, version control. In the community version, the focus was on several visual improvements. November 2013: PDI 5.0 was released, offering better previewing of data, easier looping, a lot of big data improvements, an improved plugin marketplace, and  hundreds of bug fixes and features enhancements, as in all releases. In its Enterprise version, it offered interesting low-level features, such as step load balancing, Job transactions, and restartability. December 2015: PDI 6.0 was released with new features such as data services, data lineage, bigger support for Big Data, and several changes in the graphical designer for improving the PDI user experience. Some months later, PDI 6.1 was released including metadata injection, a feature that enables the user to modify Transformations at runtime. Metadata injection had been available in earlier versions, but it was in 6.1 that Pentaho started to put in a big effort in implementing this powerful feature. November 2016: PDI 7.0 emerged with many improvements in the enterprise version, including data inspection capabilities, more support for Big Data technologies, and improved repository management. In the community version, the main change was an expanded metadata injection support. November 2017: Pentaho 8.0 is released. The highlights of this latest version are the optimization of processing resources, a better user experience, and the enhancement of the connectivity to streaming data sources—real-time processing. Using PDI in real-world scenarios Paying attention to its name, Pentaho Data Integration, you could think of PDI as a tool to integrate data. In fact, PDI does not only serve as a data integrator or an ETL tool. PDI is such a powerful tool that it is common to see it being used for these and for many other purposes. Here you have some examples. Loading data warehouses or data marts The loading of a data warehouse or a data mart involves many steps, and there are many variants depending on business area or business rules. However, in every case, with no exception, the process involves the following steps: Extracting information from one or more databases, text files, XML files, and other sources. The extract process may include the task of validating and discarding data that doesn't match expected patterns or rules. Transforming the obtained data to meet the business and technical needs required on the target. Transforming includes such tasks such as converting data types, doing some calculations, filtering irrelevant data, and summarizing. Loading the transformed data into the target database or file store. Depending on the requirements, the loading may overwrite the existing information or may add new information each time it is executed. Kettle comes ready to do every stage of this loading process. The following screenshot shows a simple ETL designed with the tool: Integrating data Imagine two similar companies that need to merge their databases in order to have a unified view of the data, or a single company that has to combine information from a main Enterprise Resource Planning (ERP) application and a Customer Relationship Management (CRM) application, though they're not connected. These are just two of hundreds of examples where data integration is needed. The integration is not just a matter of gathering and mixing data; some conversions, validation, and transfer of data have to be done. PDI is meant to do all these tasks. Data cleansing Data cleansing is about ensuring that the data is correct and precise. This can be achieved by verifying if the data meets certain rules, discarding or correcting those which don't follow the expected pattern, setting default values for missing data, eliminating information that is duplicated, normalizing data to conform to minimum and maximum values, and so on. These are tasks that Kettle makes possible, thanks to its vast set of transformation and validation capabilities. Migrating information Think of a company, any size, which uses a commercial ERP application. One day the owners realize that the licenses are consuming an important share of its budget. So they decide to migrate to an open source ERP. The company will no longer have to pay licenses, but if they want to change, they will have to migrate the information. Obviously, it is not an option to start from scratch or type the information by hand. Kettle makes the migration possible, thanks to its ability to interact with most kind of sources and destinations, such as plain files, commercial and free databases, and spreadsheets, among others. Exporting data Data may need to be exported for numerous reasons: To create detailed business reports To allow communication between different departments within the same company To deliver data from your legacy systems to obey government regulations, and so on Kettle has the power to take raw data from the source and generate these kinds of ad hoc reports. Integrating PDI along with other Pentaho tools The previous examples show typical uses of PDI as a standalone application. However, Kettle may be used embedded as part of a process or a data flow. Some examples are preprocessing data for an online report, sending emails in a scheduled fashion, generating spreadsheet reports, feeding a dashboard with data coming from web services, and so on. Installing PDI In order to work with PDI, you need to install the software. Following are the instructions to install the PDI software, irrespective of the operating system you may be using: Go to the Download page at http://sourceforge.net/projects/pentaho/files/DataIntegration. Choose the newest stable release. At this time, it is 8.0, as shown in the following Screenshot: Download the available zip file, which will serve you for all platforms. Unzip the downloaded file in a folder of your choice, as, for example, c:/util/kettle or /home/pdi_user/kettle. And that's all. You have installed the tool in just few minutes. We learnt about installing and using PDI. You can know more about extending PDI functionality and Launching the PDI Graphical Designer from Learning Pentaho Data Integration 8 CE - Third Edition.        

(For more resources related to this topic, see here.) Many applications involve several screens of data that a user might want to browse or flip through to view each screen. As an example, think of an application where we list a catalogue of books with each book in the catalogue appearing on a single screen. A book's screen contains an image, title, and description like the following screenshot: To view each book's information, the user needs to move to each screen. We could put a next button and a previous button on the screen, but a more natural action is for the user to use their thumb or finger to swipe the screen from one edge of the display to the other and have the screen with the next book's information slide into place as represented in the following screenshot: This creates a very natural navigation experience, and honestly, is a more fun way to navigate through an application than using buttons. Summary Fragments are the foundation of modern Android app development, allowing us to display multiple application screens within a single activity. Thanks to the flexibility provided by fragments, we can now incorporate rich navigation into our apps with relative ease. Using these rich navigation capabilities, we're able to create a more dynamic user interface experience that make our apps more compelling and that users find more fun to work with. Resources for Article : Further resources on this subject: So, what is Spring for Android? [Article] Android Native Application API [Article] Animating Properties and Tweening Pages in Android 3-0 [Article]

  ASP.NET Site Performance Secrets Simple and proven techniques to quickly speed up your ASP.NET website Speed up your ASP.NET website by identifying performance bottlenecks that hold back your site's performance and fixing them Tips and tricks for writing faster code and pinpointing those areas in the code that matter most, thus saving time and energy Drastically reduce page load times Configure and improve compression – the single most important way to improve your site's performance Written in a simple problem-solving manner – with a practical hands-on approach and just the right amount of theory you need to make sense of it all. The reader can benefit from the previous articles on Pinpointing bottlenecks for better Database Access in ASP.Net and Fixing bottlenecks for better Database Access in ASP.Net. Locking In this section, you'll see how to determine which queries are involved in excessive locking delays, and how to prevent those delays from happening. Gathering detailed locking information You can find out which queries are involved in excessive locking delays by tracing the event "Blocked process report" in SQL Server Profiler. This event fires when the lock wait time for a query exceeds the "blocked process threshold". To set this threshold to, for example, 30 seconds, run the following lines in a query window in SSMS (locking.sql in the downloaded code bundle): DBCC TRACEON(1222,-1) Then, start the trace in Profiler: Start SQL Profiler. Click on Start | Programs | Microsoft SQL Server 2008 | Performance Tools | SQL Server Profiler. In SQL Profiler, click on File | New Trace. Click on the Events Selection tab. Select Show all events checkbox to see all events. Also select Show all columns to see all the data columns. In the main window, expand Errors and Warnings and select the Blocked process report event. Make sure the checkbox in the TextData column is checked—scroll horizontally if needed to find it. If you need to investigate deadlocks, also expand Locks and select the Deadlock graph event. To get additional information about deadlocks, have SQL Server write information about each deadlock event to its error log, by executing the following from an SSMS query window: ALTER DATABASE mydatabase SET SINGLE_USER WITH ROLLBACK IMMEDIATE; ALTER DATABASE mydatabase SET READ_COMMITTED_SNAPSHOT ON; ALTER DATABASE mydatabase SET MULTI_USER; Uncheck all the other events, unless you are interested in them. Click on Run to start the trace. Save the template, so that you don't have to recreate it the next time. Click on File | Save As | Trace Template. Fill in a descriptive name and click on OK. Next time you create a new trace by clicking on File | New Trace, you can retrieve the template from the Use the template drop-down. Once you have captured a representative sample, click File | Save to save the trace to a trace file for later analysis. You can load a trace file by clicking on File | Open. (Move the mouse over the image to enlarge.) When you click a Blocked process report event in Profiler, you'll find information about the event in the lower pane, including the blocking query and the blocked query. You can get details about Deadlock graph events the same way. To check the SQL Server error log for deadlock events: In SSMS expand the database server, expand Management and expand SQL Server Logs. Then double-click on a log. In the Log File Viewer, click on Search near the top of the window and search for "deadlock-list". In the lines that chronologically come after the deadlock-list event, you'll find much more information about the queries involved in the deadlock. Reducing blocking Now that you identified the queries involved in locking delays, it's time to reduce those delays. The most effective way to do this is to reduce the length of time locks are held as follows: Optimize queries. The lesser time your queries take, the lesser time they hold locks. Use stored procedures rather than ad hoc queries. This reduces time spent compiling execution plans and time spent sending individual queries over the network. If you really have to use cursors, commit updates frequently. Cursor processing is much slower than set-based processing. Do not process lengthy operations while locks are held, such as sending e-mails. Do not wait for user input while keeping a transaction open. Instead, use optimistic locking, as described in: Optimistic Locking in SQL Server using the ROWVERSION Data Type http://www.mssqltips.com/tip.asp?tip=1501 A second way to reduce lock wait times is to reduce the number of resources being locked: Do not put a clustered index on frequently updated columns. This requires a lock on both the clustered index and all non-clustered indexes, because their row locator contains the value you are updating. Consider including a column in a non-clustered index. This would prevent a query from having to read the table record, so it won't block another query that needs to update an unrelated column in the same record. Consider row versioning. This SQL Server feature prevents queries that read a table row from blocking queries that update the same row and vice versa. Queries that need to update the same row still block each other.Read versioning works by storing rows in a temporary area (in tempdb) before they are updated, so that reading queries can access the stored version while the update is taking place. This does create an overhead in maintaining the row versions—test this solution before taking it live. Also, in case you set the isolation level of transactions, row versioning only works with the Read Committed isolation mode, which is the default isolation mode.To implement row versioning, set the READ_COMMITTED_SNAPSHOT option as shown in the following code (locking.sql in the downloaded code bundle). When doing this, you can have only one connection open—the one used to set the option. You can make that happen by switching the database to single user mode; warn your users first. Be careful when applying this to a production database, because your website won't be able to connect to the database while you are carrying out this operation. select is_read_committed_snapshot_on from sys.databases where name='mydatabase' To check whether row versioning is in use for a database, run: SET LOCK_TIMEOUT 5000 Finally, you can set a lock timeout. For example, to abort statements that have been waiting for over five seconds (or 5000 milliseconds), issue the following command: SET TRANSACTION ISOLATION LEVEL REPEATABLE READ BEGIN TRAN SELECT Title FROM dbo.Book UPDATE dbo.Book SET Author='Charles Dickens' WHERE Title='Oliver Twist' COMMIT Use 1 to wait indefinitely. Use 0 to not wait at all. Reducing deadlocks Deadlock is a situation where two transactions are waiting for each other to release a lock. In a typical case, transaction 1 has a lock on resource A and is trying to get a lock on resource B, while transaction 2 has a lock on resource B and is trying to get a lock on resource A. Neither transaction can now move forward, as shown below: One way to reduce deadlocks is to reduce lock delays in general, as shown in the last section. That reduces the time window in which deadlocks can occur. A second way is suggested by the diagram—always lock resources in the same order. If, as shown in the diagram, you get transaction 2 to lock the resources in the same order as transaction 1 (first A, then B), then transaction 2 won't lock resource B before it starts waiting for resource A. Hence, it doesn't block transaction 1. Finally, watch out for deadlocks caused by the use of HOLDLOCK or Repeatable Read or Serializable Read isolation levels. Take for example the following code: SET TRANSACTION ISOLATION LEVEL REPEATABLE READ BEGIN TRAN SELECT Title FROM dbo.Book WITH(UPDLOCK) UPDATE dbo.Book SET Author='Charles Dickens' WHERE Title='Oliver Twist' COMMIT Imagine two transactions running this code at the same time. Both acquire a Select lock on the rows in the Book table when they execute the SELECT. They hold onto the lock because of the Repeatable Read isolation level. Now, both try to acquire an Update lock on a row in the Book table to execute the UPDATE. Each transaction is now blocked by the Select lock the other transaction is still holding. To prevent this from happening, use the UPDLOCK hint on the SELECT statement. This causes the SELECT to acquire an Update lock, so that only one transaction can execute the SELECT. The transaction that did get the lock can then execute its UPDATE and free the locks, after which the other transaction comes through. The code is as follows: SELECT b.Title, a.AuthorName FROM dbo.Book b JOIN dbo.Author a ON b.LeadAuthorId=a.Authorid WHERE BookId=5

In this article written by Omar Khedher, author of Mastering OpenStack, we will explore the various aspects of networking and security in OpenStack. A major part of the article is focused on presenting the different security layouts by using Neutron. In this article, we will discuss the following topics: Understanding how Neutron facilitates the network management in OpenStack Using security groups to enforce a security layer for instances The story of an API By analogy, the OpenStack compute service provides an API that provides a virtual server abstraction to imitate the compute resources. The network service and compute service perform in the same way, where we come to a new generation of virtualization in network resources such as network, subnet, and ports, and can be continued in the following schema: Network: As an abstraction for the layer 2 network segmentation that is similar to the VLANs Subnet: This is the associated abstraction layer for a block of IPv4/IPv6 addressing per network Port: This is the associated abstraction layer that is used to attach a virtual NIC of an instance to a network Router: This is an abstraction for layer 3 that is used to perform routing between the networks Floating IP: This is used to perform static public IP mapping from external to internal networks Security groups Imagine a scenario where you have to apply certain traffic management rules for a dozen compute node instances. Therefore, assigning a certain set of rules for a specific group of nodes will be much easier instead of going through each node at a time. Security groups enclose all the aspects of the rules that are applied to the ingoing and outgoing traffic to instances, which includes the following: The source and receiver, which will allow or deny traffic to instances from either the internal OpenStack IP addresses or from the rest of the world Protocols to which the rule will apply, such as TCP, UDP, and ICMP Egress/ingress traffic management to a neutron port In this way, OpenStack offers an additional security layer to the firewall rules that are available on the compute instance. The purpose is to manage traffic to several compute instances from one security group. You should bear in mind that the networking security groups are more granular-traffic-filtering-aware than the compute firewall rules since they are applied on the basis of the port instead of the instance. Eventually, the creation of the network security rules can be done in different ways. For more information on how iptables works on Linux, https://www.centos.org/docs/5/html/Deployment_Guide-en-US/ch-iptables.html is a very useful reference. Manage the security groups using Horizon From Horizon, in the Access and Security section, you can add a security group and name it, for example PacktPub_SG. Then, a simple click on Edit Rules will do the trick. The following example illustrates how this network security function can help you understand how traffic—both in ingress/egress—can be controlled: The previous security group contains four rules. The first and the second lines are rules to open all the outgoing traffic for IPv4 and IPv6 respectively. The third line allows the inbound traffic by opening the ICMP port, while the last one opens port 22 for SSH for the inbound interface. You might notice the presence of the CIDR fields, which is essential to know. Based on CIDR, you allow or restrict traffic over the specified port. For example, using CIDR of 0.0.0.0/0 will allow traffic for all the IP addresses over the port that was mentioned in your rule. For example, a CIDR with 32.32.15.5/32 will restrict traffic only to a single host with an IP of 32.32.15.5. If you would like to specify a range of IP in the same subnet, you can use the CIDR notation, 32.32.15.1/24, which will restrict traffic to the IP addresses starting from 32.32.15.*; the other IP addresses will not stick to the latter rule. The naming of the security group must be done with a unique name per project. Manage the security groups using the Neutron CLI The security groups also can be managed by using the Python Neutron command-line interface. Wherever you run the Neutron daemon, you can list, for example, all the present security groups from the command line in the following way: # neutron security-group-list The preceding command yields the following output: To demonstrate how the PacktPub_SG security group rules that were illustrated previously are implemented on the host, we can add a new rule that allows the ingress connections to ping (ICMP) and establish a secure shell connection (SSH) in the following way: # neutron security-group-rule-create --protocol icmp –-direction ingress PacktPub-SG The preceding command produces the following result: The following command line will add a new rule that allows ingress connections to establish a secure shell connection (SSH): # neutron security-group-rule-create --protocol tcp –-port-range-max 22 –-direction ingress PacktPub-SG The preceding command gives the following output: By default, if none of the security groups have been created, the port of instances will be associated within the default security group for any project where all the outbound traffic will be allowed and blocked in the inbound side. You may conclude from the output of the previous command line that it lists the rules that are associated with the current project ID and not by the security groups. Managing the security groups using the Nova CLI The nova command line also does the same trick if you intend to perform the basic security group's control, as follows: $ nova secgroup-list-rules default Since we are setting Neutron as our network service controller, we will proceed by using the networking security groups, which reveals additional traffic control features. If you are still using the compute API to manage the security groups, you can always refer to the nova.conf file for each compute node to set security_group_api = neutron. To associate the security groups to certain running instances, it might possible to use the nova client in the following way: # nova add-secgroup test-vm1 PacktPub_SG The following code illustrates the new association of the packtPub_SG security group with the test-vm1 instance: # nova show test-vm1   The following is the result of the preceding command: One of the best practices to troubleshoot connection issues for the running instances is to start checking the iptables running in the compute node. Eventually, any rule that was added to a security group will be applied to the iptables chains in the compute node. We can check the updated iptables chains in the compute host after applying the security group rules by using the following command: # iptables-save The preceding command yields the following output: The highlighted rules describe the direction of the packet and the rule that is matched. For example, the inbound traffic to the f7fabcce-f interface will be processed by the neutron-openvswi-if7fabcce-f chain. It is important to know how iptables rules work in Linux. Updating the security groups will also perform changes in the iptable chains. Remember that chains are a set of rules that determine how packets should be filtered. Network packets traverse rules in chains, and it is possible to jump to another chain. You can find different chains per compute host, depending on the network filter setup. If you have already created your own security groups, a series of iptables and chains are implemented on every compute node that hosts the instance that is associated within the applied corresponding security group. The following example demonstrates a sample update in the current iptables of a compute node that runs instances within the 10.10.10.0/24 subnet and assigns 10.10.10.2 as a default gateway for the former instances IP ranges: The last rule that was shown in the preceding screenshot dictates how the flow of the traffic leaving the f7fabcce-finterface must be sourced from 10.10.10.2/32 and the FA:16:3E:7E:79:64 MAC address. The former rule is useful when you wish to prevent an instance from issuing a MAC/IP address spoofing. It is possible to test ping and SSH to the instance via the router namespace in the following way: # ip netns exec router qrouter-5abdeaf9-fbb6-4a3f-bed2-7f93e91bb904 ping 10.10.10.2 The preceding command provides the following output: The testing of an SSH to the instance can be done by using the sane router namespace, as follows: # ip netns exec router qrouter-5abdeaf9-fbb6-4a3f-bed2-7f93e91bb904 ssh cirros@10.10.10.2 The preceding command produces the following output: Web servers DMZ example In the current example, we will show a simple setup of a security group that might be applied to a pool of web servers that are running in the Compute01, Compute02 and Compute03 nodes. We will allow inbound traffic from the Internet to access WebServer01, AppServer01, and DNSServer01 over HTTP and HTTPS. This is depicted in the following diagram: Let's see how we can restrict the traffic ingress/egress via Neutron API: $ neutron security-group-create DMZ_Zone --description "allow web traffic from the Internet" $neutron security-group-rule-create --direction ingress --protocol tcp --port_range_min 80 --port_range_max 80 DMZ_Zone --remote-ip-prefix 0.0.0.0/0 $neutron security-group-rule-create --direction ingress --protocol tcp --port_range_min 443 --port_range_max 443 DMZ_Zone --remote-ip-prefix 0.0.0.0/0 $neutron security-group-rule-create --direction ingress --protocol tcp --port_range_min 3306 --port_range_max 53 DMZ_Zone --remote-ip-prefix 0.0.0.0/0 From Horizon, we can see the following security rules group added: To conclude, we have looked at presenting different security layouts by using Neutron. At this point, you should be comfortable with security groups and their use cases. Further your OpenStack knowledge by designing, deploying, and managing a scalable OpenStack infrastructure with Mastering OpenStack

(For more resources related to this topic, see here.) Veeam Backup & Replication v7 for VMware is a modern solution for data protection and disaster recovery for virtualized VMware vSphere environments of any size. Veeam Backup & Replication v7 for VMware supports VMware vSphere and VMware Infrastructure 3 (VI3), including the latest version VMware vSphere 5.5 and Microsoft Windows Server 2012 R2 as the management server(s). Its modular approach and scalability make it an obvious choice regardless of the environment size or complexity. As your data center grows, Veeam Backup & Replication grows with it to provide complete protection for your environment. Remember, your backups aren't really that important, but your restore is! Backup strategies A common train of thought when dealing with backups is to follow the 3-2-1 rule: 3: Keep three copies of your data—one primary and two backups 2: Store the data in two different media types 1: Store at least one copy offsite This simple approach ensures that no matter what happens, you will be able to have a recoverable copy of your data. Veeam Backup & Replication lets you accomplish this goal by utilizing the backup copy jobs. Back up your production environment once, then use the backup copy jobs to copy the backed-up data to a secondary location, utilizing the Built-in WAN Acceleration features and to tape for long-term archival. You can even "daisy-chain" these jobs to each other, which ensures that as soon as the backup job is finished, the copy jobs are fired automatically. This allows you to easily accomplish the 3-2-1 rule without the need for complex configurations that makes it hard to manage. Combining this with a Grandfather-Father-Son (GFS) backup media rotation scheme, for tape-based archiving, ensures that you always have a recoverable media available. In such a scheme, there are three, or more, backup cycles: daily, weekly, and monthly. The following table shows how you might create a GFS rotation schedule: Monday Tuesday Wednesday Thursday Friday         WEEK 1 MON TUE WED THU WEEK 2 MON TUE WED THU WEEK 3 MON TUE WED THU WEEK 4 MON TUE WED THU MONTH 1 "Grandfather" tapes are kept for a year, "Father" tapes for a month, and "Son" tapes for a week. In addition, quarterly, half-yearly, and/or annual backups could also be separately retained if required. Recovery point objective and recovery time objective Both these terms come into play when defining your backup strategy. The recovery point objective (RPO) is a definition of how much data you can afford to lose. If you run backups every 24 hours, you have, in effect, defined that you can afford to lose up to a day's worth of data for a given application or infrastructure. If that is not the case, you need to have a look at how often you back up that particular application. The recovery time objective (RTO) is a measure of the amount of time it should take to restore your data and return the application to a steady state. How long can your business afford to be without a given application? 2 hours? 24 hours? A week? It all depends, and it is very important that you as a backup administrator have a clear understanding of the business you are supporting to evaluate these important parameters. Basically, it boils down to this: If there is a disaster, how much downtime can your business afford? If you don't know, talk to the people in your organization who know. Gather information from the various business units in order to assist in determining what they consider acceptable. Odds are that your views as an IT professional might not coincide with the views of the business units; determine their RPO and RTO values, and determine a backup strategy based on that. Native tape support By popular demand, native tape support was introduced in Veeam Backup & Replication v7. While the most effective method of backup might be disk based, lots and lots of customers still want to make use of their existing investment in tape technology. Standalone drives, tape libraries, and Virtual Tape Libraries (VTL) are all supported and make it possible to use tape-based solutions for long-term archival of backup data. Basically any tape device recognized by the Microsoft Windows server on which Backup & Replication is installed is also supported by Veeam. If Microsoft Windows recognizes the tape device, so will Backup & Replication. It is recommended that customers check the user guide and Veeam Forums (http://forums.veeam.com) for more information on native tape support. Veeam Backup & Replication architecture Veeam Backup & Replication consists of several components that together make up the complete architecture required to protect your environment. This distributed backup architecture leaves you in full control over the deployment, and the licensing options make it easy to scale the solution to fit your needs. Since it works on the VM layer, it uses advanced technologies such as VMware vSphere Changed Block Tracking (CBT) to ensure that only the data blocks that have changed since the last run are backed up, ensuring that the backup is performed as quickly as possible and that the least amount of data needs to be transferred each time. By talking directly to the VMware vStorage APIs for Data Protection (VADP), Veeam Backup & Replication can back up VMs without the need to install agents or otherwise touch the VMs directly. It simply tells the vSphere environment that it wants to take a backup of a given VM; vSphere then creates a snapshot of the VM, and the VM is read from the snapshot to create the backup. Once the backup is finished, the snapshot is removed, and changes that happened to the VM while it was backed up are rolled back into the production VM. By integrating with VMware Tools and Microsoft Windows VSS, application-consistent backups are provided if available in the VMs that are being backed up. For Linux-based VMs, VMware Tools are required and its native quiescence option is used. Not only does it let you back up your VMs and restore them if required, but you can also use it to replicate your production environment to a secondary location. If your secondary location has a different network topology, it helps you remap and re-IP your VMs in case there is a need to failover a specific VM or even an entire datacenter. Of course, failback is also available once the reason for the failover is rectified and normal operations can resume. Veeam Backup & Replication components The Veeam Backup & Replication suite consists of several components, which in combination, make up the backup and replication architecture. Veeam backup server: This is installed on a physical or virtual Microsoft Windows server. Veeam backup server is the core component of an implementation, and it acts as the configuration and control center that coordinates backup, replication, recovery verification, and restore tasks. It also controls jobs scheduling and resource allocation, and is the main entry point configuring the global settings for the backup infrastructure. The backup server uses the following services and components: Veeam Backup Service: This is the main components that coordinates all operations, such as backup, replication, recovery verification, and restore tasks. Veeam Backup Shell: This is the application user interface. Veeam Backup SQL Database: This is used by the other components to store data about the backup infrastructure, backup and restore jobs, and component configuration. This database instance can be installed locally or on a remote server. Veeam Backup PowerShell Snap-in: These are extensions to Microsoft Windows PowerShell that add a set of cmdlets for management of backup, replication, and recovery tasks through automation. Backup proxy Backup proxies are used to offload the Veeam backup server and are essential as you scale your environment. Backup proxies can be seen as data movers, physical or virtual, that run a subset of the components required on the Veeam backup server. These components, which include the Veeam transport service, can be installed in a matter of seconds and are fully automated from the Veeam backup server. You can deploy and remove proxy servers as you see fit, and Veeam Backup &Replication will distribute the backup workload between available backup proxies, thus reducing the load on the backup server itself and increasing the amount of simultaneous backup jobs that can be performed. Backup repository A backup repository is just a location where Veeam Backup & Replication can store backup files, copies of VMs, and metadata. Simply put, it's nothing more than a folder on the assigned disk-based backup storage. Just as you can offload the backup server with multiple proxies, you can add multiple repositories to your infrastructure and direct backup jobs directly to them to balance the load. The following repository types are supported: Microsoft Windows or Linux server with local or directly attached storage: Any storage that is seen as a local/directly attached storage on a Microsoft Windows or Linux server can be used as a repository. That means that there is great flexibility when it comes to selecting repository storage; it can be locally installed storage, iSCSI/FC SAN LUNs, or even locally attached USB drives. When a server is added as a repository, Veeam Backup & Replication deploys and starts the Veeam transport service, which takes care of the communication between the source-side transport service on the Veeam backup server (or proxy) and the repository. This ensures efficient data transfer over both LAN and WAN connections. Common Internet File System (CIFS) shares: CIFS (also known as Server Message Block (SMB)) shares are a bit different as Veeam cannot deploy transport services to a network share directly. To work around this, the transport service installed on a Microsoft Windows proxy server handles the connection between the repository and the CIFS share. Summary In this article, we will learned about various backup strategies and also went through some components of Veeam® Backup and Replication. Resources for Article: Further resources on this subject: VMware vCenter Operations Manager Essentials - Introduction to vCenter Operations Manager [article] Use Of ISO Image for Installation of Windows8 Virtual Machine [article] An Introduction to VMware Horizon Mirage [article]

(For more resources related to this topic, see here.) Integrating StyleCop analysis results in Jenkins/Hudson (Intermediate) In this article we will see how to build and display StyleCop errors in Jenkins/Hudson jobs. To do so, we will need to see how to configure the Jenkins job with a full analysis of the C# files in order to display the technical debt of the project. As we want it to diminish, we will also set in the job an automatic recording of the last number of violations. Finally, we will return an error if we add any violations when compared to the previous build. Getting ready For this article you will need to have: StyleCop 4.7 installed with the option MSBuild integration checked A Subversion server A working Jenkins server including: The MSBuild plug in for Jenkins The Violation plug in for Jenkins A C# project followed in a subversion repository. How to do it... The first step is to build a working build script for your project. All solutions have their advantages and drawbacks. I will use MSBuild in this article. The only difference here will be that I won't separate files on a project basis but take the "whole" solution: <?xml version="1.0" encoding="utf-8" ?> <Project DefaultTargets="StyleCop" > <UsingTask TaskName="StyleCopTask" AssemblyFile="$(MSBuildExtens ionsPath)..StyleCop 4.7StyleCop.dll" /> <PropertyGroup> <!-- Set a default value of 1000000 as maximum Stylecop violations found --> <StyleCopMaxViolationCount>1000000</StyleCopMaxViolationCount> </PropertyGroup> <Target Name="StyleCop"> <!-- Get last violation count from file if exists --> <ReadLinesFromFile Condition="Exists('violationCount.txt')" File="violationCount.txt"> <Output TaskParameter="Lines" PropertyName="StyleCopMaxViola tionCount" /> </ReadLinesFromFile> <!-- Create a collection of files to scan --> <CreateItem Include=".***.cs"> <Output TaskParameter="Include" ItemName="StyleCopFiles" /> </CreateItem> <!-- Launch Stylecop task itself --> <StyleCopTask ProjectFullPath="$(MSBuildProjectFile)" SourceFiles="@(StyleCopFiles)" ForceFullAnalysis="true" TreatErrorsAsWarnings="true" OutputFile="StyleCopReport.xml" CacheResults="true" OverrideSettingsFile= "StylecopCustomRuleSettings.Stylecop" MaxViolationCount="$(StyleCopMaxViolationCount)"> <!-- Set the returned number of violation --> <Output TaskParameter="ViolationCount" PropertyName="StyleCo pViolationCount" /> </StyleCopTask> <!-- Write number of violation founds in last build --> <WriteLinesToFile File="violationCount.txt" Lines="$(StyleCopV iolationCount)" Overwrite="true" /> </Target> </Project> After that, we prepare the files that will be scanned by the StyleCop engine and we launch the StyleCop task on it. We redirect the current number of violations to the StyleCopViolationCount property. Finally, we write the result in the violationsCount.txt file to find out the level of technical debt remaining. This is done with the WriteLinesToFile element. Now that we have our build script for our job, let's see how to use it with Jenkins. First, we have to create the Jenkins job itself. We will create a Build a free-style software project. After that, we have to set how the subversion repository will be accessed, as shown in the following screenshot: We also set it to check for changes on the subversion repository every 15 minutes. Then, we have to launch our MSBuild script using the MSBuild task. The task is quite simple to configure and lets you fill in three fields: MSBuild Version: You need to select one of the MSBuild versions you configured in Jenkins (Jenkins | Manage Jenkins | Configure System) MSBuild Build File: Here we will provide the Stylecop.proj file we previously made Command Line Arguments: In our case, we don't have any to provide, but it might be useful when you have multiple targets in your MSBuild file Finally we have to configure the display of StyleCop errors. This were we will use the violation plugin of Jenkins. It permits the display of multiple quality tools' results on the same graphic. In order to make it work, you have to provide an XML file containing the violations. As you can see in the preceding screenshot, Jenkins is again quite simple to configure. After providing the XML filename for StyleCop, you have to fix thresholds to build health and the maximum number of violations you want to display in the detail screen of each file in violation. How it works... In the first part of the How to do it…section, we presented a build script. Let's explain what it does: First, as we don't use the premade MSBuild integration, we have to declare in which assembly the StyleCop task is defined and how we will call it. This is achieved through the use of the UsingTask element. Then we try to retrieve the previous count of violations and set the maximum number of violations that are acceptable at this stage of our project. This is the role of the ReadLinesFromFile element, which reads the content of a file. As we added a condition to ascertain the existence of the violationsCount.txt file, it will only be executed if the file exists. We redirect the output to the property StyleCopMaxViolationCount. After that we have configured the Jenkins job to follow our project with StyleCop. We have configured some strict rules to ensure nobody will add new violations over time, and with the violation plugin and the way we addressed StyleCop, we are able to follow the technical debt of the project regarding StyleCop violations in the Violations page. A summary of each file is also present and if we click on one of them, we will be able to follow the violations of the file. How to address multiple projects with their own StyleCop settings As far as I know, this is the limit of the MSBuild StyleCop task. When I need to address multiple projects with their own settings, I generally switch to StyleCopCmd using NAnt or a simple batch script and process the stylecop-report.violations.xml file with an XSLT to get the number of violations. Summary This article talked about integrating StyleCop analysis in Jensons/Hudkins. This article helped in building a job analysis for our project. Resources for Article : Further resources on this subject: Organizing, Clarifying and Communicating the R Data Analyses [Article] Generating Reports in Notebooks in RStudio [Article] Data Profiling with IBM Information Analyzer [Article]

OpenThreads basics OpenThreads is a lightweight, cross-platform thread API for OSG classes and applications. It supports the fundamental elements required by a multithreaded program, that is, the thread object (OpenThreads::Thread), the mutex for locking data that may be shared by different threads (OpenThreads::Mutex), barrier (OpenThreads::Barrier), and condition (OpenThreads::Condition). The latter two are often used for thread synchronization. To create a new thread for certain purposes, we have to derive the OpenThreads::Thread base class and re-implement some of its virtual methods. There are also some global functions for conveniently handling threads and thread attributes, for example: The GetNumberOfProcessors() function gets the number of processors available for use. The SetProcessorAffinityOfCurrentThread() function sets the processor affinity (that is, which processor is used to execute this thread) of the current thread. It should be called when the thread is currently running. The CurrentThread() static method of OpenThreads::Thread returns a pointer to the current running thread instance. The YieldCurrentThread() static method of OpenThreads::Thread yields the current thread and lets other threads take over the control of the processor. The microSleep() static method of OpenThreads::Thread makes the current thread sleep for a specified number of microseconds. It can be used in single-threaded applications, too. Time for action – using a separate data receiver thread In this example, we will design a new thread with the OpenThreads library and use it to read characters from the standard input. At the same time, the main process, that is, the OSG viewer and rendering backend will try retrieving the input characters and displaying them on the screen with the osgText library. The entire program can only quit normally when the data thread and main process are both completed. Include the necessary headers: #include <osg/Geode> #include <osgDB/ReadFile> #include <osgText/Text> #include <osgViewer/Viewer> #include <iostream> Declare our new DataReceiverThread class as being derived from OpenThreads::Thread. Two virtual methods should be implemented to ensure that the thread can work properly: the cancel() method defines the cancelling process of the thread, and the run() method defines what action happens from the beginning to the end of the thread. We also define a mutex variable for interprocess synchronization, and make use of the singleton pattern for convenience: class DataReceiverThread : public OpenThreads::Thread { public: static DataReceiverThread* instance() { static DataReceiverThread s_thread; return &s_thread; } virtual int cancel(); virtual void run(); void addToContent( int ch ); bool getContent( std::string& str ); protected: OpenThreads::Mutex _mutex; std::string _content; bool _done; bool _dirty; }; The cancelling work is simple: set the variable _done (which is checked repeatedly during the run() implementation to true) and wait until the thread finishes: int DataReceiverThread::cancel() { _done = true; while( isRunning() ) YieldCurrentThread(); return 0; } The run() method is the core of a thread class. It usually includes a loop in which actual actions are executed all the time. In our data receiver thread, we use std::cin.get() to read characters from the keyboard input and decide if it can be added to the member string _content. When _done is set to true, the run() method will meet the end of its lifetime, and so does the whole thread: void DataReceiverThread::run() { _done = false; _dirty = true; do { YieldCurrentThread(); int ch = 0; std::cin.get(ch); switch (ch) { case 0: break; // We don't want '' to be added case 9: _done = true; break; // ASCII code of Tab = 9 default: addToContent(ch); break; } } while( !_done ); } Be careful of the std::cin.get() function: it firstly reads one or more characters from the user input, until the Enter key is pressed and a 'n' is received. Then it picks characters one by one from the buffer, and continues to add them to the member string. When all characters in the buffer are traversed, it clears the buffer and waits for user input again. The customized addToContent() method adds a new character to _content. This method is sure to be called in the data receiver thread, so we have to lock the mutex object while changing the _content variable, to prevent other threads and the main process from dirtying it: void DataReceiverThread::addToContent( int ch ) { OpenThreads::ScopedLock<OpenThreads::Mutex> lock(_mutex); _content += ch; _dirty = true; } The customized getContent() method is used to obtain the _content variable and add it to the input string argument. This method, the opposite of the previous addToContent() method, must only be called by the following OSG callback implementation. The scoped locking operation of the mutex object will make the entire work thread-safe, as is done in addToContent(): bool getContent( std::string& str ) { OpenThreads::ScopedLock<OpenThreads::Mutex> lock(_mutex); if ( _dirty ) { str += _content; _dirty = false; return true; } return false; } The thread implementation is finished. Now let's go back to rendering. What we want here is a text object that can dynamically change its content according to the string data received from the main process. An update callback of the text object is necessary to realize such functionality. In the virtual update() method of the customized update callback (it is for drawables, so osg::NodeCallback is not needed here), we simply retrieve the osgText::Text object and the receiver thread instance, and then reset the displayed texts: class UpdateTextCallback : public osg::Drawable::UpdateCallback { public: virtual void update( osg::NodeVisitor* nv, osg::Drawable* drawable ) { osgText::Text* text = static_cast<osgText::Text*>(drawable); if ( text ) { std::string str("# "); if ( DataReceiverThread::instance()->getContent(str) ) text->setText( str ); } } }; In the main entry, we first create the osgText::Text drawable and apply a new instance of our text updating callback. The setAxisAlignment() here defines the text as a billboard in the scene, and setDataVariance() ensures that the text object is "dynamic" during updating and drawing. There is also a setInitialBound() method, which accepts an osg::BoundingBox variable as the argument. It forces the definition of the minimum bounding box of the drawable and computes the initial view matrix according to it: osg::ref_ptr<osgText::Text> text = new osgText::Text; text->setFont( "fonts/arial.ttf" ); text->setAxisAlignment( osgText::TextBase::SCREEN ); text->setDataVariance( osg::Object::DYNAMIC ); text->setInitialBound( osg::BoundingBox(osg::Vec3(), osg::Vec3(400.0f, 20.0f, 20.0f)) ); text->setUpdateCallback( new UpdateTextCallback ); Add the text object to an osg::Geode node and turn off lighting. Before starting the viewer, we also have to make sure that the scene is rendered in a fixed-size window. That's because we have to also use the console window for keyboard entry: osg::ref_ptr<osg::Geode> geode = new osg::Geode; geode->addDrawable( text.get() ); geode->getOrCreateStateSet()->setMode( GL_LIGHTING, osg::StateAttribute::OFF ); osgViewer::Viewer viewer; viewer.setSceneData( geode.get() ); viewer.setUpViewInWindow( 50, 50, 640, 480 ); Start the data receiver thread before the viewer runs, and quit it after that: DataReceiverThread::instance()->startThread(); viewer.run(); DataReceiverThread::instance()->cancel(); return 0; Two windows will appear if you are compiling your project with your subsystem console. Set focus to the console window and type some characters. Press Enter when you are finished, and then press Tab followed by Enter in order to quit the receiver thread: You will notice that the same characters come out in the OSG rendering window. This can be treated as a very basic text editor, with the text source in a separate receiver thread, and the drawing interface implemented in the OSG scene graph: What just happened? It is very common that applications use separate threads to load huge files from disk or from the Local Area Network (LAN). Other applications use threads to continuously receive data from the network service and client computers, or user-defined input devices including GPS and radar signals, which is of great speed and efficiency. Extra data handling threads can even specify an affinity processor to work on, and thus make use of today's dual-core and quad-core CPUs. The OpenThreads library provides a minimal and complete object-oriented thread interface for OSG developers, and even general C++ threading programmers. It is used by the osgViewer library to implement multithreaded scene updating, culling, and drawing, which is the secret of highly efficient rendering in OSG. Note here, that multithreaded rendering doesn't simply mean executing OpenGL calls in different threads because the related rendering context (HGLRC under Win32) is thread-specific. One OpenGL context can only be current in one thread (using wglMakeCurrent() function). Thus, one OSG rendering window which wraps only one OpenGL context will never be activated and accept OpenGL calls synchronously in multiple threads. It requires an accurate control of the threading model to make everything work well.

You may be a seasoned or novice web developer, but no matter your level of experience, you must always be able to set up a basic MVC application. This two part series will briefly show you how to use Koa, a bleeding edge Node.js web application framework to create a web application using MongoDB as its database. Koa has a low footprint and tries to be as unbiased as possible. For this series, we will also use Jade and Mongel, two Node.js libraries that provide HTML template rendering and MongoDB model interfacing, respectively. Note that this series requires you to use Node.js version 0.11+. At the end of the series, we will have a small and basic app where you can create pages with a title and content, list your pages, and view them. Let’s get going! Using NPM and Node.js If you do not already have Node.js installed, you can download installation packages at the official Node.js website, http://nodejs.org. I strongly suggest that you install Node.js in order to code along with the article. Once installed, Node.js will add two new programs to your computer that you can access from your terminal; they’re node and npm. The first program is the main Node.js program and is used to run Node.js applications, and the second program is the Node Package Manager and it’s used to install Node.js packages. For this application we start out in an empty folder by using npm to install four libraries: $ npm install koa jade mongel co-body Once this is done, open your favorite text editor and create an index.js file in the folder in which we will now start our creating our application. We start by using the require function to load the four libraries we just installed: var koa = require('koa'); var jade = require('jade'); var mongel = require('mongel'); var parse = require(‘co-body'); This simply loads the functionality of the libraries into the respective variables. This lets us create our Page model and our Koa app variables: var Page = mongel('pages', ‘mongodb://localhost/app'); var app = koa(); As you can see, we now use the variables mongel and koa that we previously loaded into our program using require. To create a model with mongel, all we have to do is give the name of our MongoDB collection and a MongoDB connection URI that represents the network location of the database; in this case we’re using a local installation of MongoDB and a database called app. It’s simple to create a basic Koa application, and as seen in the code above, all we do is create a new variable called app that is the result of calling the Koa library function. Middleware, generators, and JavaScript Koa uses a new feature in JavaScript called generators. Generators are not widely available in browsers yet except for some versions of Google Chrome, but since Node.js is built on the same JavaScript as Google Chrome it can use generators. The generators function is much like a regular JavaScript function, but it has a special ability to yield several values along with the normal ability of returning a single value. Some expert JavaScript programmers used this to create a new and improved way of writing asynchronous code in JavaScript, which is required when building a networked application such as a web application. The generators function is a complex subject and we won’t cover it in detail. We’ll just show you how to use it in our small and basic app. In Koa, generators are used as something called middleware, a concept that may be familiar to you from other languages such as Ruby and Python. Think of middleware as a stack of functions through which an HTTP request must travel in order to create an appropriate response. Middleware should be created so that the functionality of a given middleware is encapsulated together. In our case, this means we’ll be creating two pieces of middleware: one to create pages and one to list pages or show a page. Let’s create our first middleware: app.use(function* (next) { … }); As you can see, we start by calling the app.use function, which takes a generator as its argument, and this effectively pushes the generator into the stack. To create a generator, we use a special function syntax where an asterisk is added as seen in the previous code snippet. We let our generator take a single argument called next, which represents the next middleware in the stack, if any. From here on, it is simply a matter of checking and responding to the parameters of the HTTP request, which are accessible to us in the Koa context. This is also the function context, which in JavaScript is the keyword this, similar to other languages and the keyword self: if (this.path != '/create') { yield next; return } Since we’re creating some middleware that helps us create pages, we make sure that this request is for the right path, in our case, /create; if not, we use the yield keyword and the next argument to pass the control of the program to the next middleware. Please note the return keyword that we also use; this is very important in this case as the middleware would otherwise continue while also passing control to the next middleware. This is not something you want to happen unless the middleware you’re in will not modify the Koa context or HTTP response, because subsequent middleware will always expect that they’re now in control. Now that we have checked that the path is correct, we still have to check the method to see if we’re just showing the form to create a page, or if we should actually create a page in the database: if (this.method == 'POST') { var body = yield parse.form(this); var page = yield Page.createOne({    title: body.title,    contents: body.contents }); this.redirect('/' + page._id); return } else if (this.method != 'GET') { this.status = 405; this.body = 'Method Not Allowed'; return } To check the method, we use the Koa context again and the method attribute. If we’re handling a POST request we now know how to create a page, but this also means that we must extract extra information from the request. Koa does not process the body of a request, only the headers, so we use the co-body library that we downloaded early and loaded in as the parse variable. Notice how we yield on the parse.form function; this is because this is an asynchronous function and we have to wait until it is done before we continue the program. Then we proceed to use our mongel model Page to create a page using the data we found in the body of the request, again this is an asynchronous function and we use yield to wait before we finally redirect the request using the page’s database id. If it turns out the method was not POST, we still want to use this middleware to show the form that is actually used to issue the request. That means we have to make sure that the method is GET, so we added an else if statement to the original check, and if the request is neither POST or GET we respond with an HTTP status 405 and the message Method Not Allowed, which is the appropriate response for this case. Notice how we don’t yield next; this is because the middleware was able to determine a satisfying response for the request and it requires no further processing. Finally, if the method was actually POST, we use the Jade library that we also installed using npm to render a create.jade template in HTML: var html = jade.renderFile('create.jade'); this.body = html; Notice how we set the Koa context’s body attribute to the rendered HTML from Jade; all this does is tell Koa that we want to send that back to the browser that sent the request. Wrapping up You are well on your way to creating your Koa app. In Part 2 we will implement Jade templates and list and view pages. Ready for the next step? Read Part 2 here. Explore all of our top Node.js content in one place - visit our Node.js page today! About the author Christoffer Hallas is a software developer and entrepreneur from Copenhagen, Denmark. He is a computer polyglot and contributes to and maintains a number of open source projects. When not contemplating his next grand idea (which remains an idea) he enjoys music, sports, and design of all kinds. Christoffer can be found on GitHub as hallas and at Twitter as @hamderhallas.